Jeremy Kahn* says Facebook has revealed artificial intelligence (AI) that is already improving Instagram video recommendations.
Facebook unveiled an ambitious project to create an artificial intelligence system that can understand videos on Friday.
Initial progress on the technology has already improved video recommendations on Instagram, Facebook said.
Applying the latest breakthroughs to the app’s Reels, the short videos users can upload and share, made suggestions about what users watch next more relevant.
The project showed a 20 per cent reduction in errors when recognising the speech in videos, Facebook said.
In the future, the company could adapt the technology to automate video captioning and flagging policy violations involving hate speech and incitements to violence, it said in a blog post announcing the effort.
Facebook said the new methods would eventually make it easier for users to search for and organise content.
An example feature could enable someone to find a specific past video by asking Facebook to “show me every time we sang to Grandma.”
Putting brains in computers
The A.I. technology behind the project uses a large neural network, a kind of machine learning software loosely modelled on the human brain.
The system teaches itself through a process called self-supervised learning.
The method involves training the A.I. by having it ingest large amounts of unlabelled or unstructured data and asking it to cluster that data into distinct groups by comparing and contrasting its intrinsic properties.
This is different from the method that is often used to train A.I. systems, particularly for commercial applications, where a large pool of human-annotated examples are needed to teach a system to make a decision.
Facebook has in recent weeks made a series of announcements about its progress using self-supervised learning.
Last week, it debuted a system called SEER (short for SElf-supERvised) that took in more than one billion variables and was trained on more than 400 million publicly available Instagram accounts.
The technology performed tasks such as object recognition and image classification better than any previous self-supervised A.I. systems, Facebook said.
SEER demonstrated other advantages too.
With a little additional training on a relatively small set of labeled data, the A.I. was able to outperform systems that had been specifically trained for those tasks with much larger sets of human-labelled data.
The future is so bright you’ll need shades
The Learning From Video project is even more ambitious since it involves moving images, instead of still ones.
Adding to the complexity, the system must incorporate audio information alongside all the extra visual data.
The A.I.’s ability to find similarities in audio proved particularly useful in improving which Reels videos to recommend to Instagram users, Facebook said.
The next step will pose a greater challenge. As Facebook seeks to build a system capable of surfacing digital “memories” based on users’ short text queries, it will have to grapple with inconsistencies between the text description of the scene the user is looking for and the audio found in the video segment.
To overcome the hurdle, the system will have to use other information, such as text found on the Internet, to learn what text descriptions might be associated with a set of images and sounds, and then match its findings accordingly.
“This work will allow us to move away from A.I. that requires people to look at and label videos by hand,” Facebook said in its blog post.
That sort of automatic categorisation of video will be crucial in a not-too-distant future when augmented reality glasses enable consumers to capture exponentially more video than they do even now with smartphones, the company said.
Eventually, people will expect A.I. systems to serve as a kind of extension of their actual memory, Facebook said.
They will expect these tools to be able to quickly pull up and play back the past on request.
Of course, it’s no coincidence that Facebook is placing a huge bet on the future of augmented reality.
The company has plans for new augmented reality glasses that it is working on in conjunction with glasses company Ray-Ban.
*Jeremy Kahn is a contributor at Fortune.
This article first appeared at fortune.com.
