A program that captions your photos

16. Researchers have come up with a system capable of automatically captioning photos, including with sentences it was not taught. Facebook is very interested. Two researchers at Idiap, a research institute in Martigny that is affiliated with EPFL, developed an algorithm that - unlike systems recently unveiled by Google and Microsoft - can describe an image without having to pull up captions that it has already learned. To do this, the researchers used a program capable of making vector representations of images and captions based on an analysis of caption syntax. "When we give it a photo, the program compares the image vector to the vector of possible words and selects the most likely noun, verb and prepositional phrases," said Rémi Lebret, a PhD student specializing in Deep Learning at Idiap. This is how the system finds the most likely description for a photo of a man skateboarding, for example, even if it has never seen a similar photo previously.