IK Prize: Fabrica uses AI to match news photos with classic paintings

Italian research house Fabrica has received the Tate IK Prize for Recognition, a project that uses artificial intelligence to pair contemporary photojournalism with paintings from the Tate’s collection

Eunuchs apply make-up before Raksha Bandhan festival celebrations in a red light area in Mumbai, India, August 17, 2016. REUTERS/Danish Siddiqui TPX Images of the Day. Sir Peter Lely Two Ladies of the Lake Family c.1660 Tate. Purchased with assistance from the Art Fund 1955.
Eunuchs apply make-up before Raksha Bandhan festival celebrations in a red light area in Mumbai, India, August 17, 2016. REUTERS/Danish Siddiqui TPX Images of the Day. Sir Peter Lely Two Ladies of the Lake Family c.1660 Tate. Purchased with assistance from the Art Fund 1955

The IK Prize awards projects that use digital technology to offer new ways of exploring the Tate’s collection. Previous winners include After Dark – a project devised by The Workers which saw robots roam the Tate’s galleries at night – and Tate Sensorium, an immersive installation that used sound and scent to heighten the experience of viewing paintings.

This year, Tate teamed up with Microsoft and set a brief to create a project using artificial intelligence. The winning proposal, Recognition, uses AI to match images from Reuters news feed with artworks from the Tate’s collection. Images are displayed online in a virtual gallery and in a digital display at Tate Britain.

The AI analyses images in various ways. Facial recognition determines the gender, age and emotional status of people in paintings and photographs. Object recognition can detect anything from tables to instruments and fruit while composition recognition analyses colours, shapes and composition. The system also analyses context, looking at image titles and tags. Images which match closely in all four categories are considered a match and displayed side by side.

Screen Shot 2016-09-02 at 16.39.56
An image of Chinese soldiers taking photographs as leaders arrive for the G20 summit in Hangzhou, taken by Aly Song, matched with Josef Herman’s Two Seated Peasants

A photo of a man searching for coins thrown into the River Ganges, for example, is paired with John Constable’s painting The Gleaners, Brighton. Both feature a bird, a beach, a man standing on the beach and a vaguely similar composition.

Alongside images with obvious similarities, the system throws up some odd couplings. Photos of football fans are paired with 17th century portraits of men in coats and tights and an image of a Southern Railway train appears alongside a painting by Eduardo Paolozzi. There are ‘mistakes’, too – one picture of an Olympian biting her gold medal is matched with a painting of a woman eating a biscuit, with the AI assuming the medal is a snack.

It is these odd couplings that prove most fascinating, encouraging viewers to consider the connections between seemingly disparate sets of images and highlighting the differences in the way humans and machines process information.

An image of lorries crossing a bridge in Malaysia, taken by Olivia Harris for Reuters, is paired with a photo of Muizenberg in South Africa by John Riddy
An image of Turkish president Tayyip Erdogan and his wife Emin Gulbaran, taken by Aly Song, is coupled with Conroy Maddox’s Passage De L’Opera

“Some matches don’t make sense to a human but that’s quite interesting,” says Angelo Semeraro, who worked with Fabrica colleagues Coralie Gourguechon, Monica Lanaro and Isaac Vallentin and AI specialists JoliBrain to create Recognition. “We can analyse this data and [work out] what the AI is thinking rather than just looking at it as a mistake.”

Semeraro says the team was initially fascinated with exploring the intersection between art and AI. “For us it was both unexplored and fascinating territory,” he told press. After looking at the Tate’s collection of 50,000 artworks, the team hit upon the idea of exploring recurrences between past artworks and current events, using AI as a creative tool while helping people understand how AI works. Microsoft provided mentorship throughout the project and the team received a £15,000 prize plus £90,000 to put the idea into production.

Recognition will run online for three months, building a vast virtual gallery. It is also on display at Tate Britain alongside an interactive one which allows visitors to select their own matches from pre-selected groups.

An image of lorries crossing a bridge in Malaysia, taken by Olivia Harris for Reuters, is paired with a photo of Muizenberg in South Africa by John Riddy
An image of lorries crossing a bridge in Malaysia, taken by Olivia Harris for Reuters, is paired with a photo of Muizenberg in South Africa by John Riddy

Tate says the project was chosen because it posed interesting questions about how our brains work and the role machines can play in helping us understand art.

Four projects were shortlisted for the prize – the runners up were The Wandering Intelligence, a project which used facial recognition and sound detection to allow artworks to respond to visitors and effectively think for themselves; OSCAR, an intelligent interactive machine that can create artwork and Texting Tate, an intelligent chat bot that can describe artwork with help from the public. The Wandering Intelligence was devised by Ross Frame and Tom Wyatt, OSCAR by Unit Lab and Texting Tate by Michael Erier.

Holiday makers swim in the Bassin d'Arcachon as warm summer temperatures continue in Arcachon, southwestern France, August 16, 2016. Reuters/Regis Duvignau, paired with Henry Scott Tuke August Blue 1893–4 Tate. Presented by the Trustees of the Chantrey Bequest 1894
Holiday makers swim in the Bassin d’Arcachon as warm summer temperatures continue in Arcachon, southwestern France, August 16, 2016. Reuters/Regis Duvignau, paired with Henry Scott Tuke August Blue 1893–4 Tate. Presented by the Trustees of the Chantrey Bequest 1894

Recognition will run until November 27. You can view the project online or at Tate Britain. For visiting details, see tate.org.uk

More from CR

How Tate Modern is using digital technology to create new experiences for visitors

Tate Modern’s long anticipated Switch House extension opens to the public this weekend. Alongside new exhibition spaces, the building features two digital installations created by Labs at Framestore. We spoke to Tate’s Susan Doyon and Framestore’s Tom Schwarz about the installations and how Tate Modern is using digital technology to engage with visitors

Senior Creative Designer

Monddi Design Agency

Head of Digital Content

Red Sofa London