AI pilot project in the MSK: start a conversation with a painting

What is AI, and what kinds of things does it make possible?

Dennis (LMLM): Artificial intelligence is a branch of science concerned with reproducing human-like intelligence as closely as possible. We focus on a specific subcategory of this: machine learning. Generally speaking, software is programmed to deal with ‘if-then’ situations. For example, how do identify something as a car? If it has four doors, a windscreen, four wheels and a motor, then it’s a car. But with machine learning you provide input in the form of 10,000 photos of cars and the software figures it out for itself.

Is data always pictures?

Dennis (LMLM): I used photos in this example but it could also be text or audio.

Dries (LMLM): There are countless applications for AI. Depending on what you plan to use it for, input could be numbers or other codes, images or text. Our application uses only text. The machine simulates language and conversations, but it can’t generate images.

What are some examples of things that AI is used for now?

Dries (LMLM): There’s already a lot of AI for text applications. I think everyone has heard of ChatGPT, for example.

Dennis (LMLM): AI has an enormous range of uses. From medical imagery, where it can sometimes see details that the human eye can’t, to the application AlphaGo, which can decisively beat even grandmasters in the game of GO.

Dries (LMLM): And for generating images, of course, which is the first thing that many people think of with AI. The images tended to look a little bit weird in the beginning, but there’s been huge progress since then.

You brought paintings to ‘life’ for the MSK. What does that mean?

Dries (LMLM): We’re experimenting with different applications with Lean Mean Learning Machine. We had been interested for some time in conversational AI, which enables naturalistic dialogue, and then we started thinking about what it could be used for. Neither of us are fans of audioguides, and we wondered whether there was some other way to access the same information without having to listen to a five-minute recording.

Madelein (MSK): We are fans of audioguides! (She laughs.) But we’re also constantly on the lookout for other ways to make information accessible in the museum. AI was on our radar, and Dries and Dennis showed up at exactly the right time.

Dries (LMLM): AI, and in particular people’s perceptions of AI’s impact on art: that’s a complex conversation. I’m really happy that the MSK is thinking about this in a broader context, and that it understands that AI can play a role in support of art rather than simply churn it out. In general I see that museums are still taking a cautious approach. We hope this project will show that AI can have a place in museums, and that it doesn’t necessarily threaten art, but can make art more visible and accessible.

Madelein (MSK): AI is already everywhere, and we’re not going to be able to keep it out of museums. So it’s better to start thinking about the best way to use it in museums.

You chose two works from the collection: from the 19th century, Jenny Montigny’s painting ‘The Cattle Stable’, and from the early 20th century, ‘Seated Woman at the Window’ by Rik Wouters. Why did you pick these?

Madelein (MSK): I chose them. These are works that are both in a good location in the museum and that are either by a woman or feature a woman. That gave us two opportunities to let a woman speak, which I think is important given that the MSK doesn’t have a lot of works by women in its collection.

Dennis (LMLM): And they’re human figures. It makes more sense for people to talk to a woman rather than a snowy landscape or an abstract shape. (He laughs.)

What information did you base your input on?

Dries (LMLM): The prompt, which is the information that you give AI so it can formulate its answers, was a concise summary of the information that we thought was most important: what do you see in the picture, who created it - that was about a page of text. We didn’t give it much about the wider historical context; that’s something the software generates itself.

What do you mean by ‘wider historical context’?

Dennis (LMLM): With the Jenny Montigny painting, that was the context of labourers and farmers in the 19th century. We gave it the century, the location and the fact that it was the beginning of the industrial age, all in one sentence, and the software extrapolated from that.

Dries (LMLM): But you can also ask AI questions about all kinds of other things and the lady in the painting will answer those too. For instance: What would you name that cow standing behind you? What’s your favourite food? Your favourite colour?

Have you done any other projects with conversational AI?

Dennis (LMLM): We did another test project involving a woman with dementia, as a way for trainee nurses to practise conversations. That means the trainer doesn’t have to participate in the conversation with the student, and it also enables more objective evaluations. With the Flemish employment office VDAB we looked at AI as a tool for practising job interviews. People often skip an interview due to nerves, but this technology can help them prepare and equip them to deal with difficult situations.

One really special thing about conversational AI is that you actually feel a personal connection to the application. Once you’ve spent 20 minutes talking to Nel or to Rik Wouters, you could almost forget it’s AI.

Kind of like in the film ‘Her’.

Dennis (LMLM): Exactly. Talking to a human being is a much more emotional experience than chatting with a computer.

What’s the funniest answer that the paintings have come up with so far?

Madelein (MSK): Someone asked one to recite all the decimals of pi. And once when I was in the gallery, the woman in Jenny Montigny’s ‘Cattle Stable’ suddenly said ‘It’s kind of stinky in here.’ After all, she was sitting in a cattle stable! (She laughs.)

Dries (LMLM): To encourage people to start a conversation, we had told AI to say something spontaneously at regular intervals.

Dennis (LMLM): And when I asked her which other painting she would like to be, she said ‘Girl with a Pearl Earring’.

Madelein (MSK): We’ll have to fix that so she chooses one from the MSK collection! (She laughs.)

To find out more, visit the Lean Mean Learning Machine web page.