Making Renoir Paint My Dog
Trying to avoid clichés when writing about DALL-E
By now, probably most of you have heard of DALL-E, the AI artiste behind such masterworks as “Armchair in the shape of an avocado” and “A teddy bear on a skateboard in Times Square.” Give it a prompt — some combination of content and style — and it will generate an often-convincing image of that thing. I put my name on the waiting list for access, and received my invitation yesterday.
Articles about new leaps in artificial intelligence have clichés of their own, and GPT-3 could probably write one without too much trouble. The format is usually something like this:
Clickbait headline: Computers Paint Renoirs Now
Artificial intelligence has progressed! Computers can now do things once considered the exclusive domain of humans.
The results are better than expected — even savvy humans have been fooled!
2a (optional) “guess the impostor” quiz
A bunch of hand-wringing about art, meaning, humanity, etc.
But don’t worry, the AI isn’t that convincing. There are still some things it definitely can’t do.
Being human and thus capable of recognizing cliché, I’ll do my best to avoid writing the piece above, though I won’t be able to avoid some parallels.
Alan Turing, the father of modern computing, proposed a test for determining whether a computer is capable of thinking like a human being. He described a setup where a human interrogator converses with two interlocutors, one of them a computer and the other a human. If the human is unable to correctly determine which is which, then the machine is indeed capable of human-like thought. Of course, the usefulness and validity of the Turing Test have often been challenged. But, Turing’s example of a conversation demonstrating what might count as a “pass” is the closest thing to a literary classic that exists in computer science scholarship.
Read this, and consider just how much a computer would need to know and understand to be able to produce responses like the following:
Interrogator: In the first line of your sonnet which reads 'Shall I compare thee to a summer's day', would not 'a spring day' do as well or better?
Computer: It wouldn't scan.
Interrogator: How about 'a winter's day'? That would scan all right.
Computer: Yes, but nobody wants to be compared to a winter's day.
Interrogator: Would you say Mr. Pickwick reminded you of Christmas?
Computer: In a way.
Interrogator: Yet Christmas is a winter's day, and I do not think Mr Pickwick would mind the comparison.
Computer: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.
Of course, the problem is that many humans wouldn’t be able to meet this standard either.
A Google engineer who believed the company’s AI chatbot had become sentient recently made the news, suggesting that a successful pass of the Turing test is closer than ever. But I found the transcripts completely unconvincing. What makes Turing’s snippet above so compelling is that the computer is able to defend itself and maintain a consistent position when challenged on points of metaphor and meaning — by contrast, the LaMDA responds to all questions with the amiable yes, yes, and yes, even when its answers are nonsensical in context or would contradict previous ones. Most of what it says (insights like “the monster represents difficulty” are not evidence of interpretive skill) could be easily copy-pasted from a Google search.
An article about the creative writing generator Sudowrite (which mostly follows my model above for these pieces) highlights the areas where these AIs are now doing very well: generating passable, if not especially meaningful, imitations of human creative products. I gave Sudowrite a spin (you can try it yourself here) and fed it the first paragraph of Swann’s Way:
For a long time I would go to bed early. Sometimes, the candle barely out, my eyes closed so quickly that I did not have time to tell myself: “I’m falling asleep.” And half an hour later the thought that it was time to look for sleep would awaken me; I would make as if to put away the book which I imagined was still in my hands, and to blow out the light; I had gone on thinking, while I was asleep, about what I had just been reading, but these thoughts had taken a rather peculiar turn; it seemed to me that I myself was the immediate subject of my book: a church, a quartet, the rivalry between François I and Charles V. This impression would persist for some moments after I awoke; it did not offend my reason, but lay like scales upon my eyes and prevented them from registering the fact that the candle was no longer burning. Then it would begin to seem unintelligible, as the thoughts of a previous existence must be after reincarnation; the subject of my book would separate itself from me, leaving me free to apply myself to it or not; and at the same time my sight would return and I would be astonished to find myself in a state of darkness, pleasant and restful enough for my eyes, but even more, perhaps, for my mind, to which it appeared incomprehensible, without a cause, something dark indeed.
Asked to continue the piece, Sudowrite produced the following:
Short as my night was, it was long enough for me to have several times seen, as I lay there with my eyes closed, strange, fleeting images which no effort of will could cause to vanish; these images seemed to consist of several superimposed layers; the first was a sort of wing, for I saw a wing as one sees a sail at a great distance; this was followed by an impression of movement, and this by a closer vision which did not seem to belong to the wing, but to be examining it; this vision in its turn vanished, to be followed by another, behind the first, of a different wing, and so on, as if several successive images could come and go behind the first; finally these images, which were taking shape so slowly that I could distinctly see them form, stopped altogether, and I sank back into my former state, and it was not till then that I was able to stir and wake up and look at my clock and see that the wings, the sail, and the examination of the wing had lasted for about five minutes.
This is a somewhat-convincing pastiche of Proust’s style, and it’s able to pick up on a few of the themes of the first paragraph. It doesn’t seem like it’s going anywhere particularly interesting, but… that’s not an uncommon experience when reading Proust. I asked Sudowrite for another paragraph:
It was almost nine o’clock, and my candle was almost out. I got up, went to look for a match, and lit another candle. Then, without putting on my slippers, I went as quickly as possible to my bookshelves, took down one of the books which I thought would least likely be missed, and carried it to my armchair, where I began to read it. This was a collection of essays by the late Charles-Edmond de Senancour, which I had found, a few days earlier, in the library of the house in which I was staying. I had already looked through it several times, but for some reason I had never been able to read more than a few pages at a time; but now that I was reading it for the fifth or sixth time, I felt a certain pleasure in being able to follow the…
Now here we see it start to break down, even though some aspects are pretty impressive, in particular the invention of the essayist Charles-Edmond de Senancour, who is entirely Sudowrite’s concoction (believe me, I googled the name) but would not be out of place in the slightest in the Proustian world. But the cracks are appearing. He appears to have pinched the book from his host, but why does he qualify it as “least likely to be missed” when he’s taking it now from his own bookshelves? Also, why is his candle still lit when he’s been trying to sleep? I asked Sudowrite for yet another paragraph, and it really started to run out of steam: of the essays of the late de Senancour, our AI writes that “what was valuable in such a book was the fact that it was well-written, that the author was a good prose writer”, and I think that Marcel would have only described someone that way backhandedly, as an insult. It did manage to have the narrator “pausing to look at a drawing or engraving”, to which I will say: nicely done.
My experiments with DALL-E were also Proust-themed (see its contribution to the Proust Extended Universe at the top of this post) and considerably goofier. I decided to ask it to combine two things I love: Marcel Proust, and my dog Walter. Most of my prompts were some variation on “Marcel Proust with his pet golden retriever, in the style of…” and in observing how the various incarnations turn out, we can see something about how DALL-E operates. For each prompt it generates four possible images, and out of those I’ve chosen the best ones.
Marcel Proust with his pet golden retriever, painted by Renoir
So, we have a man with a prominent mustache and a dog. The man is seated and resting his head in his hand (as Proust appears in many of the photos of him). The color palette is golden; this will be a theme. The painting has the softness of a Renoir but not much else; none of that painter’s characteristic subjects are present, nor his luminous portrait backgrounds. But perhaps if you told someone with only vague ideas about Renoir that this was painted by him, they’d believe you.
Marcel Proust with his pet golden retriever, painted by Whistler
DALL-E gave me a few options for the Whistler version, and all of them featured Proust in a simple wooden chair, in profile. Clearly, one particular painting dominates DALL-E’s understanding of Whistler’s oeuvre. The dog appears to be sitting on some kind of raised platform? I like the wallpaper, though.
Marcel Proust with his pet golden retriever, painted by Caravaggio
I give DALL-E points for attempting chiaroscuro here, as well as swapping the top hat for a more period-appropriate one and including some drapery in the background. The face here is rather mangled, but I’ve come to believe that this is intentional. DALL-E, if you give it an existing image to play with, refuses to accept any pictures with realistic faces in them, presumably to avoid the potential for abuse that comes with generating convincing but false images of real people.
We start to see more similarities and can start to make guesses about the AI’s process. From the prompt, it determines content — a man with his pet — and can start filling in specifics based on associations. Proust has a mustache and was often photographed seated with his head resting in his hands, so that’s how he’s posed. It gives Proust a hat appropriate to his own time, except when the setting demands something different. It varies the type of chair according to the prevailing style. And a painter’s style is reduced to superficial visual characteristics only: lines, lighting, and sometimes composition. Marcel always appears in these pictures as rather portly — I’m not sure where DALL-E is getting that from.
Marcel Proust with his pet golden retriever, drawn by a four-year-old
This one is very charming but was clearly drawn by an adult for a four-year-old. Or a computer that thinks four-year-old style amounts to “cartoon with crayons.”
Children’s cartoon of Marcel Proust with his pet golden retriever
Absolutely unhinged. The dog has three front legs. I assume that this is what it sounds like inside Walter’s head. A+
Interestingly, the children’s cartoon versions all had Proust sitting on the floor, not in a chair.
Marcel Proust and his pet golden retriever, painted by Basquiat
Bonus points for the signature crown, but nothing else in this image bears any resemblance to a Basquiat. I’m not going to be fraudulently exhibiting this one anytime soon.
Marcel Proust with his pet golden retriever by Aubrey Beardsley
Not bad! I like this version of the chair.
New Yorker cartoon depicting Marcel Proust with his pet golden retriever
This is a dud. They’re not even getting the font right.
Medieval religious painting of Marcel Proust and his pet golden retriever
They didn’t even try to make him look like Proust. And Walter should get a halo. Otherwise not bad.
Page from a graphic novel about Marcel Proust and his pet golden retriever
Surprisingly expressionist. This graphic novel clearly has a dark storyline. Perhaps I’ll ask Sudowrite to fill in the text.
Tattoo of Marcel Proust with his pet golden retriever
Honestly, this one is pretty clever.
Slim Aarons color photograph of Marcel Proust and his pet golden retriever
OH MY GOD
The other Slim Aarons variations are too good not to share:
What becomes clear is that DALL-E does pretty well with the simpler styles and artists who can be most easily boiled down to a few key characteristics. The results for great painters are, unsurprisingly, the least convincing — partly because they are more difficult to caricature stylistically, and partly because the ideas behind the style are missing. I tried the same prompt with Picasso and Munch, and the results were exceedingly dull (although the Munch version of the dog had intriguingly wild eyes). Caravaggio would have made the dog symbolize something; in the computer version, he’s just a large mass. The results were very disappointing when I asked for classical marble statues; none of them even tried to depict the man locked in a mythological struggle with the dog, and all of them were clothed.
The AI does reveal something like a collective unconscious: knowing that Proust in a children’s book would sit on the floor and not in a chair, and that a high society portrait would need to feature some striking decor. But these images can’t satisfy any need beyond novelty. The stock photo business may suffer some competition; however, Renoir has nothing to worry about. One thing that an AI like this absolutely can’t do — almost by definition — is detect and avoid cliché.
Such a fantastic read, I’ll have to check this out!