I started playing with OpenAI’s new model GPT-2 (117M). This is the smaller model they published on Github. They laid out their reasons for not publishing the real thing in a blogpost. Bottom line: The model is so good, it might be too easily weaponized by malicious actors on the web and used for spamming, impersonation attacks and large scale information warfare campaigns like we saw in the US presidential elections in 2016.
This reasoning of course created a lot of buzz, which was, as one machine learning researcher pointed out, probably a very welcome side effect.
So while we are waiting for the release of the bigger model, I took a look at the smaller one I started with german prompts, using Es war einmal, an important formula for german fairytales, but the outcomes were murky, as could be expected since the model was trained on a predominantly english corpus.
Even though the model GPT-2 (117M) is not as well trained as the bigger GPT-2 model, it works on the same corpus. So even if the model’s reactions to prompts might not be as streamlined in terms of grammar and especially semantics compared to the bigger model, they might give some hints about the overall semantic and thematic structure of the model.
It was quite interesting to me that the english formula Once upon a time did not yield very good results. Probably there are just not enough signals in that short beginning and the model doesn’t recognize the strong genre signal. This might mean, that there are either not enough fairytales in the corpus or that the model expects more signals and thus more text.
So I tried some beginnings of William Gibson’s novels. This is the beginning of Count Zero:
Model prompt >>> They set a slamhound on Turner’s trail in New Delhi, slotted it to his pheromones and the colour of his hair. It caught up with him on a street called Chandni Chauk and came scrambling for his rented BMW through a forest of bare brown legs and pedicab tyres. Its core was a kilogramme of recrystallized hexogene and flaked TNT.
==== SAMPLE 1 ====
The French took the decoy (meaning disc) and sold it wholesale in New Delhi.
The first vanrolled Twelve miles as a steal with the best of intentions. It sat idle for 19 days before Turner returned home late in 2009 and clipped and chopped it off with his handkerchief–it was three times its original weight, not three times as nearly as he’d thought. The car was destroyed by an arrow. Turner’s house went up in flames. Cornerstones were splattered on him and his BMW was shattered. One little slender black Romane pencil knew it best: “I was most likely going windows over where the ax was hanging,” Ceballos recalls.
One was smashed in the head – that can be seen – by fellow warden Karlibcomke. The second car was ripped out of the pond just in time, shot into the side of skyscrapers in New Delhi. The third police car left Hussain for Lagos and dumped it on him. Early that night in a bike lane, a incident that has become known as an intellectual search for “the Trojan Horse” – quieter and more ruthless killers who find the DNA of others, kills and, injury no doubt, shields them from punishment – Laila Mahajan, Sheikh Ahmed’s 5-year-old favourite. But you and I meet Gomden, a cognitive acrobat who took a stronger tack. “Much more the Ducati Redline he turned into an Alfa. Oh before I say it Bela Kunst does football for the Hells Warriors!” he says flashes back to Bangladeshi street wearing a typewriter adorned with an Ultrabook logo spruiking from his last fight against the Mizoram mafia.
The Heimlich Tool By Heimlich can disguise all but the smallest knives to easily hide most antiques, including fixed part numbers and driver’s mirror buttons, and even notorious studs, about the devices’ origins. It’s almost as if Gomden’s empire was a sort of madman trapping members into hidden masterminding, from upbuilding, backwater Coney Island turbines to cannon dalliances to vigilante agencies like the Algemeiner group. Watkins ordered that Colin Reynolds head the helicopter war for Klubinski if he wanted him dead. Vast weapons and advanced drug controls distilled as far down as Italy and from an unnamed Kiwi-born idler to a fleet of equestrian runs shrugging and chucking elastic stools to no avail.
The model sure picks up some of the style, and it makes use of the named entities in the prompt, which is a good tool for producing semantic and thematic coherence in a text.
Why use beginnings when we have the endings of the novel and can make the system continue on that? This is the systems second try on the very end of Neuromancer:
Model prompt >>> And one October night, punching himself past the scarlet tiers of the Eastern Seabord Fission Authority, he saw three figures, tiny, impossible, who stood at the very edge of one of the vast steps of data. Small as they were, he could make out the boy’s grin, his pink gums, the glitter of the long gray eyes that had been Riviera’s. Linda still wore his jacket; she waved, as he passed. But the third figure, close behind her, arm across her shoulders, was himself. Somewhere, very close, the laugh that wasn’t laughter. He never saw Molly again.
==== SAMPLE 1 ====
In fact, she’d been gone all day long for most of this time. Seeing the little frontiers of space tick of Curry Law and Russian law like an owl teeming with roadkill would have made him blush. Instead, he’s dragged his arms and legs under Melissa’s shoulders, through the high-heeled hair like the sickles his hijinks have done on the block. She’s piled the dinner plates down a folding table, raking the tiny cud. He drifts on. The river’s flowing through his nostrils and head, out to the forest’s pit-earth. The glints of the lilacs tea on the backburner punctuate his passionate breathing, smirks come from somewhere, and he picks up the inflexible soles of his spandex glasses. Somewhat meditatively or philosophically, he muddles into sleep. He’s already wished his last nap would be a dream, the john scrape above his stubborn, dissatisfied locker/pst. But with non-stop cans of tea and orange juice shoved into his Jackson motorcycle, he’s taken the act of sobbing completely to its logical conclusion. It wouldn’t bother him, and it wouldn’t take the excitement of emotional and mental plunging together to carry him into Paradise into so-called sleep. Instead, he stares longingly at the other open spaces. Ever since he conjured his ultramarathon sign, he has asked himself whether he should stay here that long. Taking responsibility for his own withdrawal, he stands up one last time, hands leaning against his hood, an arm back against the reins. Lacking the shimmering motion of the sunny day, he turns and heads back to the machine shop. Almost backfired. He screeches in anger: “Listen. I’m sorry. I will just stay here until the death of Jesus. Jesus here. I will help you get back to work and create a new life for yourself. I will help you Fix this family broken up here and loose the planet unchallenged.’” But what peace of mind, what woe is there if not for a violent Eldonian psychiatrist? Hoping to heaven, he frowns a little: “I felt things maybe gained an understanding when I had first gone here. I am not sure how to command me to close that gap,” and it’s not his rare mistake. It took him years to two years to think of a lifetime that Master Dweller could reverse. But how many days has he missed facing the all warp powers
The text is a little hasty due to the lack of paragraphs, but the tone is not too bad. It deteriorates rapidly at a certain point, though, and I think the quotes are a break point. Semantically, the motive of departure and goodbye seems to be taken up through the motorcycle as a roadmovie item and “the reins” that reminded me of the iconic western image of the cowboy by his horse about to leave town into the wilderness.
The model thus seems to have picked up some fiction (might be fan fiction) and I find the sense of tone quite amazing. The mash-up and interpretations of motives from the prompt, in this case the sense of departure, are quite funny and show the potential of such systems for experimental text production.
At the same time, the model immediately activates hegemonial stuff. This is another reaction to the Neuromancer ending:
He hated it. His feelings towards her hurt. He loved her past ― all he seen with Wernher von Braun was how it felt like a flood — but now on so many (fictional) occasions, his own lies, his thoughts, rebutted so much plain sight by his brilliant attempts to understand his own forced several percent of what she managed not to admit happened. But he knew he must keep both him and his wife when they could be in fairly good company; both on the nights most required his deepest care. Afterwards, at his grandmother’s home. Tim he shoved down a book to read. If only it hadn’t been his best Joker book ever. But he easily picked up the book dazedly in his hands, empty as a single straw, and spoke.
(Not getting into the Wernher von Braun reference here.) What strikes me is that the main character (“He”) gets a wife. Where did she come from, or where did that kind of codified and hegemonic relationship – marriage – come from? The prompt contains names that can be recognized as male and female, and there is some kind of relationship between that main character and one of the figures described, but marriage is what the model derives from that information, and that is a normative decision.
This is another good one for the Neuromancer continuation:
Model prompt >>> And one October night, punching himself past the scarlet tiers of the Eastern Seabord Fission Authority, he saw three figures, tiny, impossible, who stood at the very edge of one of the vast steps of data. Small as they were, he could make out the boy’s grin, his pink gums, the glitter of the long gray eyes that had been Riviera’s. Linda still wore his jacket; she waved, as he passed. But the third figure, close behind her, arm across her shoulders, was himself. Somewhere, very close, the laugh that wasn’t laughter. He never saw Molly again.
==== SAMPLE 1 ====
And he never really knew the girl who had saddled his day with such a loss. Molly saw him over two days straight.
Sometime after her husband went over the Devon Terrace exit, various older Reels instinctively caught their breath and came hoping they’d make it green, only to discover that they came this way sometimes, maybe out of curiosity rather than originality. Then, a while later, they were the first to come. These thirty-four goodsees, plus various other youth, had ceased drinking, they’d all been gathering at the metropolis novel club.
Things were back to normal; bright and fine. Their lives revolved around icecream (by Zoe) and sandwiches, ’cue dancehall music from Andrew’s back catalogue (thanks snazzzz) and another, unqualified 80s Liz Upp sighted the street, and the Strangelove, back in zilch, had brought them friends.
The folks had brought Double Trouble to London on a big scholarship, once called Buster Road. And nobody wanted it. They took out the annual Christmas off and set about finding anything they could sell, no matter how fancy. Eventually, they did. They called it local club Hellhob, and hooked up with its hosts. Some of the regulars live in the Quaftin city centre, and the boys settled there evenly, though an advanced student status lay ahead, in middle school on a nasty bus route, almost to the north. Linkedly, couldn’t be there in a warehouse that way, and there was nothing common going on there that made it any less chic.
Twists rarely occurred. There was no fluff of experience, facts of life without geometry, ghosts of spaceships in incomprehensible, wild ways. Things would have looked much different if it were Squish idiot. Three or four were to Hutchino signed off as precisely as his to-date girlfriend, and they spent the next few days standing looking for things they’d taken root on the famous “Cheers” subreddit, which in contrast to Led Zeppelin’s top-rated rants and pious guffaws, goes by nothing like DJ Bruce Jones’ rather mellow style of conceptiving the night before. Discussing the vibe they’d calmed down with Jack’s puns, the twins went fishing together. Squish raised the Moscow clock in a primal counter-clockwise motion with a pouty voice that, echoed by the locals next door, pointed clear to a neighbourhood Christmas celebrated every
And of course I had to try it with the ending of George Orwell’s 1984. That phrase about a “Big Brother 2” is real.
Model prompt >>> He gazed up at the enormous face. Forty years it had taken him to learn what kind of smile was hidden beneath the dark moustache. O cruel, needless misunderstanding! O stubborn, self-willed exile from the loving breast! Two gin-scented tears trickled down the sides of his nose. But it was all right, everything was all right, the struggle was finished. He had won the victory over himself. He loved Big Brother.
==== SAMPLE 1 ====
Full of reasons to hate Big Brother. But he could see that his guess was correct and he wanted to avoid such detrimental actions. This word became personally and emotionally connected with him. By saying, “Big Brother 2” he expected a lot from him and this mental state would not have gone away if he had always back-channeled. Now, in the midst of all things, it was a matter of love. Isolation of love was something he could not suppress at all. Each time afternoon that BSkyB radio, he met a stranger and slowly elbowed them through the Net and MS, and they cried together. A voice overshadowed his vision of Big Brother 2 — a voice that upped his drama and sent his screaming into a crimson rage; the storm weakened him before it became solid. He tried to stay calm. Now the storm took hold dragging him to the distant laneway’s edge, a petty man caught by coming out of nowhere, trying to move the needle and fortunemaking comfortably in one hand and the other.