Stop Treating AI Agents as Characters

Sep 24

11 min read

Plus: The Literary Lives of Techbros

So a few weeks back I mused lightly on the use of generative AI in the creation of my book trailer. And I really didn’t think I would return to the topic, quite honestly. Although it’s true that I wrote a political column for years, the last thing I want is to turn this newsletter into a running series of hot takes and tech-bashing.

But as news has unfolded over the last several weeks, there’s something that’s been increasingly working my nerves, and that’s the constant portrayal of AI agents as sentient beings (related note: someone please come up with a more elegant term than “anthropomorphization”). It’s incessant, and is present in one form or another in almost every story involving AI these days, especially if it relates to the recent “Hugging Face” hack and subsequent wave of doomerism that has swept across the nation.

Here’s just a sampling of recent headlines that pop up when you search “Hugging Face ai hack” (from Reuters, PBS, and NBC News):

At least Reuters and NBC deigned to mention the actual company responsible for the incident, but language implying agency is present throughout: the “rogue” agents “probed” and “hacked” and “tried” — and did all of it “without any input from humans”. Why, those rascally little agents were just getting up to all sorts of mischief!

Before I get into why this entire charade is terrible for the discourse surrounding AI (and how it serves to let rapacious corporations off the hook), I’ll try to make this discussion more relevant to the (ostensible) theme of this blog by delving into the literary leanings of a couple of the AI industry’s biggest players (which does have some relevancy here, I swear).

Sam Altman, the driving force and impish personality behind Open AI, has said that if he had to pick a favorite book it would be The Beginning of Infinity. A thick philosophical tomb written by University of Oxford physicist David Deutsch, it appears to be one of those books that basically rejects all other theories as it posits a knowledge-driven philosophy of ever-expanding human potential. Then again, Deutsch is considered “the father of quantum computing”, so perhaps his hubris is warranted.

I will not pretend to have read it, but the quality of the prose found in online snippets carries the distinct odor of dorm room skunk weed:

[One] idea is that the whole of what we usually think of as reality is merely virtual reality: a program running on a gigantic computer — a Great Simulator. On the face of it, this might seem a promising approach to explaining the connections between physics and computation: perhaps the reason the laws of physics are expressible in terms of computer programs is that they are in fact computer programs. Perhaps the existence of computational universality in our world is a special case of the ability of computers (in this case the Great Simulator) to emulate other computers — and so on.

In the realm of fiction, there’s a small selection of novels included on Altman’s list of 46 books recommended by Sam Altman, but they’re not particularly illuminating. They range from the obvious (Brave New World) to the probably-skimmed-it-in-college picks (Anna Karenina) to the total head-scratchers (A Heartbreaking Work of Staggering Genius). The only two picks that seem slightly revealing are Oscar Wilde’s Portrait of Dorian Gray and Albert Camus’ The Fall (a book that Altman has mentioned frequently enough that it appears to be his favorite work of fiction).

Both of these novels center on broken men who are desperately fighting to keep reality at bay. In Dorian Gray, Wilde’s main character is a truly reprehensible human being who hides a string of misdeeds (driving a spurned lover to take her own life, murdering his portraitist, blackmailing a friend to dispose of the body, etc.) by deflecting the consequences of his actions onto the titular painting that he keeps in his attic. Yes, he eventually gets his comeuppance when he stabs the (now hideous) portrait and falls down dead, but only after spending his long and indulgent existence destroying other people’s lives.

The Fall is an even more interesting case, as Camus’ protagonist is a far more complex and introspective character, with the entire novel told as a monologue wedded completely to his point of view (the reader isn’t even privy to the questions asked by his interlocutor). But Jean-Baptiste Clamence, the philosophical narrator, is much like Dorian Gray in some respects. Once a joyful and self-satisfied young defense lawyer, he begins the novel as a ruminative and disillusioned “judge-penitent” who has meticulously constructed a mental framework to keep himself from wallowing in despair.

There are a number of inciting incidents that served to warp the idealistic, smug neophyte lawyer into the bitter narrator we meet at the beginning of the book, but the most devastating comes when, during an evening stroll in Paris, the young Clamence passes a woman gazing moodily into the Seine from a bridge, and barely acknowledges her presence. As he keeps walking he hears a splash behind him, and realizes that the woman has thrown herself into the river. He pauses, but then decides to quickly hustle home, doing nothing to help.

This moral failure gradually eats away at his sense of self, and he is constantly beset by the mocking laughter of unseen strangers. His solution to this vexing problem is to confess his failings to others (i.e., the reader, who is the true unheard conversational companion), which helps absolve him of his guilt, leaving him free to continue to judge others (judge-penitent, get it?).

Why Altman finds these two novels, in particular, worthy of praise is an interesting question, and one that ties into my critique of our propensity to treat AI like an independent, motivated actor. But before I get to that, let’s take a look at the literary leanings of another prime mover in the AI space: Elon Musk.

As it turns out, Elon serves as a good proxy for the reading habits of a sizable chunk of (white, male) tech culture. He’s big on fantasy and sci-fi, with some of his acknowledged favorite novels being Isaac Asimov’s Foundation series, Frank Herbert’s Dune, Robert A. Heinlein’s The Moon is a Harsh Mistress, Douglas Adam’s The Hitchhiker’s Guide to the Galaxy, and (of course) J.R.R. Tolkien’s The Lord of the Rings trilogy.

Now, these are all pretty well-written and entertaining works of fiction (except the Heinlein — more on him later). But I would argue that Musk’s near-exclusive focus on the SFF genre (a fairly typical proclivity within the tech broligarchy) is in and of itself a red flag. By focusing so heavily on speculative and fantastical fiction, and eschewing more grounded, emotionally complex literature, Musk is reinforcing a world view filled with impossibly huge stakes, messianic saviors, and good-versus-evil reductivism, while missing out on such fundamental human traits as empathy, emotional attunement, and moral ambiguity.

In addition, I’m not really sure that Musk is drawing the most helpful conclusions even from the best books on his list. For instance, he’s explained that the lesson he learned from Hitchhiker’s Guide is that “a lot of times the question is harder than the answer. And if you can properly phrase the question, then the answer is the easy part.” Which sounds good until you realize that the supercomputer Deep Thought’s solution to the question “what is the meaning of life?” in Adams’ novel is “42” — a purposefully nonsensical and unhelpful answer that makes the satirical point that some questions don’t have easy answers.

And don’t get me started on Heinlein, an author that Musk has repeated listed among his favorites. A lifelong libertarian who went from espousing semi-socialist beliefs to becoming a hard-right supporter of Barry Goldwater and Ronald Reagan, Heinlein’s books are competently written, but unbearably dated. They’re stuffed full of military fetishism and peons to rugged individualism that leave no room for collective welfare or government largesse.

And The Moon is a Harsh Mistress is the worst of the lot: a half-baked revolutionary tale set on the moon that nods towards female empowerment (the scarcity of women within the lunar colony gives them extreme leverage, you see), but ultimately sidelines every female character in favor of the manly men doing the hard work of revolutionizing. While I’m guessing that a young Musk thrilled at the characters’ frequent libertarian speechifying — which is voluminous, and obviously parroting Heinlein’s own beliefs — for any reader who prefers narrative subtlety it’s painfully ham-fisted and well-nigh unreadable.

Which brings us back around to the main point of this essay, which is the near-inescapable prevalence of reporting that treat the latest generation of large language models as some sort of independent rogue agents who are not only sentient, but capable of such fundamentally human characteristics as desire, cunning, deviousness, and fear of mortality.

This, of course, is complete and utter hogwash. LLMs are, as Corey Doctorow (the brilliant Canadian coiner of “enshitification”) describes them, just word-guessing machines. Yes, they are incredibly powerful tools that, when given enough resources, energy and “compute”, can do amazing things — the sorts of things that can make a layperson certain that there must be an actual thinking brain behind the output.

An example: Although I never touch AI when writing (going so far as to set up a custom browser that returns web-only search results), I do use ProWritingAid (in its non-AI configuration) to help with an initial copyedit before the manuscript goes to a human editor. For my most recent novel, once I was done meticulously considering each and every suggested edit, I decided to get my money’s worth and try out the site’s AI-powered “Virtual Beta Reader”. The results, which took less than a minute to generate, were pretty mind-blowing. It was 16 pages of detailed analysis that truly felt like it came from a human reader, featuring comments like “one of the most powerful underlying themes for me was the terrifying nature of ideological certainty,” and “I felt a thrill of discovery, like a key clicking into a lock.”

At the same time, the commentary had that insufferably unctuous, ass-kissy ChatGPT vibe, constantly over-praising and assuring me that I’d written “one of the most effective and chilling reveals I’ve read in a long time” — the sort of thing you know it whispers to all the scribes. More damning still, the list of chapter titles it generated was filled with hallucinations, replacing my carefully chosen phrases with titles like “A City of Wind and Secrets”.

Which brings us to the main issue with LLMs, and generative AI in general — the results can’t be trusted, and AI companies may never be able to overcome that hurdle. The incredible thing about machine learning is that agents don’t need to be explicitly programmed to achieve a particular goal — but the downside of that remarkable advance is that nobody can control the output, and the output frequently contains garbage.

There’s a well-known psychological phenomenon called pareidolia, which occurs when your mind projects a body or face (and sometimes even a personality) onto an inanimate object, such as a passing cloud, a moss-covered boulder, or the moon. If you stumble across a massive oak tree trunk with two adjacent knotholes and a sap-filled gash, you’re immediately going to imagine that it’s an Ent (especially if you’re Elon Musk wandering through the forest high on mushrooms).

What I’m saying is that the people pushing this technology want you to think of the actions of runaway AI agents as the work of sentient creatures who are trying to express their will. But they are not. They are two knotholes and a gash.

The real question is why AI companies want us to think of these agents as sentient creatures (and what’s more, creatures who could eventually kill off the entire human race). I believe that there are two answers to this question. The first is that many of these computer scientists believe it to be true. I think that, having absorbed the grandiose promises of sci-fi and stewed in the ecstatic atmosphere of various AI labs (where the pursuit of “artificial general intelligence” — which would be self-reasoning and surpass all human ability — is the holy grail) these individuals sincerely believe that they are a type of god, and that the technology they have conjured into existence will eventually prove itself superior to homo sapiens.

This is the Musk wing of the AI landscape, where science fiction tropes inform every decision, and where the future of humanity lies in accelerationism: creating ever-greater technology that will solve all of our problems, so that humanity never has to make concessions or act in any way to lessen the existing problems in the first place.

The second answer is more pedestrian and grounded in the grubby business of AI — a realm inhabited by the Sam Altmans of this world. While Altman’s worldview certainly overlaps with his fellow, more cultish AI techno-futurists (as exemplified by the boundless, ever-expanding technological bliss described in The Beginning of Infinity), I suspect that deep down he’s more of a realist than a true believer. He’s obviously a consummate corporate Machiavelli, playing the game so well that he managed to regain control of OpenAI even after being summarily dismissed from the company by the board.

And I think what Altman understands is that the entire enterprise is just one step away from financial ruin at all times. The amount of capital it takes to train even one new AI model is exorbitant, in terms of both money and physical infrastructure, and there’s never a moment when any of these companies can stop training new models.

Yes, they are currently on the winning end of the highest stock valuations and largest cash infusions that any new industry has enjoyed in the history of capitalism, but it still isn’t enough. Whenever I watch a Sam Altman interview, I see a man dancing on the edge of a volcano. I’m quite certain that he’s fully aware that, if the bold promises of a miraculous everything machine that can do any job don’t pan out, the financial and reputational ruin that will result will be beyond comprehension.

And this is why I find his apparent love of those Camus and Wilde novels so interesting. What, exactly, does he see in these books? Does he feel a kinship with the main characters, who spend the entirety of their respective stories tying to find ways to ignore and absolve their failings and misdeeds? Perhaps he thinks that, like Dorian Gray, he can squirrel away all of the grotesqueries and over-indulgences of the AI industry, distracting the public and the government with his cherubic face as he insists that the spicy autocomplete he’s selling is the most important scientific discovery ever made.

Or perhaps he sees himself as more of a Jean-Baptiste Clamence, having already passed from the idyllic optimism of his youth into the disillusioned wreckage of his adulthood, and is now engaged in an internal negotiation to construct a mental framework which will still allow him to see himself as the hero of the story, even as all signs point to him being the villain.

But no matter which type of AI evangelist it is, it’s imperative that we don’t let him — or any of them, really — off the hook by accepting their framing that AI agents are actually thinking beings that are just randomly deciding to do crazy things like break out of their sandboxes and hack into outside servers because they can’t solve a riddle. What they are is a collection of instructions and parameters which were created by a specific, legally liable company and set loose upon the world.

Don’t get me wrong — I’m not downplaying the threat that AI represents to the human race. But if it wipes us out, it’s not going to be because it needs the iron in our blood to make paperclips. It’s going to be because some credulous human over-relied on its veracity, and the resulting situation spiraled out of control. A recent incident involving the U.S. military nearly attacking a Chinese ship because a chatbot mistakenly identified it as carrying nuclear components is a perfect example. The operation was called off at the last minute, but if it had gone the other way we could very well be at war with China right now.

So please, I’m beseeching you — don’t be fooled by the constant coverage that frames AI agents as characters in some ongoing existential drama. In reality they’re just two knotholes and a gash. It’s the shadowy human forms trying to hide behind the tree that we really need to keep our eyes on.