There’s a line in Minority Report that’s stuck with me. After looking over the crime scene, the detective character turns to another cop. “This is what we’d call an orgy of evidence. Know how many orgies I had as a homicide cop? None.” The logic behind it is simple: the real world is messy, so when you stumble on something that conforms exactly to your biases, be skeptical.
In every sufficiently large business we have observed (say, with 500+ employees), we have noted that continued advancement, and increasingly continued employment, has started to require repeated professions of belief in the transformative power of AI for said business. I am not talking about providing ideas about how to use AI in the business – I mean religious profession, declarations of faith. Overwhelmingly these statements are made by non-technicians, though it is not uncommon for technicians to emit deranged statements to curry favour.
I just stumbled into an orgy, and so my instinct is to take a step back and see if I’ve missed anything.
To say someone is in a cult is an easy insult. You’ve cleaved the world into two types of people, and the type you’re not holds to some irrational belief. It also ends the conversation; you cannot reason with an unreasonable entity, thus your tactics for dealing with them necessarily shift towards isolation and manipulation. Going for the person instead of the idea is uncomfortably fascist.
But I am also an atheist, surrounded by religious people of various stripes. I have taken the time to understand their beliefs and arguments, and I have found them wanting. If a cult is merely a religion with baggage, then I’ve been brushing shoulders with cult members my entire life.
Now that we know models are getting dangerous, we can do some extrapolating. …
I am now in the camp who believe that we are only at most two or three model generations away from AI finally being controlled like nuclear weapons. Only a few will have access to superintelligence above the classes of models we’re seeing this year. As far as I can tell, most Fortune 500 companies will either not have access at all, or it will be tightly controlled for only a small subset of the company. And it will be supervised.
Steve Yegge is the first point of comparison I can think of, and sure enough it’s trivially easy to scan his writing and find parallels. Here, we have two specific irrational ideas: the notion of “general intelligence,” which has not been shown to exist; and that LLMs are dangerous, when they instead seem merely incompetent.
On several occasions, we’ve been exposed to folks that have been sort of lukewarm on our main offerings, but they really, really wanted to use AI to perform a natural language query on their data. And we thought “Okay, if you really want to see it, maybe we can caveat this appropriately and show you what it might look like.”
This was a terrible mistake. … every lukewarm client that saw the chatbot in action, even with us telling them that it was not going to accomplish what they wanted, wanted to buy it immediately. Every other consideration, including millions of dollars that we could plausibly help them achieve by non-AI means, was swept aside. It was like a dark and terrible force seized control of their limbs, plunged their hands into their own chests, and presented their still-beating credit cards to us in grim supplication. We were so mortified by the inexplicable shift in energy that we (wisely) declined to take the money and ended the sales process, and soon thereafter removed Cortex from our list of demonstrations.
I caught that reference. Besides the examples of irrationality I quoted above, Yegge also argues that the latest Claude model is a step above other LLMs (not really), and that exponential growth is possible (not with finite resources).
Many of you have been expecting the hockey-stick AI advancement curve to level out soon, refusing to believe that it’s truly on an exponential curve that could lead to it being so much smarter than humans. You predicted AI would not be able to replace human engineers. In a way, you turned out to be right. …
We are seeing a plateau in intelligence. It is artificial: the exponential increase continues behind the scenes, gated away from you. And at some point you won’t be able to tell it’s getting better, even if you could see it. The intelligence curve is as real as the Earth is round, but just as flat from where you stand. Welcome to the Flat Curve Society.
The prophecy was correct, Jesus did in fact return, but not in a way your mortal brain can comprehend. This is so close to how religious adherents rationalize away contradictions and failed prophecies that I can’t help but burst out laughing. It’s no wonder the original article hit my heart.
These mandates have led to extremely strange places. Several of my peers now “AI-wash” their work, meaning that even when they can perfectly competently execute on their jobs to the satisfaction of their management teams, said managers are unhappy if the engineers haven’t used AI in the work… so now they’re lying about using LLMs even in contexts where their professional judgement is that they aren’t the appropriate tool. They just do the work, the same way they have for decades, and say Claude did it. Others are being measured on their AI bills with “token leaderboards”, where higher is better because I have evidently fallen into the pocket of Hell where the demons torment me by doing elaborate impressions of absolute fucking morons, so the people hired for their freakish ability to perform system optimisation do the obvious thing. They set the LLMs prompting themselves in a semi-plausible loop in case someone inspects the token consumption and then they watch Netflix.
Recruiting new members is a preoccupation of smaller cults, they tend to rely heavily on deceit and emotional manipulation. Larger ones like Christianity instead bend the rules to their favour and make themselves impossible to avoid. But if you force your irrational belief system on people, you make it easier to spot the internal contradictions. Force everyone to use LLMs, regardless of how useful they are, and you’ll push people to fudge the truth. Misattributing authorship reinforces the belief that the LLM is competent, providing precious evidence to the true believer, while also permitting the job to get done.
I watched a mind-blowing presentation in April from Ezra Savard, who ran a training study/experiment at Netflix from December through March. … The study’s goal was to train Netflix engineers on agentic coding, and measure the impact. … measuring your employees’ token spend at a coarse level can provide powerful insight into where your organization stands on AI literacy, and how much training lies ahead of you. …
What’s the right training setup, you ask? Ezra’s team spent considerable effort honing the formula. The training must be done a team at a time, with 5 to 10 people, including their manager. The manager must opt the team in, during regular work hours, as “blessed” company time. The trainees must bring their actual work, and the instructor(s) will help them learn how to do it with agents. … the exact curriculum barely matters; you can teach it however you like. And then you get everyone through it, 5 hours and ten people at a time.
If the curriculum doesn’t matter, then you’re not actually teaching people. What you’re instead doing is locking them in a room with true believers and the person who can fire them, and not letting everyone out until they can “prove” they believe. That proof does not come in the form of how many lines of code they’ve written, nor how well they can explain the concepts you’ve taught, but the number of tokens they’ve generated via an LLM. That encourages conformity and groupthink, yet is trivially easy to cheat. This has some obvious problems, of course, not least of which is that OpenAI, Anthropic, and Microsoft are being forced to remove token subsidies in order to avoid bankruptcy, leading to pressure on businesses to reduce token usage.
Once you’ve taught everyone how to spend tokens, your second culture problem emerges, which is teaching people how NOT to spend tokens. Token efficiency is a fairly advanced topic. There are many, many ways that models can steer you wrong, and the most efficient agentic coders focus on maximizing their outcomes for a given token budget. …
So at some point you will probably want to have a third training course, this one on efficiency techniques and good token hygiene. Then, give your newly AI-savvy people budgets. Make them earn budget increases with real outcomes. However you do it, measuring outcomes is going to become critically important, so you can differentiate your effective builders from your vanity builders.
The solution is to flip LLM usage from being a virtue to a vice, and switch your performance metrics back to something sane. Liars who didn’t really need an LLM will flourish and get elevated to “experts” by Yegge, while the true believers struggle and remain “beginners.” It’s a difficult stance to justify if you believe the number of tokens generated is proportional to the quality of the output, but it makes sense if you have to balance getting people to conform to your cult with maintaining a profitable business.
There have been several occasions where I have seen someone, apropos of nothing, blurt out almost word-for-word “AI is changing everything”, only to concede moments later that their organisation does not currently use LLMs for anything, and indeed, that they cannot name a single thing that has changed other than they get some use out of ChatGPT (frequently the free-tier). In one extreme case, I have seen an executive confess that they had never even used ChatGPT or any AI tool in their life, immediately after producing a technical strategy for an organisation with $2B+ in revenue which was entirely centered around AI.
Initially these statements were so absurd on their face that I thought it was some cynical ploy to achieve thought leader status, and there are certainly some people doing this – I have had it admitted to me. But the broader reality is so much worse: people who have no background in the technology at all actually believe what they are saying. As a general rule you should avoid getting into business with a liar, but if you must, you can at least reason with them even if only in private. A true believer is much more threatening because they are impervious to even inducement by self-interest.
That last paragraph is an eerie mirror of my own thoughts on the love-hate relationship between religion and true believers. They make for wonderful foot soldiers and enforcers, gleefully handling messy or ridiculous tasks, but they’re also the first to schism your church and point out your lack of commitment to the bit. Thus you tend to get an endless parade of “reasonable moderates,” who disclaim some of the excesses of the true believers but nonetheless demand you conform. Like the down-home environmentalist who isn’t an LLM fanboy, yet just happens to use LLMs on an hourly basis and somehow never seems to acknowledge most data centre water usage comes from power generation. Instead, the “environmentalist” encourages his followers to post his misleading water usage calculator but without crediting him, “because my name’s become somewhat toxic in the debate as more people accuse me of being a shill.” That’s not the tactics of someone who is confident the facts are on their side.
The net result of this is that almost every large organisation that I am aware of is no longer able to focus on anything important, unless they are one of the (very) few organisations where AI happens to address their highest priorities. They cannot buy sensible software, hire competent talent, communicate honestly with executives about the state of projects, or undertake any sort of sensible initiative.
And yet I find myself back in that orgy. I’ve been scouring that post for evidence I may have missed, and yet the best I can do is hunches and vibes. Take the paragraph I just quoted; every large organization has been rendered senseless by AI mania? C’mon, there must be some level of exaggeration going on there. The lack of names both lets people talk freely, but could permits the creation of convincing forgeries. You should always keep an eye out for a profit motive, and there’s a plausible one for the executive director of an “ethical” freelance software company who’s most famous for threating violence against LLM bosters.
Do not question the broadest claims about AI. I cannot emphasize this enough. If someone says “AI is changing everything”, just let it pass if your goal is to fix an object-level problem rather than challenge the reality at the institution. The challenge can only come after you have gained the trust of the most senior person involved. Trust is gained over a meal in private where you assuage their anxieties, not by embarrassing them in front of peers.
Meanwhile, paragraphs like this nail me square in the chest. I’ve both had to be careful about who I mention my atheism to, as well as watch how I talk about LLMs around other people. The end of the post is exactly the advice I would give other atheist for coping in a religious world, and precisely the wisdom I’d find most useful for navigating a workspace infested with LLMs.
Maybe the resolution of the case is that orgies of evidence do exist, they’re just rare. Or that I’ve been on this beat for too long, and become blind to the obvious. Or maybe the investigation is still ongoing; that line happens halfway through the movie, before all the evidence is in. Still, I can at least take away this: be mindful of what you say and who you can trust., especially when your livelihood is on the line.