I’ve outlined how LLMs work, how they differ from Markov chains but also what they have in common. I’ve demonstrated that commonality, in great detail and with graphs. For the third and final part, I apply those previous two to a series of posts by Ranum and whip up a holiday feast for everyone.

Table-setting: On Philosophy

Ranum’s first post marks his series as “a deeper discussion regarding machine intelligence, in the sense of strategy and philosophy.” When I think of philosophy, I hold up Judith Jarvis Thomson’s “A Defense of Abortion” as a great example of how to do it right, provided you ignore the bit about Kitty Genovese.

The article’s name is misleading, however, because Thomson’s defense comes with strings attached. The paper assumes that a fetus is a full legal person, with all the same rights as an adult; if your argument against abortion does not rely on that assumption, Thomson’s “Defense” is no defense at all. Nor does she defend abortion in all cases, for instance Thomson claims at one point it “would be indecent in the woman to request an abortion, and indecent in a doctor to perform it, if she is in her seventh month, and wants the abortion just to avoid the nuisance of postponing a trip abroad.” You might also be surprised at how often she argues against her points; sometimes she uses this to bring in another counter-point, but sometimes Thomson just accepts the critique and moves on.

It seems to me that the argument we are looking at can establish at most that there are some cases in which the unborn person has a right to the use of its mother’s body, and therefore some cases in which abortion is unjust killing. There is room for much discussion and argument as to precisely which, if any. But I think we should sidestep this issue and leave it open, for at any rate the argument certainly does not establish that all abortion is unjust killing.

That can feel weird and unsatisfying, but that’s because many people think of philosophy as persuasion and conclusion. It is not, it is instead about illumination through exploration. Philosophy is not the tasty pile of empty calories you pick up after beeping your card against the machine, it is when you take a set of ingredients into the kitchen to see what you can create. It is the joy and laughter of the kitchen, it is the agony of mixing in the wrong ingredient and having to bin the batch, it is the anticipation of a timer at 2:57. When done right, how the dishes taste is surprisingly unimportant, the key part was the process of creation.

Appetizer: Creativity

It might seem that Ranum is also mucking about in the kitchen, but look a bit closer. Take his section on creativity:

ChatGPT: Machine creativity is real because machines produce novel structures that did not exist before and could not have been predicted from the inputs alone. That is the definition of creativity in any domain.

Ranum: I agree with that. Novel structures imply differential survival (“success”). I have always been OK with the idea that creationists are beligerently stupid (“Oi! You! Do you know who I AM!?”) and I am OK with the idea that creativity is an evolutionary process. … But if that’s right – and I’m not saying it is – human creativity isn’t any greater than machine creativity at best.

Ranum is admitting he could be wrong, like Thomson, but unlike her he’s merely pushing the reheated slop around on his plate, like a kid picking at their peas to convince their parents they’re eating. What happens if we instead take this ingredient into the kitchen and cook?

import numpy as np
rng = np.random.default_rng( 183634071391190024553394930478166269482 )

def softmax( lnlikes:np.ndarray, count:int=1, temp:float=1, rng=np.random.default_rng() ):
    """Picks an index from an array of unnormalized log likelihoods."""

    m       = np.max( lnlikes ) / temp
    choices = np.exp( lnlikes/temp - m )
    return rng.choice( len(choices), p=choices / np.sum(choices), size=count )

1. In the above code, the pseudo-random number generator is seeded with a very large value, one that’s almost certainly never been used before. We have no way to predict the choice the function will make, as a result, beyond actually executing it. When run repeatedly, this code is a machine that produces a novel structure (the resulting sequence of numbers) that did not exist before and could not have been predicted from the inputs alone, and therefore it is capable of creativity.
2. Mass-produced paper is made by shredding logs, adding water and chemicals, then baking everything into a paste that’s often bleached. The paste is passed into a multi-story forming machine that breaks it up, then presses it into flat sheets. There are many ways to arrange the long wood fibres that paper is made out of, enough we can be pretty confident every sheet of paper has a unique layout of fibre. Thus, a paper-forming machine produces novel structures that did not exist before and could not have been predicted from the inputs alone, and therefore it is capable of creativity.
3. Attach one pendulum to the end of another, place an LED light at the base of the second, and set up a camera to record that light. A double pendulum system is chaotic, so one real-world version of it behaves quite differently from another, and the noise inherent to reality prevents you from starting the system at the same exact point twice, so you will always get a unique output. Thus, this double pendulum system in motion produces novel structures that did not exist before and could not have been predicted from the inputs alone, and therefore it is capable of creativity.

Maybe I’ve convinced you ChatGPT’s definition of creativity is inadequate, but maybe I’ve handed you more evidence it isn’t. Maybe you’ve spotted an error or disanalogy in one or more of the machines I constructed above; maybe that proves some of them are not creative, or maybe the problem can be fixed. By increasing the size of the menu, though, I’ve greatly clarified the problem. There’s an actual dialogue going on here, one that takes a modest step towards clarifying what “creativity” means.

Main Course: Free Will

If I want to start interrogating an AI about free will, I don’t want to put my thumb on the scale by assuming I have free will, and then pretend to be a skeptic by trying to get an AI to show me its free will.

Ranum teases a promising discussion about “free will,” but unfortunately doesn’t deliver. We’re grown adults with our own kitchen, though, so we can cook without him. Let’s borrow from Thompson, and just outright assert LLMs have “free will.” Based on that assumption, we’ll try to figure out what part of it possesses free will, based on how LLMs are constructed and operated.

Obviously, one major obstacle is the definition of “free will.” There’s easily two thousand years of debate around that, far too much to include in a single blog post. So, we won’t: instead of crafting a reasonable definition of “free will,” we’ll start with a terrible definition: whenever reality offers multiple possible outcomes, “free will” is the ability to choose one of those outcomes. As we experiment, we’ll improve that definition.

The heart of an LLM is the transformer side of things. A little thought suggests this part cannot have free will, because (as designed) it doesn’t eliminate any possibilities. As mentioned in part one, the output here isn’t one or a handful of tokens but a probability distribution for all possible tokens, and no such distribution should assign zero probability to a token.

One way to wiggle out of that is to point out that “as designed” and “should” are guidelines. They don’t prevent the GPT from assigning zero probability to tokens, thus narrowing the range of possible choices and qualifying for free will. Alas, there is no easy way for me to access the full probability distribution at each iteration for any model I run, so I can’t completely block this escape. What I can say is that if the GPT portion wants to insulate itself against the randomness injected by what I call “the second system,” a lot of tokens must be assigned zero probability. Thanks to a desire to analyze the non-determinism in the GPT portion, I’ve currently gathered up 1,114,694 potential output tokens, across multiple LLMs and temperature settings. The number of zero-probability tokens present is zero.

Another escape is to argue that the GPT portion does narrow the possibilities, because every one of the 200,000+ potential token outputs goes from having an unknown probability before the transformer does its work to a specific probability. This sort of “free will” would also declare the Gaussian distribution to have “free will,” as a consequence. You’re free to define terms however you like in philosophy, but you also can’t blame people for getting upset when you start calling salt “sugar.”

If you do not think that free will exists in the GPT portion, then the next obvious place to look is in the second system, the part that samples from the probability distribution of possible tokens to assert one token as “canonical.” But remember that Python example I tossed out when discussing “creativity?” That three-line function is the core of the second system! You need a bit more code to set up and manage the transformer portion of the LLM, but all of that either has no effect the output or merely moves values from one place to another. That extra code cannot plausibly grant “free will.”

Now might be a good time to update our definition of “free will” by introducing the concept of “determinism.” The output of a deterministic systems is controlled entirely by the input state, and nothing else. By way of example, the Gaussian distribution is deterministic. You can also see some similarities between it and that definition of “creativity,” but there’s a key difference: “creativity” required the system outputs to be “predictable,” presumably without “activating” the system to read off the output, while this “determinism” trades that restriction for an assertion that links inputs to outputs. A double pendulum system is deterministic, but only in theory; in practice, it is highly unlikely that two input states are ever repeated and thus the system appears to be non-deterministic.

Why introduce this concept? Because it is common to assert that if a system is deterministic, it cannot possess free will. There are alternative views that state determinism and free will are compatible with one another, but I find them to be a tangled mess of spaghetti. Thankfully, there’s a way to route around the mess: I’ll assert that if I feed the same input state into a system two or more times, and each time got back the exact same output, I am justified in saying that system lacks free will. Most of the complications come from dealing with systems where that isn’t a practical scenario, like people or double pendulums. Here, we only care about computer programs and mathematical systems, and here we can actually bake this scenario.

By this modified definition, that three-line function does not have free will. Since there is no other part of the second system that could plausibly produce free will, that means the entire second system lacks free will.

Again, there are ways to wiggle out of this. The seed value is fixed for the example I gave, but why not mandate that the seed be changed any time we could repeat a prior input state? This is trickier than it sounds, because we must start the LLM at some fixed initial state. Your plan is merely to increment the seed in those situations? Then I start the LLM from scratch, restoring the seed’s initial value and earning me a repeat. Note that my assertion is atemporal, so it doesn’t matter if the same input state leads to a different outcome in the future or past, all I need is to observe one successful repeat to rule out free will. This sneaks around the non-deterministic behaviour I’ve previously noted. Even with these tricks, there’s still a wiggle route: state that the second system only has free will if the seed is drawn from a non-deterministic source outside the user’s control, such as a quantum mechanical source. If the maximum size of the seed is sufficiently large, then even though it’s theoretically possible for a seed to repeat it’s so unlikely that we’ll never actually observe it.

This solution still tastes a bit funny, though. That big number I used as a seed in my code? It was drawn from the very source I just linked to! Just drawing from a high-quality quantum mechanical source isn’t sufficient, we also have to stipulate how that draw is performed. Plus, the LLM itself doesn’t care where I’m drawing random seeds from. There’s nothing stopping me from modifying the second system to record the seeds it chose and when it used them. Those modifications do not alter the LLM’s outputs in any way. If I ever create a copy of the LLM that is identical except for the ability to modify the seeds, then re-run it from the same input state and swap in the same seeds at the same times, I’ll earn a repeat. Does that mean the original LLM lacks free will, or did the fact that I tampered with the seeds prevent me from making that declaration?

This might seem to exhaust the possibilities, as an LLM consists only of these two systems, but there’s a third possibility: emergence. In a later post, Ranum makes the puzzling assertion that “we don’t know shit about how neurons work.”

Theoretical neurophysiology rests on certain cardinal assumptions. The nervous system is a net of neurons, each having a soma and an axon. Their adjunctions, or synapses, are always between the axon of one neuron and the soma of another. … From the point of excitation the impulse is propagated to all parts of the neuron. The velocity along the axon varies directly with its diameter, from less than one meter per second in thin axons, which are usually short, to more than 150 meters per second in thick axons, which are usually long. … Between the arrival of impulses upon a neuron and its own propagated impulse there is a synaptic delay of more than half a millisecond. During the first part of the nervous impulse the neuron is absolutely refractory to any stimulation. Thereafter its excitability returns rapidly, in some cases reaching a value above normal from which it sinks again to a subnormal value, whence it returns slowly to normal. Frequent activity augments this subnormality.

McCulloch, Warren S., and Walter Pitts. “A logical calculus of the ideas immanent in nervous activity.” The bulletin of mathematical biophysics 5.4 (1943): 115-133.

As of the 1940’s, biologists had a pretty good understanding of how a neuron behaved. In fact, the primary motivation for early research into neural networks was the contrast between the simple behaviour of individual neurons and the complex behaviour that emerges when you put them in networks! To be charitable to Ranum, though, he may have meant “neurons” as in “neural networks” instead of “individual neurons.” Whatever the case, we could argue that even though neither transformers nor this second system possess free will, one emergent property of combining them is free will.

Slight problem: when I observed deterministic behaviour, I observed it from the LLM as a whole. While I have a pretty good idea of how the GPT and second systems are constructed, enough to describe them in isolation, I didn’t have the time nor patience to fully separate them. Thus the “no free will for observed atemporal determinism” assertion can be invoked for the complete LLM, and not just the second system.

There’s also the problem that, as I said in part one, the GPT depends on the second system in order to create any output, and modifies its future output based on the choices made by that system. Conversely, I could plug the second system into a proper Markov chain instead of a GPT and it would work A-OK; or I could swap “baby ChatGPT” for Qwen or DeepSeek or Llama or any other transformer and it would still work; or I could manually alter or rearrange the probabilities and yet the second system would continue baking as it always has. Emergence comes from the feedback between different parts of the system, but it’s hard to argue the second system accepts any feedback.

Again, none of my baking conclusively proves LLMs lack free will, but my time in the kitchen should have clarified things. At best, if you stipulate that all random seeds are chosen by a non-deterministic source according to some strict criteria, the user cannot even in principle alter this seed, and the user cannot in principle alter the temperature, then I think you’ve got a chance of successfully arguing LLMs have free will.

Dessert: Cognition

By now, you might think I disagree with everything Ranum believes. Not so! Remember my point about using pairs of words instead of letters to create more coherent output? Ranum got there first: “… instead, you make your analysis tree include trigrams or digrams.” His “my goal here is to illuminate not argue” sounds an awful lot like my “illumination through exploration.” I also hold to “substrate independence,” which Ranum describes as “conflating algorithms with implementations.” Put another way, it doesn’t matter if your computer is made of silicon or DNA or water; so long as all three systems are carrying out the same algorithm, what they’re made of is irrelevant.

While there’s a lot of overlap between our beliefs, a critical difference between us is that I have a better idea of how LLMs work, and the receipts to back that up. This gives me more tools in my kitchen, and lets me pull off dishes that Ranum currently cannot.

One theme Ranum comes back to is “human supremacy.”

Let me throw out an example of what I mean by a “human-supremacist” point of view: we humans have something special about us, let’s call it a “soul” that is a fount of emotion, creativity, humor, and other things that make us discernably human. Things that have no “soul” are not “beings” they’re just things – like a toaster, or a 5-ton chunk of exploded steam boiler – we don’t assume they have “will” or “intent” or “creativity” – i.e.: the chunk of steam boiler is not anything I’d think to blame, it just appeared in the kitchen through the hole in the wall.

I should type up a separate blog post on that, but in the meantime here’s a much-too-short retort: “difference” and “supremacy” are not synonyms, but treating them as such is a convenient way to paint your opponents as bigots and dismiss their arguments without a second thought.

Is there something magical about human neurons implementing associative visual memory? What about human neurons implementing plain old memory? What about human neurons implementing a wetware-version of a neural probability Markov-chain walk?

Humans are not Markov chains, for the same reason LLMs aren’t. But let’s boil away the extra ingredients and concentrate on the concentrate: if we hold to substrate independence, but do not think LLMs can achieve what biological neural networks can, then we must be able to produce some “magic” difference between the two.

Finding such a difference is quite a challenge, especially if we’re properly tinkering in the kitchen. As I’ve pointed out, LLMs are trained by predicting the next token in the sequence; human beings instead learn language by breaking down its components into a hierarchy and working upwards from the most primitive. But is this a true difference, or do both achieve the same ends via different means? Cooking on that would make this blog post seem brief.

Nonetheless, there is one thing that might qualify. The “neurons” in an artificial neural net communicate with one another via continuous values. Biological neurons instead use discrete “impulses.”

At any instant a neuron has some threshold, which excitation must exceed to initiate an impulse. This, except for the fact and the time of its occurrence, is determined by the neuron, not by the excitation. … No case is known in which excitation through a single synapse has elicited a nervous impulse in any neuron, whereas any neuron may be excited by impulses arriving at a sufficient number of neighboring synapses within the period of latent addition, which lasts less than one quarter of a millisecond. Observed temporal summation of impulses at greater intervals is impossible for single neurons and empirically depends upon structural properties of the net. McCulloch (1943)

By itself, this is not enough. Early researchers decided to simplify neural networks by focusing on the rate of impulses over time, which captures how excited any one neuron is better than discrete impulses. This may still perfectly describe the long-term behaviour of the net. The process of picking a “canonical” token isn’t a problem, either: if we imagine each of the 200,000+ outputs firing discrete impulses at a rate proportional to their probability, then observing the output at a random time until one impulse arrives is no different from the second system. What seemed like a true difference can be smoothed out with some time in the mixer.

You’ve probably discarded “temperature” as a candidate for the one thing, thinking it’s similar to taking hallucinogens or drinking alcohol or sleep deprivation or any other action that dis-inhibits people. My intuition says they aren’t analogous, but I can’t give a plausible argument so I’ll just concede. “Random seeds” might seem promising, but we can handle that in the same way we handled picking a discrete token: the seed just slightly modifies the time we started observing each of the outputs. It relocates and collects the local non-determinism of individual neurons into a global non-determinism, but it’s not clear that’s a meaningful difference.

No, the missing ingredient is evidence that the short-term timing of those discrete impulses effects an important attribute of biological neural networks.

Epilepsy has been historically seen as a functional brain disorder associated with excessive synchronization of large neuronal populations leading to a hypersynchronous state. Recent evidence showed that epileptiform phenomena, particularly seizures, result from complex interactions between neuronal networks characterized by heterogeneity of neuronal firing and dynamical evolution of synchronization. Desynchronization is often observed preceding seizures or during their early stages; in contrast, high levels of synchronization observed towards the end of seizures may facilitate termination. In this review we discuss cellular and network mechanisms responsible for such complex changes in synchronization. Recent work has identified cell-type-specific inhibitory and excitatory interactions, the dichotomy between neuronal firing and the non-local measurement of local field potentials distant to that firing, and the reflection of the neuronal dark matter problem in non-firing neurons active in seizures. These recent advances have challenged long-established views and are leading to a more rigorous and realistic understanding of the pathophysiology of epilepsy.

Jiruska, Premysl, et al. “Synchronization and desynchronization in epilepsy: controversies and hypotheses.” The Journal of physiology 591.4 (2013): 787-797.

Seizures often effect cognition, and yet by definition their short-term nature cannot be described by long-term rates of firing. This gives us reason to think no transformer-based neural network could have “cognition,” while everything from humans to worms could. Other artificial neural networks might not have this difference, and they could become the basis of an LLM, but any empiric evidence for that is a long way off. Reverting back to discrete impulses in an artificial neural network would dramatically cut performance, because a probability of one in ten trillion translates into waiting roughly ten trillion steps to observe one discrete impulse. You can imagine the effect this would have on training times!

Is that great evidence for difference? Beats me! I’ve merely laid out a festive meal for your enjoyment, and tried to give you a wide menu to choose from. If it wasn’t to your taste, you’re free to tinker with the ingredients in your own kitchen.