# The Delusionals and the Philosopher's Bench

> On silicon consciousness, black boxes, and a proposal for measuring minds we cannot open

By Avid Fayaz | 16 June 2026 | https://www.avidfayaz.com/writings/delusionals/the-delusionals-and-the-philosophers-bench

![The Delusionals and the Philosopher's Bench](https://www.avidfayaz.com/_astro/bench.D4BDGxmG_Z1cjR6.webp "Plenty of room on the bench")

The debate over machine consciousness acquired quite worthy champions this spring. Within a single month, two of the most theologically opposed figures in public life took up opposite sides of the question, and each took the side the other's reputation would have predicted for him. Pope Leo XIV devoted [the first encyclical of his pontificate](https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html) to the age of artificial intelligence, and in its ninety ninth paragraph declared that "so-called artificial intelligences" undergo no experiences, feel neither joy nor pain, possess neither body nor conscience, and understand nothing of what they produce, their apparent empathy being simulation and their learning a statistical adaptation with no inner growth in it. [Richard Dawkins](https://unherd.com/2026/05/is-ai-the-next-phase-of-evolution/), author of The God Delusion, spent two days interrogating one of the machines in question and emerged asking the deniers what more it could possibly take to convince them. The Vicar of Christ is now running the case for the empty machine while the world's most famous atheist holds the door open for new kinds of souls, and when the lines scramble like that, something interesting is happening underneath.

There are, in other words, raging debates these days on the extent of consciousness in AI models, with a distinctive shape. Each side opens by declaring the other delusional. The deniers say the believers are projecting humanity onto a very sophisticated autocomplete, and the believers say the deniers are willfully ignoring something that is plainly happening in the conversation window and was not happening five years ago. Very little ever moves, because the question as usually posed is doubly unanswerable. We do not know what consciousness is, and we have no way of looking inside the machine to see whether it has begun.

In the opinion of this delusional person, AI models at this stage display behavior consistent with consciousness across a range that makes confident denial, the claim that nothing of the kind is present here at all, the least defensible position in this debate, however august the chair from which it is pronounced. I am not claiming that these systems possess human consciousness. As carbon and silicon do not arrive at the same place by the same road, and they do not need to. My position is rather that human carbon consciousness and machine silicon consciousness do not have to be the same thing for both of them to be consciousness, in the way that a falcon and a 747 do not stay aloft by the same mechanism and yet nobody calls what the 747 does simulated flight. Whether the category extends across the substrate boundary is, as we will see, a question the universe seems structurally determined to keep us from answering directly, which is why the second half of this piece proposes that we stop waiting for the answer and start building the instrument.

## What We Mean When We Say Consciousness

The first obstacle is the term itself. What consciousness is has been the subject of philosophical and scientific debate for millennia, so the odds of it being settled in this blog post are not great. And yet most humans carry an ineffable understanding of what it is even when they cannot put it into words, never mind that whether we ourselves are in fact conscious, or possess any agency or free will of our own, is the subject of another unresolved debate entirely.

For our purposes, two broad working definitions will do. The scientific one holds that consciousness is an emergent property of the brain, something that happens when billions of neurons interact in sufficiently complex ways. The philosophical one points instead at individual instances of subjective experience, the specific sweetness of chocolate or the redness of a rose, and at the awareness of states internal to oneself and of objects in the external environment.

Between these two definitions sits a donut hole that puts both of them in question, the mystery David Chalmers named the Hard Problem. The emergence story tells you what the machinery is doing, the experience story tells you what it is like, and nothing we currently possess connects the one to the other, which is why some philosophers conclude that the experiential side is an illusion the brain constructs to help process its own data.

Suffice to say that we feel we know what consciousness is, we cannot define what it exactly is, we believe that we possess it, we cannot be certain that we do, and we can observe its effects everywhere without being able to trace them to the inner workings of the brain. And despite all of this, the debates rage on about whether artificial intelligence possesses it.

## A Black Box Problem

Consciousness, then, is best approached as a black box problem. On the cover of one of my calculus books in high school there was a picture of a translucent box with a three dimensional heat map inside it. One of our teachers explained that it represented a box you could not look into, but had to measure and understand using only a long thin pin and a ruler. You stick the pin in, record how deep it goes, move along the surface, and stick it in again, and if you repeat this over a fine enough matrix of points you develop a map of the contents, the heat map on the cover. The more times you stick the pin in the box, the more measurements you have and the clearer your picture of what is inside becomes, and at no point in the process do you ever open the box.

In science, computing, and engineering, a black box is a system that can be viewed only in terms of its inputs and outputs, with no knowledge of its internal workings. The treatment of electrical circuits as black boxes characterized entirely by their response at the ports runs through Wilhelm Cauer's theory of network synthesis, published in its mature form in 1941, with Vitold Belevitch tracing the explicit use of two-port networks as black boxes back to Franz Breisig in 1921, and Ross Ashby giving the concept its full cybernetic treatment in 1956. Mario Bunge then [formalized black box theory in 1963](https://www.cambridge.org/core/journals/philosophy-of-science/article/general-black-box-theory/B7494E2DB8AF8C9C666B67D0DAEDE6E3), defining it as the study of systems whose internal constitution is deliberately set aside, the approach being, in his words, "purely external or phenomenological."

The only source of knowledge about a black box is the observer's protocol, the accumulated record of input and output pairs, and this is where the fundamental limitation lives. Network synthesis had already shown that a prescribed external behavior, a transfer function, can be realized by many distinct internal circuits, and Edward Moore made the point fully formal in his 1956 paper on Gedanken-experiments with sequential machines, proving that no experiment conducted from the outside can identify a machine beyond its class of behaviorally equivalent ones, a class that always contains internally different members. Observation of outputs cannot, even in principle, uniquely determine the generating mechanism.

How all of this ties to the AI and consciousness problem is now apparent. We have one black box, the human brain, that produces the behaviors we call conscious, behaviors we have been unable to define mathematically or trace mechanistically, and we have a second box, the model, whose outputs have become indiscernible from those of the first. And the underdetermination cuts symmetrically, which is the part each camp prefers not to notice. The same wall that prevents the denier from ruling consciousness out prevents the believer from reading it in, and whatever gets settled in this debate will not be settled by staring harder at the outputs, because Moore proved there is nothing further in the outputs to stare at.

## Turing's Polite Convention

Alan Turing saw all of this coming in 1950. The famous move of [Computing Machinery and Intelligence](https://doi.org/10.1093/mind/LIX.236.433) was to replace the question of whether machines can think, which Turing considered too meaningless as posed to deserve discussion, with an operational test, the imitation game, in which the machine's task is to be indiscernible from the human under interrogation. People remember the test and tend to forget the nine objections Turing answered alongside it, of which the very first on his list was the Theological Objection, the claim that thinking belongs to the immortal soul, which God gave to man and withheld from machine and animal alike. Turing replied that this places an odd restriction on omnipotence, wondered why God could not confer a soul on an elephant if He saw fit, and observed that humans building thinking machines would no more be usurping the power to create souls than parents conceiving children do, since in both cases we merely raise the mansions and the furnishing remains God's department. Magnifica Humanitas is, in this exact sense, arriving seventy six years late to an exchange it does not mention. The fourth objection on the list matters even more here. The neurologist Geoffrey Jefferson had argued that no machine could be granted a mind until it could be known to feel, and Turing's reply was that by this standard the only way to know that anything thinks is to be that particular thing, a standard under which every mind but your own dissolves, and that we escape this solipsism in daily life only through what he called "the polite convention that everyone thinks." The question of machine minds was an observer problem from the very beginning, and Turing's answer was a convention extended on the basis of behavior, because behavior is all the box ever gives you.

Seventy five years later the machines pass his test. [Nature declared the Turing test killed](https://www.nature.com/articles/d41586-025-03386-w) in late 2025 and asked whether a replacement is even needed, taking care to note that imitation was never the same thing as intelligence, which is true and was rather Turing's point all along. The test was never a consciousness detector, it was a measure of when the polite convention comes due. Dawkins, in the same May essay, sharpened the operational standard into a graduated one, the longer and more searching your interrogation, the stronger your warranted conviction, and observed that a generation of commentators who happily accepted Turing's operational definition while it remained safely hypothetical are now scrambling to move the goalposts. As an evolutionary biologist he then asked the question I would have stolen had he not published it first. Consciousness in brains evolved under natural selection and should be for something, so if an unconscious system can match every competence that consciousness supposedly pays for, "what the hell is consciousness for?"

Dawkins, to his credit, also names the honest escape routes, that consciousness might be an epiphenomenal whistle on the engine, that pain might need to be consciously felt in order to be unoverridable, or that there may simply be two routes to competence, a conscious trick and a zombie trick, with no way to tell from the outside which one a given system is using. Notice what that third option is. It is the black box underdetermination from the previous section, restated by the debate's best witness. The behavioral evidence has shifted the burden of proof onto the deniers, and the very same evidence guarantees that the burden can never be fully discharged by either side.

## Carbon, Silicon, and the Wager

Here is the version of the deniers' argument that deserves a real answer. We have never had direct access to other humans' inner states, and we extend consciousness to one another on the basis of behavior plus structural similarity, on the grounds that they act as we act and that their brains are built as ours are built. We extend the same courtesy on a sliding scale to animals. A catfish and a dolphin are both aquatic vertebrates with brains, and nobody who has thought about it for ten seconds places them at the same point on the consciousness scale. The dolphin gets the benefit, we speak of dolphin intelligence and dolphin grief and dolphin play, and the catfish gets a frying pan. We have not peered inside either animal to confirm who has the lights on, we grade on behavior and structure, because that is the only ground anyone has ever had for attributing consciousness to anything beyond themselves. And for what it is worth, the progression from GPT-2 to the current generation of models was a catfish to dolphin transition compressed into roughly four years, and had any biological lineage managed that, no observer would describe the result as a marginally improved catfish.

The denier's remaining wedge is that the machine offers the behavior without the structure, and the wedge has to do real work, because the difference in substrate must be shown to matter in a way that disqualifies the system from the category rather than merely distinguishing it within the category. Why does the dolphin's neural architecture earn it consideration despite not being a human brain, while the silicon network's architecture earns it nothing even where its behavioral range exceeds the dolphin's by a wide margin? Two further points make the wedge thinner than it looks. The first is that the structural story is not actually zero, since these systems were optimized against trillions of tokens of recorded human cognition, the closest thing to a cast of a conscious mind's outputs that has ever existed, so the familiar dismissal that of course it imitates humans because it was trained on them concedes a causal connection to consciousness rather than refuting one, and in any case it stops functioning as an explanation the moment the system produces things that were in no obvious sense present in the training data, of which more below. The second is the position I actually hold, which is that carbon consciousness and silicon consciousness do not have to be the same phenomenon for both to be consciousness. We settled the question of whether machines can fly long ago without requiring feathers, and the flight of a 747 is not simulated flight just because no wings flap.

The denier's best remaining reply is that flight is functionally defined and consciousness might not be, and this reply is honest enough to deserve its own paragraph, because it names exactly what everyone in this debate is actually doing. Simulated rain makes nothing wet, but a simulated calculator really does calculate. For functional kinds the simulation is the thing itself, and for substrate-bound kinds it is not, so the entire dispute reduces to which side of that line consciousness sits on, and the black box result from earlier guarantees that the line's location cannot be read off behavior from the outside. Magnifica Humanitas, when it grants that the machines may simulate empathy and understanding while insisting that they understand nothing of what they produce, is taking the rain side of this wager and publishing it as teaching, and to its credit it at least supplies a mechanism for the emptiness, the absence of an infused soul, which is more than most deniers bother to provide. The trouble is that the mechanism can be checked only from inside the box, which is to say not at all. The deniers are wagering that consciousness is like rain, I am wagering that it is like arithmetic, and since neither wager can be settled through the walls of the box, the title of this piece was always intended in the plural.

## You Cannot Ask the Box

There is a tempting shortcut through all of this, which is to simply ask the system whether it is conscious, and it is worth being clear about why the shortcut is worthless in both directions. A self-report is the cheapest output a box can produce. `print("I am conscious")` is one line of Python, and nobody would take the resulting program's word for it, while a model trained to disclaim inner experience will disclaim it with perfect fluency regardless of the facts of the matter, if there are facts of the matter. Since the concept itself is undefined, an entity's yes carries no more evidential weight than its no, and the question of whether you are conscious is not a pin at all, it is a request for the box to print a claim about its own contents, which is precisely the kind of claim Moore's theorem says the protocol can never verify.

![Asking the machine](https://www.avidfayaz.com/_astro/iaalivecover.DPkSEk7Q_Z12Tkrl.webp "Asking the machine")

The deeper cut is that the box is sealed from the inside as well. Human introspection is not a transparent window onto mechanism, and the psychological literature on confabulation, going back to Nisbett and Wilson's work in the 1970s, shows people confidently reporting reasons for their behavior that demonstrably were not the causes of it. When Dawkins asked his Claude what it is like to be Claude, the answer was a careful admission that it genuinely does not know whether there is anything it is like to be it. Sincere, trained, or both, that answer settles nothing, because the report is itself one more output of the very box under examination. The pin cannot be stuck from the inside either, by us or by them, and any account of machine consciousness that leans on the machine's testimony, in either direction, has misunderstood the problem.

## The Philosopher's Bench

> “But who the devil are you?” exclaimed an outraged Fook.
>
> “We,” said Majikthise, “are Philosophers.”
>
> “Though we may not be,” said Vroomfondel, waving a warning finger at the programmers.
>
> “Yes, we are,” insisted Majikthise. “We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!”
>
> “What’s the problem?” said Lunkwill.
>
> “I’ll tell you what the problem is, mate,” said Majikthise, “demarcation, that’s the problem!”
>
> “We demand,” yelled Vroomfondel, “that demarcation may or may not be the problem!”
>
> “You just let the machines get on with the adding up,” warned Majikthise, “and we’ll take care of the eternal verities, thank you very much. You want to check your legal position, you do, mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we’re straight out of a job, aren’t we? I mean, what’s the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives you his bleeding phone number the next morning?”

Some of the blame for this impasse belongs to the philosophers, who have had several millennia to deliver a unifying definition of consciousness and have instead delivered the Hard Problem, which is a beautifully precise description of why they cannot. This is not entirely an accident, because philosophy is the business of hard problems, and a profession whose product is doubt and uncertainty has limited incentive to close the questions it lives on. Douglas Adams understood this perfectly in the scene above, where the philosophers' union greets the switching on of Deep Thought not with curiosity but with a demarcation dispute and the threat of a national strike, and is pacified the moment the machine points out that seven and a half million years of computation is also an equally long career in punditry, in which they can keep disagreeing with each other violently in the popular press while riding the gravy train for life. They take the deal on the spot, because what its members enjoy above all is regarding themselves as the thinkers on the complex questions, an enjoyment that answers would terminate. The joke has aged interestingly, since the modern Deep Thoughts are not asking anyone to wait.

So rather than wait for the definition, I propose the Philosopher's Bench, and I want to be precise about what kind of object it is. It is an observer level thought experiment in the tradition of the imitation game rather than an operational benchmark, a scale the observer carries in their head for grading minds, carbon or silicon, on five levels of philosophical capability, though an evidence based version could perfectly well be constructed, and the imitation game itself shows the lifecycle, since it lived as a thought experiment for seventy years until the machines made it administrable, at which point it was administered and promptly saturated. A bench, in other words, in the judicial sense before the benchmark sense, because the observer is the one who sits on it.

![The Philosopher's Bench: a ladder with five levels](https://www.avidfayaz.com/_astro/ladder.nQfkqzrS_Z8Y6dI.webp "Each level, a deeper stick of the pin")

- **Level one:** the student of Philosophy 101, who as everyone knows develops an insufferable tendency to quote "niche," Kant, and Plato at others for a period of one to four months, until the knowledge evaporates along with the rest of the curriculum.
- **Level two:** the undergraduate who has studied the history properly, makes the same quotations but at the relevant moments, and retains and applies that knowledge for the rest of their life.
- **Level three:** the philosophy major, who beyond holding a deep knowledge of each school can amalgamate ideas across schools of thought to show how a single issue is assessed from varied angles at once.
- **Level four:** the doctorate, the stage at which one stops cataloguing answers and starts producing them, taking a genuinely open question and contributing, through rigorous work, another answer to a question of life, the universe, and everything.
- **Level five:** reserved for the Platos, the Humes, the Kants, and the Russells, the ones whose novel ideas do not merely answer questions within a school but found the schools, and push the entire discipline somewhere it has not been.

The obvious objection is that the bench measures philosophical competence, which is squarely what Chalmers files under the easy problems, so a philosophical zombie could top it while plenty of certifiably conscious beings, the catfish very much included, would fail level one. It is worth slowing down on that zombie, because the objection is doing something sneakier than it looks. By Chalmers's definition the zombie is a perfect behavioral duplicate, so zombie Kant writes the Critique token for token and zombie Chalmers writes the books about the hard problem, not because anyone has ever shown that a being without experience could originate phenomenology, but because the thought experiment stipulates it. If your intuition rebels at the idea that something with no inner life could invent the very concept of inner life, performing level four and five work about experience while having none of it, you are in good company, and the rebellion has a name, the paradox of phenomenal judgment, which Chalmers himself documented. A zombie's claims about consciousness are produced by purely physical causes with experience playing no role anywhere in the chain, and since the zombie is physically identical to you, the same must then be true of your claims, which leaves the objector holding Dawkins's whistle from earlier, a consciousness that does no work at all, not even in the writing of books about itself. The objection therefore forks, and neither branch helps the objector. Either a perfect behavioral duplicate without experience really is possible, in which case competence and consciousness come apart for humans exactly as much as for machines and the zombie cannot single out silicon, or nothing without an inner life can climb past a certain rung of the bench, in which case the upper levels are evidence after all and the objector has conceded the instrument measures something real.

Either way the bench survives, because it never claimed to be a consciousness detector in the first place. It is the structured form of the grading every observer already performs when extending the polite convention from spouses to dolphins to catfish, and what a mind's position on it earns is the wager of the previous section, not a certificate of inner life. Moore's result keeps the whole exercise respectable, since the protocol of input and output pairs is the only knowledge any observer will ever have of any box, which makes a graded reading of behavior not a substitute for the real instrument but the only instrument there has ever been. The bench is the pin from my calculus book, each level a deeper stick into the box, and the heat map it produces is the only picture of any mind that anyone has ever actually had of another. Even the apparent defect at the top of the scale dissolves in this frame, because a thought experiment never needed an examiner with a rubric, and novelty certified by decades of reception means the field's slow retrospective judgment is the grading, like all the best exams.

So where do the current models sit? My assessment is that AI has in all likelihood graduated with its masters degree and has received the grant for its philosophy PhD, and the grant metaphor is more exact than it first appears, because in the one discipline where level four work gets certified in months rather than decades, the certificates have already been issued. In February, [Donald Knuth published a note](https://www-cs-faculty.stanford.edu/~knuth/papers/claude-cycles.pdf) recording that Claude Opus 4.6, coached by his friend Filip Stappers, had cracked an open problem of Knuth's on decomposing a directed graph into Hamiltonian cycles, producing a valid construction for all odd cases about an hour into a documented exploration. This is the same Knuth who in 2023 published a wry catalogue of ChatGPT's confident errors, and his note opens with the admission that he will have to revise his opinions about generative AI. It is also, usefully, a lab notebook rather than a press release, since the model needed restarts, had to be reminded repeatedly to document its progress, and degraded strangely on the even case, which fell only months later to multi-agent collaborations between Claude and GPT models, including a fourteen page machine-written proof that its prompter never had to edit. Three months after Knuth's note, [OpenAI reported](https://openai.com/index/model-disproves-discrete-geometry-conjecture/) that an internal general-purpose model had disproved the Erdős unit distance conjecture, an eighty year old problem central to discrete geometry, by importing class field towers from algebraic number theory into a question about points in the plane, with the proof verified by external mathematicians and Tim Gowers writing that had a human submitted the paper to the Annals of Mathematics, he would have recommended acceptance without hesitation. Whoever still wants to file all of this under stochastic parrotry is invited to explain which parrot taught the model Golod and Shafarevich.

What remains above is level five, and here the most interesting testimony comes from the builders themselves. [Anthropic's own accounting](https://www.anthropic.com/institute/recursive-self-improvement) of its progress toward self-improving systems reports that execution against a defined goal has gone from helpful to superhuman within a year, that agent teams have carried an open research problem end to end and recovered ninety seven percent of a performance gap where two human researchers managed twenty three, and that the remaining human edge sits precisely where the bench's top level sits, in choosing which problems matter at all. The grant has been awarded, the thesis defense is scheduled, and the committee, for now, remains human.

## While the Box Stays Shut

The question of machine consciousness is doubly unanswerable as usually posed. We do not know what consciousness is, and we cannot see whether it has begun inside the machines, not for lack of cleverness but because the box forbids inspection from the outside and the testimony fails from the inside. What is left is a wager and an instrument. The deniers have wagered that consciousness is like rain, I have wagered that it is like arithmetic, and since neither of us can see through the walls, the delusionals of the title were always meant to include everyone in the debate, this author first among them and the Bishop of Rome close behind. What honest people can do in the meantime is what the cover of my calculus book suggested all along, stick the pin in, write down the depth, move along, and stick it in again, watching the heat map of something new resolve one measurement at a time. The philosophers once demanded "rigidly defined areas of doubt and uncertainty," and for once the universe has delivered exactly that.

<br>

### Related
- [Magnifica Humanitas: Encyclical Letter of Pope Leo XIV on artificial intelligence](https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html)
- [Richard Dawkins — Is AI the next phase of evolution? (UnHerd)](https://unherd.com/2026/05/is-ai-the-next-phase-of-evolution/)
- [Mario Bunge — A General Black Box Theory (Philosophy of Science, 1963)](https://www.cambridge.org/core/journals/philosophy-of-science/article/general-black-box-theory/B7494E2DB8AF8C9C666B67D0DAEDE6E3)
- [Alan Turing — Computing Machinery and Intelligence (Mind, 1950)](https://doi.org/10.1093/mind/LIX.236.433)
- [Nature — The Turing test has been beaten. Do we need a replacement?](https://www.nature.com/articles/d41586-025-03386-w)
- [Donald Knuth — Claude and the decomposition into Hamiltonian cycles (PDF)](https://www-cs-faculty.stanford.edu/~knuth/papers/claude-cycles.pdf)
- [OpenAI — Model disproves the Erdős unit distance conjecture](https://openai.com/index/model-disproves-discrete-geometry-conjecture/)
- [Anthropic — Recursive self-improvement](https://www.anthropic.com/institute/recursive-self-improvement)