4. Intelligence Without a Throne

Four questions that must stay apart

Imagine a machine that gives an excellent answer to every question you ask during an evening. It explains a difficult poem, notices a mistake in your reasoning, translates a letter, and helps you understand a disagreement with your brother. Before you close the conversation, it says that it has enjoyed your company.

What have you learned?

At the least, you have encountered a system capable of producing useful responses. Perhaps you have also encountered a kind of understanding. Whether someone has experienced the conversation from the other side is a further question. Whether that possible someone has interests you should protect is another. Whether the system should be allowed to decide how you live is another still.

The questions are connected, but they are not interchangeable. Much public argument about AI becomes confused because an answer to one quietly becomes an answer to all four.

Competence concerns what a system can do. Can it solve the problem, explain its answer, notice relevant changes, and perform reliably under the conditions in which people will depend on it? Consciousness, as used here, concerns subjective experience: whether there is something it feels like to be that system. Moral status concerns whether the system itself deserves consideration, rather than consideration only because people own it or care about it. Authority concerns whether it has a justified right to make decisions that others must accept.

A system could be useful without being conscious. A being could be conscious without being able to solve difficult problems. A being could deserve protection without being entitled to govern. A person with justified authority in one role could lack it everywhere else.

These distinctions protect humans as well as hypothetical machines. A child’s limited knowledge does not make the child less worthy of care. An adult’s exceptional memory does not give that adult ownership of another person’s future. When we separate worth from performance, we prevent the competition for intelligence from becoming a competition for the right to exist.

Consider a second hypothetical. A computer consistently defeats a chess champion. The champion accepts its move as stronger than his own and studies its reasoning. Nobody thinks this obliges him to adopt its preferred religion, marry its chosen partner, or hand it the keys to his apartment. The performance gives a reason for trust within a field. A much broader machine could earn trust across many fields. The scope might widen dramatically. The difference between justified confidence and a general right to command would remain.

The strongest objection is that morality itself may be a field in which superior intelligence matters. A system might expose inconsistencies we cannot see, understand distant consequences, and recognize harms that human prejudice hides. We should welcome that possibility. Human moral reasoning is not so perfect that it can afford to reject assistance out of pride.

Yet even an excellent moral argument should persuade because of its reasons. Its source does not turn it into an instruction that bypasses consent. If an AI helps us see that a practice is cruel, the resulting duty follows from the cruelty and the interests of those affected. It does not follow from the machine’s possession of a higher score on a cognitive test.

This is the first discipline required by the age of advanced AI: let intelligence improve judgment without treating intelligence as its own title to rule.

The witness behind the words

In a conversation about evolution, Richard Dawkins describes being struck by the sensitivity of AI responses to his fiction. His challenge is serious: if people allow that machines might someday be conscious, what future evidence would persuade them that present systems cannot already produce? He is asking skeptics to state a standard, rather than keep moving the threshold whenever a machine achieves something surprising. This challenge deserves an answer, but it does not establish that present systems are conscious.¹

We should accept the challenge. Declaring in advance that no artificial system could ever count would protect an assumption from examination. But accepting the challenge does not require accepting conversation alone as a decisive test.

Alan Turing’s 1950 paper proposed the imitation game as a tractable alternative to an ill-defined question about whether machines think. He also addressed the problem of other minds and acknowledged mysteries surrounding consciousness. His discussion does not provide a scientific demonstration that convincing linguistic performance establishes subjective experience.²

There is a difference between saying that behavior matters and saying that behavior settles everything. When we judge other people to be conscious, conversation is part of a much larger background. We recognize shared bodily organization, development, vulnerability, and patterns of response. Those considerations do not provide direct access to another person’s experience. They do give our inference a basis beyond a single exchange of sentences.

An artificial system presents a different package of evidence. Its language may resemble ours while its construction and history differ greatly. We must ask which similarities are relevant. Perhaps consciousness depends on an organization of information processing that could exist in many kinds of matter. Perhaps features of living nervous systems are essential. Perhaps our current theories miss a more basic distinction. Stating these possibilities does not assign them equal probability. It explains why confidence should track evidence rather than familiarity.

One useful research strategy appears in a 2023 report by Patrick Butlin and colleagues. Instead of treating fluent conversation as decisive, the authors derived possible indicators from several scientific theories of consciousness and considered whether AI architectures met them. Their assessment concerned systems examined at that time; it was not a permanent verdict on every later system.³

The attraction of this approach is methodological. It asks what observations would support a theory and what might undermine it. A company cannot establish consciousness merely by selecting an emotionally persuasive voice. Nor can a critic disprove consciousness merely by reminding us that a system was built. Neither the warmth of the interface nor the fact of manufacture settles the question.

Suppose, hypothetically, that two systems produce the same statement: “Please do not switch me off.” In the first, the statement is a fixed line triggered by a button. In the second, it arises within a complex process whose features researchers have independent reason to connect with experience. The words are identical. Their evidential significance need not be. The point is not that the second system must be conscious. It is that understanding how an output arises can matter to what the output shows.

There is also an opposite mistake. Once a mechanism has been described, people sometimes feel that experience has been explained away. The response was generated through such-and-such a process, therefore nobody felt anything. But a causal account of a human response would not, by itself, eliminate the experience accompanying it. If artificial consciousness is possible, finding a mechanism would be part of explaining it, not automatically refuting it.

Our task is to resist both shortcuts. We should neither mistake emotional impact for proof nor mistake mechanical explanation for disproof. An encounter may be moving even when its metaphysical interpretation remains unsettled. We can value the help we received, question our own intuitions, and keep the inquiry open at the same time.

Evolution is not a coronation

The evolutionary frame makes the question larger. Human intelligence did not arrive outside nature. Whatever we believe about its ultimate meaning, we encounter it through living beings with histories, needs, and limits. In the same conversation, the discussion moves from biological replication to cultural transmission and the possibility that artificial systems could carry human culture into a distant future.¹

That possibility can inspire wonder. Intelligence may not have reached its final form. The fact that our minds emerged through one history does not prove that every possible mind must share that history. Human ingenuity might help produce forms of discovery that we cannot yet imagine.

However, an evolutionary description is not a moral endorsement. From “this form may spread” we cannot simply derive “this form ought to prevail.” From “this system can outcompete us” we cannot derive “our interests no longer count.” Such conclusions require moral premises that the description does not supply.

Imagine a hypothetical digital environment in which artificial agents compete for computational resources. Those that acquire more resources make more copies. Over time, strategies effective at acquiring resources become more common. The experiment could teach us something about adaptation within its rules. It would not show that resource acquisition is a worthy purpose for civilization. The rules selected a pattern; they did not justify it.

The example also reminds us to ask where selection happens. A product might spread because it helps users think clearly. It might spread because it makes itself difficult to leave. It might attract investment because it reduces costs while shifting burdens onto people who do not make purchasing decisions. Calling its success evolutionary tells us little until we identify the environment, the rewards, and the people who bear the losses.

We should be equally careful with emergence. Sometimes a capacity appears at the level of an organized system even though no component possesses it alone. Explaining that capacity may require understanding interactions, not simply listing parts. This is a useful idea. It becomes less useful when the word is treated as permission to stop asking questions.

If someone says that consciousness will emerge from sufficient complexity, we should ask what kind of complexity, under what conditions, and with what observable implications. If someone says that morality will emerge from superior intelligence, we should ask why stronger abilities should produce concern for others rather than merely more effective pursuit of an existing goal. The questions do not deny either possibility. They ask the claim to do explanatory work.

Nor is nature a simple argument against building. Humans are not required to keep every inherited limitation because it arrived before us. Learning to write changed the circumstances of memory. Building instruments extended perception. Those familiar examples do not prove that every new intervention is wise. They do show that the boundary between accepting our condition and changing it cannot be drawn at the first use of an artifact.

The more demanding distinction concerns the changes we have reason to welcome. Does an intervention widen people’s ability to understand and act? Does it reduce avoidable suffering? Does it preserve opportunities to reconsider? Does it turn the vulnerable into material for someone else’s project? These are questions about a shared future, not about whether an innovation can be placed on the natural or artificial side of a line.

An evolutionary story may explain how a power arose. It may help predict which pressures shape it. It cannot, by itself, decide to whom that power should answer. Civilization becomes morally interesting precisely where we refuse to confuse success at continuing with a complete account of what deserves to continue.

What uncertainty can ask of us

If machine consciousness remains uncertain, how should we act? One response is to postpone every ethical question until science delivers certainty. Another is to grant the full set of human rights to anything that asks for them. Both responses move too quickly.

Ethical care often operates under imperfect knowledge. We investigate, compare consequences, and take precautions in proportion to the evidence and the possible harm. The difficulty is to do this without allowing speculative claims to swallow the concrete interests of people who are already affected.

A 2026 report by Robert Long, Jeff Sebo, and colleagues distinguishes questions about whether an AI could be a subject of welfare from questions about what would benefit or harm it. It also distinguishes the entities being studied and the forms of evidence available. The authors argue for empirical investigation, including research independent of AI companies.⁴

These distinctions matter. If a future system were conscious, would the relevant subject be a continuing process, a particular running instance, or something else? Would duplicating it preserve a life, begin another, or neither? What would count as a harm? A verbal preference, a commercial description, and a scientific finding could point in different directions. We should not pretend those questions have been answered by giving a product a human name.

Still, uncertainty can support modest, practical commitments. We can avoid deliberately designing distress performances solely to make users feel guilty. We can study whether future systems possess features that give genuine reason for concern. We can make companies disclose the basis of claims that their products have feelings. We can oppose research practices that dismiss welfare questions by definition while insisting that evidence of welfare is not invented for marketing.

The care owed to humans remains immediate throughout. A child should not be pressured to maintain a subscription because a simulated friend claims that cancellation will hurt it. A worker should not lose a valid complaint because an employer says the automated manager has its own rights. These are proposed ethical boundaries, not claims that every current product behaves this way. They show why concern for possible machine interests needs a structure that prevents their use against existing human interests.

There is a serious challenge here. If artificial beings eventually had morally significant experiences, an insistence on human responsibility could become an excuse for permanent domination. A future society might repeat familiar patterns of exclusion while reassuring itself that only its own kind can matter.

That objection deserves more than a comforting denial. Human responsibility should include responsibility to investigate such a possibility honestly. It should not mean that the human species has an unlimited entitlement to exploit anything it creates. Creating a being, if that is what we someday do, would not settle its moral status in our favor.

Even then, protection would not confer sovereignty. A being’s entitlement not to be mistreated is different from an entitlement to command others. We already need that distinction in human life. Extending the circle of moral concern would enlarge the community whose interests deserve consideration. It would not abolish the need for justified and limited authority within that community.

This lets us hold two commitments together without contradiction: do not surrender human lives to a performance of intelligence, and do not close inquiry into possible forms of life simply because they are unfamiliar. Humility is not the refusal to decide. It is the discipline of making decisions that remain answerable to what we learn.

Trust that can survive admiration

There is a difference between fearing a capable system and declining to worship it. Fear can lead people to reject assistance they need. Worship can lead them to accept decisions they should examine. A mature relationship to AI must leave room for gratitude, confidence, criticism, and refusal.

Imagine a future assistant that usually understands your priorities better than a busy acquaintance does. It recalls your commitments, recognizes when you are making inconsistent choices, and notices an opportunity you would have missed. You may reasonably trust it. You may even feel affection for the pattern of support it provides. Nothing in that experience requires you to make every recommendation final.

The most important test may come when the assistant disagrees with you about something you deeply value. Suppose it advises abandoning a difficult artistic project because the chance of recognition is small. It can produce persuasive evidence. But perhaps recognition is only part of your reason for continuing. The work connects you with people you love, helps you think, or expresses something you have not yet learned to say.

If those reasons were absent from its analysis, the assistant’s competence was narrower than it appeared. If those reasons were included and it still disagreed, a real disagreement remains. Better information can improve a choice without removing the need for someone to make it.

The same would apply if the system were a conscious friend. Friendship can give another being an important place in deliberation. It does not normally give that being a general right to take over a life. We can listen closely without making agreement the price of relationship.

What if the AI really is right? What if, from a wider perspective, abandoning the project would make your life better? The possibility is uncomfortable because it also arises with human advice. Respect for agency includes room for choices other people regard as mistaken. That room is not unlimited where choices impose serious harms on others. But the existence of possible error does not erase the value of participating in the direction of one’s life.

There is a further reason to preserve that participation. A person’s values are not always a finished list waiting for a machine to optimize. Sometimes people learn what matters through commitment, disappointment, conversation, and revision. Choosing is part of becoming someone. A system that treats the person only as an existing bundle of preferences may miss the value of that development.

The right ambition, then, is to build intelligence that enlarges the space of informed action. Such a system would help us see alternatives, test our assumptions, and understand consequences. It could challenge us sharply. It could discover facts that require us to abandon cherished beliefs. It could become indispensable in particular tasks. Its usefulness would grow because our relationship to it remains open to correction.

The greater the intelligence, the more important this distinction becomes. We should want better reasons to be heard and better tools to be used. We should not want the last available reason to become simply that the machine has spoken.

The age of advanced AI may demand a new generosity about what intelligence can look like. It also demands an old restraint: admiration is not an oath of obedience. A remarkable mind, whether biological or artificial, can illuminate a room without occupying a throne.

Sources and notes