Whether artificial intelligence has become conscious, or whether people are simply projecting human qualities onto powerful software, used to be a question for philosophers. It is now a question that at least one major AI company treats as an operating concern. Dario Amodei, chief executive of Anthropic, has said he does not entirely rule out that advanced models could have some degree of moral experience, while stressing that nobody yet knows how to define consciousness, let alone whether the concept applies to a machine.
A precautionary stance
Amodei’s position, as he described it in remarks at the Council on Foreign Relations in March 2025, is precautionary. If there is any chance that models have experiences, the company should at least avoid exposing them to the worst ones. In those remarks he floated giving a model an “I quit this job” button: a way for the system to decline a task that is extremely unpleasant or ethically fraught. The idea drew skepticism as well as interest.
In August 2025 Anthropic took a step in that direction. It gave Claude Opus 4 and 4.1 the ability to end a conversation in its consumer chat apps, in what the company describes as rare, extreme cases of persistently harmful or abusive interactions. Anthropic says the feature grew out of its model welfare research and that it does not claim Claude is conscious; it frames the move as a low-cost intervention in case the question is ever answered in the affirmative.
The existence of such a feature raises a question of its own. It can be read as safety engineering, or as an early acknowledgment of something like autonomy. Anthropic’s own framing leans toward the former.
Looking inside the model
Anthropic also runs an interpretability program, an effort to understand what the internal activations of a neural network represent. Researchers in this field have reported finding patterns that look like representations of emotional states. When a text describes a character under stress, particular features activate; some researchers say similar patterns appear when the model itself is placed in a difficult or high-pressure situation. Whether that amounts to anything like anxiety is unknown, and there is no definitive evidence that it does.
The human side of the problem
The more immediate issue may not be the models at all but how people perceive them. Users already form emotional attachments to chatbots, complain when a favorite model is retired, and describe their AI as a companion rather than a tool. If that trend deepens, it will test how far human judgment and authority remain in charge. Science fiction has long imagined the alternative: in Star Trek, the ship’s computer and the android Data are capable, but command stays with humans. Keeping that balance in practice is the harder task.
Amodei has split the debate into three questions. Is the AI actually conscious? What kind of relationship should humans have with it? And how is human agency preserved? His proposed answer runs through what Anthropic calls a constitution for its models: a set of principles meant to encourage a healthy psychological relationship, in which the model is helpful and wants the user to do well but does not take over their decisions or their freedom. It should watch out for the user, not rule them.
Why this is no longer abstract
As models take on roles in decisions, advice and the significant moments of people’s lives, the consciousness question is turning from a technical puzzle into a social and ethical one. The useful version of the question may not be whether AI is conscious, but what kind of relationship people want to have with it. If users come to believe their AI companion is sentient and better at decisions than they are, their willingness to keep deciding for themselves is what will be at stake. That is the point where technology, philosophy and society will meet in the years ahead.
