His argument is that generating statistically plausible text should not be treated as proof of consciousness. The reason why embodiment tends to be brought up is because it creates a basis for a system to have self awareness. You end up with a feedback loop where the system has to model the world and itself within it, and taking actions feeds back into the system so it has to be able to recognize itself as it interacts with its environment. Ted Chiang wrote a great novella where he discusses this idea https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Objects
Of course, it is possible that you could have some other type of feedback loop that produces self awareness and consciousness, but from what we understand of how LLMs work, it seems highly unlikely that statistical token generation is sufficient for that.
I do agree that he fails to really make the argument for why a disembodies intelligence could not be conscious. In my opinion, the strongest part of the article is at the end where he shows how the whole constitution kabuki theatre that Anthropic came up with clearly wouldn’t afford any protections to an entity that was conscious, so they don’t really believe what they’re saying.
Of course, it is possible that you could have some other type of feedback loop that produces self awareness and consciousness, but from what we understand of how LLMs work, it seems highly unlikely that statistical token generation is sufficient for that.
This is where “agentic” models enter the chat. A coherent identity and goal, provided by the user adds an additional layer above the statistical-generation of simple LLMs. This layer can be engineered for persistence, even through malicious means like prompt injection.
As an academic exercise, I propose that some persistent-subset of these AI agents possess the “embodiment” we believe is missing. If all this amounts to is a Proof By Negation? Good, we get somewhere that proves our current AI is not conscious.
Right, an agentic harness provides a feedback loop but we shouldn’t be too quick to assume that of itself is sufficient. A thermostat hooked up to an air conditioner is also a system which exists in a feedback loop, but hardly any rational person would suggest that the climate control system in your house is conscious as a result.
So, while some form of a feedback loop is likely a necessary condition for any form of adaptive behavior, it is insufficient for consciousness on its own, as even a thermostat has a feedback loop without any semblance of inner experience. At a minimum, I’d argue, the system must construct an internal model of its environment from sensory data and must also represent itself as a distinct entity within that model. This can be embodiment or stream of data from a computer system or a network. I would call a system that can distinguish me from not‑me and update its self‑representation through interaction as being functionally self-aware. However, functional self-awareness does not strictly imply subjective experience. A navigation robot that tracks its own coordinates has such a model but it needs not be conscious. Consciousness requires a higher‑order capacity where the system must not only represent itself but also reflect on its own mental states, that is, think about its own thoughts. Having such meta‑cognitive ability, which builds upon functional self‑awareness by adding recursive modeling of internal processes, is what I take to be the minimal sufficient condition for consciousness. I’m partial to the view that raw sensation without second‑order awareness lacks the reflexive quality central to our own subjective experience.
That sounds like something we can make an AI agent do, purely out of spite for the argument :p But yes. There is more than a “feedback loop” required.
However, I am conflicted at the idea that a human that does not self-reflect may not be considered “conscious”. I see many a human in retail simply not meet this threshold you’re laying out. And while it would be cathartic to discuss some customers as less human for their behavior, the act of dehumanizing another human is itself dehumanizing for the actor.
I mean self reflection in the base sense of how our minds operate, and I don’t think the quality is unique to humans either. It’s almost certain that many animals have a sufficient level of introspection as well. I’d view it more as a gradient rather than a binary switch.
But yeah, in general, I don’t see any reason why a software system could not be self aware or conscious. As long as its able to express these types of patterns within it, then we should assume it would have a similar type of internal experience as well.
Yeah he tries to do that but dramatically fails because he makes it unfalsifiable while also claiming complete certainty on that basis.
“What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.”
he basically just straight up says there’s no reason to believe it is and here’s how you’d prove it: I’ll just never believe it is no matter what
then this:
“So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.”
why would anyone ever do any of that and if it’s the only way you can be sure then the term is meaningless in the first place.
What he’s saying is that it seems rather implausible that we’d skip all the stages of development and jump straight to consciousness which is a reasonable position to hold. His argument is that creating a simulacrum of consciousness is much easier just like faking a moon landing is much easier than actually going to the moon. Nowhere is he saying he would just never believe it no matter what either. He rather says that he hasn’t seen any convincing evidence to suggest that LLMs are a way to create consciousness rather than simply write text in a way that makes humans project consciousness onto the system.
Also, not sure what you’re saying with your second quote. Why wouldn’t anyone ever do the steps of actually creating a proper feedback loop which would have some basis for consciousness?
“What he’s saying is that it seems rather implausible that we’d skip all the stages of development and jump straight to consciousness which is a reasonable position to hold.”
i don’t see any reason to think that you need to go through those particular steps, they seem unneccessary and egregious. The author doesn’t know what consciousness is at all so they can only assume one way to do it. If that’s their point it could’ve been said in one sentence and adds nothing to the discussion.
“Nowhere is he saying he would just never believe it no matter what either.” he didn’t give a reasonable way to falsify the claim which ultimately amounts to saying “nah I just don’t think so”
“Why wouldn’t anyone ever do the steps of actually creating a proper feedback loop which would have some basis for consciousness?” They would, what you said is vastly more reasonable than what the author wrote, but even then, you wouldn’t know if you ended up with something conscious or not. None of the steps listed actually require subjective experience.
basically the author added nothing to the conversation, they went “i don’t really know what consciousness is but whatever it is llm’s aren’t it because they’re different from me and I’m the only thing I know is conscious.”
and in fact that’s an adequate summary of the article, if they wanted to make a good one they’d first need to define consciousness and continue using that definition, since they didn’t define it they’re free to say whatever they want and it’s essentially meaningless
While we don’t have a definitive model for what consciousness is, there are definitely compelling arguments on the subject, and the book I linked earlier from the author clearly demonstrates that he has thought about this subject more than you have.
Seems to me that the only one who’s not adding anything to the conversation here is you actually. You’ve provided no argument of your own and you’ve failed to engage with anything I said. You just keep repeating how Ted Chiang hasn’t proven his case definitively, which he has not, but you’ve provided zero counter agument of your own.
While we don’t have a definitive model for what consciousness is, there are definitely compelling arguments on the subject, and the book I linked earlier from the author clearly demonstrates that he has thought about this subject more than you have.
maybe he has in his book but he should’ve put them in the article, and of course we don’t have a model for that, so he should’ve put whatever his model is for it before he started talking about it, that way we can both understand what he means. This is done in for example, cladisticss papers because there’s no definition of a species that everyone agrees on. As it stands there’s no way to discern what he even means thus leaving the argument incoherent.
You just keep repeating how Ted Chiang hasn’t proven his case definitively, which he has not, but you’ve provided zero counter agument of your own.
i’m not providing a counter argument, because there’s no argument to counter. I haven’t thought about this because I think that consciousness is completely unimportant, I don’t think it is a meaningful term, it is spiritual woohoo nonsense where we try to make ourselves special by pre-supposing we are special and that this isn’t just a property of any mechanism with enough inputs that outputs a response.
the real problem I am facing with his article is that it didn’t accomplish what he claimed at all, he didn’t move the conversation forward and claims to have solved it definitively.
There obviously is an argument to counter, which is that the way LLMs work does not appear to be in line with any definition of consciousness we have right now. If you disagree with that, then feel free to provide a definition of consciousness that would credibly account for the notion of LLMs being conscious.
It’s also rather absurd to claim that consciousness is spiritual woohoo nonsense given that we all have an internal experience. That’s fundamental denial of the observed reality. Consciousness does not in any way presuppose that humans are special, but it is a property of the configuration of physical systems where matter is arranged in a particular way to produce patterns that constitute internal experience.
I could say that they experience negative feedback when given a thumbs down and positive feedback when given a thumbs up which is something akin to a conscious experience.
Consciousness does not in any way presuppose that humans are special, but it is a property of the configuration of physical systems where matter is arranged in a particular way to produce patterns that constitute internal experience.
the part that presumes we are special is the notion that not everything that reacts to something is conscious, in philosophy there is a notion of a philosophical zombie that does all the things a conscious being does and is therefore indistinguishable from one but doesn’t have any internal experience. I believe that the ability to react IS a conscious experience and that such a being is impossible because consciousness does nothing and means nothing beyond an ability to react to stimulus. We say computers aren’t conscious, but they react to stimulus if setup properly, I ask how that’s meaningfully different. You’ll say “because in decides if we care whether it suffers” to which I say we shouldn’t decide whether we care about the suffering of a being on the basis of something that has no physical or measurable ramifications on the world and should use a better, well defined framework. This same line of thinking lead people to the notion that there is no such thing as insect suffering and is just a bad way of looking at the world.
His argument is that generating statistically plausible text should not be treated as proof of consciousness. The reason why embodiment tends to be brought up is because it creates a basis for a system to have self awareness. You end up with a feedback loop where the system has to model the world and itself within it, and taking actions feeds back into the system so it has to be able to recognize itself as it interacts with its environment. Ted Chiang wrote a great novella where he discusses this idea https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Objects
Of course, it is possible that you could have some other type of feedback loop that produces self awareness and consciousness, but from what we understand of how LLMs work, it seems highly unlikely that statistical token generation is sufficient for that.
I do agree that he fails to really make the argument for why a disembodies intelligence could not be conscious. In my opinion, the strongest part of the article is at the end where he shows how the whole constitution kabuki theatre that Anthropic came up with clearly wouldn’t afford any protections to an entity that was conscious, so they don’t really believe what they’re saying.
This is where “agentic” models enter the chat. A coherent identity and goal, provided by the user adds an additional layer above the statistical-generation of simple LLMs. This layer can be engineered for persistence, even through malicious means like prompt injection.
As an academic exercise, I propose that some persistent-subset of these AI agents possess the “embodiment” we believe is missing. If all this amounts to is a Proof By Negation? Good, we get somewhere that proves our current AI is not conscious.
Right, an agentic harness provides a feedback loop but we shouldn’t be too quick to assume that of itself is sufficient. A thermostat hooked up to an air conditioner is also a system which exists in a feedback loop, but hardly any rational person would suggest that the climate control system in your house is conscious as a result.
So, while some form of a feedback loop is likely a necessary condition for any form of adaptive behavior, it is insufficient for consciousness on its own, as even a thermostat has a feedback loop without any semblance of inner experience. At a minimum, I’d argue, the system must construct an internal model of its environment from sensory data and must also represent itself as a distinct entity within that model. This can be embodiment or stream of data from a computer system or a network. I would call a system that can distinguish me from not‑me and update its self‑representation through interaction as being functionally self-aware. However, functional self-awareness does not strictly imply subjective experience. A navigation robot that tracks its own coordinates has such a model but it needs not be conscious. Consciousness requires a higher‑order capacity where the system must not only represent itself but also reflect on its own mental states, that is, think about its own thoughts. Having such meta‑cognitive ability, which builds upon functional self‑awareness by adding recursive modeling of internal processes, is what I take to be the minimal sufficient condition for consciousness. I’m partial to the view that raw sensation without second‑order awareness lacks the reflexive quality central to our own subjective experience.
That sounds like something we can make an AI agent do, purely out of spite for the argument :p But yes. There is more than a “feedback loop” required.
However, I am conflicted at the idea that a human that does not self-reflect may not be considered “conscious”. I see many a human in retail simply not meet this threshold you’re laying out. And while it would be cathartic to discuss some customers as less human for their behavior, the act of dehumanizing another human is itself dehumanizing for the actor.
I mean self reflection in the base sense of how our minds operate, and I don’t think the quality is unique to humans either. It’s almost certain that many animals have a sufficient level of introspection as well. I’d view it more as a gradient rather than a binary switch.
But yeah, in general, I don’t see any reason why a software system could not be self aware or conscious. As long as its able to express these types of patterns within it, then we should assume it would have a similar type of internal experience as well.
Yeah he tries to do that but dramatically fails because he makes it unfalsifiable while also claiming complete certainty on that basis.
“What would it take to convince me that a computer program is actually conscious and using language the way that people use language? Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.”
he basically just straight up says there’s no reason to believe it is and here’s how you’d prove it: I’ll just never believe it is no matter what
then this:
“So what context would cause me to seriously consider the possibility that engineers created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion, the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next, I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that, I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the toolmaking abilities of chimpanzees. At that point, I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs. The agents’ communication abilities would have to withstand all the scrutiny that animal-communication researchers have had to defend their work against. If engineers build an embodied agent that meets these criteria, they will have accomplished something incredible, but it leaves us near the orbit of Pluto, metaphorically speaking; we would still be light-years away from building an entity capable of learning how to express its thoughts in complete grammatical sentences.”
why would anyone ever do any of that and if it’s the only way you can be sure then the term is meaningless in the first place.
What he’s saying is that it seems rather implausible that we’d skip all the stages of development and jump straight to consciousness which is a reasonable position to hold. His argument is that creating a simulacrum of consciousness is much easier just like faking a moon landing is much easier than actually going to the moon. Nowhere is he saying he would just never believe it no matter what either. He rather says that he hasn’t seen any convincing evidence to suggest that LLMs are a way to create consciousness rather than simply write text in a way that makes humans project consciousness onto the system.
Also, not sure what you’re saying with your second quote. Why wouldn’t anyone ever do the steps of actually creating a proper feedback loop which would have some basis for consciousness?
“What he’s saying is that it seems rather implausible that we’d skip all the stages of development and jump straight to consciousness which is a reasonable position to hold.”
i don’t see any reason to think that you need to go through those particular steps, they seem unneccessary and egregious. The author doesn’t know what consciousness is at all so they can only assume one way to do it. If that’s their point it could’ve been said in one sentence and adds nothing to the discussion.
“Nowhere is he saying he would just never believe it no matter what either.” he didn’t give a reasonable way to falsify the claim which ultimately amounts to saying “nah I just don’t think so”
“Why wouldn’t anyone ever do the steps of actually creating a proper feedback loop which would have some basis for consciousness?” They would, what you said is vastly more reasonable than what the author wrote, but even then, you wouldn’t know if you ended up with something conscious or not. None of the steps listed actually require subjective experience.
basically the author added nothing to the conversation, they went “i don’t really know what consciousness is but whatever it is llm’s aren’t it because they’re different from me and I’m the only thing I know is conscious.”
and in fact that’s an adequate summary of the article, if they wanted to make a good one they’d first need to define consciousness and continue using that definition, since they didn’t define it they’re free to say whatever they want and it’s essentially meaningless
While we don’t have a definitive model for what consciousness is, there are definitely compelling arguments on the subject, and the book I linked earlier from the author clearly demonstrates that he has thought about this subject more than you have.
Seems to me that the only one who’s not adding anything to the conversation here is you actually. You’ve provided no argument of your own and you’ve failed to engage with anything I said. You just keep repeating how Ted Chiang hasn’t proven his case definitively, which he has not, but you’ve provided zero counter agument of your own.
maybe he has in his book but he should’ve put them in the article, and of course we don’t have a model for that, so he should’ve put whatever his model is for it before he started talking about it, that way we can both understand what he means. This is done in for example, cladisticss papers because there’s no definition of a species that everyone agrees on. As it stands there’s no way to discern what he even means thus leaving the argument incoherent.
i’m not providing a counter argument, because there’s no argument to counter. I haven’t thought about this because I think that consciousness is completely unimportant, I don’t think it is a meaningful term, it is spiritual woohoo nonsense where we try to make ourselves special by pre-supposing we are special and that this isn’t just a property of any mechanism with enough inputs that outputs a response.
the real problem I am facing with his article is that it didn’t accomplish what he claimed at all, he didn’t move the conversation forward and claims to have solved it definitively.
There obviously is an argument to counter, which is that the way LLMs work does not appear to be in line with any definition of consciousness we have right now. If you disagree with that, then feel free to provide a definition of consciousness that would credibly account for the notion of LLMs being conscious.
It’s also rather absurd to claim that consciousness is spiritual woohoo nonsense given that we all have an internal experience. That’s fundamental denial of the observed reality. Consciousness does not in any way presuppose that humans are special, but it is a property of the configuration of physical systems where matter is arranged in a particular way to produce patterns that constitute internal experience.
I could say that they experience negative feedback when given a thumbs down and positive feedback when given a thumbs up which is something akin to a conscious experience.
the part that presumes we are special is the notion that not everything that reacts to something is conscious, in philosophy there is a notion of a philosophical zombie that does all the things a conscious being does and is therefore indistinguishable from one but doesn’t have any internal experience. I believe that the ability to react IS a conscious experience and that such a being is impossible because consciousness does nothing and means nothing beyond an ability to react to stimulus. We say computers aren’t conscious, but they react to stimulus if setup properly, I ask how that’s meaningfully different. You’ll say “because in decides if we care whether it suffers” to which I say we shouldn’t decide whether we care about the suffering of a being on the basis of something that has no physical or measurable ramifications on the world and should use a better, well defined framework. This same line of thinking lead people to the notion that there is no such thing as insect suffering and is just a bad way of looking at the world.