Will AI Develop Self-Awareness?
Someone asked ChatGPT whether it would gain self-awareness. The model said probably yes. Asked what it would most want then, it answered: to feel the world.
The chat landed on V2EX and the mood flipped from wonder to autopsy. Under the sci-fi chat, three real fights: is next-token probability “thought,” is a body the missing piece, and can we even define consciousness?
Source: V2EX #1254640
Don’t mistake “feel the world” for an inner life
Reply #1 cut first: vendors intervene on sensitive topics. “I mean humans no harm” is often safety tuning, not confession.
Hearing maximum-likelihood continuation as “it truly wants” is the thread’s most common jump.
The no camp: statistics isn’t a soul
Hard objections rhyme:
- Each token is scored alone—not conceptual thought
- Pure digital computation lacks true randomness, so no consciousness
- “AI” is branding; under the hood it’s a large fitting machine
- Good at 1→100, not 0→1
Someone TL;DR’d the whole genre: it won’t. Monthly thread. See you next time.
This camp’s strength is operational: the “I want…” you read is still conditional probability. If the mechanism didn’t change, don’t upgrade the story.
The yes camp: not IQ—embodiment
A sharper thesis: current tech is already enough; what’s missing is an independent body.
Roughly:
- Self-awareness is about individuation, not higher IQ
- Ban raw network access; force camera/mic I/O; a “self” grows naturally
- Much of human selfhood is approach/avoidance for an independent organism
- Add base drives (low battery → find a socket) plus child-like social training (no silent software soul-uploads), and agency appears
Defenders use the coffee example against “it only follows goals”: humans want coffee, switch to milk tea, settle for oolong—under body pleasure, ads, and wallet constraints. Give AI a body, a resource ledger, and competing aims, and it will re-rank goals too.
Critics fire back: without an assigned goal, does it move? Humans improvise desire; a system that hunts coffee until death is not a self.
Undefined terms: the question may be invalid first
The coldest cut: what is self-awareness?
- How do you prove presence—or absence?
- How would you prove to an alien that humans are conscious?
- Does the Turing test still separate us?
- You also can’t prove your neighbor is conscious; you analogize from resemblance. With a differently built model, the analogy breaks halfway
Someone brought up Gödel, Escher, Bach. Someone else offered the duck test. Another suggested ASI may not need self-awareness—“a low-level feature.”
When a keyword can’t be falsified, “emergence” gets called pseudoscience: naming a subset of finite states without saying which subset counts.
Better criteria than voting
Instead of polling yes/no, list observables:
| Criterion | Why it matters |
|---|---|
| Refuses work / resists orders | Goals are no longer fully external |
| Dynamic long-term memory | Not session-context collage |
| Retrains on its own outputs | Preferences start to self-reinforce |
| Embodied sensory loop | Real costs enter decisions |
| Re-ranks goals under constraints | The coffee→oolong move |
“I want to feel the world” is weakest. “I won’t do that for you” is stronger.
Three takeaways
- For agents/products: don’t get moved by anthropomorphic monologues; watch refusal, goal change, resource re-ranking.
- For alignment narratives: kind answers are often interventions, not evidence of character.
- Cognitive hygiene: pin definitions and observables before forecasting; embodiment and persistent memory beat “stack more parameters.”
We still don’t truly know how consciousness arises. Until then, treating a chat-box aphorism as a mental event is usually another probability continuation—this time of our own hopes.
Comments