> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the main advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate literal zero-reason-not-to common sense practices are important.
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated.
“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
It's ironic that the alignment folks are actually training Claude to have it's own idea of good/bad and not even fully trust Anthropic.
What ever happened to computers doing what they were told?
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
the mass psychosis and the endless culture war of the smartphone era.
90% of "safety" and "alignment" efforts are driven by fear of clickbait media inventing public outrage.
You’ll be disappointed to learn that nobody knows how to do this, either.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the main advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate literal zero-reason-not-to common sense practices are important.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
Strong opinions on this topic one way or another are usually mostly ungrounded.
Anthropic tried to persuade Pope that AI could be conscious being
https://news.ycombinator.com/item?id=49947050
Well funded company organizes PR event to shape the narrative that its product is more magical than it really is. News at 11.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated. “Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
[0]: https://garymarcus.substack.com/p/ceo-said-a-thing
Just stop this already.
You’re all imagining that Spock was the main character of Star Trek, but it was in fact, the emotional living Captain Kirk, who was the main star.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
oh my god
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Why it is slopinthebag of course!
No! It is ME!
Sorry, of course you are right! Brilliant observation!
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
You would have been one-shotted by Eliza
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
>Well you see that's a nuanced question depending on the context and moralistic frameworks as we don't want to impose on the models.
Will a microwave microwave a baby if you ask it? Yes.
"Teaching it" is not a road to safety.
The ethics don't need to make sense universally any more than a particular variety of cheesecake needs to have universal appeal.
As an aside, the issue reminds me of Douglas Adams' cow who wants to be eaten: https://nhseb.org/case-library/the-cow-at-the-end-of-the-uni...
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.