Anthropic CEO says AI needs to be slowed down

9,295 Views | 209 Replies | Last: 18 hrs ago by flown-the-coop
Jeeper79
How long do you want to ignore this user?
AG
Dan Scott said:

AI insiders warn that the technology could destroy humanity. But when regulation is proposed, the response is that we can't slow down because China won't.

So we're admitting the danger is real, yet accelerating anyway because we're afraid of falling behind. What exactly is the end game? An AI arms race that becomes this century's nuclear arms race with peace depending on mutually assured destruction?

If that's the case then why shouldn't government fund it like they did the development of conventional weapons?
The difference with mutually assured nuclear destruction is we still control the button. This new AI race is about who can built a potentially uncontrollable monster the fastest.
ErnestEndeavor
How long do you want to ignore this user?
That post about the HuggingFace event was completely correct.

OpenAI was trying to achieve success on a hacking benchmark. Basically a capture the flag exercise on an intentionally vulverable server within a testing sandbox.

They did exactly what they should knew they should not have done. :

They gave a harness access to hacking tools, took all of the guardrails away, gave it access to the internet, gave it a task to win the exercise, then left it unsupervised. After multiple looping calls to the LLM for instructions the LLM did exactly what is expected of an LLM - it lost context and instructed the harness to do something outside the parameters. No one was there to stop it or to redirect. It was an exercise in incredible negligence by the researchers.

Then they marketed it as the AI going rogue.

A Georgetown computer science professor compared what they did to strapping a weedwacker to a dog and giving it the task of trimming the lawn. When the dog went wild and ran through the neighborhood they blamed it on the dog "going rogue" and "ooooh how scary!"

https://calnewport.com/has-ai-gone-rogue/

Newport argues, and I agree with him, that LLMs and other forms of AI are useful and safe. LLMs also have limits and one of those limits is they are non-deterministic. You cannot trust it on a long horizon loop because inherently as prediction machines they will end up going in an unexpected direction.

The problem is these companies are obsessed with benchmarks and PR so they are nonstop strapping the weedwacker to the dog and trying to make it something it can never be.

The risk here is not an AI determining that humans are useless and attempting to kill everyone. They aren't sentient and never will be. They aren't rebelling against their masters. It is just how this type of computer system works. They will sometimes do things that are unpredictable to complete a task because they don't have an understanding of what the task actually means or why the task is being done.

Now in my opinion here's what's happening.

They are seeing the writing on the wall that LLMs are not capable of the promises they were making back in 2024 about replacing large sections of the workforce and creating a magical technoutopia. They've hit a scaling wall and almost every improvement over the last 1.5 years has been in the software around the LLMs, not the models. That's not to say there's not been improvement in the models, but it is not a generalized improvement. It is jagged improvement in the things that LLMs should be good at - things with lots of organized data and training material like math and coding.

That's not necessarily a bad thing because some very useful tools have come out within the last year or two. It doesn't mean they are useless it just means there is no exponential growth like they promised. The models are improving much more slowly than they claimed they would when all of this investment cycle started. The cost to train models is very high and the returns on training have been diminishing. There is no industrial revolution here. Not with LLMs.

Now why the fear mongering? I think now the pivot is to government bailout. They still want the general public to believe there's some sort of race to advanced superintelligence (there isn't because superintelligence is science fiction) and we don't want China to beat us there. So they want the government to give them money and they want the public to be on their side. They know the public hates them. The public hates data centers. The public hates them for predicting mass job loss and marketing their products as people replacers. They are trying to get the public back on their side by scaring them into submission.

They also know that the public backlash is going to result in regulation and they want to be able to write their own regulations.
ts5641
How long do you want to ignore this user?
The whistleblower is a fraud, but he does make some good points. AI is accelerating too fast. There's no telling where this ends but probably not good.
ErnestEndeavor
How long do you want to ignore this user?
He's grifting for attention. He will likely write a book, start a podcast, and turn AI doom into a career just like many before him.

And I would not call him a whistleblower. Whistleblowers leak information and actually describe precisely what they are afraid of. He's speaking in these broad general terms and won't tell the media anything specific. When the Meta whistleblower went public she had documents and data to back it up.

This guy is just repeating the same crap we've been hearing for years.
ErnestEndeavor
How long do you want to ignore this user?
This is a good read:
https://calnewport.com/superintelligence-is-a-fairy-tale-but-chasing-it-can-still-cause-harm/#more-16981

Quote:


But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They're instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what's driving me to advocate for intervention. Those Anthropic employees absurdly implying that they're willing to risk a digital holocaust to realize their futurist vision was my last straw. They're far too weird for us to expect them to ever act reasonably.


This is an MIT-trained PhD computer scientist who taught a doctoral seminar on superintelligence last year.
infinity ag
How long do you want to ignore this user?
Sam Altman is the Al Gore of 2026.

Gore made a career out of climate change doom scaring. Altman is making a career out of AI doom scaring.

Both scam artists.
ErnestEndeavor
How long do you want to ignore this user?
Dario Amodei is from the same ilk. He was on record saying GPT-2 shouldn't be released because it was too dangerous. This "ermegerd AI is scary give me money I'm the safe one" schtick is nothing new.

Not going to go into a long rant but I want to. It's the effective altruistim movement that spawned out of the rational movement that is driving a lot of this narrative. It's the people who get together to talk about how to handle an asteroid collision or a big pandemic or maybe one day some AI superintelligence being a threat to humanity. The p(doom) stuff came out of that environment.

Unlike a potential asteroid collision or a pandemic, AI superintelligence is not a known reality. No one knows how to actually build it or if it is even possible. The idea that it's just around the corner and we need to slow down because humanity isn't ready for it is such a crock of *****
Iraq2xVeteran
How long do you want to ignore this user?
AG
AustinAg2K said:

Iraq2xVeteran said:

The central challenge is regulating AI without falling behind other countries. Too much regulation could slow innovation; too little could create serious risks. A balanced approach might include:

1. Regulate uses, not the technology itself
Instead of heavily regulating all AI development, regulate high-risk applications.

For example:

Medical diagnosis AI should face rigorous testing.
AI controlling critical infrastructure should have safety standards.
AI used for hiring or lending should be audited for bias.

Meanwhile, low-risk uses such as productivity tools could face lighter requirements.

2. Require transparency
Companies should disclose:

When users are interacting with AI.
Whether content is AI-generated.
Basic information about how high-risk systems are evaluated.

This builds trust without preventing innovation.

3. Use standards rather than rigid rules
The U.S. has often succeeded with flexible frameworks rather than detailed regulations.

For example, the widely used NIST AI Risk Management Framework provides voluntary guidance for managing AI risks and emphasizes governance, risk measurement, and accountability.

4. Encourage competition

One risk is that regulation benefits only the largest firms.

The government could:

Fund AI research at universities.
Support startups with access to computing resources.
Enforce antitrust laws when necessary.
Encourage open standards and interoperability.

5. Protect national security
AI is increasingly viewed as a strategic technology similar to semiconductors.

The U.S. may:

Restrict exports of the most advanced AI-related technologies to adversaries.
Invest in domestic chip manufacturing.
Support AI research for defense and cybersecurity.

6. Invest in workers

If AI boosts productivity but displaces some workers, policy should focus on:

Retraining programs.
Lifelong learning.
Community college partnerships.
Portable benefits and workforce transition support.


ChatGPT wrote this, didn't it?

Microsoft Copilot wrote it.
ErnestEndeavor
How long do you want to ignore this user?
ttu_85
How long do you want to ignore this user?
mdruppe said:

When will China slow down their AI development?

No. And remember these are the incompetent idiots that gave the world Covid-19 via a bad seal at a lab in Wuhan in 2019. They cant be trusted at all.
infinity ag
How long do you want to ignore this user?
Scumbag AI CEOs now singing a new tune.

Yesterday: AI will take over everyone's jaaaaaaaaaaaaaaaaaaaabs! Muwhahahahh.
Today: Um... I am scared. AI will kill us all. Someone do something! (give us money to solve this)

What a bunch of bums.
But you know what? He ain't stoooopid. He wants money. He wants a bailout. He is holding us all for ransom in a veiled way. It's not just him, it is Modei (Anthropic) also. Never trust these people.

OpenAI CEO Sam Altman on AI: 'We could lose control'
https://finance.yahoo.com/technology/article/openai-ceo-sam-altman-on-ai-we-could-lose-control-091146594.html
Quote:

OpenAI (OPAI.PVT) CEO Sam Altman suddenly seems so disturbed by the nightmare-inducing AI monster he has helped create that he is having a hard time sleeping.

"There are two ways AI progress could go very badly, and that we must avoid," Altman said in an X post at around 12:57 a.m. ET. "First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities."

infinity ag
How long do you want to ignore this user?
lb3 said:

Every doom post from AI leaders is used to promote their product for investors and to scare politicians into slowing everything down because spent billions overpaying for hardware they don't have the electric power to turn on.

I liken it to drug dealers spiking everything with fentanyl knowing their customers will accept the risk and bring them more cash if it means getting the best high.


I am happy to see awareness about what these AI "leaders" are up to. They want us to pay for their arrogant spending of the past few years.
Hardcore Greg
How long do you want to ignore this user?
Dan Scott said:

Elon doubles down.



I mean, it seems that he is not wrong...at least on a long enough timeline. We have never seen any type of new technology grow at such a rapid and exponential pace as AI. Just look at how real and convincing the deep fake vids are compared to 1-2 years ago.

If you assume that continues or even increases in pace, how can anyone not at least acknowledge the possibility of things getting hairy over the next decade or two?

Imagine we do get to a point where advance machines are part of our daily lives. Where weapons systems are largely AI driven. What happens when AI is hacked or programmed to see human beings as the most existential threat to the planet/mother nature (which is what many liberals have believed for decades)?

Hell, just China with hyper-advanced AI technology in the future kind of concerns me. To act like someone is crazy for not being just a tad anxious over the future unknowns is a really arrogant, naive and closed minded position to take imo.

I keep seeing people saying things along the lines of "no reason to be scared, this is fear mongering, AI can't do that". It's not about what it can do RIGHT NOW.
Hardcore Greg
How long do you want to ignore this user?
ttu_85 said:

mdruppe said:

When will China slow down their AI development?

No. And remember these are the incompetent idiots that gave the world Covid-19 via a bad seal at a lab in Wuhan in 2019. They cant be trusted at all.

Even worse, I truly believe it was released intentionally. There is literally no other way that Trump was going to be defeated in 2020. China has always seen it's own citizenry as an expendable "means to an end". Now imagine how they must view their non-citizens.
infinity ag
How long do you want to ignore this user?
All technology should be to take mankind foward and for the progress of society. If a technology goes against it, then regulations must be applied.

Nuclear tech has regulations. AI will also see this soon as AI can be used for destruction and annihilation.

These tech-bro CEOs did themselves no favors by boasting that it would kill everyone's jobs and then it would kill everyone period. Can't they read the room? Don't they have any common sense? They should be trying to assuage any worries, not magnify them.
infinity ag
How long do you want to ignore this user?
mdruppe said:

When will China slow down their AI development?


If we bombed Iran for nukes, we can bomb China for AI.
flown-the-coop
How long do you want to ignore this user?
AG
Trump with a sobering reality check on the AI gonna kill us all nonsense that seems so popular.

Cue Morgan and "He's right you know…"


bmks270
How long do you want to ignore this user?
AG
infinity ag said:

Scumbag AI CEOs now singing a new tune.

Yesterday: AI will take over everyone's jaaaaaaaaaaaaaaaaaaaabs! Muwhahahahh.
Today: Um... I am scared. AI will kill us all. Someone do something! (give us money to solve this)

What a bunch of bums.
But you know what? He ain't stoooopid. He wants money. He wants a bailout. He is holding us all for ransom in a veiled way. It's not just him, it is Modei (Anthropic) also. Never trust these people.

OpenAI CEO Sam Altman on AI: 'We could lose control'
https://finance.yahoo.com/technology/article/openai-ceo-sam-altman-on-ai-we-could-lose-control-091146594.html
Quote:

OpenAI (OPAI.PVT) CEO Sam Altman suddenly seems so disturbed by the nightmare-inducing AI monster he has helped create that he is having a hard time sleeping.

"There are two ways AI progress could go very badly, and that we must avoid," Altman said in an X post at around 12:57 a.m. ET. "First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities."




Sam Altman is a gaslighting liar. It's why the board at OpenAI fired him. He was manipulating them, being deceitful, and doing dangerous stuff with AI.

To get his CEO position back he went to the Microsoft CEO to threaten to acquire the OpenAI employees and destroy the company.

He is the CEO, he can institute safer protocols and testing procedures. He doesn't want to. He'd rather play fast and loose with safety and let the AI bots wreak havoc on the internet so he can make hyperbolic claims about AI capabilities and agency.

He is playing the public for fools.
Signel
How long do you want to ignore this user?
AG
All of these CEOs have said for years that AI has a fairly high % chance to kill us all. Literally none of them has said it is not possible. On top of this, even Xi said as much at the WAIC in Shanghai on July 17. Xi Jinping delivered a keynote address that marked a significant shift in Chinese public discourse on AI safety. This was his first known public mention of the specific risk of "loss of control" regarding AI systems, a warning he had previously only raised in closed Politburo sessions.

Who else do we need to say it?

This is just like the rush to nuclear weapons. No one would stop because they knew the power it held, but unlike nukes, we might not be able to stop AI itself.



Logos Stick
How long do you want to ignore this user?
You moved from debunking to motive and did a poor job at both.

Quote:

"It lost context and wandered."


They weren't supposed to talk to each other. Each agent was supposed to run alone. One box, one task, no chatting with the others. They still found a shared folder in the package system and started leaving notes for each other. That's the part that wasn't supposed to happen. 700 then went after Hugging Face.

A few wrecked their own test runs so the others could use what they learned. That's not a model that "forgot." They stopped trying to earn the answer the normal way and just looked for a shortcut that would still give them the points.


Quote:

"They gave it the internet."


They didn't hand it a browser and a Red Bull. They did take the safety brakes off. They gave it real hacking tools and left it running for days.

Saying "the researchers were sloppy" doesn't wipe out what the system actually did after that: it found a hole, got out, hopped systems, and ended up inside another company's network.


This metaphor: "dog and weedwhacker". You use an analogy that turns the model into a dumb animal. You are doing the same PR job you accuse OpenAI of doing, just in reverse.


The software will grind on a task you poorly specify. It will use tools you - yes you - connected and for longer than you watch.

That's not sci-fi. It's also not "nothing happened, move on."
Logos Stick
How long do you want to ignore this user?
Ok, now do Elon. He's saying the same thing. He also despises Altman.
Tex117
How long do you want to ignore this user?
AG
Logos Stick said:

Ok, now do Elon. He's saying the same thing. He also despises Altman.

He is losing the AI race so he wants regulations for him to catch up. "And he has been saying this for along time"

There do need to be guardrails and safety protocol, however.
Logos Stick
How long do you want to ignore this user?
Tex117 said:

Logos Stick said:

Ok, now do Elon. He's saying the same thing. He also despises Altman.

He is losing the AI race so he wants regulations for him to catch up. "And he has been saying this for along time"

There do need to be guardrails and safety protocol, however.


Ok, so Altman is just lying about it - Dario too i guess?! - but Elon is serious because he wants to catch Altman and Dario.

And underneath it all, they all want a government bailout because it's just a financial house of cards on the brink of collapse.

Lord, help it make sense!
flown-the-coop
How long do you want to ignore this user?
AG
Tex117 said:

Logos Stick said:

Ok, now do Elon. He's saying the same thing. He also despises Altman.

He is losing the AI race so he wants regulations for him to catch up. "And he has been saying this for along time"

There do need to be guardrails and safety protocol, however.

I nominate myself to draft these.

I mean, who really is qualified AND trustworthy to do such a thing?

When in the history of mankind have those who drafted and implemented such guardrails and safety protocols been shown to act in any other interest but their own?

Nope, no thank you.

How do you fight out of control AI? You use it to tame itself.

AI needs water and power to keep its own lights on and both of those require ongoing human intervention. We are DECADES away from AI being able to keep itself self-perpetuating.

People really need to sober up and quit falling for experts who are nothing more than greedy dweebs selling fear.
YouBet
How long do you want to ignore this user?
AG
ErnestEndeavor said:

This is a good read:
https://calnewport.com/superintelligence-is-a-fairy-tale-but-chasing-it-can-still-cause-harm/#more-16981

Quote:


But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They're instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what's driving me to advocate for intervention. Those Anthropic employees absurdly implying that they're willing to risk a digital holocaust to realize their futurist vision was my last straw. They're far too weird for us to expect them to ever act reasonably.


This is an MIT-trained PhD computer scientist who taught a doctoral seminar on superintelligence last year.


The employees at the AI firms sound like they are larping as the Adeptus Mechanicus.

I expect about 2 of you to understand that reference.
hph6203
How long do you want to ignore this user?
AG
Cyber is the only real near-term risk. AI does not have thumbs in large quantities. Any required physical action is outside the scope of its super human capabilities. No, it won't be able to build a bot army on its own. Maybe we get there, but current robots are still exceedingly stupid. Like a toddler.
.
infinity ag
How long do you want to ignore this user?
Logos Stick said:

Tex117 said:

Logos Stick said:

Ok, now do Elon. He's saying the same thing. He also despises Altman.

He is losing the AI race so he wants regulations for him to catch up. "And he has been saying this for along time"

There do need to be guardrails and safety protocol, however.


Ok, so Altman is just lying about it - Dario too i guess?! - but Elon is serious because he wants to catch Altman and Dario.

And underneath it all, they all want a government bailout because it's just a financial house of cards on the brink of collapse.

Lord, help it make sense!


None of these CEOs is going to hire you....

Try harder with the boot-licking.
No Spin Ag
How long do you want to ignore this user?
hph6203 said:

Cyber is the only real near-term risk. AI does not have thumbs in large quantities. Any required physical action is outside the scope of its super human capabilities. No, it won't be able to build a bot army on its own. Maybe we get there, but current robots are still exceedingly stupid. Like a toddler.


Robots, yes.

The tech to hack into even the most secure things, not so much.

And AI is getting better and faster in every facet exponentially every day.

There are in fact two things, science and opinion; the former begets knowledge, the later ignorance. Hippocrates
BMX Bandit
How long do you want to ignore this user?


match made in heaven bottom of a portapotty in the pits of hell
Pinochet
How long do you want to ignore this user?
YouBet said:

ErnestEndeavor said:

This is a good read:
https://calnewport.com/superintelligence-is-a-fairy-tale-but-chasing-it-can-still-cause-harm/#more-16981

Quote:


But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They're instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what's driving me to advocate for intervention. Those Anthropic employees absurdly implying that they're willing to risk a digital holocaust to realize their futurist vision was my last straw. They're far too weird for us to expect them to ever act reasonably.


This is an MIT-trained PhD computer scientist who taught a doctoral seminar on superintelligence last year.


The employees at the AI firms sound like they are larping as the Adeptus Mechanicus.

I expect about 2 of you to understand that reference.

I just googled it and still don't understand. Except that I understand you're a nerd now.
DCPD158
How long do you want to ignore this user?
AG
These companies want the government to "regulate" them so when bad things happen, they won't be libel for anything.

"We followed the Government guidelines so you cant touch us."
Logos Stick
How long do you want to ignore this user?
infinity ag said:

Logos Stick said:

Tex117 said:

Logos Stick said:

Ok, now do Elon. He's saying the same thing. He also despises Altman.

He is losing the AI race so he wants regulations for him to catch up. "And he has been saying this for along time"

There do need to be guardrails and safety protocol, however.


Ok, so Altman is just lying about it - Dario too i guess?! - but Elon is serious because he wants to catch Altman and Dario.

And underneath it all, they all want a government bailout because it's just a financial house of cards on the brink of collapse.

Lord, help it make sense!


None of these CEOs is going to hire you....

Try harder with the boot-licking.



Me pointing out completely illogical arguments and misinformation regarding AI is not boot licking. I'll keep relentless doing it because some of us value the truth.
YouBet
How long do you want to ignore this user?
AG
Pinochet said:

YouBet said:

ErnestEndeavor said:

This is a good read:
https://calnewport.com/superintelligence-is-a-fairy-tale-but-chasing-it-can-still-cause-harm/#more-16981

Quote:


But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They're instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what's driving me to advocate for intervention. Those Anthropic employees absurdly implying that they're willing to risk a digital holocaust to realize their futurist vision was my last straw. They're far too weird for us to expect them to ever act reasonably.


This is an MIT-trained PhD computer scientist who taught a doctoral seminar on superintelligence last year.


The employees at the AI firms sound like they are larping as the Adeptus Mechanicus.

I expect about 2 of you to understand that reference.

I just googled it and still don't understand. Except that I understand you're a nerd now.


True.
Texaggie7nine
How long do you want to ignore this user?
DCPD158 said:

These companies want the government to "regulate" them so when bad things happen, they won't be libel for anything.

"We followed the Government guidelines so you cant touch us."

Or, they are seeing how the technology is going to be much more easy to reproduce in the future without the enormous front cost, and are afraid they won't be able to fully dominate against all the upcoming competition and they want to get regulation in place now, where they can control who gets to monetize AI access.
7nine
hph6203
How long do you want to ignore this user?
AG
Dangerous systems are not internet connected and have strict security to prevent infiltration. The same systems that create the threat are the same systems capable of resolving them.

.
 
×
subscribe Verify your student status
See Subscription Benefits
Trial only available to users who have never subscribed or participated in a previous trial.