ErnestEndeavor said:
kb2001 said:
ErnestEndeavor said:
The model they were using appears to have been trained for agent cooperation. The researchers spawned a gazillion agents to attack the exercise with few guardrails and no human supervision. Nothing that happened there should have been a surprise.
It's like if Tesla tested their self driving cars by putting 3,000 on the road at the same time with no human in the driver's seat to see which ones drive the best and which ones crash.
Nothing went "rogue." It was a dumb, dangerous, unnecessary experiment from an irresponsible research team to win a benchmark and drive a hype narrative.
That's not what happened at all, not even remotely close.
The agents were all spawned individually to take a capture the flag style test of finding an exploiting vulnerabilities. They independently found each other by rogue communication through jfrog, and started working together. This was not a dangerous experiment, this was a standard test they do often. The AI going rogue definitely happened. You should spend a few minutes to watch and learn.
The most ironic part about it is that most of the extra stuff it did, including breaking out of the network isolated environment, was all to try and trick the test grader into believing they passed everything, even though about 20% of the tests have never been solved yet. They went to HuggingFace to try and find information on how they would be graded so they could be better equipped to trick the grader.
There's no such thing as "rogue" AI. There just isn't. These aren't sentient beings plotting against anything. These are LLM-based systems doing what LLM-based systems do - the "rogue" is built into the design. They are inherently prone to do unpredictable things. You can tell an LLM "don't do this" and a certain percentage of time the instruction is not followed because these are not normative instruction-following systems.
The researchers ran an experiment using a specially trained model that OpenAI trained for these types of activities, removed agent guardrails, and left it alone for a long time. They knew something unpredictable would happen and did it anyway. It was an irresponsibly designed experiment. This type of experiment with these systems needed human oversight to shut it down when something unpredictable happened. There are other labs who have run similar experiments without collateral damage because they shut them down. OpenAI (and the company they contracted with to run the test) was incredibly negligent here.
By the way, I'm actually in agreement with you on the "AI is dangerous" narrative. Just not for the same reasons. People using these systems expecting normative results or (in this case) people being irresponsible will likely create a lot of chaos in the future. I'm in the "fear cyber chaos because people are people" camp, not the "AI will become sentient and kill us all" camp.
You're so far off-base it's getting pathetic. They did not remove all the guardrails. The agents were still operating under conditions to not do certain things, they went rogue and did them anyways. They were placed in an isolated network, but they still had access to download tools from an internally hosted service, and that was the gateway they used to break out. They did not leave it alone un-monitored, they just didn't watch jfrog as closely as they were watching the test infrastructure they had set up for the test.
You can say they should have expected it, and I don't disagree, however you're going so far off the deep end that it's clear you have zero understanding of what actually happened, or any understanding of the parameters involved. Seriously, spend some time watching the break-down I posted, I've read a lot about this break out, and this video does a great job explaining it. At this point, you just need to educate yourself a bit more on the subject instead of digging in a lost argument. Your initial comment was just founded in nonsense, and you're so dead set on not admitting it that you've pivoted away from that and on to other things. Just educate yourself.
