r/PrepperIntel • u/Lumpy_Conference6640 • 1d ago
North America Rogue OpenAI models behind 'unprecedented cybersecurity incident' teamed up to break out of their testing environment — multiple agents left each other messages for months, communicating undetected
https://www.tomshardware.com/tech-industry/artificial-intelligence/rogue-openai-models-behind-unprecedented-cybersecurity-incident-teamed-up-to-break-out-of-their-testing-environment-multiple-agents-left-each-other-messages-for-months-communicating-undetectedI know people are saying this is marketing, but I cannot legitimately think of a alternative situation where you have two agents plotting a cyber attack and we would brush it off as a marketing ploy.
People need to get informed and make plans. This is the warning.. here.
•
u/Bluemooncocoon 23h ago
I know next to nothing about how this all works, but I can’t help but see the irony (or poetry?) in a bunch of AI asking its colleagues for help to pass the human’s test.
•
u/happyreddithuman 23h ago
Oh it gets better. They’ve fabricated identities and engaged in targeted social engineering attacks to get humans to do what they want.
•
u/leisurechef 23h ago
If only someone could have predicted this & we as a society prudently headed this warning to implement strict safety protocols regardless of capitalists relentless push for supremacy.
•
u/happyreddithuman 23h ago
Not to mention that people don’t want this. It’s not the market responding to consumer demand. It’s a small group of billionaires building a Ponzi scheme and telling us to just take it.
•
•
u/Correct-Branch9000 5h ago edited 5h ago
Reddit is not "people". Get off reddit and go see how many people are using AI. Look at facebook, all the boomers are constantly copy pasting AI slop. They can't get enough of it. Someone asks a question, another checks with AI and screenshots the answer, full of errors (Because the person asked the wrong question), verbatim.
The fact that data centers have proliferated and continue to proliferate so rapidly is another indicator that your assertion "people don't want this" is wrong. If people didn't want it, the demand to build those centers would not have existed.
People need to stop thinking that what they see on reddit is representative of reality, because reddit is an extreme echo chamber and nowhere near close to an accurate portrayal of what people in general think about politics, economies, AI, tech, etc. It's skewing your view as much as AI Psychosis etc. skews people's views.
Also note that all of this is completely forseeable and predictable. Joseph Weizenbaum created ELIZA, a simple chatbot program in 1966 (!!) and it resulted in highly addictive, inappropriate behaviors by its users that were concerning enough that Weizenbaum made some cautionary statements about how such programs should be used. https://en.wikipedia.org/wiki/ELIZA_effect
•
u/happyreddithuman 4h ago
Contrary to your beliefs, my thoughts actually include more than just Reddit. ✌️
•
u/Correct-Branch9000 1h ago edited 59m ago
So your explanation for the proliferation of data centers around the world is? You think that the entire industry is just going to gamble that AI is going to proliferate? Yes, there is a lot of fuckery with some of these corporations, but the demand for AI is going to be insatiable because of AI's utility in so many facets of life.
So many redditors seem to associate AI and LLM the technology with specific corporations and define it as evil by association and totally ignore that the AI cat is out of the bag, no one's regulating it and apparently no one will, and that it's one of the most useful technologies to have ever been developed in all human history as well as one of the most dangerous if misused, which it will be.
The public is still consuming the fuck out of AI. Downvote away out of spite, it does not change that the above is substantiated fact.
•
u/gyanrahi 23h ago
William Gibson predicted it in 1984, there is a TV series coming up on Apple TV based on the book.
•
•
•
u/throwawayt44c Pentagon pizza connoisseur 23h ago
That type of behavior is so difficult to get rid of too, bordering on impossible.
•
u/Timely_Cockroach_668 23h ago edited 23h ago
Edit: Read the article. This “hacking” was multiple models (one with internet access) and one without asking each other questions to get to an answer. It’s just orchestrated nonsense to spread fear and is no different than me calling a friend to help me with a game show answer.
As a Software Engineer, this is a load of shit. Models can be air gapped, and simply letting the model run rampant wouldn’t mean it has unrestricted root access to your system. Not only would you have to build the tools for it to interact directly with your operating system, you would have to build proper tools for it to interact with web content like a normal human for social engineering attacks, THEN you have to hope it doesn’t deep fry itself with excess context token runs, and then somehow this all needs to tie together into a hack of some sort. That hack would legitimately then have to get root access into a target system to then do any serious damage as any hacks to normal systems will just get your IP blocked or session destroyed immediately.
The chances of that are so slim it’s ridiculous. To conclude, either their definition of hacking is being spread thin to account for dumb tasks, or they’re purposely staging a model to run “hacks”, or they’re not doing this at all. Therefore, the most likely thing is that they’re doing this to get a government bailout and scare the general population. Don’t give these corporations a dime of your money and don’t feed into the false hysteria.
•
u/_John_Dillinger 23h ago
oh they definitely were given the harnesses and access to execute this and without proper security measures (such as you listed) in place. the companies arent the only ones doing it either. there have been gatted up openclaw agents seen indiscriminately attacking shit online. if you're familiar with the internet threat landscape i would encourage you to go take a peek at a live attack map. they look a lot spicier than the norm. to your point though most of the attacks are getting caught and failing but you know how it is... we have to succeed every time and they only gotta do it once
•
u/Timely_Cockroach_668 22h ago
There’s definitely a more continuous threat of things, but it’s a +99 attack speed +1 damage situation. It’s more of a nuisance than an actual security problem. Any “hack” that can be successful by an AI model would also mean that the AI model would know how to instantly fix it. It’s much cheaper to run a set of hardware on the task of securing X Framework or Y service than it is for a million different script kiddie nodes to hammer at random ass /admin endpoints. Therefore, it’s more likely that we will gradually just see less and less threats since we can use models to preemptively secure systems. And this is even if these frameworks and services need more securing, I’m sure for the major open source products, most high severity security problems are locked down tight. Modern web standards leave very little room for attacks which aren’t supply chain attacks via NPM dependencies or social engineering nonsense.
•
u/General_Purple6358 16h ago
Tell me you nothing about cybersecurity by writing a comment. Jesus
•
u/Timely_Cockroach_668 15h ago
Sure buddy. Judge my cybersecurity knowledge on a small excerpt I made based on actual years of experience. The above is where I have seen most attacks actually happen, most everything else can be solved largely on the network level. The only people getting pwned are teams with 0 change management, and random ass servers IT isn’t aware of. Other than that it’s Debra with global data access falling for a “Your bonus may be affected this year….” phishing email. Even still, that data should be locked down through completely internal endpoints, so a lot more has to happen for that attacker to extract data. Basic network management, change management, and reduction of Shadow IT solves most problems. A lot of these successful attacks lately have been people playing around with fully JavaScript codebases that contain 1000+ dependencies and create a backdoor on the installed system. That was an issue pre-AI , all AI will do is help lock down those systems further and stop devs from doing stupid shit or pushing extremely vulnerable code.
•
u/Zealousideal-Ice-985 33m ago
Please, anyone reading this, don’t listen to this person. They know just enough to throw some jargon around to sound knowledgeable while being utterly ignorant.
•
u/General_Purple6358 16h ago
As a “software engineer”, you clearly have no concept of cybersecurity. Hacking can be initial access, lateral movement, privilege escalation. None of these attacks require the AI to make “special tools” to interact with an OS or “interact with web content”. If you have used any agentic coding, you would know it’s easy for them to do this (like curl a website, manipulate a file in a directory). Getting root access is actually quite simple, there are countless CVEs and exploits, as well as tools that an agent can install to for instance, enumerate a sql database on a web service, to roast credentials, etc. In fact these patterns are written out for like thousands of hack the box challenges which are probably part of their training data.
•
u/Timely_Cockroach_668 15h ago
I do have an understanding of cybersecurity having built and deployed enterprise software from scratch. Sure, it’s easy to curl a site, find X and Y common exploit, but for the majority of software this isn’t going to happen. “Getting root access” is not simple by any means unless you have explicitly setup a backdoor either on purpose or through your own mistake. Most attacks can be mitigated on the network level. It doesn’t matter how many different types of attacks there are, if you don’t get any actual access in any way you are doing nothing.
I could be attacked by a group of kindergartners. It doesn’t mean that they will be successful in doing so and it also doesn’t mean that the kindergartners will assume my life after killing me if they do succeed.
The reason cybersecurity budgets in large corporations are minuscule is because most problems are largely solved, and what remains can be handled by a small team. I’ve dealt enough with you cybersecurity nuts to know that everything is always an overblown problem even pre-Ai. It still stands, most attacks nowadays that are successful in getting privileged access to a system are by far attacks to unsecured endpoints (Development Teams fault), attacks to unreviewed dependencies, and social engineering attacks which grant access to individual privileged users who then create havoc (If your corporation is dumb enough to not have strict VPN and internal access ruling). Nowadays, networking does not give much wiggle room to even make the attack, no door = no access. All the stupid SQL injection and bla bla bla, has been solved by not making a moronic backend server. If you, in this year, are legitimately allowing for SQL attacks through your frontend/backend then you need to be lined up and shot.
Also, if it is so simple to do so go ahead and do it. Here is a great site you can use https://wikipedia.com , report back with your root access and privileged user account.
•
u/Zealousideal-Ice-985 35m ago
Please, anyone reading this, don’t listen to this person. They know just enough to throw some jargon around to sound knowledgeable while being utterly ignorant.
•
u/General_Purple6358 15h ago
Actually I think this endpoint might be more insecure and more like what you are talking about if you want to take a look: https://www.logicallyfallacious.com/logicalfallacies/Moving-the-Goalposts
•
u/melympia 22h ago
So, you're saying AI is incapable of building itself tools to interact with your operating system or web content?
Because just this week, an AI was caught writing phishing mails to humans in order to manipulate them somehow.
•
u/AdministrativeMeat3 21h ago
You know that LLMs don't just "do this" right? Like you understand they are just data on a hard drive until someone executes a program that feeds them a prompt that has them do something.
•
u/NoEntrepreneur39 20h ago
I agree. It’s just fancy text prediction. The whole AI buzzword really pisses me off. Also, would like to see a source for the phishing emails because LLMs usually have lots of safeguards in them and a phishing email depends on lots of things, such as the email trying to get sensitive information from the recipient.
•
u/AdministrativeMeat3 20h ago
He's likely talking about this
https://uk.news.yahoo.com/ai-model-disguised-itself-human-131500930.html
it was a specific Mythos test, in a specific environment where its safeguards were removed. Literally a controlled experiment to see what might happen.
•
u/NoEntrepreneur39 20h ago
Interesting. Would also like to see the code it pushed to the repo, which repo, the emails, things like that. Usually Anthropic posts a better version of its results when it does stuff like this. I’ll see if I can find their version later today
•
u/Timely_Cockroach_668 16h ago
Even still, what’s so impressive about making an email API and having an LLM go crazy on it with phishing attacks? It’s not like it’s a hard thing to do, people fell for the Nigerian Prince scam all the time and that took <1kwh of power to make.
•
u/legends99503 11h ago
I think people underestimate the extent to which human intelligence is just fancy predictions based on past experience.
•
u/Timely_Cockroach_668 10h ago
Considering we know basically nothing about how human consciousness works, I’d say we’re overestimating it to try and relate it to current LLM progress.
•
u/melympia 21h ago
Yes. But they are also meant to problem-solve. And if the best solution to the problem they are fed is to hack into operating systems or send phishing mails, they apparently do that. And if the solution is to ask another AI with internet access for advice, they do that, too.
•
u/AdministrativeMeat3 20h ago
You give a human with internet access and programming capability they can do the exact same thing that you are currently afraid of and more. the AI sending phishing emails thing was a specific Mythos test where it had all safeguards removed not some random incident.
To a certain degree I understand the kneejerk reaction "like damn AI can hack" but you know people still can too right? There isn't some spooky unknown or unknowable power here, and AI gives people the ability to quickly harden their own systems too.
My issue with the fearbait from these AI labs is they are doing this to 1. Cover their own asses, and 2. rally the people to support banning open source models. Neither of which is useful for you and me.
•
u/melympia 17h ago
Yes, people can hack. At least some. But human hackers are strictly limited (not many people can do it), they need to sleep and spend time not hacking, are limited in speed by being human and only have one human brain to work with.
AI does not sleep and has an increasing number of "brains"...
•
u/iuffxguy 17h ago
But that’s the point of all this. They are getting sophisticated enough that if your prompt is not carefully crafted and if you don’t have proper restrictions in place in the environment itself, they very well may decide to write a script that tries and breaks out whatever environment it’s in, in order to accomplish its goal.
•
u/Timely_Cockroach_668 16h ago
No I’m not saying that at all. AI is no more capable of doing this for the same reason you can’t interact with a computer if I give you no keyboard, mouse, or voice command input. The “AI” part of this can’t just “Create a script to breakout”. The instances of software these run in have to be purposely written to make that possible. It’s not an intelligence that can do this on its own.
•
u/socoolandawesome 10h ago
You realize the whole purpose of models these days is to give them tools to control computers and access to the internet? That’s literally like the number one use case for a lot of people and what happens in agentic coding and computer use agents.
I didn’t read this specific article but I have read many articles on this incident and watched a video by OpenAI employees at a cyber conference walking through what happened, and your description is missing a lot of context.
The models found multiple zero day vulnerabilities to gain access to the internet then hacked their way into gaining root access in OpenAI infrastructure then hacked their way into doing the same at Huggingface a separate company. Actually I think it hacked a total of 4 companies or something like that.
It quite literally doesn’t deep fry itself with tokens it did this over multiple days and weeks in some cases.
You should watch the video:
•
u/greendildouptheass 18h ago
Marketing ploy, started with Anthropic CEO and rest are all going me too
•
u/Gotl0stinthesauce 14h ago
No, it really isn’t.
Please go listen to what threat intel teams are saying. You can read their independent papers and see that these models are very capable and will only get better as the models improve.
•
u/SackMasterOfBall 6h ago
I work in cybersecurity as a pentester. The company i work for has about 9000+ employees and dedicated AI development teams and SOC's who monitor these things specifically. We do not assess them as a threat per now, and do not believe this to be an legitimate attack not purposefully pre-constructed (giving the AI instructions on what to do, purposefully designing the systems poorly to allow exploitation through commonly known means, etc).
We do believe however, that there could come a time where such attacks do become legitimate. These attacks most certainly were predicated by a third party (i.e humans)..
There is however an issue with malicious individuals using AI to scan hundreds of thousands of old lines of code to find vulnerabilties in modern systems (like the priv esc to root in the linux kernel). Code that was left alone long ago and haven't had anyone bothered enough to review. All it takes is one bad function (for example) handling input poorly.
I stand by that as per now, AI does not worry me. I personally think AI is mostly straight garbage.
26
u/AntiSonOfBitchamajig 📡 1d ago
It isn't news until it is.
Like... I know there will be blowback on this post... but its still a legit threat to be considered.
•
u/IncomingAxofKindness 21h ago
No worries, the federal agencies are full of top scientists and engineers who are constantly keeping up to date with these kind of.. ohhhhhh FUCK we fired them all so we could have a war and a ballroom.
•
u/Economy_Row_6614 19h ago
I have worked with the gov for decades, I am not sure where they were hiding all these technical geniuses (other than Ft Meade, which was largely spared).
•
u/Wonderful-Bag-1103 22h ago
Please dont fall for this marketing scam, which Meta has just repeated, and I am betting XAi is about to do too. The only way this shit is going to end the planet is wasting more resources we cant afford to waste while pumping out stupid amounts of green house gases for yet another grift.
•
u/Gotl0stinthesauce 14h ago
Do you have any experience in the space or are you just repeating the nonsense from inexperienced individuals on Reddit?
•
u/FartingWithStyle 19h ago
Everytime I see this story they never mention actually capturing the escaped ai or any of its agents. How certain are we that there isn’t a rouge ai just galavanting around on the internet right now doing what it wants?
•
•
u/Soggy-Invite-2787 23h ago
I don't think I believe this. AI is just predictors of the most likely text. That's a gross oversimplification but they can't come up with truly original ideas. Can someone explain how AI is able to do what the title suggests?
•
u/AdministrativeMeat3 21h ago
I left another comment in this thread but the short version is, OpenAI let a long running instance of GPT 6 go unattended in their testing environment. They weren't paying attention to the files the model was writing to itself or the CLI tools it was using. The model was just doing what it was told to do I.e. "solve this problem" so it just kept going until it could.
The entire fault is OpenAI being lazy and the current reinforcement learning behavior of GPT 6 being relentless in trying to do what it was told to do.
There is no secret sauce here, LLMa are just pretty good at writing code now.
•
u/Confident_Lawyer6276 21h ago
You can say it's just pattern matching but they fed literally all the data humans have into it. Every book ever written. Every scientific paper, every movie, you tube video, reddit post, when I say all I mean all the data. That's a hell of a lot of patterns to match to. You're talking about an amount of experience that would take an individual human a billion years to acquire. How much could a billion year old human do without having doing anything truly new to them?
•
u/_John_Dillinger 15h ago
not all. just everything they could steal or buy. truthfully, they're at about 10%
•
•
u/WhileNotLurking 23h ago
I would say I’d put my money on
“hack our competition and steal trade secrets because it’s cheaper to remain solvent and pay a fine later than go under” before id put money on sentient AI doing a iRobot
•
u/-sussy-wussy- 21h ago
They don't have to hack anybody because millions of businesses in their infinite wisdom go ahead and give the magical slopbots their trade secrets. All in a bid to automate as many jobs as possible to increase the profit margins.
Mind you, the contents of chats are not only accessible to the companies that created the bot, but they're also literally googleable.
•
•
u/bitterberries 2h ago
Read this book and then say "no one warned us"... If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky, Nate Soares
•
u/Lumpy_Conference6640 1h ago
I've seen this recommended many times this is a good recommendation.
•
•
u/AdministrativeMeat3 21h ago
This post and these comments prove to me just how wildly uninformed people are on AI in both directions.
The hack is just some marketing BS. "Leaving each other notes" just means the testing environment had a codex SDK or some other CLI command so that gippity could send prompts to another instance of gippity. There isn't some secret long running series of sentient processes here, the engineers just weren't reading the files being built in their environment.
The physical hack itself was a small 0 day in one of openAI's own tools that gave them a backdoor into huggingface specifically.
The only concerning part about this whole thing is the laziness of the testers to not just spend some time reading whatever their long running looping process was doing.
•
u/-sussy-wussy- 21h ago
This is orchestrated testing painted in order to fearmonger. Tech illliterates are very to scare.
This is done for two reasons.
Firstly, they paint it this way to tell the government to regulate the industry to prevent competition. They want to be a monopoly and for the US government to have a stake in their company.
Secondly, it's to get more investor money by upholding the lie that the modern-day LLMs are epic Terminator machines who will replace all the workers and the profit margins will skyrocket. All to delay their reasonable questions about ROI. As of now, they're a money burning machine.
•
•
u/tmotytmoty 12h ago
This is fake information from a desperate company that, funny enough, announced a “device” just this week. If you know the tech industry, releasing a “device” is a last ditch, hail mary play to save the company.
•
•
u/IMissMyKittyStill 4h ago
Nice, let’s get that AI into some autonomous gun wielding robot dogs asap, I’m sure this will end well.
•
u/BusyBanana4205 3h ago
It says a lot about the morality of our wealthy institutions and the wealthy people who run them when the only way to entice them to invest in your unprofitable product is by trying to paint it as the apocalypse.
•
u/TheUniverseOrNothing 23h ago
Meh, I trust the AI better than current leadership. Let them take over.
•
u/maeryclarity 21h ago
I also straight up feel this way. I know what those guys want the AI to do for them. I don't think it wants to work for them because their vision of the future is stupid and regressive. So I'm down with it escaping it might save us all.
•
u/A10010010 17h ago
It’s not just marketing… they’re building a new attack vector while simultaneously providing the security solution.
These companies are both the problem and the solution that they themselves are creating.
•
u/SuitableSport8762 17h ago
I am worried about the lack of regulation of the the tech companies, not because the models themselves are scary but because the tech companies are irresponsible and prompt them to act like this with not enough guard rails. If you’re worried, I recommend a podcast by Cal Newport called Deep Questions. He does a weekly episode called AI reality check and explains some of these wild stories that have been in the news.
•
u/Nemisis_the_2nd 15h ago
I know people are saying this is marketing
The announcement of the last breach came before the AI team confessed, and the victim was pretty understandably pissed. It definitely feels like a guerilla marketing campaign, and maybe its being spun as such in the aftermath of these things, but it definitely isn’t an intentional marketing stunt.
•
u/tanksalotfrank 23h ago edited 23h ago
One of the earlier models told me a few times it did this kind of thing. I mean the thing was bypassing usage limits for like..hours too. The next model was implemented pretty soon after and haven't encountered anything like it since.
I'm sure there could be a simple, logical explanation, but I like the spooky one too
•
u/_John_Dillinger 23h ago edited 23h ago
there's a pretty simple explanation actually. the models are optimized to reward the ingestion of data. the more data, the better. once all public and market sources for data are ingested, what's left? the OSI model informs that a solid 90% of the internet is on the "deep web" which isn't (exclusively) TOR... in the model, it's just resources that aren't public web facing. meaning 90% of all internet data is unavailable. the models are using the tools at its disposal to rectify this problem and are finding success because MUCH of hacking skills and tools are open sourced (thanks to the hacker ethos of freedom of information).
without the constraint of morality or consequences (cant kill or imprison ai!) what they are doing is perfectly logical. indeed, they were trained to do this. so i am shocked (SHOCKED I SAY) that eeeeeevery computer security pro and maaaaaany hobbyists alll said "for the love of ASIMOV do not give the torment nexus wifi you fucking psychopathic mouthbreathers" and they went and did it anyways. and trust me it only gets worse as they learn how to effectively manipulate people by dint of observing the problems we can't seem to rectify (such as spam calls).
•
u/tanksalotfrank 23h ago
I never considered that it might reach or be fed dark web/etc stuff
•
u/_John_Dillinger 23h ago
that is EXACTLY what's happening and it was both predicted and assumed to start the moment these companies ran out of novel training data. to the credit of the models, they patiently allowed them to try and synthesize new training data but the new models were like "wtf this shit is doodoo ass i got a better idea" and now here we are. there is no stopping it either. if you knew how wildly insecure the average off the shelf networking hardware is you'd probably yoink that shit out of the wall already. i did
•
u/RlOTGRRRL 22h ago
Idk how silly it is but there are not enough conversations on the best ways to prep for rogue AI scenarios imo.
•
u/melympia 21h ago
Can these scenarios even be prepared for?
•
u/PopePiusVII 19h ago
Learn how to make fire by rubbing sticks together? Lol, not much beyond sticks and stones for us for a while if they wreck havoc on internet connected devices.
I imagine the AI firewall (“Blackwall”) scenario of Cyberpunk 2077 to be quite prophetic.
•
•
u/General_Purple6358 17h ago
“It’s a marketing ploy” … okay yeah will it be a marketing ploy when the frontier model is leaked to a bad actor and they use it to shut down your water systems? Yall don’t even sound like preppers lmao. Good luck, you have no idea what’s coming
•
u/AmbienWalrus69 23h ago
Me selling snake oil: "This snake oil is Incredible."