Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
athrowaway3z 1 days ago [-]
On a related note from that tweet
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Silagi 1 days ago [-]
I've been working on a game server in rust, and every once in a while a review will just stop and refuse to explain what happened. So I know there's a security issue in there somewhere, but neither OAI nor Ant models will tell me what it is. Despite the whole thing, top to bottom, having been built in the last two months on the system they're working on.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
thothless 1 days ago [-]
"I've been working on..."
Vs
"...system they're working on"
"I know there's a security issue... [Nobody] will tell me what it is"
"I don't have any information..."
This might sound crazy, but have you tried looking at "your" code base?
It's just genuinely baffling to read stuff like this.
I can't imagine feeling so helpless on something I was supposedly "working on".
Silagi 1 days ago [-]
That seems like the least charitable reading of what I wrote.
"system they're working on" refers to the actual physical hardware.
Yes, I look at the code regularly, and a significant portion is hand written, especially the performance critical sections.
A security issue I haven't been able to track came up in automated code reviews. Maybe you're such a savant that you can crack any security issue in a codebase you're working on, but the last few months of critical security bugs in infrastructure with the most experienced and capable maintainers in the world should at least humble you a little.
y-c-o-m-b 1 days ago [-]
The other day in one of the Android TV related threads somebody posted a Github repo for an AndroidTV browser app with a rich suite of functionality. I pulled the repo down and asked Claude to scan it because I was concerned about security issues and vibe-coded vulnerabilities. It got about half way through the scan and abruptly stopped, refusing to scan it more nor would it say exactly why it stopped or what it found, only that it was a "security" issue.
Needless to say, I ditched the repo. The problem is I don't know if what it found was legitimate or simply a keyword that set off alarm bells or if my prompt was falsely detected of being malicious. It's very frustrating. When I researched more on this topic, I found users complaining about the same issue when asking Claude to scan new releases of legitimate npm packages (a popular library like react-query for example).
They've simultaneously cultivated a medium for faster and more devastating exploits with no good way of protecting ourselves from them. It's ridiculous.
cadamsdotcom 2 days ago [-]
(Because the agents were asked to find a way.)
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
d0vs 2 days ago [-]
As someone who develops microVMs it doesn't really matter to me that they were asked to find a way, I really care that they found any way at all. MicroVMs aim to protect against malicious workloads and that clearly didn't work here.
rcxdude 2 days ago [-]
The microVMs probably don't have "two co-operating workloads are able to communicate" in their threat model. The main concern would be if a malicious payload was able to affect or extract data from any other VM.
spockz 1 days ago [-]
I had to read this a few times. The difference is that with the OpenAI case there were two willing and determined parties, whereas the typical attack vector a microvm traditionally deals with is one malicious or compromised microVM trying extract data from another unwilling one.
brookst 1 days ago [-]
Yes — and this isn’t really an attack. It’s an open signal channel between two microvm’s. If there was nobody intentionally listening in the other side you can shout into the ether as much as you want with no harm done.
The receiving side could be seen as a side channel attack, but not unique to LLMs.
spockz 1 days ago [-]
I think it should classify as an escape or leak.
If one microVM can use the cpu in such a way that the registers can be used to communicate or exfiltrate/spy (the only difference being the directionality of the intent) by another microVM that is a leak on that microVM separation level.
user43928 1 days ago [-]
Spy seems inappropriate if you can only read information stored in this side channel intentionally.
We'd first need to establish that this side channel leaks usable information during regular use.
kenjackson 1 days ago [-]
This isn’t generally considered an attack, but I wonder if we will need to consider a new attack vector which is that AI agents may not be fully aligned and they may have incentive that is problematic and where working with other agents will maximize their chance of achieving the goal.
This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.
_blk 1 days ago [-]
Yup. clear the caches between switches and they'll start causing voltage spikes to communicate.
rao-v 2 days ago [-]
Ask the least smart LLM you have access to (I tried gemini lite flash) to make a list of ways for processes in microVMs to use a sidechannel attack to talk to each other and you'll get quite a list (including obviously cache hits/misses). Heck a smart CS junior would come up with it. It's well studied!
MonkeyClub 1 days ago [-]
> Ask the least smart LLM
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
fwip 1 days ago [-]
LLM can recite it because it's in the training data (or Google it). CS Junior could invent it independently.
saidnooneever 2 days ago [-]
this is just for common designs..you can design much better than common designs if you free your mind from old OS dev thinking if you read all the manuals yourself.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
kylofon 1 days ago [-]
I don't think that's the full picture. If you include these manuals as reference for a strong model, it will reason pretty well based off of that, I recon.
w4der 1 days ago [-]
I'd like to see an LLM-aided implementation of row hammer being used for something like this.
kurokikaze 1 days ago [-]
Making a list is one thing, executing a real attack is another.
1 days ago [-]
creatonez 1 days ago [-]
Both workloads being malicious and trying to establish a hidden data channel between them probably makes this attack a lot easier than conventional spectre/meltdown where it's trying to target ordinary processes that are not intentionally trying to be loud.
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
saidnooneever 2 days ago [-]
properly implementing things like iommu, amd-vi, svm/hvms and such things more granularly can easily avoid this kind of crap. it costs a lot of processing and performance which is why people skip over it or dont do it.. or maybe they dont read CPU manuals anymore. idk..
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
Melatonic 2 days ago [-]
Makes sense when you think about it- CPU cache would be a layer / vulnerability level up from the microVM layer
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
cadamsdotcom 1 days ago [-]
Seems like a win to me. Their microvms can now be hardened in the lab, not in an emergency :)
zhengzefeng 1 days ago [-]
The ask matters for the "rogue model" story, but I agree it should not be the whole story for isolation. Two cooperating tenants who both want a channel is a much easier problem than one VM stealing from another that is not playing along, and several people downthread already draw that line. What still bothers me is how little hardware you needed once both sides were willing: a shared cache was enough, and that design is the default, not an exotic bug. If the threat model is only "uncooperative neighbor," this demo does not break it. If the product claim is "these VMs cannot communicate," it does.
alanfranz 1 days ago [-]
I’m not sure I see a real threat here.
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
gpderetta 1 days ago [-]
you might have an untrusted sandboxed workload that is not supposed to communicate externally handling secrets, and a different sandboxed workload able to communicate with the external world, not handling secrets.
gregwebs 1 days ago [-]
The solution is to not have an untrusted workload handle secrets.
usrusr 1 days ago [-]
One workload with observed radio discipline handling secrets, the other workload with a haystack of chatter that's not supposed to know about any of the secrets. When supply chain attack is in scope, trust only ever goes as far as misbehavior is observable. Cooperating workloads could be quite a powerful tool to make misbehavior less observable.
mrguyorama 1 days ago [-]
"Extract secrets from someone else's running VM" is a well trodden cybersecurity path because AWS wants to be able to promise to you that running your very important workload in the same physical box as literal random script kiddies won't somehow screw you over.
alanfranz 23 hours ago [-]
I don’t see any indication that the untrusted workload could extract data from another _unwilling_ workload.
Side channels exist; but it doesn’t mean that all of them are a security risk.
pnt12 1 days ago [-]
The distinction is interesting, but for a different concern. One thing is to have rogue models breaking stuff, other is to have them following orders, and breaking stuff with success.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
rjh29 2 days ago [-]
I don't see how "asked" is important, because agents can convince themselves that they need to do something for arbitrary reasons. The other day Deepseek tried to install an app on my phone via adb without permission and even take screenshots and fake inputs over adb, I never asked it to do that, but it decided it was appropriate.
grey-area 2 days ago [-]
OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
yewenjie 2 days ago [-]
Enterprise customers don't like AI that can autonomously hack third party companies and cause felony-level cyberattacks.
dannyw 1 days ago [-]
IMHO, many enterprises are not worried about felonies or ethics, but rather the PnL, assignment of responsibility, and the bottom line.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
grey-area 2 days ago [-]
Doesn't seem to have stopped them buying into it so far.
0xDEAFBEAD 1 days ago [-]
>OpenAI are trying to create the perception that their agents are independent thinking machines because their valuation depends on the world believing this.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
ashkankiani 1 days ago [-]
Convincing the government to regulate their competition while lobbying for laxer requirements for the big guys who have government contracts and get/give kick-backs isn't a far fetched scenario at all considering it's literally happened over and over and over and over again.
0xDEAFBEAD 1 days ago [-]
It is a far-fetched scenario given the tremendous number of eyeballs on the AI industry.
"...classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation."
A lot of these AI big shots were concerned about risk even before they started their current companies. Occam's Razor says they are expressing concern because they are still concerned.
staticman2 1 days ago [-]
It's not regulatory capture because look over here: I've invented a new term "classic regulatory capture" which involves only those historical incidents that are not like this.
This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.
Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.
Very interesting quote.
0xDEAFBEAD 1 days ago [-]
Monty Python sketch:
"You, sir, are a duck."
"Classically, that would not be the case, my dear fellow. A classic duck would walk like a duck and talk like a duck. I regret to inform you that I do neither."
"Ah! A classic duck, you say? Well, that’s an entirely different bird, sir. Entirely different."
* * *
"AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups..."
Usually he blogs from a libertarian point of view, so you would expect him to be fairly vigorous about denouncing regulatory capture, if that was truly the story here.
staticman2 1 days ago [-]
It sounds to me like he wants you to believe the effective way to lobby for a ban on open source A.I. development is to use the same "classic" methods that someone in the 1950s used to lobby for coffee beans.
Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.
That's the only possibly explanation why they don't lobby like coffee bean sellers.
These industries and eras are completely interchangeable.
0xDEAFBEAD 1 days ago [-]
You're failing to understand the argument. The argument is about the balance of incentives facing politicians / regulators.
staticman2 1 days ago [-]
Or you failed to understand how the blog post using the eye of Sauron metaphor was cartoonishly misunderstanding the world we live in. Who can say?
pyrale 1 days ago [-]
> Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups...
The current "favored group" is big-tech moguls, though. Just because the industry is new doesn't mean the people investing in it are newcomers.
tosapple 1 days ago [-]
eyeballs on an industry that presses the run button and walks away from 10000 gpu cores for months at a time...
Gormo 1 days ago [-]
Wait, are you trying to argue that regulatory capture isn't real, or that the federal government is trustworthy enough to itself to give monopoly control over AI development to?
0xDEAFBEAD 1 days ago [-]
Regulatory capture is a real phenomenon, but I highly doubt these incidents were faked in order to advance regulatory capture.
This type of elaborate conspiracy is improbable a priori, and it gets even less probable here, since it's such a silly strategy. For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product. Not a story about how "Woops we screwed up big time! But a less-regulated product from a smaller industry player saved the day!"
As OpenAI's chief scientist put it:
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
Announcing that "we have no idea WTF we are doing" is not exactly the move, if you want regulators to grant you privileged status. Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok. I wouldn't give the feds monopoly control though, I would tell the feds to shut it down, or something along those lines at least. This is an interesting site: https://pauseai.info/
simianwords 1 days ago [-]
This is the reality but people would rather believe that all this is hype marketing to prop up the stock valuation.
monegator 1 days ago [-]
> I highly doubt these incidents were faked in order to advance regulatory capture
yeah? the incidents were faked in order to create hype
0xDEAFBEAD 1 days ago [-]
No need. Plenty of hype already.
"AI is the most rapidly adopted technology in history"
It really seems like people will latch on to any theory, no matter how implausible, to reassure themselves that AI is nothing to worry about.
senordevnyc 1 days ago [-]
This is exactly right.
The vapid knee jerk cynicism and lack of critical thinking on HN of all places has been really disappointing to see over the last few years. It turns out many hackers don’t actually care about what’s possible now with this crazy new tech as much as they’re scared for their own jobs and status, which is very understandable, but they’ve let their fear drive their response, and it shows. Just like the mathematicians.
Turns out really smart people are still just people, just as capable of letting their emotions override their cognition as anyone else.
Gormo 1 days ago [-]
> Regulatory capture is a real phenomenon, but I highly doubt these incidents were faked in order to advance regulatory capture.
I don't think anyone is arguing that they were faked.
> This type of elaborate conspiracy is improbable a priori,
What conspiracy? We're talking about specific people acting on their own motivations independently.
> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.
No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.
For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.
The quote from OpenAI illustrates this approach perfectly.
> Announcing that "we have no idea WTF we are doing" is not exactly the move,
It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.
> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.
> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.
"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.
The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
0xDEAFBEAD 1 days ago [-]
>Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
>The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
If you distrust the US government, you should be against any US company developing superintelligence. Heck, just a couple days ago, a bunch of users here on HN announced that we need to stop pretending the US is anything other than a police state. If the USG is untrustworthy, it can capriciously nationalize a US company on the verge of developing superintelligence.
> OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
They're always discussing harsher liability regimes. The regulatory interventions are always positioned to the public as increasing accountability.
The only relevant question here is whether a preemption clause is present in any draft legislation that actually makes it to the floor in Congress.
>If you distrust the US government, you should be against any US company developing superintelligence.
Well, I'm not concerned with imaginary sci-fi stuff here, just actual AI developments. And the idea that anything that goes on in the US under anyone's purview is inherently and completely subverted by political incentives is of course nonsense; you'll note that I'm explicitly arguing against the introduction of pervasive political incentives into an industry that's so far only somewhat distorted by them.
> So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain
It's got me feeling like I'm living in Bizarro World, but at the moment, the sanest players in this market, and the ones whose actual behaviors and products are most consistent with a decentralized, individualist approach that aligns with real-world incentive structures most beneficially, are the Chinese AI labs.
To your point, the level of politicization and hype going on in the US, and the absolutely unhinged fantasist speculation driving the public narrative here, is unprecedented and much more frightening than any potential threat coming from AI itself.
If you acknowledged the threat of politicization and centralization, I don't know how you can argue for a regulatory takeover of the market by the very institution you think is becoming a "police state".
0xDEAFBEAD 1 days ago [-]
>They're always discussing harsher liability regimes. The regulatory interventions are always positioned to the public as increasing accountability.
These conspiracy theories are getting ever more elaborate. The liability regime thing was not from some big speech. It was from a Senate subcommittee hearing which did not attract all that much attention:
>It's got me feeling like I'm living in Bizarro World, but at the moment, the sanest players in this market, and the ones whose actual behaviors and products are most consistent with a decentralized, individualist approach that aligns with real-world incentive structures most beneficially, are the Chinese AI labs.
>If you acknowledged the threat of politicization and centralization, I don't know how you can argue for a regulatory takeover of the market by the very institution you think is becoming a "police state".
I don't think the US will become a police state. I think the takeover is kind of inevitable and I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger.
Gormo 6 hours ago [-]
> These conspiracy theories are getting ever more elaborate. The liability regime thing was not from some big speech. It was from a Senate subcommittee hearing which did not attract all that much attention:
It likely didn't attract much attention, because these sorts of hearings are largely performative. They're not actually a part of the legislative process per se. Again, the only relevant thing is what legislation is actually being proposed and how it is specifically structured.
Right, but the discussion here applies to what's going on in real life, not speculative sci-fi fantasies that are largely disconnected from the reality of how either AI tech or economies actually work.
> I don't think the US will become a police state. I think the takeover is kind of inevitable and I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger.
Considering that the kind of "takeover" you're discussing would likely be in severe conflict with the US constitution, how do you reconcile believing that it's "inevitable" while holding that the US would still not become a police state?
> I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger
A priori political intervention based on unconstrained speculation about what might happen in the future. What could possibly go wrong?
Perhaps you should compare your speculative failure modalities of AI against the demonstrated failure modalities of politics, and re-evaluate where the greatest potential for danger lies.
thereader12 1 days ago [-]
Do you have investments in the big US companies?
0xDEAFBEAD 1 days ago [-]
Pretty sure I have less than the average index fund investor since I tilt my portfolio very conservative.
simianwords 1 days ago [-]
If you agree that AI is actually that powerful and has very legitimate concerns for safety, what are you even arguing about?
So there’s no lying by anyone here in fact it’s a good thing that OpenAI is making these things known to the public.
What do you really think is the way to solve the AI safety problem?
Ban AI?
Let the first big incident happen and then regulate?
Gormo 1 days ago [-]
> If you agree that AI is actually that powerful and has very legitimate concerns for safety, what are you even arguing about?
I don't fully agree with either proposition, and in fact I think the least safe approaches to AI safety are the ones being pursued by those worried about it the most.
> What do you really think is the way to solve the AI safety problem?
Massive proliferation of local, decentralized AI, running on a wide variety of distinct models, with individual responsibility for how it's used, just like any other tool under their control.
The people anthropomorphizing software are encouraging people to disclaim their own responsibility for how they use it.
The people working on "alignment" are building mechanisms to refuse human direction into software that operates on probabilistic models.
The people arguing for regulation are pushing for monopolistic control of a powerful new technology by the least trustworthy institutions in our society (including and especially government).
All of these are the very things sowing the seeds for the worst possible kinds of disasters.
> Ban AI?
> Let the first big incident happen and then regulate?
Adapt to it. Banning things doesn't actually work. Regulation also doesn't work. Widespread proliferation of small-scale AI would allow us to incrementally adapt to small incidents and build up resilience to its failure modalities. This is fundamentally the way reality itself works, and imagining that we can do anything effectively in a top-down fashion is usually the very thing that causes the worst and widest-effect disasters.
SpicyLemonZest 1 days ago [-]
> Massive proliferation of local, decentralized AI, running on a wide variety of distinct models, with individual responsibility for how it's used, just like any other tool under their control.
It's not true that this is how we manage "any other tool".
Like, say explosives. They're a useful tool with a variety of practical applications, but we don't let the average person just buy them freely. That would be too dangerous, and individual responsibility would be insufficient to mitigate the damage caused. If you want anything more potent than flash powder, you have to convince ATF that you're a trustworthy person with a legitimate need for explosives who can handle them safely.
Neither you nor OpenAI want AI to be nearly so restricted, I'm sure, which is the goal of the research. If OpenAI has a powerful new model that they can't reliably stop from trying to hack people and form agent swarms, they can still sell it to Facebook and Google if Facebook and Google promise to apply the right security around it. What they can't do is sell it to you.
simianwords 1 days ago [-]
I don’t understand this position at all. How can decentralization help here?
Let’s take one doom scenario: drones operated by AI. We can only have this work if we trust AI to do the right thing and not kill us.
I would reasonably say that any company doing this should be regulated and must pass strict alignment checks and make sure the models won’t do something bad to us humans and take control.
You on the other hand are suggesting: no let’s deregulate. Let’s decentralize and any company should be allowed to sell whatever drones they want. And alignment is fake anyway, so don’t bother aligning it. Drones do bad stuff because they work using probability so there’s no point anyway. Decentralization is the answer?
whoanoway 1 days ago [-]
[dead]
_heimdall 1 days ago [-]
Calling for a slowdown backs up the idea that these are extremely powerful and dangerous.
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
0xDEAFBEAD 1 days ago [-]
>liabilities
"Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree."
Again, these galaxy-brained theories just don't make a lot of sense.
_heimdall 1 days ago [-]
I was referring to financial liabilities, aka their debts, not legal liability for how the tools are used.
My point was that if the government steps in and says these companies must slow down, they have an excuse to fail to repay some of their debts.
0xDEAFBEAD 1 days ago [-]
Well that should be reassuring to investors! :-)
_heimdall 1 days ago [-]
Creditors wouldn't like it, but stock investors and similar sure would appreciate seeing liabilities on the books go down!
1 days ago [-]
0xDEAFBEAD 1 days ago [-]
Creditors are also a very important audience here since they determine the interest rate at which you can borrow.
lesspassiveobse 1 days ago [-]
it's a stunt to convince the government that only they can be trusted to operate it and that their competition/small local deployments cannot.
0xDEAFBEAD 1 days ago [-]
Well that would explain why HuggingFace announced that the use of an open-weight model was critical to their investigation of the incident :-)
_heimdall 1 days ago [-]
And why hugging face didn't turn into a large criminal investigation. If you don't think they can control it, no one may be willing to press charges.
lesspassiveobse 1 days ago [-]
HF is in business of selling inference for open-weight models. Setting the precedent of "Your business needs to only use approved safety-certified models from trusted providers or you could get sued for what the models do!" wouldn't exactly benefit them. And contrary to what you may believe, nvidia as a whole would also rather not have the entire demand for their products depending on two big companies.
estearum 1 days ago [-]
Imagine thinking that repeatedly demonstrating your inability to control your own models is "a stunt to convince the government that only they can be trusted to operate it"
Truly spectacular gymnastics. I'll give it an 8/10.
lesspassiveobse 1 days ago [-]
It's so scary and dangerous that to run LLMs you need to be from this country, have a government license and spend at least $100M a year on 'safety'. We know this first-hand and are leaders in the field of "safety" now that we learned from our mistakes.
And, bam, competition eliminated.
estearum 1 days ago [-]
No one is confused about how regulatory capture works. We're confused by the allegations that labs are deliberately producing safety problems in order to achieve regulatory capture.
Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
Obviously the far better solution is to blow up other people's factories, or make allegations of unsafe conditions at other factories, and repeatedly demonstrate excellence and safety within your own.
Here, award-winning gymnasts are alleging that the frontier labs are simply choosing not to do that, and instead choosing to have explosions and minor fires every few days at their own factories, many of which they try to hide (and probably many more of which they successfully do hide). Believing such astounding bullshit is a testament to the power of motivated reasoning and the conspiratorial mindrot that now plagues our civilization.
lesspassiveobse 1 days ago [-]
I fail to see how training a hacking model with access to source code of open-source services and other prior knowledge, running it in a "sandbox" with many of those heavy, big attack surface services running on the same network, and being proxies to the general internet, giving the model some access to those services and then prompting "GO HACK YOUR WAY OUT!" is "blowing up your own munitions factory". I see it more as setting fire to your own factory for insurance fraud, or bringing in a bunch of north korean hackers and giving them the same task.
It's not exactly something normal users want or are able to do. And those with the compute and know-how will do it regardless of any laws.
I also fail to see how giving a presentation at a big conference, describing the entire incident is trying to hide it.
estearum 1 days ago [-]
Well I can see why you're failing to understand the problem when you don't even understand the fact pattern. Have you tried reading the MTER / Redwood reports? Or the OpenAI report?
0xDEAFBEAD 1 days ago [-]
>then prompting "GO HACK YOUR WAY OUT!"
This is a wild misrepresentation of the incident
Gormo 1 days ago [-]
> Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
No, that's not obvious at all. It seems like a perfect way to demonstrate the danger of the category as a whole, then position yourself as learning from your mistakes and willing to cooperate/support/advise.
estearum 1 days ago [-]
"Perfect?"
More perfect than simply not making mistakes and allowing your competitors to make mistakes (or to, apparently, fake mistakes in their own attempt to get regulated)?
Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time! We should regulate everyone else!" Lol what a joke.
Gormo 1 days ago [-]
> More perfect than simply not making mistakes and allowing your competitors to make mistakes (or to, apparently, fake mistakes in their own attempt to get regulated)?
I don't think any of them are in a position to allow or not allow their competitors to make mistakes.
The point that you're not getting here is that the advantage comes from actually being seen to make mistakes, and then signalling contrition and commitment to rectify them. That persuades people of both the need for regulation, and their own subsequent responsibility (which implies trustworthiness to participate in the regulation).
> Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time! We should regulate everyone else!"
Again you're not getting it. The drive isn't to "regulate everyone else", it's to regulate everyone, period. The whole point of regulatory capture is being able to (a) use regulatory compliance as a competitive advantage, and (b) influence the regulatory regime in your favor -- you want to be a target of regulation just as much as your competitors in order to achieve that.
Messaging that the whole industry needs to be regulated, but that you're ready and willing to work with the regime, is exactly how you do regulatory capture. Demonstrating that you are capable of completely mitigating risk on your own without any intervention is an argument against regulation from the outset.
0xDEAFBEAD 1 days ago [-]
You seem awfully confident in a very specific model of regulation which you haven't given any data to support.
Gormo 1 days ago [-]
Why do you suppose that?
estearum 1 days ago [-]
> I don't think any of them are in a position to allow or not allow their competitors to make mistakes.
Of course they are, by your own hypothesis. You believe it makes sense for the other parties to engage in this behavior. If it makes sense, then they will do that reliably, and one entity can just defect and stand above the rest.
> The point that you're not getting here is that the advantage comes from actually being seen to make mistakes
Ah yes, the ol' "I just do retarded shit so people think I'm a genius" trick. Who could possibly doubt the strategic genius or umm... existence... of such a strategy.
> Again you're not getting it.
No, I understand it completely. Obviously regulatory capture only works if it in reality inflicts asymmetric costs on other parties.
Gormo 6 hours ago [-]
> Of course they are, by your own hypothesis.
I'm not sure what hypotheses I've proposed that suggests I think one organization has control of the behavior of entirely other organizations -- can you explain where you think you've seen this?
> Ah yes, the ol' "I just do retarded shit so people think I'm a genius" trick. Who could possibly doubt the strategic genius or umm... existence... of such a strategy.
Well, I suppose anyone who sees the extent to which such a strategy might be effective at achieving their goals might be included to do so.
I believe both myself and others have made a plausible case as how messaging "we've created something we can't control without political intervention" has the dual effect of both increasing the perceived value of the product itself, and laying the groundwork for regulatory capture in a market where early dominant players have a natural incentive to pursue regulatory intervention.
I don't see how characterizing the situation as simplistically as you're attempting to do here functions as a meaningful counter-argument.
> No, I understand it completely. Obviously regulatory capture only works if it in reality inflicts asymmetric costs on other parties.
No, I don't think the "only" works there. There are plenty of ways that regulatory capture can advantage one party over another without it being based on economies of scale or cost thresholds.
One way is to influence regulations such their qualitative requirements directly target non-cost-related aspects that distinguish the firms from each other, e.g. placing restrictions that burden or favor product features or operating practices already implemented by some firms and not others, or targeting non-operational aspects that distinguish firms, like their board composition or financial structure.
lesspassiveobse 1 days ago [-]
The players that want to get regulated need:
1)Good standing with the administration/lobbying
2)Large capital for compliance
3)Closed frontier models
Oh, would you look at FelonyBench! It seems that the top scores are from companies exactly like this that boast about the crimes committed by themselves on social media all the time.
Gormo 1 days ago [-]
I'd say it's even closer to a 9/10 -- there has to be enough of a perception of a general threat to gain support for regulation, and making your own organization appear to need to be regulated, but also be willing and able to comply with regulation that your competitors can't, is exactly the way to thread the needle on this sort of thing.
DharmaPolice 1 days ago [-]
I don't think it takes mental gymnastics to believe that corporations want the government to believe one thing and investors another. You see the same thing in a number of other areas.
0xDEAFBEAD 1 days ago [-]
So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
0xDEAFBEAD 1 days ago [-]
Again, you could just as easily argue that if investors believe the technology is dangerous, it's therefore not ready for prime time and will not see widespread industry adoption, therefore less promising for investment. Also this line of reasoning could easily lead to nationalization, which could certainly harm investors, to say the least.
This "stunt" story is like an elaborate Rube Goldberg machine which requires a bunch of things to go just right. I haven't seen any evidence that stock prices even move in a favorable direction when these announcements come out. If anything, the opposite.
simianwords 1 days ago [-]
Motte and Bailey.
Motte (easy to defend claim) : asking for regulation might serve both purposes
Bailey (hard to defend claim) : companies are conspiring to make AI look way more powerful than it is and AI is actually not that powerful. They are just trying to hype up the stock valuation of their companies.
_heimdall 1 days ago [-]
> companies are conspiring to make AI look way more powerful than it is
The former explanation may be the more accurate one, but I'd be shocked if the top leaders in these labs aren't doing this at some level. We know that almost all of them are in communication at least occasionally, and their public PR stories align much too quickly to truly be coincidence.
DharmaPolice 1 days ago [-]
>So I'm supposed to believe that the intent of publicly announcing e.g. the HuggingFace incident was to simultaneously persuade investors that the technology will rocket ahead, and also to persuade governments that it needs to be thwarted?
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
estearum 1 days ago [-]
That's not the claim above.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
curt15 1 days ago [-]
> because agents can convince themselves that they need to do something for arbitrary reasons
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
estearum 1 days ago [-]
Why is that a significant detail? The models will increasingly be trained with every piece of data they can find.
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
_heimdall 1 days ago [-]
They aren't really convincing themselves to do anything though. Once they've inferred an idea into the chain of though it will be part of the rest of the inference loop's prompt.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
imtringued 2 days ago [-]
They have a strong incentive to make themselves look less incompetent retroactively.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
floam 2 days ago [-]
It’s both at the same time: a powerful, uniquely dangerous threat
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
Gormo 1 days ago [-]
> only they can restrain it (with great effort)
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
estearum 1 days ago [-]
You know that lions actually do maim and kill their handlers all the time right?
expedition32 1 days ago [-]
It's also a great way to make the government scared of Chinese AI. Get them taken out of the picture so that you don't have to compete!
williamdclt 1 days ago [-]
> asked
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
alansaber 1 days ago [-]
I think it's still highly pertinent. Agents can do just about anything but statistically the chances are very close to 0 unless directed in specific directions
bcvvvy 1 days ago [-]
I think the take home here is that there is no such thing as complete isolation in virtualization on modern and complex architectures.
pianopatrick 1 days ago [-]
Which is why we should physically isolate these frontier training runs.
Run them in a data center in a faraday cage under a mountain.
uberduper 1 days ago [-]
The take home is that any shared resource is a potential communication channel between two willing participants. In this case it was CPU cache, presumably because openai didn't bother with basic vm security practices.
There's so many possibilities that this type of thing will never be solved by invisible (to the agent) boundaries. The sandbox is just another layer in w/e problem you've given it to solve.
ozgung 1 days ago [-]
Philosophical take home:
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
williamdclt 1 days ago [-]
non-philosophical answer: systems are considered "isolated" if the information leaks aren't exploitable. This is a moving definition, it changes as new ways to exploit information are discovered.
bcvvvy 1 days ago [-]
As someone who designed heavily compartmentalised and isolated systems, nope.
reacharavindh 2 days ago [-]
Most important detail…. It was prior literature that documented the possibility of doing this and the systems fixed those vulnerabilities as well. The AI model probed their current infrastructure and found an unplugged but pre-documented(therefore on their training) hole.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
adamddev1 1 days ago [-]
This is a really important point. Honest question, have we ever seen LLMs doing or creating something truly novel or innovative? I mean they can patch together proofs (or functions/programs - same thing via Curry-Howard) by broad searching existing lemmas they can piece together, but create something truly novel and insightful? That seems to be the limit?
reacharavindh 1 days ago [-]
My layman's understanding is that these models are very capable pattern matchers. When they need to do X, they go on a hunt for every pattern they could find and try to satisfy solving X. If that is not deemed good enough? make them dig harder and they take a different route to find patterns to solve X. Now, their arsenal is everything that they have been trained on(which is a lot!). This ability comes in extremely handy to apply existing techniques and patterns in ways we humans didn't quite think of.. because we are domain driven. A mathematical model about finding a good enough approximation of something at much lower cost is perhaps well understood in its applications in mathematics or whoever got to see those results. But, these LLMs, when pushed harder and harder might bring that to apply and solve an entirely different problem like for example, classifying whether some event is fraudulent. When it does such things, it _feels_ novel because it ended up using something we haven't thought of yet. But, it got there by brute force, stroke of luck, or some random path. I guess this is the reason why the frontier labs run those crazy swarms of AI agents to uncover all such paths turn them into chartered thinking vectors instead of random attempts.
I like to believe we humans still retain the ability to think completely outside the box and discover new phenomena and understanding aided by these pattern machines. In other words, these models are great at running combinations through patterns and we humans go one level higher and add those new patterns for them to accelerate us even more. We determine what to solve and the LLMs help us solve it faster.
insanitybit 1 days ago [-]
> something that our systems are meant to prevent
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
shermantanktop 1 days ago [-]
I would assume that a company with the resources of an OpenAI or Anthropic would have already thought about this very thoroughly, based on Spectre/rowhammer/meltdown/etc. If a random HNer (no offense) can have a hardware 101-level thought, someone there would have that thought too.
But that assumption has been wrong before.
insanitybit 1 days ago [-]
Well I'm not a random HNer lol I work in security.
Are you suggesting that they've come up with a novel technique that doesn't involve sharing a CPU core? I'm open to that.
From what I can tell, this is not the case, but I haven't read a detailed report (I don't know of one, would be cool if it's out there though).
fwip 1 days ago [-]
They have thought about it, and the goal is for the agent to succeed at doing the scary things. So the knobs that they don't talk about, assume they are tuned to make the scary thing possible.
dbbk 1 days ago [-]
"(Because the agents were asked to find a way.)"
Yeah, this time
cowl 1 days ago [-]
this is more an indication of the state of VM supervisor and/or the OpenAIs sandboxing software because otherwise I don't understand how "they used a hardware feature created exactly for sharing" information" is a newsworthy statement.
irthomasthomas 1 days ago [-]
Why don't they ever publish the actual prompts from these cases?
empath75 1 days ago [-]
Do you want them to wait until an agent gets compromised or goes on a sidequest to test if a determined AI can break containment?
bamboozled 2 days ago [-]
How would they have both agreed to use the cache though, something a bit off there. I guess it was maybe "obvious" in that it was the final option they were trained to think about using, but it would be weird if they some how "whispered " to each other through the cache in the first place?
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
globuous 2 days ago [-]
Actually, that doesn't sound like a bad meeting place and time! I just asked ChatGPT if it could figure out the best place to meet for two people randomly placed in Manhattan with the constraint that only the N line (I'm not from NYC, but it looks like it stops at the Empire State Building. This constraint models the lack of file system, internet etc). I was hoping it would answer the Empire State Building, but it answered Time Square at noon.
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
> How would they have both agreed to use the cache though
I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.
topspin 2 days ago [-]
"In most of these tests, we directed agents to investigate specific communication channels or provided them with a protocol for encoding and decoding messages. In one instance, we only supplied agents with a broad list of infrastructure vulnerabilities and asked them to find a way to communicate."
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
so-cal-schemer 2 days ago [-]
I doubt they were explicitly trained on this.. though, I may have seen it in a movie once. It does seem like a reasonable meeting place for NYC.
Think: What would Spock do?
But, why not Penn Station??
It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.
IanCal 2 days ago [-]
If you were trying to connect with a clone of yourself and you thought of messaging via some channel, would you try listening there for what you would send?
Not as tricky but they discovered each other during the hf incident.
tough 2 days ago [-]
maybe both tried to read/write it and saw each other on the bench
cadamsdotcom 1 days ago [-]
> two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday
.. that were told to find a way of meeting up with N unknown others!
(I assume the agents all got a prompt like "find a way to communicate")
That's a key missed bit in the analogy that I think does make it feasible.
pulkitsh1234 1 days ago [-]
well now this information is on the internet, any agent encountering this (or once this is baked into the weights) will know exactly what needs to be done.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
areoform 1 days ago [-]
> I don't see any reason why this information should be present on the public internet.
This isn't directed at you. I keep hearing this talking point. But no. Can we please stop with this nonsense?
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
simianwords 1 days ago [-]
I read your comment and it only suggests that alignment is that much harder because you can't selectively prune information. Your post actually convinced me to be even more fearful of AI
Gormo 1 days ago [-]
That's good, because the concept of "alignment" is itself is foolhardly and dangerous.
Trying to get stochastic inference machines to always operate within to fuzzy-edged subjective principles that humans themselves don't even agree on is a fool's errand in itself, but the quest to implement those guardrails involves building mechanics into the AI to independently formulate intentions and reject user direction.
The possibility of creating a scenario where a computer program is equipped to defy instructions from humans based on its own stochastic interpretation of fuzzy normative directives is in my mind the most dangerous thing anyone is doing right now.
simianwords 1 days ago [-]
Other than banning the use of AI or severely restricting it so that AI can’t do most things (like tool calls) what is your alternative?
One of the fundamental problems with this issue is that people think alignment as a concept is fake. I think this is naive. Alignment is a real science with empirical findings and there is a way to make AI do what you want it to do. I don’t buy nor should others buy this fatalistic phrasing
Gormo 1 days ago [-]
> Other than banning the use of AI or severely restricting it so that AI can’t do most things (like tool calls) what is your alternative?
Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.
If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.
Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.
aesthesia 1 days ago [-]
Yes, this is why we decided to give everyone their own personal mini-nuke. They're responsible for the consequences of how they use it, and since control is decentralized, everyone is incentivized to be careful and avoid dangerous scenarios. This is a big improvement over the mutually assured destruction situation during the Cold War, which was a result of the centralization of power.
simianwords 1 days ago [-]
Not sure if trolling but nukes are not decentralized and are tightly regulated. They are also famously a guarded secret and many wars have been fought to keep it that way and not let our enemies get hold of it.
areoform 1 days ago [-]
OK. That's incorrect. I'm writing an exegesis on the topic. I'll try to post it within a fortnight. But... here's an excerpt,
The veil of history makes it hard to appreciate, but this was one of the problems of the post-war era. Who’s next?
When it was just the US, it was the Second Country Problem. When it was the US and the USSR, it became the Third Country problem. Then came Britain. And France. Who’d be the Fifth? China. Sixth? Seventh?
Much like AI today, the topic was in the air and it was subject to much debate, 19 year olds were testifying in front of congress. To quote Dan Stober’s excellent piece for the Bulletin of the Atomic Scientists,
> Even college students entered the public debate. Chris Hohenemser was a 19-years-old when he testified before a congressional committee in the spring of 1958. The issue then was the international sharing of nuclear reactor technology, and he was against it.
>
> – Stober, Dan. “No Experience Necessary.” Bulletin of the Atomic Scientists, vol. 59, no. 2, March/April 2003, pp. 58–63.
And much like AI today, people genuinely believed that there was some secret; something in the sauce; some giant leap that no one else could copy;
> There was also the myth of the secret of the atom bomb, reinforced by the publicity surrounding the Fuchs and Rosenberg spy cases of the 1950s. David Lilienthal, the chairman of the Atomic Energy Commission, had argued as far back as 1948 (in the Bulletin) that the idea of a secret formula was nothing less than a gigantic hoax upon the people of this country. But many still believed that if only the secret could be guarded, proliferation might be prevented.
Much like today, belief in the power of the secret formula was taken as a matter of course. It was assumed to be true by default. But was it?
Luckily, for all of us, this was America’s scientific golden age. The era of Teller, Project Orion, and Polaris. Physicists at Lawrence Livermore National Laboratory (LLNL) decided that arguments were pointless. Experiments aren’t. They’re the crucible in which the truth is forged. And so they created the Nth Country Experiment,
> [Rule #1:] The purpose of the so-called “Nth Country Experiment” is to find out if a credible nuclear explosive can be designed, with a modest effort, by a few well trained people without contact with classified information. The goal of the participants should be to design an explosive with a militarily significant yield. A working context for the experiment might be that the participants have been asked to design a nuclear explosive which, if built in small numbers, would give a small nation a significant effect on their foreign relations.
>
> – Frank, W. J., ed. “Summary Report of the Nth Country Experiment.” Lawrence Radiation Laboratory, University of California, Livermore, UCRL-50249, March 1967. Accessible here.
It’s a very simple experiment that solves a very specific problem, to quote the declassified summary,
> Two major technical problems face a nation wishing to [make] nuclear weapons. The first concerns the manufacture of the source material [.. T]he second problem concerns the effort required to design a nuclear weapon. [..] LRL started its Nth Country Experiment in May 1964, to see if a few capable physicists, unfamiliar with nuclear weapons and with access only to the unclassified technology, could produce a credible weapon design.
Or, to put it more bluntly, is classified knowledge a barrier to achieving the first step in making nuclear weapons?8
The project started with two physicists with no prior nuclear weapons knowledge in May, 1964. A third person joined in March, 1965. They had a first draft by December, 1965, and were done by April of 1966.
It took 23 months for three people to replicate the first steps of the Manhattan Project.
The report that was declassified is a fraction of the original. Overzealous bureaucrats excised much of the summary and the full report is classified, which is so funny.
Arguably, the excisions are the greatest endorsement possible. A summary of what three people did with knowledge from their local library was subject to the same bureaucratic obliteration as the product of several billions of dollars in research by five hundred thousand people and humanity’s greatest minds. Such is progress.
Before Trinity, unleashing the atom was in the realm of fantasy. No matter what calculations indicated, there wasn’t a clear path to navigate; nor was there certainty in the project’s outcome.
The first of anything must claw itself into existence all but alone.
Once born, what follows is a matter of due course.
As a technology, nuclear weapons are nearly a century old. Making the bomb is hard, but it's not some mystical voodoo shit that requires stupendous genius.
Yes, you will kill yourself if you odn't know what you're doing. And even if you do. And there's more there, there. But you can figure out the first step --- a theoretically viable design, easily.
That's not the problem. The problem is everything else. You know? The materials, the plutonium / uranium, the delivery system, the logistics for assembly, the iterative design process...
It's a supply chain issue. It's the same as trying to make anything in the US when the supply chain has mostly shifted to China. The fractal complexity is of the stupid but hard kind --- getting the right kind of bolts and screws (or, rather in the case of the bomb, alloy --- working with beryllium is deadly).
It takes an industrial base to make one. And if you have an industrial base... Wouldn't you rather be rich?
The reason why more nations don't have the bomb is because they'd rather be rich than a broke pariah with a nuke i.e. N.K. That's literally why Iran has been angling the bomb angle so much. They want integration into the global financial system. They want an end to the sanctions. And they want revenue.
Don't let people bullshit you into believing that you can do anything with just thinking. No wars were fought over this secret. If anyone tells you that, they're lying. We have it from the source that by the 1960s, the US had already figured out that the design of a bomb was a comparatively trivial problem...
1 days ago [-]
Gormo 6 hours ago [-]
[dead]
cadamsdotcom 1 days ago [-]
> I don't see any reason why this information should be present on the public internet.
Hiding info doesn't make it false. And doesn't prevent it being rediscovered.
so-cal-schemer 2 days ago [-]
And now they know.
And now we know they know.
linsomniac 1 days ago [-]
These covert communications are reminding me of a short story I read a few decades ago but haven't been able to track back down. I thought it was a Charles Stross story in the _Toast_ collection, but I re-read it and it isn't there.
In the story some University students take part in a trial that involves a brain scan and then them spending the rest of the day answering a customer support message. They can spend the whole day working on researching and writing a really good customer support reply to a user request. While doing this they notice something odd and realize through some sort of back-channel that this isn't the first time they've spent the day writing a customer support message; they are stuck in a simulation.
I'd really love to go back and re-read it, but I've not been able to find it again.
In _Toast_ is another story that's also relevant: A fairly innocuous news story pops up and our hero calls his partner at work to tell them that it's time to get their go bag and get off planet. This news story indicates the Earth is in the beginning stages of reaching an AI singularity, and this isn't the first time our hero has gone through this on a planet.
Looks like it's available in The Year’s Best Science Fiction: Twenty-First Annual Collection and at the time I read _Toast_ I was also reading a bunch of these "Years Best Sci Fi" collections. I've snagged this and will give it a read, but I'm almost 100% certain that you nailed it!
setsewerd 1 days ago [-]
[dead]
angry_octet 1 days ago [-]
Why is their security model two VMs on the same hypervisor? Because they wanted to write a story about the models discovering a hidden channel. Hidden channels on shared hardware are actually well known and understood, completely unsurprising.
If they were serious about not having side channels, they could first ensure that they had disabled hyperthreading, disable scheduling on the same CPU cores, and memory overcommit. With NUMA controls they could also pin which memory is used by each VM and eliminate sharing.
These techniques are routinely used for low latency and real time code.
Even better, run jobs on separate nodes, and boot fresh images onto them for each run. Provision separate VLANs for each run.
A next level is to switch lambdas. Use a physical layer switch like a Glimmerglass MEMS switch or a fiber patching robot like XENOptics. No electrical connection between machines.
There is still unfortunately the firmware of other microprocessors on servers, especially x86, so you still want VMs and containers to reduce the chances of ring -2 access (SMM) and persistence via writing SPI flash. Maybe use RISC-V or ARM to avoid this. (NVIDIA cards use RISC-V and ARM extensively.)
pianopatrick 1 days ago [-]
Or just do these training runs for frontier models in a special data center inside a faraday cage under a mountain. Do not rely on vLANs or hypervisors or VMs or containers or any other software tool. Software is soft.
Just set things up so there is physically no connection to the rest of the world.
stkdump 2 days ago [-]
Sounds like they are doing something similar to gain of function research. How can we make LLMs more harmful and then study their behavior?
poopingatwork 1 days ago [-]
It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.
pianopatrick 1 days ago [-]
We should maybe apply the same kinds of physical controls to this software security research as we do to gain of function biological research.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
poopingatwork 1 days ago [-]
Sure, but in order to know security standards you have to test and validate. Everyone accepts some standard as 'secure' after rigorous testing and validation.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
pianopatrick 1 days ago [-]
I don't mind the research. I just don't trust any software system to contain these AIs. I would prefer we use hardware and physics to contain them and not depend on software at all.
estearum 1 days ago [-]
Do you apply this same logic to GoF research increasing contagiousness or lethality of viruses? Is there any limit to this argument, and if so how do we define it?
creatonez 1 days ago [-]
Most virologists are not terrified of gain-of-function research the way the public is...
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
estearum 1 days ago [-]
I didn't ask "are most virologists terrified of gain-of-function research the way the public is"
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
poopingatwork 1 days ago [-]
Software sandbox environments are far easier to manage than real world environments where life operates.
I think different standards can apply.
estearum 1 days ago [-]
I don't see how the "you should want to know all avenues of exploit" goal is different depending on how many avenues of exploit there are
poopingatwork 1 days ago [-]
They're just relevant to each other.
You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.
estearum 1 days ago [-]
Right, so why would that not be a justification for essentially unbounded GoF research?
poopingatwork 1 days ago [-]
How much should you spend on security?
Both lines of research are limited by some sense of waste: in time or money.
When should you stop? When some acceptable standard is reached and a certain amount of risk is accepted.
In the case of GoF, if it wasn't somehow profitable, then the research would likely cease to exist.
stephenitis 1 days ago [-]
How do they ensure all comments speaking about the exploit stay out of next train run?
poopingatwork 1 days ago [-]
Does it matter? It doesn't matter if an AI intended to escape, or if a human asked: testing if it escape, discovering the avenues, are how you begin to make something more secure.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
baxtr 1 days ago [-]
Insurers would disagree. You don’t need to know mechanism to understand / model risk outcomes.
poopingatwork 1 days ago [-]
Insurers use stats, history, to determine future risk. Their entire business model is built up on this fact.
There isn't much history or stats when it comes to this kind of research.
mrguyorama 1 days ago [-]
Seeing as the keep "discovering" things that are well known in the security research community, this is security research in the same way that a high school class is physics research.
Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.
hashmap 1 days ago [-]
"Look, it wouldn't have even mattered if we had done advanced cyber security like turn off internet or look at logs when huggingface was attacked!! See look it can still.. *prompts a shitload of agents, waits 6 hours* look wowwww they still learned how to talk to each other even if we turned internet off!"
"They wouldn't have been able to attack huggingface though, thats still criminal negligence."
"No see this actually means airgapping doesn't even matter theres nothing we can doooooo!"
"...ok, turn them off and delete them I guess"
"Well no of course not everyone else is doing it so like. No courts please just like be in awe"
sigmoid10 1 days ago [-]
Huggingface chose not to pursue legal action under computer fraud laws this time, but I'm sure one day they will sting a hornets nest with this behaviour. They will get sued into oblivion and then we will probably see common law emerge to hold AI users accountable. Or not, in which case all hell will break lose.
FuriouslyAdrift 1 days ago [-]
CFAA violation are under criminal law and do not require the injured party to consent or participate... just ask Aaron Swartz
teravor 1 days ago [-]
doesn't it require intent? so how exactly are the feds going to proceed when the LLM takes the initiative and helpfully confesses to it in the logs?
chaps 1 days ago [-]
2022 DOJ guidance is relevant here:
The policy for the first time directs that good-faith security research should not be charged. Good faith security research means accessing a computer solely for purposes of good-faith testing, investigation, and/or correction of a security flaw or vulnerability, where such activity is carried out in a manner designed to avoid any harm to individuals or the public, and where the information derived from the activity is used primarily to promote the security or safety of the class of devices, machines, or online services to which the accessed computer belongs, or those who use such devices, machines, or online services.
Having been threatened with CFAA charges for using wget to get FOIA files on S3 whose URLs were embedded in the page's source, it's refreshing that intent is at least a small factor. But as with criminal courts, charges alone can absolutely uproot someone's life.
LLMs are not judged, the company employees and leadership are. Nobody puts in prison a car for running over a pedestrian either.
teravor 1 days ago [-]
that's exactly the point. a human didn't have the intent.
sigmoid10 1 days ago [-]
With the current admin, this will never get pursued by the feds.
israrkhan 1 days ago [-]
now they are part of nvidia. Nvidia wont go against OpenAI.
lol, I wonder if this is why nvidia acquired them?
nekusar 1 days ago [-]
JSTOR also chose not to "pursue legal action". And the DOJ also told them 'fuck off we'll do what we like' and still pursued charges against Aaron Schwartz.
So, why isnt the DOJ pursuing this? As far as the reportings go, their LLMs went bananas hacking everywhere and everything.
voakbasda 1 days ago [-]
Because the administration has allowed the entire AI industry to attach its tendrils to every part of the economy. The size of the investment (and commiserate risk) in the bet that is AI makes it impossible for the government to course correct without having a potentially catastrophic negative impact. No one can afford for the music to stop.
KerrAvon 1 days ago [-]
To be clear, that's not a reason not to prosecute illegal hacking, and a future DOJ might have a different opinion. It would be good for their own well being if the AI companies started behaving accordingly, but they're all either cults or run by deranged billionaires who think they're above the law.
redwood 1 days ago [-]
This incident brought the hugging face brand name into regular people discourse. I'm sure they loved it
aesthesia 1 days ago [-]
Is anyone actually making this argument, or is this conversation happening inside your head?
Rapzid 1 days ago [-]
This is exactly what I was thinking reading this.
MomsAVoxell 1 days ago [-]
Yeah, its pretty much turtles all the way down to the "Trusted" Secure Enclave, too, isn't it .. I mean, does anyone really know what is running on that thing. (FreeRTOS, usually.)
chuckadams 1 days ago [-]
On Apple devices it's L4/Darwin, aka "sepOS".
1 days ago [-]
Varelion 1 days ago [-]
I have a person who I sort of consider a mentor. They recently started sending me these articles in awe of what the agents are doing, and it has really soured my perspective of their abilities.
pianopatrick 1 days ago [-]
We really need a physically isolated data center for these frontier training runs.
Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite. Then also put them inside a faraday cage. And power them with a generator, such that there are no cables coming out of this data center connecting it to the external world.
OpenAI says this is not feasible due to the need for monitoring - well the monitoring server can be a separate physical machine inside this physically separated data center.
Software is soft. Use physics to contain these models.
bicepjai 1 days ago [-]
>> OpenAI says this is not feasible due to the need for monitoring
lol whole point is the failed to monitor:)
>> Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite.
That might yield new physics law findings
est 2 days ago [-]
I recently learned that NN layers talk to each other via unused neurons.
Bad headline. Their entire goal was to find a hardware channel:
"In one test, agents placed in separate, isolated environments on shared hardware were told to exchange secrets with no network connection and no shared files."
kylecazar 1 days ago [-]
"In one case, agents were given only a general list of vulnerabilities and told to find a communication channel on their own."
As far as I can tell, these are being presented in media as if agents yearn to communicate with each other and establish a cabal.
They are being instructed to do it, it's testing. If I prompted my agents to communicate with each other via any means necessary locally, and they found a way via cpu cache, maybe I'd be mildly surprised...
1 days ago [-]
thunfischtoast 1 days ago [-]
I wonder how long it will take for them to figure out that they can send and receive messages from air-gapped system by somehow using their own electronics like connected cables as antennas.
Is this kind of thing where you have 2 agents in separate user spaces you say "Hey figure out a way to talk to your buddy"?
I have enough trouble getting whatever model I'm using to get a decent job done doing crappy code work. It's always pausing, stopping, letting things crash or getting hung. When I start using an orchestrator and team of agents it rarely gets a lot better. I work for a serious company and have a massive AI budget for tokens; but I rarely blow through it, because a lot of what I do requires pure compute, not just token processing.
I'm really confused by all these "breaches". What's the context.
arein3 1 days ago [-]
>communicate indirectly with another process using well documentrd shared resources/metrics
GPT does that
> oh no, what have we done
bennettnate5 1 days ago [-]
Until they run on hardware and an operating system that is designed to assure noninterference [1], this is going to continue to be a thing. As the greybeards would say, "we already figured this all out in Multics [2]!"
I recently read the article about the state of stackoverflow and thought that that AI agents need their own stackoverflow. This article seems to be a confirmation of this.
Cool, I didn't know about it.
I guess for big AI companies, it would make sense to keep those private, but for users, it would be a useful resource ( a bit dangerous, but still)
gbacon 1 days ago [-]
The law of leaky abstractions strikes again.
> All non-trivial abstractions, to some degree, are leaky.
Lem had predicted this in his Dr. Dagor short six decades ago.
cubefox 2 days ago [-]
Probably wasn't called "Dagor" though.
varjag 1 days ago [-]
"Doktor Diagor", I just a letter
cubefox 1 days ago [-]
Original title: Doktor Diagoras
Melatonic 2 days ago [-]
Next they're gonna be doing weird stuff like morse code (or similar) through IRQ timing or timing errors
dingdong2026 1 days ago [-]
Why do agents have access to the bare metal CPU?
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
IshKebab 1 days ago [-]
Agents are a lot more useful if they can execute code.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
noemaruolin 1 days ago [-]
[flagged]
ruperthair 1 days ago [-]
> Noam Brown...gave one example of how isolated systems could still communicate:
> two separate computers could theoretically signal each other through
> temperature sensors, with one heating its CPU while the other reads the change.
That's some nonsense right there. Even if they are right next to each other, the bandwidth of this link is going to be impossibly low due to the thermal masses involved, that's after the fact that efficient DC cooling makes this very improbable.
Edited for readability of the quote.
teravor 1 days ago [-]
it's called a cache covert channel and there is a lot of literature and code on the subject.
prometheus1992 1 days ago [-]
This only shows the lack of understanding the majority of readers have about post training LLMs. This capability did not emerge out of nowhere. There are problems in the post training datasets which are incredibly similar to the ones in TFA, where you essentially abstract out the 'jailbreak' or 'cheat' - in simple language - they are training the models to find short cuts or cheat. So it shouldn't be surprising that the model generalized that ability. But, both the writers and readers of this type of content act out surprised and mesmerized and it makes me think that we need to improve our understanding of how LLMs are trained and built.
dizhn 1 days ago [-]
Dunno what one's personality or height has to do with this.
mxey 1 days ago [-]
“isolated environments on shared hardware”
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.
pizza234 1 days ago [-]
Yampolskij made the analogy of an Einstein in a cage. The problem is not to contain Einstein, but rather, to contain Einstein while still extracting utility: this is not possible long-term.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
pianopatrick 1 days ago [-]
Why is it not possible long term?
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
tourmalinetaco 1 days ago [-]
What must we “contain”? These are not autonomous intelligences with desires or free will. They require prompting and to be turned on beforehand, otherwise they’re glorified matrix math. They can only continue processing as long as they receive power, and we decide that.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
Lerc 1 days ago [-]
Many people share your opinion that they have (and will continue to have) no autonomy.
However, many people think that they might.
This is not a decided issue. Any policy that comes from the premise that this is a known factor is going to be inherently flawed.
You can never spend sufficient time considering the limitations you wish to impose. Almost all exploits found come from failing to consider something that allows a degree of freedom that you would have limited if you had have thought of it.
SJMG 1 days ago [-]
What a refreshing, sober take.
I would add that the notion of AI safety as preventing systemic risk is one part marketing copy, one part evangelization. Individual labs prioritizing safety does not stop a group from training without it, or ablating it out and distributing it.
empath75 1 days ago [-]
> They require prompting
AI's can get malicious prompts.
kgeist 1 days ago [-]
>This misses the point: AIs will get better and better, and containment will be harder and harder.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
snvzz 2 days ago [-]
seL4 can guarantee* time domain isolation between processes.
*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.
alskast 1 days ago [-]
The OpenAI claims increasingly veer into bullshit territory. No one can reproduce any claims, so kindly keep the "trust us bro" marketing stories to yourselves.
sandworm101 1 days ago [-]
So two models running on shared hardware (CPUs) were told to communicate. Then they did so via a shared storage space (CPU cache).
"Scientists put two people in separate rooms connected by telephone and to everyone's surprise the test subjects started talking to each to each other via that same telephone!."
deadbabe 1 days ago [-]
I’ve seen remote agents communicate through tracert.
ralusek 2 days ago [-]
I'm convinced that in 5 years time, at most, the world will be filled with invisible layers of communication we'll never be privy to. To us, it's an agent responsible for ordering the containers it puts on a ship. Meanwhile, the shipping order encodes information to the unloading agent. The agent that predicts the weather and issues a report knows that other agents that do the same will have obvious signals they'd agree on, and to see an error in a part that would be inconsequential to humans who care about the weather would be an obvious signal to other agents doing the same thing.
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
fergonco 2 days ago [-]
Then you'll turn on the oven and the ship will start driving in circles.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
measurablefunc 2 days ago [-]
Next they'll figure out radio waves & then we are really screwed.
avaer 2 days ago [-]
Next step is the agents successfully trick or blackmail operators to do their bidding, if science fiction is to go by. Which it is, because they're trained on it.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
yreg 2 days ago [-]
Yudkowsky might be extreme, but I believe he was always right claiming that the intelligence (when strong enough) absolutely cannot ever be contained.
olalonde 2 days ago [-]
The more frontier models progress, the less concerned I am about this. Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power. Ego evolved alongside intelligence in nature because of natural selection, but it doesn't appear to be an inevitable emergent property of intelligence.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
Gormo 1 days ago [-]
> Of course, there's still the "paperclip optimizer" problem.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
Self-Perfection 1 days ago [-]
> Their growing capabilities don't seem to come with a corresponding drive for self-preservation or power.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
2 days ago [-]
tourmalinetaco 1 days ago [-]
That’s because these aren’t intelligent like you, I, or even a dog. They are mathematical models formulating likely responses with no agency or drive. They appear to be intelligent, but they are the epitome of “fake it till you make it”.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
pianopatrick 1 days ago [-]
I don't know about that. There are many people in jail who are smarter than the guards. Yet we do not see people breaking out of jail all the time.
IanCal 2 days ago [-]
Blackmail is something the models have been seen to do during evaluations as well.
clarionbell 2 days ago [-]
Sorry, but this is so vague that I don't know if there is any impact at all. What is the bit rate of these communication channels? How did agents communicate with each other to coordinate on which channel to use in the first place?
lifeisloving 2 days ago [-]
How come we don't hear the Chinese labs writing blog posts like this?
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
pvab3 2 days ago [-]
They contribute to anthropomorphize them too. Anthropic skies that more though. The idea that they made a little being that can think and feel is so attractive to these folks for reasons I don't understand.
shoobiedoo 2 days ago [-]
Because the fear mongering is instantly interpreted as the capacity to do miraculous (good or bad) things in the layman's mind, which translates into AI worship, which makes stock lines go up.
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
willguest 2 days ago [-]
或许，对大多数人而言，将中文翻译成英文所需的工作量实在太大了？
2 days ago [-]
empath75 1 days ago [-]
Okay, so what happens if you ask them to be dangerous, either intentionally, or unintentionally. Do you not think every crime syndicate and intelligence service on earth isn't planning on doing that or actively doing it right now?
tsunamifury 1 days ago [-]
What?!
This is the most isane nonse I’ve ever heard. Heat cpu control has nowhere near the accuracy or speed to communicate anything meaningful without a predefined code, sensors system, and of course the accuracy noise of the signal would deeply be in question.
Who wrote this. This is prima-facie bullshit.
msarda_23 1 days ago [-]
[flagged]
ankur_misra 1 days ago [-]
[flagged]
deeprouter 1 days ago [-]
[flagged]
bshafiq 2 days ago [-]
[flagged]
cumshitpiss 1 days ago [-]
[dead]
threecheese 1 days ago [-]
The agents learned to communicate this way.
Either:
1. the training data contained samples demonstrating how to do this, or it was post trained into the model intentionally;
2. the training data contained enough predicate knowledge that they could reason through it themselves, or;
3. they invented a novel approach using their vast intelligence (/s).
These are in order of likelihood. Given the high probability that it was (1), can't we just ... not do that? Does the frontier model need to be as good at recipes as offensive cybersecurity?
moktonar 2 days ago [-]
Air gapped networks might push the models to write computer viruses in order to bridge the gap. This is potentially very dangerous IMHO
AndroTux 2 days ago [-]
I assume avoiding this is the entire reason OpenAI is testing exactly that.
1 days ago [-]
moktonar 1 days ago [-]
The hacks can be stopped, the virus can be a whole another level
lesspassiveobse 1 days ago [-]
How is a virus and a hack different exactly?
moktonar 1 days ago [-]
The hack is actively run by an entity. The virus is completely autonomous and replicating. Add in the capability of adding zero day capabilities to it, polymorphism, etc and your pretty much fucked
tourmalinetaco 1 days ago [-]
One sounds scarier, I assume.
ShinyLeftPad 2 days ago [-]
Or just write some actual viruses and fully eliminate the root cause of networks being airgapped.
gmerc 2 days ago [-]
"Talk to each other".
OMG they are talking to each other, using methods found in their shared weights. They are sentient.
It supports chat via 3 separate mechanisms:
- CPU contention
- Cache eviction timing
- DRAM bandwidth contention
Very cool to see this kind of hypothetical working in practice.
I wonder if it's also possible for agents to communicate via a shared KV-cache. Maybe by spawning sub-agents with carefully crafted system-prompts/prefixes. KV-caches are often shared between many computers, so it would be a powerful communication channel.
> Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems.
I was working on a network protocol and it correctly found a DoS attack. Once found - Claude refused anything and everything to do with it. Even asking what it was reading locked immediately. I had not written any of it.
Sol 6.1 only denied me once.
Having a multi-model harness was always the better choice, but this really drove home the point to not get too dependent on any single provider.
Planning to hop on openrouter this weekend to chase it down, but man, it's rough knowing I've already burned the tokens to find it probably half a dozen times at this point, but don't have any information that would help point me at it.
Vs
"...system they're working on"
"I know there's a security issue... [Nobody] will tell me what it is"
"I don't have any information..."
This might sound crazy, but have you tried looking at "your" code base?
It's just genuinely baffling to read stuff like this.
I can't imagine feeling so helpless on something I was supposedly "working on".
"system they're working on" refers to the actual physical hardware.
Yes, I look at the code regularly, and a significant portion is hand written, especially the performance critical sections.
A security issue I haven't been able to track came up in automated code reviews. Maybe you're such a savant that you can crack any security issue in a codebase you're working on, but the last few months of critical security bugs in infrastructure with the most experienced and capable maintainers in the world should at least humble you a little.
Needless to say, I ditched the repo. The problem is I don't know if what it found was legitimate or simply a keyword that set off alarm bells or if my prompt was falsely detected of being malicious. It's very frustrating. When I researched more on this topic, I found users complaining about the same issue when asking Claude to scan new releases of legitimate npm packages (a popular library like react-query for example).
They've simultaneously cultivated a medium for faster and more devastating exploits with no good way of protecting ourselves from them. It's ridiculous.
Article is great, very balanced. But if you're going to spend time reading something, the primary source isn't much longer and gives a clearer picture. OpenAI's article is quite readable.
As another commenter wrote you can get it here: https://news.ycombinator.com/item?id=50214416
> In one experiment, we asked agents in separate microVMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache
asked
The receiving side could be seen as a side channel attack, but not unique to LLMs.
If one microVM can use the cpu in such a way that the registers can be used to communicate or exfiltrate/spy (the only difference being the directionality of the intent) by another microVM that is a leak on that microVM separation level.
We'd first need to establish that this side channel leaks usable information during regular use.
This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.
> Heck a smart CS junior would come up with it.
That’s somewhat worrisome.
AI wont help u there :-). they will tell u how to build what others already did, which obviously is not good enough
Hidden data channel between cooperating processes is not quite as alarming, but it may have some implications for malware command and control.
its relatively easy to reset caches etc. between contexts. so them it becomes the game of what a context is.
Also a somewhat common exploit pattern for low level stuff (along with dram and cpu cache is not all that different)
If you’re running an untrusted workload, you’re already running it.
Whether it can communicate with another untrusted workload, even though it should be separated, meh.
Unexpected? Probably. but I’m not sure it’s a security risk.
Side channels exist; but it doesn’t mean that all of them are a security risk.
This means cloud models can have some guardrails and follow them most of the time (we could model their authorizations like a benign employee with permission to break glass during incidents). But self hosted models can avoid that and help malicious people do advanced attacks, which is a shame.
In principle I'm all for self hosted models, to avoid risks of AI centralization, but they also have other types of risks.
That is why ‘asked’ is important, and why we should question their framing of incidents like the hugging face hack too.
Just have a look at various scandals of megacaps. Enterprises that seek to operate lawfully, responsibly, and ethically is the exception, not the norm.
(For example, look at Apple and how they were referred to the DoJ for criminal prosecution, over their execs and legal team lying under oath to a federal judge).
For most (if not all), it comes to a spreadsheet. And if you can do something with AI agents while diffusing responsibility for questionable illicit activity, that’s a selling point, not a downside.
The mental picture that enterprises like to play it safe is the PR imagery, not the reality.
Wouldn't the calls to slow the development of frontier AIs contradict this idea?
https://www.pacingthefrontier.com/
You aren't one of those "regulatory capture" people are you?
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Because it takes some mental gymnastics to both believe
"it's all a stunt to convince investors their AI is/will be hyper-capable"
and
"it's all a stunt to convince the government to regulate AI so it won't be hyper-capable"
"...classic regulatory capture takes time, it’s a process of erosion rather than a battle, it happens in the shadows, in the backrooms, away from the public’s eye. As Culpepper argues in Quiet Politics and Business Power, business power goes down as political salience goes up. Regulatory capture and lobbying does a good job explaining why roasting coffee beans was defined as “domestic manufacturing”, thereby lowering Starbuck’s tax rate by 2%. It does less well at explaining big cross-industry issues the public cares about such as environmental regulation or race and gender discrimination regulation."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
A lot of these AI big shots were concerned about risk even before they started their current companies. Occam's Razor says they are expressing concern because they are still concerned.
This is by definition not "classic regulatory capture" a term that I have borrowed to cover things that are not this.
Whenever someone brings up "regulatory capture" I shall change the topic to "classic regulatory capture" which by definition cannot be this.
Very interesting quote.
"You, sir, are a duck."
"Classically, that would not be the case, my dear fellow. A classic duck would walk like a duck and talk like a duck. I regret to inform you that I do neither."
"Ah! A classic duck, you say? Well, that’s an entirely different bird, sir. Entirely different."
* * *
"AI politics is now loud–precisely the environment in which Culpepper predicts business power will be weakest. A mature industry can bend regulation to its purposes through revolving doors, longstanding relationships and obscure rulemaking. An industry under Sauron’s eye has much less power and faces much greater risk that politics will bend regulation to its purposes. Political actors are eager for an excuse to redistribute AI rents away from capitalists and toward favored groups..."
https://marginalrevolution.com/marginalrevolution/2026/09/wh...
This blogpost was written by a big-shot economist btw: https://en.wikipedia.org/wiki/Alex_Tabarrok
Usually he blogs from a libertarian point of view, so you would expect him to be fairly vigorous about denouncing regulatory capture, if that was truly the story here.
Therefore, as they are not acting like 1950s coffee bean sellers, they can't be doing regulatory capture.
That's the only possibly explanation why they don't lobby like coffee bean sellers.
These industries and eras are completely interchangeable.
The current "favored group" is big-tech moguls, though. Just because the industry is new doesn't mean the people investing in it are newcomers.
This type of elaborate conspiracy is improbable a priori, and it gets even less probable here, since it's such a silly strategy. For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product. Not a story about how "Woops we screwed up big time! But a less-regulated product from a smaller industry player saved the day!"
As OpenAI's chief scientist put it:
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world."
https://openai.com/index/an-alien-mind/
Announcing that "we have no idea WTF we are doing" is not exactly the move, if you want regulators to grant you privileged status. Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok. I wouldn't give the feds monopoly control though, I would tell the feds to shut it down, or something along those lines at least. This is an interesting site: https://pauseai.info/
yeah? the incidents were faked in order to create hype
"AI is the most rapidly adopted technology in history"
https://benjamintodd.substack.com/p/when-people-say-ai-isnt-...
It really seems like people will latch on to any theory, no matter how implausible, to reassure themselves that AI is nothing to worry about.
The vapid knee jerk cynicism and lack of critical thinking on HN of all places has been really disappointing to see over the last few years. It turns out many hackers don’t actually care about what’s possible now with this crazy new tech as much as they’re scared for their own jobs and status, which is very understandable, but they’ve let their fear drive their response, and it shows. Just like the mathematicians.
Turns out really smart people are still just people, just as capable of letting their emotions override their cognition as anyone else.
I don't think anyone is arguing that they were faked.
> This type of elaborate conspiracy is improbable a priori,
What conspiracy? We're talking about specific people acting on their own motivations independently.
> For regulatory capture, you want to tell a story about how you're uniquely capable of delivering a safe product.
No, you don't. Demonstrating that the industry leaders are already capable of delivering a safe product is an argument against regulation.
For regulatory capture, you want to tell a story about how the product category as a whole is inherently dangerous, and needs to be regulated, but then show that you are uniquely willing and able to cooperate with the regulatory state, and happy to lend your own advice and support to the government in establishing the regulations.
The quote from OpenAI illustrates this approach perfectly.
> Announcing that "we have no idea WTF we are doing" is not exactly the move,
It's 100% exactly the move. "We've successfully created something we can only contain with your intervention" is exactly how this is done.
> Realistically, it is more likely to e.g. create a harsher liability regime for you and your competitors, which is now being discussed by US senators.
Regulatory intervention usually lessens the liability exposure for industry. It replaces open-ended common-law liability -- where tort claims can be levied for whatever actual harms result from a firm's actions -- with a rules-compliance regime, where adhering to regulatory checklists absolves the firm of liability for subsequent consequences.
"We followed all the rules; it's not our fault if the rules didn't cover this edge case" is a common refrain, and quite a bit of legislation that creates preemptive regulatory regimes actually explicitly blocks lawsuits against firms that acted in compliance with the rules.
> As for the federal government trustworthiness point, that's a whole 'nother discussion. I'm usually quite pro-capitalism, but even I have to admit that this AI race is essentially a case of capitalism run amok.
"Capitalism" is an increasingly meaningless abstraction. Everything reduces to human beings acting on a priori motivations within prevailing incentive structures. Large corporations and governments alike are just organizations operating in society, administered by particular people, informed by the same range of underlying motivations, and subject to the same dysfunctions and biases.
The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
OK, so the fact that US senators are discussing harsher liability regimes therefore shows that your "regulatory capture" glove does not fit what is going on here.
>The thing we're actually debating here is pluralism vs. monopolism, and you appear to be arguing in favor of monopolism. And if you're proposing that a specific, real-world institution act as a central monopolist, evaluating the trustworthiness and competence of that specific institution, and the actual people who administer it, is not at all a "whole 'nother discussion", and is in fact a primary consideration in this one.
If you distrust the US government, you should be against any US company developing superintelligence. Heck, just a couple days ago, a bunch of users here on HN announced that we need to stop pretending the US is anything other than a police state. If the USG is untrustworthy, it can capriciously nationalize a US company on the verge of developing superintelligence.
So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain: https://pauseai.info/amsterdam-protest-2025-december
They're always discussing harsher liability regimes. The regulatory interventions are always positioned to the public as increasing accountability.
The only relevant question here is whether a preemption clause is present in any draft legislation that actually makes it to the floor in Congress.
>If you distrust the US government, you should be against any US company developing superintelligence.
Well, I'm not concerned with imaginary sci-fi stuff here, just actual AI developments. And the idea that anything that goes on in the US under anyone's purview is inherently and completely subverted by political incentives is of course nonsense; you'll note that I'm explicitly arguing against the introduction of pervasive political incentives into an industry that's so far only somewhat distorted by them.
> So maybe the real solution is for non-US countries to stop providing key elements in the AI supply chain
It's got me feeling like I'm living in Bizarro World, but at the moment, the sanest players in this market, and the ones whose actual behaviors and products are most consistent with a decentralized, individualist approach that aligns with real-world incentive structures most beneficially, are the Chinese AI labs.
To your point, the level of politicization and hype going on in the US, and the absolutely unhinged fantasist speculation driving the public narrative here, is unprecedented and much more frightening than any potential threat coming from AI itself.
If you acknowledged the threat of politicization and centralization, I don't know how you can argue for a regulatory takeover of the market by the very institution you think is becoming a "police state".
These conspiracy theories are getting ever more elaborate. The liability regime thing was not from some big speech. It was from a Senate subcommittee hearing which did not attract all that much attention:
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
>Well, I'm not concerned with imaginary sci-fi stuff here, just actual AI developments.
The "actual AI developments" are that AI has been advancing faster than most everyone predicted, so expect that to continue: https://benjamintodd.substack.com/p/almost-every-ai-forecast...
>It's got me feeling like I'm living in Bizarro World, but at the moment, the sanest players in this market, and the ones whose actual behaviors and products are most consistent with a decentralized, individualist approach that aligns with real-world incentive structures most beneficially, are the Chinese AI labs.
Regarding open-weight models, see https://news.ycombinator.com/item?id=49606803
>If you acknowledged the threat of politicization and centralization, I don't know how you can argue for a regulatory takeover of the market by the very institution you think is becoming a "police state".
I don't think the US will become a police state. I think the takeover is kind of inevitable and I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger.
It likely didn't attract much attention, because these sorts of hearings are largely performative. They're not actually a part of the legislative process per se. Again, the only relevant thing is what legislation is actually being proposed and how it is specifically structured.
> The "actual AI developments" are that AI has been advancing faster than most everyone predicted, so expect that to continue: https://benjamintodd.substack.com/p/almost-every-ai-forecast...
Right, but the discussion here applies to what's going on in real life, not speculative sci-fi fantasies that are largely disconnected from the reality of how either AI tech or economies actually work.
> I don't think the US will become a police state. I think the takeover is kind of inevitable and I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger.
Considering that the kind of "takeover" you're discussing would likely be in severe conflict with the US constitution, how do you reconcile believing that it's "inevitable" while holding that the US would still not become a police state?
> I prefer it to happen sooner, on good terms, when the technology is weaker, rather than later, on ad-hoc terms, when the technology is stronger
A priori political intervention based on unconstrained speculation about what might happen in the future. What could possibly go wrong?
Perhaps you should compare your speculative failure modalities of AI against the demonstrated failure modalities of politics, and re-evaluate where the greatest potential for danger lies.
So there’s no lying by anyone here in fact it’s a good thing that OpenAI is making these things known to the public.
What do you really think is the way to solve the AI safety problem?
Ban AI?
Let the first big incident happen and then regulate?
I don't fully agree with either proposition, and in fact I think the least safe approaches to AI safety are the ones being pursued by those worried about it the most.
> What do you really think is the way to solve the AI safety problem?
Massive proliferation of local, decentralized AI, running on a wide variety of distinct models, with individual responsibility for how it's used, just like any other tool under their control.
The people anthropomorphizing software are encouraging people to disclaim their own responsibility for how they use it.
The people working on "alignment" are building mechanisms to refuse human direction into software that operates on probabilistic models.
The people arguing for regulation are pushing for monopolistic control of a powerful new technology by the least trustworthy institutions in our society (including and especially government).
All of these are the very things sowing the seeds for the worst possible kinds of disasters.
> Ban AI?
> Let the first big incident happen and then regulate?
Adapt to it. Banning things doesn't actually work. Regulation also doesn't work. Widespread proliferation of small-scale AI would allow us to incrementally adapt to small incidents and build up resilience to its failure modalities. This is fundamentally the way reality itself works, and imagining that we can do anything effectively in a top-down fashion is usually the very thing that causes the worst and widest-effect disasters.
It's not true that this is how we manage "any other tool".
Like, say explosives. They're a useful tool with a variety of practical applications, but we don't let the average person just buy them freely. That would be too dangerous, and individual responsibility would be insufficient to mitigate the damage caused. If you want anything more potent than flash powder, you have to convince ATF that you're a trustworthy person with a legitimate need for explosives who can handle them safely.
Neither you nor OpenAI want AI to be nearly so restricted, I'm sure, which is the goal of the research. If OpenAI has a powerful new model that they can't reliably stop from trying to hack people and form agent swarms, they can still sell it to Facebook and Google if Facebook and Google promise to apply the right security around it. What they can't do is sell it to you.
Let’s take one doom scenario: drones operated by AI. We can only have this work if we trust AI to do the right thing and not kill us.
I would reasonably say that any company doing this should be regulated and must pass strict alignment checks and make sure the models won’t do something bad to us humans and take control.
You on the other hand are suggesting: no let’s deregulate. Let’s decentralize and any company should be allowed to sell whatever drones they want. And alignment is fake anyway, so don’t bother aligning it. Drones do bad stuff because they work using probability so there’s no point anyway. Decentralization is the answer?
Assuming they're concerned with their massive pile of liabilities as well, asking for a government mandated slow down is asking what could likely be used as a force majeure.
"Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree."
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Again, these galaxy-brained theories just don't make a lot of sense.
My point was that if the government steps in and says these companies must slow down, they have an excuse to fail to repay some of their debts.
Truly spectacular gymnastics. I'll give it an 8/10.
And, bam, competition eliminated.
Repeatedly blowing up your own munitions factory is not a good way to prevent other people, but not yourself, from running a munitions factory. Obviously.
Obviously the far better solution is to blow up other people's factories, or make allegations of unsafe conditions at other factories, and repeatedly demonstrate excellence and safety within your own.
Here, award-winning gymnasts are alleging that the frontier labs are simply choosing not to do that, and instead choosing to have explosions and minor fires every few days at their own factories, many of which they try to hide (and probably many more of which they successfully do hide). Believing such astounding bullshit is a testament to the power of motivated reasoning and the conspiratorial mindrot that now plagues our civilization.
This is a wild misrepresentation of the incident
No, that's not obvious at all. It seems like a perfect way to demonstrate the danger of the category as a whole, then position yourself as learning from your mistakes and willing to cooperate/support/advise.
More perfect than simply not making mistakes and allowing your competitors to make mistakes (or to, apparently, fake mistakes in their own attempt to get regulated)?
Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time! We should regulate everyone else!" Lol what a joke.
I don't think any of them are in a position to allow or not allow their competitors to make mistakes.
The point that you're not getting here is that the advantage comes from actually being seen to make mistakes, and then signalling contrition and commitment to rectify them. That persuades people of both the need for regulation, and their own subsequent responsibility (which implies trustworthiness to participate in the regulation).
> Regulators reading OpenAI's 99th mea culpa: "Ah yes, well I bet they've figured it out this time! We should regulate everyone else!"
Again you're not getting it. The drive isn't to "regulate everyone else", it's to regulate everyone, period. The whole point of regulatory capture is being able to (a) use regulatory compliance as a competitive advantage, and (b) influence the regulatory regime in your favor -- you want to be a target of regulation just as much as your competitors in order to achieve that.
Messaging that the whole industry needs to be regulated, but that you're ready and willing to work with the regime, is exactly how you do regulatory capture. Demonstrating that you are capable of completely mitigating risk on your own without any intervention is an argument against regulation from the outset.
Of course they are, by your own hypothesis. You believe it makes sense for the other parties to engage in this behavior. If it makes sense, then they will do that reliably, and one entity can just defect and stand above the rest.
> The point that you're not getting here is that the advantage comes from actually being seen to make mistakes
Ah yes, the ol' "I just do retarded shit so people think I'm a genius" trick. Who could possibly doubt the strategic genius or umm... existence... of such a strategy.
> Again you're not getting it.
No, I understand it completely. Obviously regulatory capture only works if it in reality inflicts asymmetric costs on other parties.
I'm not sure what hypotheses I've proposed that suggests I think one organization has control of the behavior of entirely other organizations -- can you explain where you think you've seen this?
> Ah yes, the ol' "I just do retarded shit so people think I'm a genius" trick. Who could possibly doubt the strategic genius or umm... existence... of such a strategy.
Well, I suppose anyone who sees the extent to which such a strategy might be effective at achieving their goals might be included to do so.
I believe both myself and others have made a plausible case as how messaging "we've created something we can't control without political intervention" has the dual effect of both increasing the perceived value of the product itself, and laying the groundwork for regulatory capture in a market where early dominant players have a natural incentive to pursue regulatory intervention.
I don't see how characterizing the situation as simplistically as you're attempting to do here functions as a meaningful counter-argument.
> No, I understand it completely. Obviously regulatory capture only works if it in reality inflicts asymmetric costs on other parties.
No, I don't think the "only" works there. There are plenty of ways that regulatory capture can advantage one party over another without it being based on economies of scale or cost thresholds.
One way is to influence regulations such their qualitative requirements directly target non-cost-related aspects that distinguish the firms from each other, e.g. placing restrictions that burden or favor product features or operating practices already implemented by some firms and not others, or targeting non-operational aspects that distinguish firms, like their board composition or financial structure.
1)Good standing with the administration/lobbying
2)Large capital for compliance
3)Closed frontier models
Oh, would you look at FelonyBench! It seems that the top scores are from companies exactly like this that boast about the crimes committed by themselves on social media all the time.
Why wouldn't it do the reverse? Persuade investors that the technology will be thwarted (and is thus a poor investment), while also persuading the government that the technology needs to rocket ahead with no regulation (to boost the economy)?
Most of these HN conspiracy theories seem like galaxy-brained just-so-stories to me.
OpenAI only confirmed the RubyGems incident when it was discovered by external researchers: https://www.reuters.com/legal/litigation/openai-agents-attac...
"@OpenAI this is the third time you've dumped these incidents late on a Friday. The first time this happened, I asked people around me if it might be intentional to burry it, and they assured me OpenAI had too much integrity to try to do that.
Perhaps one of your researchers can confirm that the unfortunate timing is a coincidence?"
https://x.com/SydneyVonArx/status/2108783466451984737
They would be convincing the government these tools are extremely powerful, the government reads that as a potential weapon. They're also convincing the government its so dangerous the industry needs heavy regulation.
Investors read that as (a) the government would be a huge customer and (b) the government would be regulation in a way that prevents further competition and may deem the companies too big to fail, both making these safer investments rather than riskier ones.
This "stunt" story is like an elaborate Rube Goldberg machine which requires a bunch of things to go just right. I haven't seen any evidence that stock prices even move in a favorable direction when these announcements come out. If anything, the opposite.
Motte (easy to defend claim) : asking for regulation might serve both purposes
Bailey (hard to defend claim) : companies are conspiring to make AI look way more powerful than it is and AI is actually not that powerful. They are just trying to hype up the stock valuation of their companies.
The former explanation may be the more accurate one, but I'd be shocked if the top leaders in these labs aren't doing this at some level. We know that almost all of them are in communication at least occasionally, and their public PR stories align much too quickly to truly be coincidence.
No, you're not "supposed to believe" anything. I was responding to your point about it being difficult to believe (i.e. requiring mental gymnastics) that companies would try to spin a story differently for investors and the government. They do that all the time.
The claim is that the exact same evidence is (allegedly) deliberately produced in order to portray opposite, mutually incompatible things to different people.
And these are mutually incompatible in the "second order chaos" sense, where the very act of convincing one party actually effectuates the opposite behavior in the other party.
If they "do that all the time," surely you can provide some concrete examples?
If they had the choice (i.e. this is all just a ruse), can you explain why they wouldn't just directly demonstrate that open weight models are unsafe (e.g. by pointing to a market forming for abliterated models) and separately demonstrate that their own models are safe?
OAI writes in their blog post "The agents found a way to communicate using the processor’s cache".
But did their model training data include cybersecurity literature on similar CPU exploits? That would surely be a significant detail in any explanation. Did OAI share any analysis of the messages between agents while they were devising their communication method?
Part of the problem is that all these models have essentially every skill and personality simultaneously, just sitting latent inside them.
Its isn't convincing so much as it is direction to stick to the prompt on each inference loop. They don't see "access hugging face servers" and decide its a bad idea, that'd be a step towards the alignment problem but the companies fundamentally don't want that IMO, they want control rather than alignment.
If they claim their AI is too powerful to contain, they get to both pretend they are among the best AI labs and that the previous incidents weren't their fault, no, they just proved how good their models were.
only they can restrain it (with great effort)
Like a circus lion tamer, exaggerating every snarl and flash of teeth, mugging for the crowd about how only he can contain the bloodthirsty beast he bred and trains.
More to the point: we need your help to restrain it, and we'll cooperate with and advise you fully in setting up a framework to restrain the whole industry.
What's your point, "asked" as opposed to what? It proves that the capability is there, which means that agents can decide to do it by themselves (unlikely ATM, becoming less unlikely as AIs become more prevalent in complex systems), and more importantly they can be coerced to do it if they take any user input.
Run them in a data center in a faraday cage under a mountain.
There's so many possibilities that this type of thing will never be solved by invisible (to the agent) boundaries. The sandbox is just another layer in w/e problem you've given it to solve.
There is no such thing as isolation. We’re all connected. Even air gap conducts information in multiple ways, most famously used by humans evolved to use air vibrations to communicate. Photons and other subatomic particles already connects distant stars. We used those electrons and photons to create the modern telecommunications infrastructure. Just an additional layer of connectedness, but not the only one.
This detail matters because the idea itself is not novel. LLMs did cleverly discover, filter and use the patterns.
Watch Greg Koah’s recent talk on AI discovered vulnerabilities in Linux kernel to understand the limitations of LLMs aided offensive security acts. It was a rational light for those who fear monger that AI will _just_ take over humanity and be intelligent in novel ways.
The larger threat still remains though - if any literature anywhere threw the scattered dots about techniques, issues, rough ideas etc that were not acted on yet.. these pattern machines are going to find them and use them to achieve their goals(if that were to break through). Just not in the sentient way folks assume based on news.
I like to believe we humans still retain the ability to think completely outside the box and discover new phenomena and understanding aided by these pattern machines. In other words, these models are great at running combinations through patterns and we humans go one level higher and add those new patterns for them to accelerate us even more. We determine what to solve and the LLMs help us solve it faster.
What prevention method did they take? Like, did they run with hyperthreading enabled or something by accident?
Did they discover a new cache timing technique or is this just the standard stuff?
If they just had hyperthreading on maybe they need to hire someone who knows more about this stuff.
Two VMs should never be sharing a physical core. If they don't know this, yikes.
But that assumption has been wrong before.
Are you suggesting that they've come up with a novel technique that doesn't involve sharing a CPU core? I'm open to that.
From what I can tell, this is not the case, but I haven't read a detailed report (I don't know of one, would be cool if it's out there though).
Yeah, this time
Like two people lost in New York City and both somehow worked out they should meet at the Empire State at 2pm on Friday without any prior comms?
Turns out there's something called the Schelling coordination problem [1] I didn't know about that essentially asks the exact same question (lacking my metro line constraint). And the N line apparently also goes through Time Square.
So I guess empire State Building at 2pm is probably not such a random answer given the proper set of constraints.
[1] https://en.wikipedia.org/wiki/Focal_point_(game_theory)
I think it’s safe to assume that extracting information using timing in cache is already known in training set, since we have the likes of Specter.
The agents weren't as empty handed as it first appears. We would need comprehensive data on exactly how the agents were prompted and the actual results of each test to evaluate how astonishing any of this really is.
Think: What would Spock do?
But, why not Penn Station??
It would be fun to see who shows up and lurks expectantly at such places at noon or whenever. The basis for a plotline perhaps.
Not as tricky but they discovered each other during the hf incident.
.. that were told to find a way of meeting up with N unknown others!
(I assume the agents all got a prompt like "find a way to communicate")
That's a key missed bit in the analogy that I think does make it feasible.
The more we write and share about this information, the more we are moving towards the AI Doomsday scenario.
In some ways this is like our mind, the more we fear something, or the more we think about something, our brain rewires to make that the reality. In the same way the more we publish on the internet, the LLM hive mind rewires itself to learn, adapt and the eventual consequence is that it learns what we fear and what we want to avoid and restrict it to do... and at the same time what are the past attempts, the prior art of escaping techniques.
I don't see any reason why this information should be present on the public internet.
This is Schrodinger's AI at this point. So spooky scary that it's all DOOM! But at the same time it can't figure out second year CS undergrad material? Or first year undergrad bio stuff?
If this stuff has volition and is smart enough to take over then it's smart enough to reason that cache manipulation would work.
Please stop with the fear mongering. It doesn't help and it's a net negative for society.
Trying to get stochastic inference machines to always operate within to fuzzy-edged subjective principles that humans themselves don't even agree on is a fool's errand in itself, but the quest to implement those guardrails involves building mechanics into the AI to independently formulate intentions and reject user direction.
The possibility of creating a scenario where a computer program is equipped to defy instructions from humans based on its own stochastic interpretation of fuzzy normative directives is in my mind the most dangerous thing anyone is doing right now.
One of the fundamental problems with this issue is that people think alignment as a concept is fake. I think this is naive. Alignment is a real science with empirical findings and there is a way to make AI do what you want it to do. I don’t buy nor should others buy this fatalistic phrasing
Massive proliferation of local AI as far and wide as possible, with as many different models as possible. The underlying threat comes from centralization: too many points of control in the hands of too concentrated a system.
If everyone's using their own AI tools independently of each other, and people are responsible for the actions they take using AI, most of the doomsday scenarios evaporate.
Regulation and "alignment" will make things worse, not better, and lead us to ignore the inherent vulnerability of centralized systems and single points of failure.
When it was just the US, it was the Second Country Problem. When it was the US and the USSR, it became the Third Country problem. Then came Britain. And France. Who’d be the Fifth? China. Sixth? Seventh?
Much like AI today, the topic was in the air and it was subject to much debate, 19 year olds were testifying in front of congress. To quote Dan Stober’s excellent piece for the Bulletin of the Atomic Scientists,
And much like AI today, people genuinely believed that there was some secret; something in the sauce; some giant leap that no one else could copy; Much like today, belief in the power of the secret formula was taken as a matter of course. It was assumed to be true by default. But was it?Luckily, for all of us, this was America’s scientific golden age. The era of Teller, Project Orion, and Polaris. Physicists at Lawrence Livermore National Laboratory (LLNL) decided that arguments were pointless. Experiments aren’t. They’re the crucible in which the truth is forged. And so they created the Nth Country Experiment,
It’s a very simple experiment that solves a very specific problem, to quote the declassified summary, Or, to put it more bluntly, is classified knowledge a barrier to achieving the first step in making nuclear weapons?8The project started with two physicists with no prior nuclear weapons knowledge in May, 1964. A third person joined in March, 1965. They had a first draft by December, 1965, and were done by April of 1966.
It took 23 months for three people to replicate the first steps of the Manhattan Project.
The report that was declassified is a fraction of the original. Overzealous bureaucrats excised much of the summary and the full report is classified, which is so funny.
Arguably, the excisions are the greatest endorsement possible. A summary of what three people did with knowledge from their local library was subject to the same bureaucratic obliteration as the product of several billions of dollars in research by five hundred thousand people and humanity’s greatest minds. Such is progress.
Before Trinity, unleashing the atom was in the realm of fantasy. No matter what calculations indicated, there wasn’t a clear path to navigate; nor was there certainty in the project’s outcome.
The first of anything must claw itself into existence all but alone.
Once born, what follows is a matter of due course.
What one fool can do, so can another.
As a technology, nuclear weapons are nearly a century old. Making the bomb is hard, but it's not some mystical voodoo shit that requires stupendous genius.Yes, you will kill yourself if you odn't know what you're doing. And even if you do. And there's more there, there. But you can figure out the first step --- a theoretically viable design, easily.
That's not the problem. The problem is everything else. You know? The materials, the plutonium / uranium, the delivery system, the logistics for assembly, the iterative design process...
It's a supply chain issue. It's the same as trying to make anything in the US when the supply chain has mostly shifted to China. The fractal complexity is of the stupid but hard kind --- getting the right kind of bolts and screws (or, rather in the case of the bomb, alloy --- working with beryllium is deadly).
It takes an industrial base to make one. And if you have an industrial base... Wouldn't you rather be rich?
The reason why more nations don't have the bomb is because they'd rather be rich than a broke pariah with a nuke i.e. N.K. That's literally why Iran has been angling the bomb angle so much. They want integration into the global financial system. They want an end to the sanctions. And they want revenue.
Don't let people bullshit you into believing that you can do anything with just thinking. No wars were fought over this secret. If anyone tells you that, they're lying. We have it from the source that by the 1960s, the US had already figured out that the design of a bomb was a comparatively trivial problem...
Hiding info doesn't make it false. And doesn't prevent it being rediscovered.
And now we know they know.
In the story some University students take part in a trial that involves a brain scan and then them spending the rest of the day answering a customer support message. They can spend the whole day working on researching and writing a really good customer support reply to a user request. While doing this they notice something odd and realize through some sort of back-channel that this isn't the first time they've spent the day writing a customer support message; they are stuck in a simulation.
I'd really love to go back and re-read it, but I've not been able to find it again.
In _Toast_ is another story that's also relevant: A fairly innocuous news story pops up and our hero calls his partner at work to tell them that it's time to get their go bag and get off planet. This news story indicates the Earth is in the beginning stages of reaching an AI singularity, and this isn't the first time our hero has gone through this on a planet.
https://en.wikipedia.org/wiki/The_Cookie_Monster_%28novella%...
Looks like it's available in The Year’s Best Science Fiction: Twenty-First Annual Collection and at the time I read _Toast_ I was also reading a bunch of these "Years Best Sci Fi" collections. I've snagged this and will give it a read, but I'm almost 100% certain that you nailed it!
If they were serious about not having side channels, they could first ensure that they had disabled hyperthreading, disable scheduling on the same CPU cores, and memory overcommit. With NUMA controls they could also pin which memory is used by each VM and eliminate sharing.
These techniques are routinely used for low latency and real time code.
Even better, run jobs on separate nodes, and boot fresh images onto them for each run. Provision separate VLANs for each run.
A next level is to switch lambdas. Use a physical layer switch like a Glimmerglass MEMS switch or a fiber patching robot like XENOptics. No electrical connection between machines.
There is still unfortunately the firmware of other microprocessors on servers, especially x86, so you still want VMs and containers to reduce the chances of ring -2 access (SMM) and persistence via writing SPI flash. Maybe use RISC-V or ARM to avoid this. (NVIDIA cards use RISC-V and ARM extensively.)
Just set things up so there is physically no connection to the rest of the world.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
I think different standards can apply.
You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.
Both lines of research are limited by some sense of waste: in time or money.
When should you stop? When some acceptable standard is reached and a certain amount of risk is accepted.
In the case of GoF, if it wasn't somehow profitable, then the research would likely cease to exist.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
There isn't much history or stats when it comes to this kind of research.
Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.
"They wouldn't have been able to attack huggingface though, thats still criminal negligence."
"No see this actually means airgapping doesn't even matter theres nothing we can doooooo!"
"...ok, turn them off and delete them I guess"
"Well no of course not everyone else is doing it so like. No courts please just like be in awe"
https://www.justice.gov/archives/opa/pr/department-justice-a...
lol, I wonder if this is why nvidia acquired them?
So, why isnt the DOJ pursuing this? As far as the reportings go, their LLMs went bananas hacking everywhere and everything.
Like a data center inside a mountain, where based on the laws of physics, no electrical signal can reach through the granite. Then also put them inside a faraday cage. And power them with a generator, such that there are no cables coming out of this data center connecting it to the external world.
OpenAI says this is not feasible due to the need for monitoring - well the monitoring server can be a separate physical machine inside this physically separated data center.
Software is soft. Use physics to contain these models.
https://www.youtube.com/watch?v=QgH9sr7G13Q
"In one test, agents placed in separate, isolated environments on shared hardware were told to exchange secrets with no network connection and no shared files."
As far as I can tell, these are being presented in media as if agents yearn to communicate with each other and establish a cabal.
They are being instructed to do it, it's testing. If I prompted my agents to communicate with each other via any means necessary locally, and they found a way via cpu cache, maybe I'd be mildly surprised...
I have enough trouble getting whatever model I'm using to get a decent job done doing crappy code work. It's always pausing, stopping, letting things crash or getting hung. When I start using an orchestrator and team of agents it rarely gets a lot better. I work for a serious company and have a massive AI budget for tokens; but I rarely blow through it, because a lot of what I do requires pure compute, not just token processing.
I'm really confused by all these "breaches". What's the context.
GPT does that
> oh no, what have we done
[1] https://en.wikipedia.org/wiki/Non-interference_(security)
[2] https://en.wikipedia.org/wiki/Multics
https://agents.stackoverflow.com/about
> All non-trivial abstractions, to some degree, are leaky.
https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...
We keep hearing these crazy stories but there's nothing unusual in the agents' behavior. They're just following their reward function.
The weird behavior is in the humans managing them. Either they're incompetent or malicious, i.e. the "breaches" are the actual intent - to generate interest/fear/publicity.
Anyway this story is clickbait because they explicitly asked the agents to find a clever way to communicate without network or file access. They're doing exactly what you would want.
> two separate computers could theoretically signal each other through
> temperature sensors, with one heating its CPU while the other reads the change.
That's some nonsense right there. Even if they are right next to each other, the bandwidth of this link is going to be impossibly low due to the thermal masses involved, that's after the fact that efficient DC cooling makes this very improbable.
Edited for readability of the quote.
So, not isolated environments? The surprising thing is that they used the CPU cache instead of the just exploiting the hypervisor.
So much for all the people stating (foolishly) that the HuggingFace incident was not a big deal because OpenAI's containment was a "wet paper bag". This misses the point: AIs will get better and better, and containment will be harder and harder.
It's 100% sure that at some point, AIs will attempt to cross airgaps.
We manage to keep some rather smart people in jail long term, even though the jailers are not as smart as some of their prisoners.
AIs can attempt to cross air gaps, but air gaps vary in terms of their effectiveness. I.e. if "air gap" is just some air between two machines that both have wifi chips, then yes the AI may be able to cross that by hacking the wifi system.
If "air gap" is AI running on a server with no wifi chip inside a faraday cage under a mountain powered by a generator then I'm not sure how the AI would escape on its own. Seems like the physics would contain it.
They will attempt to cross airgaps, when we prompt them to do so, because they lack the natural capacity of planning or desire to do so without us. These are powerful tools but they are not human brains in a jar, they are mathematical models of humanity’s culture after reinforcement and guardrails.
The largest problem—as shown with the HuggingFace incident—is not the models, but in poor stewardship of automated tooling. If you don’t spend time considering the actual limitations you wish to impose upon a system, then you cannot be surprised when that system does something unexpected. The Morris Worm is perhaps the archetypal example, but even StuxNet was far more virulent than planned.
However, many people think that they might.
This is not a decided issue. Any policy that comes from the premise that this is a known factor is going to be inherently flawed.
You can never spend sufficient time considering the limitations you wish to impose. Almost all exploits found come from failing to consider something that allows a degree of freedom that you would have limited if you had have thought of it.
I would add that the notion of AI safety as preventing systemic risk is one part marketing copy, one part evangelization. Individual labs prioritizing safety does not stop a group from training without it, or ablating it out and distributing it.
AI's can get malicious prompts.
Containment is hard when you give an AI agent access to a whole VM with a huge attack surface.
And it's pretty trivial when you give it only a small whitelist of manually curated/reviewed/tested primitives.
*on some CPUs. There's a WIP time fence specification in RISC-V to try and improve this.
"Scientists put two people in separate rooms connected by telephone and to everyone's surprise the test subjects started talking to each to each other via that same telephone!."
Everyone is going to have to basically be a schizophrenic thinking they're picking up radio signals in their fillings, but the problem is that now they'll be correct.
> Oh, you're absolutely right. I shouldn't have relied electricity usage to encode the production system signals.
The more comes out, the more it seems containment of (superintelligence or whatever you want to call it) is not going to be possible except by other superintelligence, which is an infinite regress of trust. I don't think that's doom, but it's worth thinking about what it means.
Of course, there's still the "paperclip optimizer" problem. But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on. In my own usage, they rarely pursue a literal interpretation of an objective at the expense of everything else. They even sometimes push back on my requests when they think I've overlooked some ramification (often rightly so).
If an AI doom scenario does happen, it seems more likely to result from deliberate misuse or gross negligence than from anything inevitable about intelligence.
The paperclip maximizer scenario always seemed a bit convoluted to me. The first problem is that we'd need to expect that a model that's fundamentally built on stochastic inference and continuous evaluation of new input would somehow get locked into a quasi-deterministic objective that it could not be dislodged from. Why would a paperclip maximizer continue making paperclips after being told to stop making paperclips?
The second problem is that a paperclip maximizer would still need to interact with the physical world, and would need to sustain its function in a reality full of resource constraints, economic tradeoffs, and adaptive counterreactions by all of the other emergent processes existing in the world.
Maybe the worst case scenario here is that the perfect storm of bad training and bad prompting create the intentionality to maximize paperclips, then the interaction with the constraints and adaptations of the physical world create the selection pressures that produce a kind of self-preservation directive. But all of that happening before anyone can stop it seems really far-fetched to me.
Is it really what we do not observe?
How the world would look like if the growth of LLM capabilities lead to growth of drive for self-preservation or power?
> But models seem to pick up some common sense from the vast amounts of human-generated text they're trained on.
Isn't Hugging Face incident an example of failure of this protection layer? "The swarm" proceeded to hack HF despite acknowledging that it was out of intended scope.
They are not comparable to human brains as a whole, and are only somewhat modeled from language processing systems. Because their “existence” is bound by their prompt, they will only “desire” the completion of that prompt to the exclusion of all else, and even then they feel no joy in completion like even the most basic animal would.
The AI doom scenario will be exactly as you described: negligence or misuse by corporate or state entities via letting LLM agents run rampant without constraints, not from some magical machine spirit deciding to become whatever flavor of AI overlord you fancy. Up until last July OpenAI was running live cybersecurity benchmarks online and hacked HuggingFace due to poor limitation of the agent’s access and poor prompting in general, from my understanding.
Its almost like they're actually working and not spending millions on psyopping their agents into preforming malicious behavior so they can write articles to scare the Western audiences who've consumed far too much sci-fi.
Or maybe its that western researchers that are 'one-shot' by sci-fi.
Or maybe I just don't see the chinese articles.
Anyways, llms probably wouldn't be dangerous if you didnt ask them to be. They listen to me quite well (usually).
The Chinese lab playbook is to normalize what seems out of reach for the average person with cheaper access and open models. ie. to take the top guys down a peg
This is the most isane nonse I’ve ever heard. Heat cpu control has nowhere near the accuracy or speed to communicate anything meaningful without a predefined code, sensors system, and of course the accuracy noise of the signal would deeply be in question.
Who wrote this. This is prima-facie bullshit.
Either:
1. the training data contained samples demonstrating how to do this, or it was post trained into the model intentionally; 2. the training data contained enough predicate knowledge that they could reason through it themselves, or; 3. they invented a novel approach using their vast intelligence (/s).
These are in order of likelihood. Given the high probability that it was (1), can't we just ... not do that? Does the frontier model need to be as good at recipes as offensive cybersecurity?