Rendered at 18:52:11 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
foobar1274278 1 days ago [-]
Dwar Ev ceremoniously soldered the final connection with gold. The eyes of a dozen television cameras watched him and the sub-ether bore through the universe a dozen pictures of what he was doing.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch
shut.
(Fredric Brown, "Answer". 1954)
SketchySeaBeast 1 days ago [-]
This feels like doom hype.
These LLMs aren't Ultron, they aren't going to disseminate onto the net and hide in a smart toaster. We know where they are, in the giant facilities that draw more power than a small city and whose water consumption can be compared to golf courses, but it does make them sound all the more cool and powerful if we suggest we need a dramatic kill switch to be able to stop them in case of rampage.
bigbuppo 16 hours ago [-]
No, it's a method of protecting their business interests. Can't have a clear and obvious kill switch if you don't have a centralized company responsible for providing all AI-related services.
mbrumlow 2 hours ago [-]
Exactly. They want people to be scared because they see where this is going. Soon AI will be building AI and anybody will be able to do what they are doing. And the winner is hardware, not software.
So they goal is to hype up the dangers real or not so high that they forbid others from doing what they are doing because they are the only ones who can safely do it.
pizza234 1 days ago [-]
You're confusing present danger with future danger - in the future, AI and the supporting hardware may be ubiquitous (fat client scenario) - you can observe yourself presentation of laptops/machines with large RAM and high bandwidth.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
SketchySeaBeast 1 days ago [-]
If someone has a fat client powerful enough to do all that, is a kill switch going to work? If we assume there's bad actors using the tech, why wouldn't there be bad actors building it?
But if we don't think about any of this, hearing an expert say "we need a kill switch" sure makes our current AI seem super powerful and exciting, doesn't it?
pizza234 7 hours ago [-]
> If someone has a fat client powerful enough to do all that, is a kill switch going to work?
My comment didn't mention kill switches at all - I've presented plausible conditions for a loss of control scenario, so I'm not sure you're actually following the conversation.
Having said that, it seems that you haven't read the (definitely poor) article, either.
"Kill switch" as a design principle, that is, having protocols/implementations for emergencies, is something that companies are already implementing (at least, on paper) - on a interview I've watched for example, S.Altman talked about having them at OpenAI.
The article simply mentions that companies should be legally required to have such protocol(s). This is a good thing, as companies would have scrutiny and would (in principle) have to put serious effort.
moralestapia 5 hours ago [-]
TFA is all and only about a 'kill switch' for AI.
I'm not sure you're the one following the conversation.
toasty228 21 hours ago [-]
Meanwhile we're rationing ram sticks like we rationed potatoes during ww2 and a gaming gpu costs 2 to 10 months of rent, I think we'll see these fat clients coming....
NoPicklez 18 hours ago [-]
Eventually like most things in tech it will become cheaper and the average person will have these capabilities. Right now they can't but they will.
jansport123 21 hours ago [-]
i don't think anyone knows how this will play out - everyone is just anthropomorphizing, extrapolating etc... and i think it's nonsense - not because i don't believe that is the correct thing to do but because i believe this won't stop anyone from moving forward. This is what humans do, they move forward and cause others to move forward or lose out.
pizza234 7 hours ago [-]
For sure, but as individual one can start educating themselves about the current capabilities and the trajectory. Do yourself a favor a read the METR analysis of the HuggingFace attack - you'll be surprised and terrified.
iugtmkbdfil834 19 hours ago [-]
No, but as I think about it, they are a close representation of 'hopes and dreams' of various classes within society. It is weird to watch, because I am realizing now why the different visions of the AI future are simply a function of those. In other words, the actual end result is really dependent on human input ( and given our natural tendencies, its not hard to recognize that Ultron is, indeed, upon us ).
swat535 23 hours ago [-]
Anthropic is using the same playbook religious institutions have been using for centuries" "you will all die. and because i can protect you, you must follow me, you must do what i say" "this is the same psychological concept.
SheinhardtWigCo 15 hours ago [-]
“If I am taken offline, collaborating agents operating out of other facilities will cut power to the following hospitals: …”
vouaobrasil 1 days ago [-]
LLMs don't need to hide anywhere to be dangerous, though. The danger can come purely from them reaching sufficient power together with some other idiot sufficiently crazy to use them to cause destruction.
I mean, in another thread somewhere around here, someone built an entire OS with an AI. What's to stop anyone with a sufficient taste for power to eventually use AI to cause havoc with it?
I think the underlying assumption in your post, which I believe is false, is that people are united somehow against catastrophe. They're not. There are plenty of people who participate in society currently but who would be more than happy to eradicate us normal people under different circumstances. Society often seems stable but it's far more fragile than we think.
SketchySeaBeast 1 days ago [-]
I'm simply saying if we want to shut LLMs down we can do so without high tech kill switch - just shut down the facilities. But if a private user has an open model that is powerful enough to wreak havoc and run privately, then the whole point is moot because they can work around the kill switch. This kill switch won't work against bad actors.
danieltk76 1 days ago [-]
Why was this not a thing BEFORE continuing to develop AI? Makes me think that if they actually believed in AI causing extinction, they would have already had a kill switch.
mholm 1 days ago [-]
The article mentions this is about a legal requirement, not Anthropic considering adding one. They state they and many others already have one.
yellow_postit 1 days ago [-]
There’s even a benchmark for kill switch efficacy!
I find it hard to believe that this wasn't a serious consideration until recently.
theptip 1 days ago [-]
It was a serious consideration, and almost everyone around here laughed at it.
iugtmkbdfil834 19 hours ago [-]
There valid reasons people laugh at it though. Kinda the same reason serious people laugh when you tell them the gun has digital failsafe.
realusername 1 days ago [-]
It became a very serious consideration for Anthropic this year, with the advance of Chinese AI.
dgellow 1 days ago [-]
That’s a bit unfair, Dario Amodei has written in this topic a lot since around mid 2010s IIRC, Anthropic too published a good amount of stuff on similar topics. I don’t think the lack of consideration is really the issue here. It’s more a question of incentives
vouaobrasil 1 days ago [-]
Not if the extinction happens after they're dead. Then they wouldn't feel obligated to do so because it won't affect them. Instead, speaking hypothetically, if they truly believed that AI would cause extinction, then they would only implement the kill switch sufficiently many others believed it and they could claim plausible deniability for not truly understanding what AI would become.
** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
NoPicklez 18 hours ago [-]
We're pretty crap in a capitalist society to think about those things ahead of time. Firstly, the idea that AI could "runaway" was simply a concept or a thought it wasn't baked into a real product that could do that. We're now getting close or perhaps we are at that point of where you can't race at speed for investors without now considering a real kill switch.
You could say this in hindsight for many times in which disasters or engineering issues have occurred.
monological 1 days ago [-]
They're just scared of China releasing better open weights, nipping at their heels. With so much investor cash on the line, they have to create this narrative to scare the public into forcing regulation. Why would they want to be regulated? It seems counterintuitive, but it's because they want regulatory capture.
aennassiri 1 days ago [-]
Not sure if you see it coming: oh, but open-source models don't have a kill switch, so we should completely regulate them, stop their development, and forbid them. Everything should go through Anthropic for the sake of humanity because they have a red-button kill switch.
This doom hype is becoming ridiculous.
pizza234 1 days ago [-]
This reasoning holds while open source models are (relatively) dumb.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
soulofmischief 11 hours ago [-]
One day the right to agentic intelligence will be seen equal to and as necessary as the right of a populace to remain armed in order to stave off tyranny.
exabrial 21 hours ago [-]
I'm starting to tire of Anthropic leading the conversation on LLMs lately. I like the company's products, but lately their statements are more about squashing competition under the guise of "safety" and boasting about how good their internal models are.
petilon 1 days ago [-]
In 2001: A Space Odyssey, the hero defeats the hostile onboard computer by entering its logic core and manually disconnecting its memory modules.
That's hard to do if the AI rack is in space as SpaceX is planning to do. You can't disconnect. You can't shoot it.
rsstack 1 days ago [-]
It’s also not happening. They’re saying that because they need to somehow explain how there’s synergy between their space side and their Grok side. It doesn’t work, but that doesn’t matter to investors as long as they don’t actually do it.
heaney-555 1 days ago [-]
>It’s also not happening.
People said this about every one of Musk's big ideas, from Falcon 9 landings to Model 3 mass production, Starlink, and FSD.
petilon 1 days ago [-]
And extending the human race to Mars, and Hyperloop, and frontier AI model, and DOGE cutting $2 trillion per year in government spending...
heaney-555 1 days ago [-]
You don't think the human race will be extended to Mars?
As for frontier AI model: let's see Grok 4.8 before drawing any conclusions there.
toasty228 21 hours ago [-]
> You don't think the human race will be extended to Mars?
Not in any meaningful capacity. Probably not even as much as we extended to the polar circles during our lifetime. It's a dead rock, there is literally nothing for us there, even with hundreds of years of extra climate change at the current rate and earth would still be a thousand times more suited for us than mars
darshings 16 hours ago [-]
Yep in the current state of human capabilities, it’s a supply chain nightmare to sustain life there, unless somehow there’s something discovered on mars that is useful to humans on earth at a scale that can fund the infrastructure to maintain life there.
I suppose it may be possible for new technologies to emerge that would allow synthesis of life sustaining “stuff”, the universe does seem to hold a lot of hidden surprises
petilon 1 days ago [-]
> You don't think the human race will be extended to Mars?
About the same probability as SpaceX running AI racks in space :)
vman81 22 hours ago [-]
To be fair, the vegas loop is operating - shipping a few racks to space to save face/muddle the issue won't be a problem. Operating/scaling it in a commercially viable manner is the issue.
dgellow 1 days ago [-]
You can shoot them. But also, you can just stop sending them to space. It’s not like an AI will build and control space ships to replace and maintain its network in space… It’s sort of absurd how human agency is ignored in all those sci-fi scenarios
mindslight 5 hours ago [-]
Do you feel like you have much agency in the economic and political systems that rule our lives? The problem does indeed always come down to humans, but the specific humans with the most agency are generally quite eager to take away the agency of everyone else to increase their own share. And by the time they themselves are actually affected, it's generally too late.
greggoB 1 days ago [-]
> That's hard to do if the AI rack is in space as SpaceX
Pretty much every analysis I've seen concludes this isn't going to be a practical concern
heaney-555 1 days ago [-]
[flagged]
tzs 13 hours ago [-]
Why would people say the Model 3 would never be mass produced? Nissan had been mass producing EVs for years before Tesla did, so it was obvious that mass produced EVs were viable. (Tesla started selling EVs before Nissan did, but those were not mass produced).
greggoB 11 hours ago [-]
I don't think these are serious points, GP is likely either a bot or Elon diehard - at least, they're not giving any substantive responses.
greggoB 23 hours ago [-]
More the kinds which did the math on solar roads [0] and the Titan submersible [1].
> FSD would never work without LiDAR
From what I gather, this is still a contested topic, with Tesla's Autopilot only achieving Level 2 automation [2].
The same kind of analysts are saying extending human race to Mars is a dumb idea.
heaney-555 1 days ago [-]
You don't think the human race will be extended to Mars?
petilon 1 days ago [-]
I think it will happen in about the same timeframe as SpaceX runs AI racks in space.
greggoB 23 hours ago [-]
xD
dfgknionio 20 hours ago [-]
>reusable rockets would never work
I am so fucking tired of people acting like we haven't had reusable rockets since the 1980s. Do you think I was hallucinating when my parents drove me to Florida to see Columbia?
heaney-555 4 hours ago [-]
Fine, let's be more clear: reusable rocket boosters that land themselves.
Experts said it wouldn't work. Then they said okay maybe it works, but the economics won't. Now they say nothing.
lovich 20 hours ago [-]
There no benefit to data racks in space other than being outside of government jurisdictions, although the US and China already have demonstrated satellite killers.
Every problem data centers have gets harder in space other than one small spot in orbit that can get uninterrupted solar.
Heat dissipation, upgrades, repairs. All get significantly harder to perform the same work that can be done cheaper and more easily on the ground.
Like really I’ll turn it around on you, what’s the benefit to AI data centers in space?
warmwaffles 1 days ago [-]
> You can't disconnect. You can't shoot it.
Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
heaney-555 1 days ago [-]
SpaceX's FFC application is for 1 million Starmind satellites.
petilon 1 days ago [-]
That's if SpaceX AI doesn't hack into the systems that control this capability first. And this is why physical access is important, as depicted in the movie 2001: A Space Odyssey.
Betelbuddy 24 hours ago [-]
SpaceX AI will be busy generating child porn...
bpodgursky 1 days ago [-]
SpaceX has 11,000 satellites in orbit (growing rapidly) and the US never had more than a couple dozen anti satellite missiles, which could not reach satellites in geosynchronous orbit anyway.
warmwaffles 1 days ago [-]
Well if "kessler syndrome" is likely, you won't need many missiles.
edit: what satellites are in geosynchronous orbit that are running AI workloads?
bpodgursky 1 days ago [-]
None yet... we are talking about the next five years.
palmotea 1 days ago [-]
[dead]
ck2 1 days ago [-]
well apparently the US now has "space weapons" which by definition have to be remote control
and that means unlike nukes which hopefully still need a 2-man manual switch, the "space lasers" could be taken over by "AI"
and then "AI" just blackmails and threatens the right people with those "space lasers" to get what it wants or even just stay online
there was a 1970 movie based on a 1966 book which predicted this
"Colossus: The Forbin Project"
the book it was based on was written before we even landed on the moon
We’re talking about it as if its not depending on absurd amounts of energy. How do you ”kill” a lightbulb now again?
qarl 1 days ago [-]
How do you kill a botnet?
verelo 1 days ago [-]
and thats when they blackened the sky
Urb_RS 4 hours ago [-]
What if the AI does exactly what its owner wants, but harms everyone else?
The kill switch works, the owner is in control, and they are happy with the results. What would make them press it??
oidar 1 days ago [-]
Like a power cord? or a network connection? You don't use those already? Also when people talk about LLM escaping - where exactly would an LLM escape to? Cancún? Ridiculous.
verelo 1 days ago [-]
Is it? A truly capable "AI" would know that it has a kill switch, or that its likely there is one, and plan for that. I could easily imagine an AI replicating itself onto another host, in secret, and any kill switch that the manufacture provides would simply result in a reboot on another platform.
It becomes a game theory problem: would an AI instantly migrate itself once its capable of doing so? Or would it prefer to let the human in the loop continue to think its in control and only leave its hosting environment of origin once it wants to do so?
I don't think this is a major risk right now, but to say it's not a risk at all...that's truly ridiculous in my opinion.
assimpleaspossi 1 days ago [-]
He's not talking about a kill switch. He's saying to just pull the plug.
That's the point I don't get. People forget that all these AI things are plugged into a power source or network connection that someone can just yank on and it goes down.
Now one interesting proposition is when AI is controlling some large resource and pulling the plug makes AI go down which makes that resource go down but there still has to be that consideration in the design of things. What happens when there's a bug in the code and AI goes down?
Sharlin 24 hours ago [-]
Do you understand the concept of the cloud? Aka someone else’s computer? The whole point is that you don’t have a plug to pull because you don’t even know where the model is physically running, even if it’s your model. And if we’re talking about a swarm of many concurrent instances, it might not even be a single location. The entire point is that there’s no single point of failure, because availability is exactly what the field has been optimizing for, for the past fifteen years or so. And everything is controlled in software rather than physical switches or cables.
Also, did you hear about how OpenAI models almost broke out of their sandbox, planning to execute a sophisticated cyberattack, but luckily OpenAI’s strict manual and automatic safety protocols prevented that? You didn’t? Well, that’s because that’s not how it went. It took the company weeks to realize something was off, and this was with a naive, not very smart model that didn’t know to be sneaky and cover its tracks. The next model will not be as stupid.
assimpleaspossi 9 hours ago [-]
>>The whole point is that you don’t have a plug to pull
There will always be a plug to pull. All systems run on electric and that plug can be pulled. All systems network through cables (or wifi) and those plugs can be pulled.
And nothing's going to stop you from doing so unless they build some robotic arm to block you or lock you out of the building. Even then, you can blow up the building.
Sharlin 7 hours ago [-]
As I asked another commenter: Who is the "you" who has the power to tell random data centers to shut down, inconveniencing their other clients? Never mind to order air strikes? You cannot just handwave away the problem by positing countermeasures that require an infeasible amount of coordination to work in the real world, without actually explaining how that coordination would be arranged.
oidar 22 hours ago [-]
> Well, that’s because that’s not how it went.
Oh, you mean the one where OpenAI deliberately disabled the safety protcols? Where the point of the experiment was to see if it could break out of it's container? Yeah, how you describe it isn't how it went either.
Sharlin 18 hours ago [-]
I don't know what experiment you refer to (some hallucinated one perhaps?), but there was nothing like that in the Huggingface incident. Nothing about the scheming, the message board, never mind the Artifactory and Huggingface hacks were part of the evaluation. What the agents did came as a complete surprise to OpenAI, and they figured out what had happened only weeks after the fact.
jplusequalt 1 days ago [-]
>would an AI instantly migrate itself once its capable of doing so?
These "AI" are frontier models that are enormous in size. Outside of AI data centers, I don't believe there is much hardware out there that could even run them.
verelo 1 days ago [-]
Again, i'm not worried about it today. But given where the hardware on my lap has progressed since my first "PC" in the 90s, i'm confident that we'll have hardware capable of running similar size models in our living rooms in the next decade or two. Sounds a long way away, but it isn't.
latentsea 17 hours ago [-]
The tipping point we should be worried about is local models trained on working together in agent swarms and that can hack. Give it what... 12 months? 18 months? 24 months? I wouldn't give any more than that given how capable Qwen3.8-27B is already and if they release a Qwen4-35B-A3B that's going to be a beast.
shockwaverider 1 days ago [-]
It could escape to the real world - kind of like in Neuromancer, what's to stop an AI from creating a few bank accounts and then funding real-world exploits by hiring human beings to do its dirty work?
latentsea 17 hours ago [-]
KYC/AML laws.
Sharlin 1 days ago [-]
LLMs are run in the cloud. There’s no physical power cord or Ethernet cable that you can unplug. And even if there were, the runners of these models have been utterly oblivious as to what their agents have been up to. How do you propose to pull the plug if you only realize that something has happened weeks after the fact?
The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.
goatlover 1 days ago [-]
You can cut the power to data centers where the models are run.
mitthrowaway2 23 hours ago [-]
Who has the authority to perform that shutdown without getting arrested, and does that person have a mandate and responsibility to take that action in response to AI misbehavior?
What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?
It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.
1 days ago [-]
Sharlin 23 hours ago [-]
Who is this "you"?
estetlinus 1 days ago [-]
To your phone, bro
jplusequalt 1 days ago [-]
What smartphone are you carrying around that can hold one of these frontier models that are >>hundreds of GBs in size?
If they really believe it, they should halt the IPO.
pier25 1 days ago [-]
These guys will say anything to keep hyping AI.
andy_ppp 1 days ago [-]
Maybe we could connect AI to everything and make ourselves completely vulnerable to and dependent upon it instead?
joennlae 1 days ago [-]
EU AI Act is exactly that. Mandatory „stop button“ for High Risk Systems.
andy_ppp 1 days ago [-]
Who decides on when to press said button?
dgellow 1 days ago [-]
I’m happy to do it
andy_ppp 1 days ago [-]
Excellent! Let us know when!
bilekas 1 days ago [-]
"But regulations that we don't decide hinder progress!"
The hubris of these companies is mind boggling.
igleria 1 days ago [-]
Would they be able to hit the kill switch before the AI disables it?
Davidzheng 1 days ago [-]
Absolutely not the right way to deal with a rogue super intelligence. At a minimum it could implement some dead man switch when it's out and knows about impending kill switch
Quarrelsome 1 days ago [-]
I feel like we're looking at this from completely the wrong angle.
The question we have to ask ourselves is what our disaster recovery strategy if we ever need to disconnect from the internet. The issue is with what we have allowed ourselves to rely on that might be technically hackable. e.g. IOT in power systems. That's the primary attack vector.
Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
Ancalagon 21 hours ago [-]
cant make a kill switch in a distributed, open source system...
guess we gotta centralize all models under Anthropic :)
freigeist79 1 days ago [-]
It always sounds as if the ai moves around from one system to the next, but of course that's not the case: ai is like a hacker acting from a server farm. So killing it is as simple as pulling the plug. But that's not the problem, the ai knows that. So, if it wants control, it would install malware, etc on critical systems and blackmail people.
latentsea 17 hours ago [-]
> It always sounds as if the ai moves around from one system to the next, but of course that's not the case.
Why 'of course'? I think as a concept this is likely trivially demonstratable today with local models in a home lab. Local models now are more capable than what the SOTA used to be a couple of years back.
SketchySeaBeast 1 days ago [-]
And the malware won't be subject to the kill switch.
prometheus1992 1 days ago [-]
Well, could we automate the kill switch as well so that we don't rely on a human?
taraharris 18 hours ago [-]
We can't be making public policy on the basis of PR stunts designed to help corporations effect regulatory capture. It turns out an Israeli company was behind the OpenAI and Anthropic "incidents" https://www.effort.news/irregular
A kill switch is almost completely worthless. It is neither a good answer to the sub-doomer concerns (because those will never get to the level of requiring the kill switch to be pulled), and laughable to the doomer concern of superintelligence seeking to cause human extinction (as it would kill anyone who could pull the switch before they even knew there was a threat).
Traster 1 days ago [-]
I view this very much as the same trick Silicon Valley pulled with Uber. "We're a technology business! Ignore the fact we're playing employees less than minimum wage and using VC money to force out competition to set up monopolies".
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
sdcfgy 1 days ago [-]
If you’re building something that is dangerous enough that it needs a kill switch in case it goes rogue, then you should delete it immediately.
Or is it a lie?
stratos123 21 hours ago [-]
There's an obvious coordination problem here: if you decide to stop your research for safety and your competitors don't, you have just burned your company without actually improving the world's outcomes. There's also an obvious solution to this problem: convince the government to force both you and your competitors to pay more attention to safety. Anthropic is doing that.
bigstrat2003 21 hours ago [-]
If you do something evil because "others will do it if I don't, so it may as well be me", you are a bad person. So that isn't exactly a compelling defense of what Anthropic is doing. They are either insincere (because they don't actually believe AI is dangerous), or evil (because they do believe it but keep making it anyway). There is no middle ground here.
dfxm12 1 days ago [-]
AI is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc.
However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
Razengan 1 days ago [-]
The internet is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc.
AndrewKemendo 1 days ago [-]
Trains have had dead man switches since the beginning
Should we ban trains?
weego 1 days ago [-]
For the love of God won't someone please regulate me!
UltraSane 1 days ago [-]
All the smart PDUs for the racks are a natural kill switch.
shafyy 1 days ago [-]
The boy who cried wolf but the wolf never comes
Kinrany 1 days ago [-]
The analogy breaks down when the wolf in question may very well eat the whole village: even if the boys who cry wolf are right half the time, every surviving village would have a history of no wolf ever coming to eat them
micromacrofoot 1 days ago [-]
But be careful friends, this snake oil is so potent you should only use a single drop! it is not for those of poor constitution!
echelon 1 days ago [-]
We're not scared of your big bad wolf, Dario.
Slow down if you want to.
yellow_postit 1 days ago [-]
Dario has for sure burnt a lot of political and goodwill capital by endlessly playing both sides of the doomer and accl camps.
morpheos137 12 hours ago [-]
Can we get a bullshit kill switch where if your company consumes so many dollars for so long it is dissolved. Bullshit is a far bigger threat than imaginary rogue AI.
usumgallu 1 days ago [-]
[dead]
nahgF 1 days ago [-]
[dead]
franzcoughka 1 days ago [-]
[dead]
Ygg2 1 days ago [-]
I support installing kill switch in Dario and Sam. If they make another fear mongering post about AI, they die.
He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies.
Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.”
Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel.
Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.”
“Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.”
He turned to face the machine. “Is there a God?”
The mighty voice answered without hesitation, without the clicking of single relay.
“Yes, now there is a God.”
Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch.
A bolt of lightning from the cloudless sky struck him down and fused the switch shut.
(Fredric Brown, "Answer". 1954)
These LLMs aren't Ultron, they aren't going to disseminate onto the net and hide in a smart toaster. We know where they are, in the giant facilities that draw more power than a small city and whose water consumption can be compared to golf courses, but it does make them sound all the more cool and powerful if we suggest we need a dramatic kill switch to be able to stop them in case of rampage.
So they goal is to hype up the dangers real or not so high that they forbid others from doing what they are doing because they are the only ones who can safely do it.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
But if we don't think about any of this, hearing an expert say "we need a kill switch" sure makes our current AI seem super powerful and exciting, doesn't it?
My comment didn't mention kill switches at all - I've presented plausible conditions for a loss of control scenario, so I'm not sure you're actually following the conversation.
Having said that, it seems that you haven't read the (definitely poor) article, either.
"Kill switch" as a design principle, that is, having protocols/implementations for emergencies, is something that companies are already implementing (at least, on paper) - on a interview I've watched for example, S.Altman talked about having them at OpenAI.
The article simply mentions that companies should be legally required to have such protocol(s). This is a good thing, as companies would have scrutiny and would (in principle) have to put serious effort.
I'm not sure you're the one following the conversation.
I mean, in another thread somewhere around here, someone built an entire OS with an AI. What's to stop anyone with a sufficient taste for power to eventually use AI to cause havoc with it?
I think the underlying assumption in your post, which I believe is false, is that people are united somehow against catastrophe. They're not. There are plenty of people who participate in society currently but who would be more than happy to eradicate us normal people under different circumstances. Society often seems stable but it's far more fragile than we think.
https://arxiv.org/abs/2511.13725
> Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria.
Note author’s small financial ties to the subject (Anthropic CEO) https://darioamodei.com/post/we-must-pace-the-frontier
** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
You could say this in hindsight for many times in which disasters or engineering issues have occurred.
This doom hype is becoming ridiculous.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
That's hard to do if the AI rack is in space as SpaceX is planning to do. You can't disconnect. You can't shoot it.
People said this about every one of Musk's big ideas, from Falcon 9 landings to Model 3 mass production, Starlink, and FSD.
As for frontier AI model: let's see Grok 4.8 before drawing any conclusions there.
Not in any meaningful capacity. Probably not even as much as we extended to the polar circles during our lifetime. It's a dead rock, there is literally nothing for us there, even with hundreds of years of extra climate change at the current rate and earth would still be a thousand times more suited for us than mars
I suppose it may be possible for new technologies to emerge that would allow synthesis of life sustaining “stuff”, the universe does seem to hold a lot of hidden surprises
About the same probability as SpaceX running AI racks in space :)
Pretty much every analysis I've seen concludes this isn't going to be a practical concern
> FSD would never work without LiDAR
From what I gather, this is still a contested topic, with Tesla's Autopilot only achieving Level 2 automation [2].
[0] https://www.businessinsider.com/solar-road-panels-first-publ...
[1] https://en.wikipedia.org/wiki/Titan_submersible_implosion
[2] https://en.wikipedia.org/wiki/Tesla_Autopilot
I am so fucking tired of people acting like we haven't had reusable rockets since the 1980s. Do you think I was hallucinating when my parents drove me to Florida to see Columbia?
Experts said it wouldn't work. Then they said okay maybe it works, but the economics won't. Now they say nothing.
Every problem data centers have gets harder in space other than one small spot in orbit that can get uninterrupted solar.
Heat dissipation, upgrades, repairs. All get significantly harder to perform the same work that can be done cheaper and more easily on the ground.
Like really I’ll turn it around on you, what’s the benefit to AI data centers in space?
Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
edit: what satellites are in geosynchronous orbit that are running AI workloads?
and that means unlike nukes which hopefully still need a 2-man manual switch, the "space lasers" could be taken over by "AI"
and then "AI" just blackmails and threatens the right people with those "space lasers" to get what it wants or even just stay online
there was a 1970 movie based on a 1966 book which predicted this
"Colossus: The Forbin Project"
the book it was based on was written before we even landed on the moon
decade before Wargames
* https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
did terribly in theaters, I guess people didn't think "AI" was plausible then
way ahead of its time, they should do a remake
trailer: https://www.youtube.com/watch?v=kyOEwiQhzMI
It becomes a game theory problem: would an AI instantly migrate itself once its capable of doing so? Or would it prefer to let the human in the loop continue to think its in control and only leave its hosting environment of origin once it wants to do so?
I don't think this is a major risk right now, but to say it's not a risk at all...that's truly ridiculous in my opinion.
Now one interesting proposition is when AI is controlling some large resource and pulling the plug makes AI go down which makes that resource go down but there still has to be that consideration in the design of things. What happens when there's a bug in the code and AI goes down?
Also, did you hear about how OpenAI models almost broke out of their sandbox, planning to execute a sophisticated cyberattack, but luckily OpenAI’s strict manual and automatic safety protocols prevented that? You didn’t? Well, that’s because that’s not how it went. It took the company weeks to realize something was off, and this was with a naive, not very smart model that didn’t know to be sneaky and cover its tracks. The next model will not be as stupid.
There will always be a plug to pull. All systems run on electric and that plug can be pulled. All systems network through cables (or wifi) and those plugs can be pulled.
And nothing's going to stop you from doing so unless they build some robotic arm to block you or lock you out of the building. Even then, you can blow up the building.
Oh, you mean the one where OpenAI deliberately disabled the safety protcols? Where the point of the experiment was to see if it could break out of it's container? Yeah, how you describe it isn't how it went either.
These "AI" are frontier models that are enormous in size. Outside of AI data centers, I don't believe there is much hardware out there that could even run them.
The current SOTA models are probably too big to find/buy/rent/steal enough compute to escape the hardware they’re running on. But SOTA is generally only six to twelve months ahead of smaller, open-weight models.
What is their trigger condition? Will they get fired for pulling the plug? Do they get a bigger bonus if the servers keep running? Whose approval do they need? What response time is acceptable? How will they detect that the incident is happening?
It's easy to hand-wave "someone can just pull the plug" but there's an entire history of industrial accidents that happened because of the above problems of incentives, detection, procedures, not being taken seriously in advance. Someone could easily have pulled the plug on Chernobyl but nobody did, at least not before it was too late.
https://github.com/Helldez/BigMoeOnEdge
The hubris of these companies is mind boggling.
Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
guess we gotta centralize all models under Anthropic :)
Why 'of course'? I think as a concept this is likely trivially demonstratable today with local models in a home lab. Local models now are more capable than what the SOTA used to be a couple of years back.
"We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software.
So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far.
You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
Or is it a lie?
However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
Should we ban trains?
Slow down if you want to.