> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.
Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.
Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.
[−]terribleperson · 2026-11-15 Sun 17:45 UTC ·
link
I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.
It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.
[−]terribleperson · 2026-11-15 Sun 19:14 UTC ·
link
I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?
We are used to Claude refusing to reverse engineer a lot of stuff. But, we are not used to Claude refusing to stop while reverse engineering something. That would be truly terrifying. You can't win - you cut the connection, you risk bricking the device, you let it continue, you're looking at Terminator, perhaps.
The guy kind of seems like he might be in a little over his head. That said I have used Claude (and other LLMs) for some similar stuff as an experiment and have seen them mess things up. Claude jacked up the canbus on a pre can gateway car while it was rolling down the road, which was funny because the whole Christmas tree in the dash lit up and my gauges dropped out (note: brakes and steering are mechanical and I knew exactly what was happening). I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS, even though a restart cleared it and I told it that this was on a controlled safe course.
In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.
> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.
Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.
finnicking... pondering... clauding..
529 please try again later
In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.