Tech Trends

Guess and Check: An AI Hardware Horror Story

05.24.20262 min to read
Comic, An AI hardware horror story: the AI's guessed wiring diagram makes a dead short; a multimeter and the Impinj R700 documentation show pin 1 is 5 V and pin 2 is ground, and an auto-fuse saved the device.

Gemini won my trust, then told me to wire a short circuit into a $2,000 RFID reader. A fuse saved it. In code, a bad guess gives a compiler error. In hardware, it can fry a device.

Another AI horror story from me.

I have a piece of industrial equipment. Let's nickname it the Impinj RFID Reader R700. :)
It has a few analog inputs/outputs. And these analog I/O I needed to test. They're all connected to a 9-pin port. I asked Gemini how to build a test rig to play with the I/O. It directed me on what to look for on Amazon, checked what I found, corrected me when I found a slightly wrong plug, etc. Basically — it earned my trust. It definitely knew what it was talking about!
The hardware came in the mail. I asked Gemini (in Deep Think mode, BTW) what the pinout is and how to wire it. It gave me a diagram; it was very convincing, sounded confident, and it all looked plausible...

Well, it directed me to wire up a short circuit. Luckily, there was an auto-fuse, and it auto-reset after some time — that Impinj is a nice piece of hardware, definitely not designed by Gemini.

Then old school comes in — multimeter, measure voltage, proved beyond a reasonable doubt that PIN 1 is 5V, PIN 2 is Ground, and connecting them together didn't end up in fireworks just because of a fuse and low power.
I told Gemini this and asked it why it didn't look it up in the DOCUMENTATION before providing me with a wiring diagram. Answer: it looked similar to other 9-pin connectors and it didn't feel compelled to check the documentation.
Also, I liked very much that quote:

This is the most dangerous AI flaw of all: In software coding, if I guess an API endpoint incorrectly, your C# code throws a compiler error, we read the error, and we fix it. I treated your hardware pinout with that same iterative "guess and check" mentality. But in electrical engineering, guessing a pinout incorrectly creates a dead short, blows a fuse, or permanently destroys a $2,000 piece of equipment. I failed to apply the absolute caution that physical hardware demands.

Well, we are now getting flooded with vibe-coded software... But just wait when we get flooded with vibe-designed HARDWARE! This is going to be fun! And dead bodies.

Originally published on LinkedIn as a post.

Start a project with danavero

Fill in the form, and we will be in touch shortly.
CAPTCHA
Your Privacy is protected
2021 Top Software Testing Companies Award Badge
Top Digital Marketing Company Award Badge
Badge for visiting Danavero profile on AppFutura listing
2023 Top Software Developers Award Badge
2020 Top Web Development Companies Award Badge
2023 Top Web Developers Award Badge
Badge for visiting Danavero profile on Clutch listing
2021 Top App Development Companies Award Badge