It all started with a phone call. "The whole network at [customer redacted] is down and they have no power - they need your help."My blood ran cold. The engineer calling me sounded panicked, and for good reason. [Customer redacted] has an enormous natural gas facility in South Texas, too far from civilization to get enough power off of the grid. We designed and built an onsite natural gas power plant for them - a big one, capable of supplying 40+MW of power at peak load. They could run the facility for a short while without the power plant, but not long - and shutting down the facility meant losing 7 figures per hour. By the time I was informed, they had 6 hours until they had to shut down.
As the guy who had designed and installed said network, I was naturally the guy to call when it had problems, which had never happened before. It was a pretty simple network, honestly - just switches, cat5 cables and fiber. Since this was the network all the PLCs, relays, meters and whatnot ran on, it was airgapped & isolated, no routers. Not much to go wrong.
I quickly get on the phone and walk the guy on their end through plugging in a laptop and running
some simple tests. Check lights on things, ping this, ping that. Everything seems good, though. The network is emphatically not down. So I send him a remote app and take control of his laptop to see for myself.
Log into switches, check things, nope, the network's not down. When I log into the HMI system, though, I see a big red error message: "Network Error: Cannot connect to database". The database server is up, though. I log into the database server (Windows Server 2012 running MSSQL) and that's where I find the problem: SQL isn't running. I try to start it and it immediately shuts back off.
