An unknown hacker virtually infiltrated the city of Oldsmar’s water treatment plant — twice — to increase its levels of sodium hydroxide (lye) before Super Bowl Sunday weekend
Tampa Bay is riding high after Sunday night’s Super Bowl IV win. But things for a lot of residents of the Bay area could have gone in a less-than-celebratory direction if it wasn’t for the keen eye of one water treatment employee.
Pinellas County Sheriff Bob Gualtieri reported a significant security breach during a news conference on Monday. An unknown hacker remotely accessed systems that control the Oldsmar water treatment plant’s operations and treatment chemical levels. The goal? To increase the plant’s level of sodium hydroxide to dangerous levels.
For those of you who bombed your high school science classes, sodium hydroxide — or what’s also known as lye — is a highly caustic chemical. In fact, it’s one of the main ingredients of drain cleaners. When used at safe levels, this corrosive material removes metals from drinking water and changes water acidity levels. But when those levels are increased, it can:
- Severely damage your lungs when inhaled,
- Burn your mouth, tongue, and esophagus when swallowed, and
- Cause significant illness, gastrointestinal perforation, and death.
This topic strikes particularly close to home seeing as how Oldsmar is about a 30-minute drive north of our company’s home office. Luckily for those potentially affected residents of Pinellas County, a plant operator noticed what was happening and managed to catch it before it could do any damage.
We’ll give you the lowdown of what occurred and what other municipalities and businesses can do to prevent similar intrusions into their computer systems.
Let’s hash it out.
Breaking Down the Situation
Although it resembles the plot of the 2005 movie “Batman Forever,” what happened on Feb. 5 was anything but fiction. In a nutshell, an unknown hacker infiltrated the city of Oldsmar’s computer system via its remote access software to increase levels of a dangerous water treatment chemical.
Like some other cities in Pinellas County, Oldsmar is one that has its own water treatment facility. This dedicated facility provides the drinking water to businesses in the area as well as the city’s nearly 15,000 residents. Other cities get their water from the County, which pipes water from Tampa Bay Water.
Here’s a quick overview of what occurred:
- At 8 a.m., a plant operator noticed someone briefly accessing the computer system he was monitoring remotely. This system controls the chemicals and water plant operations. This type of activity didn’t really stand out because the system was set up to allow remote access by authorized users. This way, they can troubleshoot issues from other locations.
- At 1:30 p.m., someone again remotely accessed the system the same plant operator was monitoring. The access lasted a total of three to five minutes. But in this time, the employee watched the mouse on their screen begin moving. The mouse accessed various software functions that control water treatment functions, including the function that controls the water’s levels of sodium hydroxide. They changed it from 100 parts per million (ppm) to 11,100 ppm.
- Once the hacker exited the system, the plant operator immediately changed the chemicals back to their normal levels. After that, he immediately notified his supervisor about what had transpired. Oldsmar City Manager Al Braithwaite says that they’ve disabled the remote access software to prevent this cyber threat from reoccurring. He also said they “are going to make some upgrades to other parts of the system to try and ensure that it doesn’t happen again.”
Gualtieri held a press conference on Monday, Feb. 8 to discuss the details of the virtual breach of this city’s critical infrastructure. In it, he offered the following reassurances:
“Because the operator noticed the increase and lowered it right away, at no time was there a significant adverse effect on the water being treated. More importantly, the public was never in danger. Even if the plant operator had not quickly reversed the increased amount of sodium hydroxide, it would have taken between 24 and 36 hours for that water to hit the water supply system, and there are redundancies in place where the water had been checked before it was released.”
Here’s a link to the press conference here so you can see it for yourself (if you’re so inclined):
What’s Still Unknown
Pinellas County Sheriff’s Office (PCSO) has launched a criminal investigation. They’re working with the FBI and the U.S. Secret Service as part of their investigation. However, some things are still unknown about the attack at this time:
- Who is responsible for the breach. Authorities say they’re uncertain about whether one person, two people, or a group of people were involved.
- How the breach occurred. They know that the access occurred via the computer system’s remote access software. However, they’ve not disclosed whether the
- Whether the attack originated in the U.S. or abroad. Sheriff Gualtieri says that it’s unknown at this time whether the attacker was someone local, elsewhere in the U.S., or if it could be an international or nation-state actor.
- No reports of any other systems being unlawfully accessed. The Sheriff and city officials say that they’re unaware of any similar attacks occurring
What Went Well in This Situation
Obviously, things have gone in an entirely wrong direction and we’d be writing a very different article right now. Thankfully, though, there was an unnamed employee who was actively monitoring the city’s computer systems. The employee recognized the unusual activity and acted quickly to mitigate the threat by reversing the attacker’s changes to the sodium hydroxide levels.
Furthermore, the city has redundancies and sensors in place that would catch the changes in the water before it made it to Oldsmar homes and businesses. So, even if the employee hadn’t observed the attack as it was happening, city official say that the water would have triggered alerts well before it would have posed a danger to residents.
There are a lot of different industries that fall under the umbrella of “critical infrastructure.” For example:
- Heathcare and emergency services
- Military and defense
- Water utilities
- Utility companies
- Cable providers
- Cellular providers
- Transportation and roadways
- Sanitation organizations
- Food and agriculture
- Building and structural organizations
A Few Key Takeaways
So, what are the big takeaways from this situation for governments, businesses and other organizations that manage our local, state, or national critical infrastructure?
- If you haven’t already done so, draft and implement computer security and monitoring protocols.
- Actively review those protocols to ensure they’re up to date and in alignment with current industry best practices.
- Have redundancies in place to serve as fail safes.
- If you’re using any remote access software, be sure to keep it patched and up to date with the latest manufacturer releases. Also be sure to use very strict access control, such as two-factor authentication (preferably PKI and/or hardware token-based).
- The same can be said with other IT hardware and software updates — don’t wait until there’s another Eternal Blue situation. Regularly perform updates and patching to keep your systems in tip-top shape.
- Run vulnerability and risk assessments to discover any exploits that cybercriminals can use to breach your network and other systems.
- Provide cyber awareness training to your employees and other system users. This way, they are less likely to fall for any phishing scams that could result in credential theft or disclosure.
To wrap up this article, it’s fitting to include one last quote from Sheriff Gualtieri to help drive home the importance of making these preparations:
“This type of activity in this type of hacking of critical infrastructure is not necessarily limited to water supply systems. It can be anything. It could be sewer systems, it could be a whole variety of things that could really be problematic, and this is where we want to make sure that we’re paying close attention to all of it.”