In the never-ending quest to save money and boost performance, many organizations are migrating their industrial control systems onto IP networks. As network engineers get to know these new systems, they must tread carefully -- one mistake can lead to disaster.
Often referred to as SCADA (Supervisory Control And Data Acquisition) systems,these industrial networks are used in a variety of industries. Utility companies use them to manage electrical grids, natural gas distribution, and sewer and water systems. Manufacturers use them to manage factory floor equipment. Construction and engineering firms often manage heavy equipment, such as cranes, with them. Nuclear power plants manage fission and power generation with these systems.
SCADA systems are dependable. Many have been running for 10 or 20 years on radio and serial network connections. Now, however, many organizations are looking to run them over IP networks in order to boost performance and save money.
"You get better performance and better costs [on IP networks]," said Dale Peterson, founder and director of the control systems security practices at consultancy Digital Bond, in Sunrise, Fla. "The serial networks have relatively low bandwidth, and the networking to the serial networks can be pretty complicated because there is often a variety of networks you could be going over: dial-up, VSATs, leased lines, radio, microwave, dark fiber. Usually, there is a front-end processor that takes packets from your SCADA server and converts them into the right-sized protocol. All that goes away with IP. You just replace it with a router. It's simpler."
Peterson said the switch to IP networks also makes sense because many sites where SCADA systems are located are already connected via an IP-based wide area network. For instance, the site may already have in place IP-based security cameras, IP telephony service and IP-based physical security systems. It often makes sense simply to add an Ethernet card to the SCADA equipment on site to save money and gain some operational efficiencies.
There are, however, plenty of risks involved with running SCADA over IP, according to Michael Markulec, COO for Lumeta Corporation, a Somerset, N.J.-based vendor of network visibility and assurance products.
"Folks face tremendous challenges securing their traditional IP networks themselves," Markulec said. "There are already Web servers, email servers, internal databases and PCs on the network. Now you add to that all these nontraditional devices on the IP networks: cameras, physical card-swiping apparatus at doors, as well as SCADA networks with monitors, programmable logic controllers, physical sensors, valves and switches that sit on your manufacturing floors that now all have IP addresses to them. And by connecting them to your IP networks, you now have a whole new set of vulnerabilities associated with them."
Organizations should take a defense-in-depth approach to SCADA, Markulec said. It starts with access control lists (ACLs) on the routers and careful placement of firewalls to make sure that only the proper traffic is reaching these systems.
"The real important part is having a control network, a network that allows you access to very different levels of secure networks, and an audit and monitoring network where you send things like logs," he said.
"To be blunt, [SCADA networks] scare the hell out of me," said Matthew Shoemaker, systems engineer for the Henry County Water and Sewerage Authority (HCWSA) in Georgia. "In a normal IT group, if the desktop team deploys a bad patch, it's, 'Oh crud, let's hurry up and rebuild the package and redeploy to the group.' If that happens on our SCADA network, literally a pipe could blow out of the ground. That would be a rare case, but the potential is there."
Shoemaker recently consolidated large portions of HCWSA's SCADA network over the authority's IP-based Metro Ethernet mesh network. He segmented the SCADA traffic from other traffic on his production network, but he was deeply concerned about security.
"It made us really nervous about having SCADA traffic on our production network because so many of these SCADA systems are fragile," Shoemaker said.
Originally, he used a series of intrusion detection system (IDS) appliances to completely segment the SCADA traffic into a closed, automated circuit, but the log information generated by the IDS equipment was difficult to correlate. And the closed circuit gave HCWSA limited visibility into how the SCADA system was performing.
"We try to keep them as segmented as possible, but business requirements dictate that there basically be some intersection [with the corporate data network], if nothing else than for information, if not control," Shoemaker said. "So it was important to have ways to monitor and correctly identify not only the type of traffic but its point of origin and how it was moving across the wire."
For instance, he needed to have visibility for maintenance issues, such as knowing when a water tank is overflowing or whether pipes are leaking.
Shoemaker ultimately started using Stealthwatch, a network behavioral analysis tool from Lancope. The appliance passively tracks the NetFlow and sFlow data generated by communication between network hosts and clients. It establishes a baseline of normal behavior, and it can alert users when anomalous network behavior occurs. Shoemaker said this helps his organization to detect whether worms or viruses or other malware are propagating on the network. But more importantly, it tracks the overall behavior of the SCADA system to make sure devices are configured properly and aren't adversely affected if and when they interact with the corporate data network. If a device starts behaving strangely, Shoemaker can use the tool to quickly isolate it from the rest of the network and determine what the problem is.
"We're using SCADA to generate telemetry that has a live business impact, everything from leak detection to helping to justify our SCADA expenditures," he said. "We're using Lancope to monitor and measure the effectiveness of the changes we're making."
In addition to security concerns, SCADA networks are also extremely sensitive to actions that many network administrators might consider routine. For instance, a faulty patch or an improperly applied software update can wreak havoc. Officials at the Hatch nuclear power plant in Baxley, Ga., learned that the hard way last March. According to the Washington Post, an engineer installed a software update on a computer that monitored chemical and diagnostic data from one of the plant's primary control systems. The update caused the computer to reboot, which reset the data on the control system. The plant's safety system interpreted a lack of data as a cooling problem and initiated an emergency shutdown of the plant that lasted for two days.
Officials said the emergency systems performed well, and there was never any breach of safety or security at the plant. However, having a nuclear power plant shut down for two days represented an obvious revenue loss to Southern Company, the plant's operator.
Peterson, of Digital Bond, said SCADA systems are also extremely sensitive to routine scanning. Many SCADA devices react in unexpected ways when they are scanned. Developers of SCADA devices and systems generally perform only positive testing of their technology. They focus on making sure that a SCADA device responds properly when it gets a legitimate request and responds in good time. He said this extensive positive testing has led to SCADA systems being very reliable, but vendors typically don't do any negative testing.
"[Vendors] have done a very bad job on negative testing," Peterson said. "What if you send it something it never expected? It's not unusual for a control system application or controller to hang or crash or shut down a service if it gets a command it wasn't expecting. And something as simple as port scans has caused serious problems. If you go in and run an nmap scan on a class C network, you could, in certain versions of control system applications, cause every one of those applications to go down – primary, backup, everything."
Peterson said IT organizations that are planning to transport SCADA traffic over the production network need to listen carefully to SCADA engineers. They need to understand the criticality of the systems and how sensitive they are to certain types of activity that routinely occur on the network.
Markulec, of Lumeta, said network administrators and IT staff generally don't understand the nature of SCADA systems, and SCADA engineers generally lack IP networking skills.
"So as organizations move through this, you need to develop some best practices," he said. "You need to know what some organizations are doing. You need to go out and ask questions. You have all these programmable logic controllers that now have TCP-IP stacks. Now who is doing the updates? Who is doing patch management for them? Who is doing vulnerability management? These are the types of questions that we take for granted on our server farms but we neglect on our manufacturing shop floors."
"I've talked to some nuclear engineers, and I've talked to other water departments and some power guys. Most of them aren't [getting any visibility into their SCADA systems]," Shoemaker said. "I don't think people truly understand how vulnerable these infrastructures are. SCADA is on nuclear power plants. SCADA is on power plants, railroads, Department of Transportation. It's scary to know that with $500 and Radio Shack, someone can replay radio traffic on some SCADA networks and literally take down large segments of their capabilities. So if you're not monitoring that or don't have a way to monitor that, you're running blind."
Let us know what you think about the story; email: Shamus McGillicuddy, News Editor