How to fix SCADA security [not]
In "A cautionary tale about nuclear change management" ComputerWorld blogger Scott McPerson discusses a few security incidents that have been linked to SCADA systems, picking out two causes: poor change management and problems with the IT architectures. If only things were so simple in Real Life.
According to Scott, the change management problem can be solved by adequate pre-release testing of patches. Mmm. OK, well let's assume a SCADA-using organization has the resources to invest in an IT test jig comprehensive enough to model the live SCADA/ICS systems, complete with real-time data feed simulators and control panels, or at least a sufficient part of the complete live system to allow representative and realistic testing. Presumably they could test the patches and software upgrades thoroughly enough to reduce the possibility of unintended consequences, but how far can or indeed should they go? Anyone who has actually tried to do exhaustive software testing, even in a very simple laboratory setting, knows that it is literally impossible to test everything in practice. With the best will in the world, the fanciest test jig that money can buy and the most competent, skilled and diligent professional testers on the job, there is always a residual risk at the declared end of testing. In real life, the end of testing is almost always declared by management well before the testers are truly happy, not least because the issues and risks that the planned software changes are supposed to fix inevitably persist at least until the fix is applied, so there are clearly competing pressures. Damned if we do, damned if we don't.
OK, I'm certainly not arguing that pre-release software testing is a waste of time on SCADA or any other IT systems, far from it. But the reality is that no matter how much testing and fixing is done, the eventual decision to implement implicitly if not explicitly accepts the residual risk. In my experience, the operational, safety and commercial risks associated with system failures on SCADA systems are so significant that the opposite situation is more of a problem, namely that SCADA systems are not patched at all, or at least not promptly, due to the extreme risk aversion. Legacy systems are the norm not the exception in SCADA/ICS-land. In the case of safety-relevant and certified systems, plus the highly specialized bespoke systems typical of controllers for complex machinery (such as, oh er, a nuclear power station), the inertia problem is even worse.
Scott's second point about IT architectural issues also seems rather glib to me. "The fact that some utilities -- including nuclear utilities -- are stupid enough to attach the servers that control and manage SCADA systems to the same Internet that runs porn and Nigerian scams and MySpace is ludicrous. It is also dangerous." That statement seriously denegrates the highly competent IT and business managers in the utilities, manufacturing and engineering companies where I have worked. Such people are far from stupid. As I said already, they are highly risk averse and do not take such decisions lightly. But again there are competing priorities. The Internet is a convenient, cheap way to access SCADA/ICS systems, networks, devices etc. for remote diagnostics and support purposes, for example, and often glues together critical business processes throughout the supply chain. Connecting the SCADA/ICS network to any other network (even the internal corporate LAN) is clearly fraught with danger so security is always a concern.
The main beef I have with you, Scott, is that you have over-simplified the problems and provided trivial solutions, as if simply saying these things will make a difference. Calling the people who are actually dealing with the risks "stupid" is hardly going to make friends and influence people.
According to Scott, the change management problem can be solved by adequate pre-release testing of patches. Mmm. OK, well let's assume a SCADA-using organization has the resources to invest in an IT test jig comprehensive enough to model the live SCADA/ICS systems, complete with real-time data feed simulators and control panels, or at least a sufficient part of the complete live system to allow representative and realistic testing. Presumably they could test the patches and software upgrades thoroughly enough to reduce the possibility of unintended consequences, but how far can or indeed should they go? Anyone who has actually tried to do exhaustive software testing, even in a very simple laboratory setting, knows that it is literally impossible to test everything in practice. With the best will in the world, the fanciest test jig that money can buy and the most competent, skilled and diligent professional testers on the job, there is always a residual risk at the declared end of testing. In real life, the end of testing is almost always declared by management well before the testers are truly happy, not least because the issues and risks that the planned software changes are supposed to fix inevitably persist at least until the fix is applied, so there are clearly competing pressures. Damned if we do, damned if we don't.
OK, I'm certainly not arguing that pre-release software testing is a waste of time on SCADA or any other IT systems, far from it. But the reality is that no matter how much testing and fixing is done, the eventual decision to implement implicitly if not explicitly accepts the residual risk. In my experience, the operational, safety and commercial risks associated with system failures on SCADA systems are so significant that the opposite situation is more of a problem, namely that SCADA systems are not patched at all, or at least not promptly, due to the extreme risk aversion. Legacy systems are the norm not the exception in SCADA/ICS-land. In the case of safety-relevant and certified systems, plus the highly specialized bespoke systems typical of controllers for complex machinery (such as, oh er, a nuclear power station), the inertia problem is even worse.
Scott's second point about IT architectural issues also seems rather glib to me. "The fact that some utilities -- including nuclear utilities -- are stupid enough to attach the servers that control and manage SCADA systems to the same Internet that runs porn and Nigerian scams and MySpace is ludicrous. It is also dangerous." That statement seriously denegrates the highly competent IT and business managers in the utilities, manufacturing and engineering companies where I have worked. Such people are far from stupid. As I said already, they are highly risk averse and do not take such decisions lightly. But again there are competing priorities. The Internet is a convenient, cheap way to access SCADA/ICS systems, networks, devices etc. for remote diagnostics and support purposes, for example, and often glues together critical business processes throughout the supply chain. Connecting the SCADA/ICS network to any other network (even the internal corporate LAN) is clearly fraught with danger so security is always a concern.
The main beef I have with you, Scott, is that you have over-simplified the problems and provided trivial solutions, as if simply saying these things will make a difference. Calling the people who are actually dealing with the risks "stupid" is hardly going to make friends and influence people.
Links to this post:
