Over the course of their careers, the members of the CIMA+ Operational and Digital Technologies team have collectively helped owners and operators of facilities and utilities in almost every industry and sector you can think of to set up and maintain their operational technology (OT) systems.
With decades of real-world design and construction experience, they understand the challenges that owners and operators face. One of those challenges is developing and implementing a facility- or utility-level master plan for managing critical computer control systems. To help owners and operators tackle this seemingly daunting challenge, the team has leveraged its experience in helping clients with their OT master plans to design a straightforward approach to completing this critical planning exercise that should be a priority for any organization relying on OT, regardless of its size. But as our Operational and Digital Technologies team demonstrates, the exercise doesn’t have to be a painful one.
What is an operational technology system and where are these systems used?
Many critical infrastructure utilities rely on operational technology (OT) automation systems. These systems go by a wide variety of names and acronyms, such as supervisory control and data acquisition systems (SCADA), process control systems (PCS), or distributed control systems (DCS). There are nuances in the specific components that go into them as well, such as the use of a programmable logic controller (PLC) instead of a remote terminal unit (RTU) or a real-time automation control (RTAC). However, despite their differences, these systems have much in common at a fundamental level. They are built with similar specialized hardware, including industrialized hardware controllers, field instruments, network switches, routers, and firewalls. They also use a combination of common business software suites and a variety of specialized software and custom applications, as well as extensive databases and both generic and proprietary communication protocols. OT systems in general are part of a broad range of technology sectors that have seen a significant emergence of smart devices as cloud-based technologies have become increasingly prevalent.
Their use in some form is ubiquitous across many industries, especially those with harsh environments. OT systems are used in our water and wastewater systems; in road-, rail- and air-based transit systems; in our electrical utilities at power generation facilities and throughout the transmission and distribution network; in oil and gas production, refining, pumping and storage; across our telecommunications systems; and in the mining and resources sectors. These systems monitor equipment status and telemetry points, measure key operational parameters, log important historical data, and provide alarms and alerts for system deviations.
OT systems are vital yet frequently overlooked elements of infrastructure
OT systems can be mission-critical assets. Many utilities and organizations could not continue day-to-day operation without them. In the design and construction of multimillion-dollar facilities, it is common practice to measure success based on how well the new system, facility, or project operates and is able to continue running quietly in the background. Despite their critical role, OT systems are rarely in the spotlight when it comes to measuring success.
In our experience, because their job is to operate behind the scenes, OT systems often do not have adequate long-term budgets allocated to them for their regular maintenance and eventual replacement. Many sectors use an accepted standard or guideline for asset management, with documented lifespans for assets, defined maintenance intervals, and capital replacement budgets. So why is it that for these critical computer control systems, it is common practice to delay or defer upkeep so long as they are still functioning, or because they were installed only a few years ago and are seemingly too new to be failing? What’s more, it’s common for a utility or facility owner to place higher importance on a production asset, such as a pump, valve, inverter or truck, than on a server or other part of the control system. The production asset may directly impact a prominent area of operation, but the server may affect the entire organization, in all areas.
What are the challenges we hear from the field?
Many organizations, for better or worse, end up adopting a passive maintenance philosophy. Without formalized plans with five- and ten-year development cycles, systems are effectively run to failure and replaced under maintenance budgets. This can be especially true for smaller organizations and utilities. A SCADA or DCS system may even be talked about as if it can be simply purchased like a car or a truck. In reality, modern OT systems have dozens of major elements and potentially hundreds of smaller subcomponents. They are highly complex, interconnected systems that are put together with considerable forethought and planning. In addition, OT system assets are subject to compatibility issues that are quite different from those of most assets that engineers and planners are accustomed to dealing with. Especially when components are a decade or more apart in age, the failure of only one of them can require the upgrade or even the replacement of the entire system.
Facility owners and operators aren’t always aware of the differences between traditional asset management plans and those that are needed for OT systems, particularly with respect to asset lifespan. It is common practice for master plans for large utilities to stretch decades into the future and to consider large-scale factors such as urban growth and population density. These plans are also typically for assets with life expectancies of half a century or more, which is simply not the case for many technology systems. In an OT system, assets can have a life expectancy of only a few years, and this often leads to asset owners feeling that a system component needs to be replaced after it has just been installed.
Another common problem for owners and operators is that OT systems are not part of their area of expertise, which means they may not know how to start a project like an OT master plan. They may be able to navigate cost estimating for hardware but not as easily the planning for the custom development effort needed to make the hardware work. Will they be able to prepare for a major software release that effectively requires the system’s redevelopment, perhaps if compatibility issues arise between the versions of the specialized controller hardware and the programming/configuration software? In this case, a software upgrade may drive a hardware upgrade, or vice versa, with significant associated costs. In addition, because of increasing cybersecurity concerns, many organizations (along with new legislation) are mandating that cybersecurity plans be put in place. This can mean decommissioning vulnerable operating systems or software revisions that are no longer receiving security updates. These software upgrades can also drive major infrastructure overhauls as the new operating system or software may not support older hardware in the system. In sum, the challenge can be daunting: How do you plan for all of this to ensure that the appropriate budgets are in place and that system components are upgraded as needed with minimal impact?
What to expect in an OT master plan project
First, it is important to understand that there is no predefined scope for an OT master plan. It is the organization’s needs, desires, complexity, capacity and budget that are the determining factors of the plan, and the development of the plan can be scaled accordingly. It is not uncommon for large utilities to have master plans that cover a decade or more and that include tens of millions of dollars in capital and operating expenditures, but that does not mean this is the only kind of master plan, or that master plans are only appropriate for large organizations. Even the smallest utility or organization using an OT system should have a budget and a plan for annual maintenance and capital investment to ensure the system stays healthy. What’s more, coming up with a plan doesn’t have to be arduous. Often, it means going through only a few distinct planning stages — and we can help you get through them.
First, you need to know where you are today — your current state — and what documentation already exists. Site visits may be necessary to determine the current age and condition of assets and to create a master asset database or a list of assets if you don’t already have one. You may conduct workshops to hear directly from affected staff about their challenges and about systems that are not fully functional. At this stage, we are not concerned with the future or how to fix problems. We are simply gathering details about current conditions.
After the first step, we encourage your organization to dream big and see the larger picture. We don’t yet challenge you with figuring out how to get to the final destination, what needs to change, how long it might take or what it might cost. This step is potentially the most transformative opportunity an organization has during a master planning exercise as you do not need to feel constrained or bound by your current operating practices. Throughout this phase, we will likely conduct many focus groups and market research. We may even visit other facilities, all with the intent of defining goals and setting a vision for the future. From here, we define a desired state and determine the finish line.
After defining a desired state, our team begins to do what we do best: critical thinking and problem solving. We conduct a gap analysis to identify the missing pieces: How do we get from where we are now to where we want to be? What are the steps? What are the projects? What needs to be upgraded? Is it a feature upgrade or a lifespan/obsolescence replacement? Typically, we do not consider budgets, timelines, or critical paths at this point; we focus on achieving each goal set in the previous step. Only after the gap analysis is complete and project specifics are in place does our team begin to develop budgets and schedules. We also reengage key stakeholders to further refine the project list and priorities, now that they can see the potential impact of different opportunities. In the last refinement, we can look critically at broader organizational connections, including project risk assessments, reviews of other initiatives, and easy wins that can be frontloaded. Project financing may also affect how the final schedule or critical path is set.
How do you start an OT master plan project?
It is important to understand that no two OT master plans are the same in scope or final deliverables. These are customized business and technology plans that require an investment of thought and due diligence to yield meaningful outcomes. The greatest benefit of a tailored master plan is that your organization can confidently make decisions, knowing that daily decisions support the desired long-term outcome. Getting started can be as easy as asking around your organization to determine if an OT master plan already exists and whether it is being followed or is overdue for an update. Depending on your organization’s technological maturity, undertaking a small exploratory or pilot study may be beneficial. The key to successfully developing a meaningful master plan is to engage a firm with OT experts who fully understand the nuances of how the elements of such an important system fit together. Whether your organization is large or small, whether you are just starting a technology transformation or are among the early adopters, our team is eager to help you undertake your next planning endeavour.
For more information, follow this link or directly contact our specialist at: adam.plumstead@cima.ca