SCADA & Automation System Administration

Key SCADA and Automation System Administration Strategies

1. Define the boundary where the SCADA or Automation System stops.
   The boundary may be clear or, in increasingly more cases, not so clear. Many SCADA systems are infringing on traditional IT turf by using the IT networks and servers as well as traditional IT technologies such as PCs, Windows 95,98, and NT, and even PC databases. As a result it may be hard to say just where the boundary between the IT organization's responsibilities stop and the I&E organization's starts concerning administration and maintenance. In the past, some legacy SCADA and Automation systems were not even connected to IT resources such as the IT network. However, in a effort to push SCADA and Automation information out to business and engineering office users (and not just operations) more and more SCADA and Automation systems are using the often in-place IT resource to accomplish this task.
2. Make a chart defining who is responsible for what.
   As simple as this sounds, many times each person's responsibilities are taken for granted. This may work okay until something critical that falls into a gray area goes wrong. Then the finger pointing may start with "I thought you were taking care of that!" "What do you mean? Last time that happened, your group handled it", etc. Sounds like a recipe for hard feelings? You bet! Define up front who is responsible for what; and put it in writing.
3. Have a system back-up and restore plan.
   Make sure your backup includes a reliable restore! I have seen some backup systems that were apparently write only! When its time to use the backup, know it is going to work by testing the restore before you need to use it. Also, know what files and databases need to be included in your backup. Don't forget about your PLC RLL and like critical support files and documents. Often SCADA systems files and databases change on a continual basis. If your backup schedule is once a week, be sure you are prepared to loose up to a week's worth historical data in the event of a disk failure. Make a schedule to run the backup and stick to it. With the advent of online backup systems, it is now feasible to do a backup on a live system in many cases. You probably also should keep an offsite backup of your critical files in the event of fire, flood, or theft.
4. Have a business resumption and contingency plan for your SCADA or Automation system.
   Suppose your facility or office has been closed for an extended period due to a serious problem. Would it be necessary to setup and run your SCADA system from an alternate facility or office? Would it be technically and economically feasible? How would an extended outage of your SCADA system impact your business? Would you incur serious and potentially very detrimental damage? If you’re the system administrator, or just the person responsible, you had better know the answer to those questions. Moreover, you had better be prepared because it most definitely can happen. I have seen an office building closed for months due to an explosion and fire of undetermined cause in an electrical equipment room. The building owner and the tenant ended up in a lawsuit with each party claiming the other was responsible. Meanwhile, the tenant was barred from re-occupying the building first by the fire marshall, then the legal staff because of the lawsuits. Fortunately, we had just moved the SCADA system to another office building across town a few weeks prior to that event.
5. Develop a PC/Server file management system for your SCADA or Automation system.
   Where is that RLL file for the safety system? How about that Excel spreadsheet with the PLC's I/O listing? And the operations users manual for the pump control system? If you’re the system administrator, or just the person responsible, you had better know the answer where to find those critical files.
   Additionally, you may need to come up with those files when things are in a crisis management mode. The PLC has gone brain dead, lost the RLL program, and the plant is shutdown because the PLC is the plant's safety system. Now just where is that RLL file? And the documentation for a black start of the plant? Again, you had better be prepared because it most definitely can happen.
   Your file management system should specify file-naming conventions that should help ID the file(s) should they be misplaced. This issue is especially important if you utilize legacy MS-DOS RLL editors that only support 8.3 file-naming restrictions. Additionally, put it in writing and identify several layers of responsibility in case you are not around when the files are needed.
6. Develop a Management of Change System for your SCADA or Automation system.
   Un-managed change can create havoc with the users and administrators of the SCADA system. Consider employing a Management of Change (MOC) system that has key players sign off and approve changes to the SCADA system. This is especially important with major upgrades, hardware changes, or anything that risks having the system unavailable for an extended time, or changes that could have a detrimental impact to the system. The review process should be efficient and not so restrictive, time consuming or difficult to employ that people will want to circumvent the MOC process. The key players who sign off on the MOC requests should include your System Administrator.
7. Train your System Administrator(s).
   When you introduce new technology in your SCADA system be sure to train the System Administrator on how to manage the newly installed technology. If you don't train the Administrator, no one may be available to support the system and restore it to operation when it fails. Remember that the System Integrator you used to put the system in may be on the other side of the world when your system hiccups. Train the System Administrator on how to manage the newly installed technology!
8. Select products that support centralized and/or remote administration.
   This will allow your Administrator to cover more ground, faster. Remote administration may mean the ability to troubleshoot, diagnose, and remediate problems with out actually making a trip to the site where the system is located. Many SCADA systems are critical systems and operate on a 24/7 basis. It helps if your Administrator has the ability to try to resolve after hour problems without driving in to the site or office during the middle of the night.
9. Avoid Islands of Automation.
   Make it easy to administer by hooking the box or system to your SCADA network. If the box or system has remote diagnostics or programming make sure your System Administrator takes advantage of this capability.
10. Use the Appropriate Level of Technology.
    Over-designed systems may be much more costly to capitalize and much more costly and complex to administer and maintain. Many times the over-designed system was actually designed to be used in a much larger application. Often these systems are sold on the basis that they are the latest and greatest and are being used in much more critical systems than ours, so the logic is that we can't go wrong by selecting this bigger system.
11. Try to be Consistent in Selecting Technology.
    I once met an I&E Engineer who was responsible for Administration and maintenance on three different DCS vendors systems, a PC/PLC based SCADA system as well as a Workstation/PLC based SCADA System. All of these systems were large systems with thousands of points and hundreds of PID loops each. I think this guy had an almost impossible job trying to successfully manage all of this diversity in systems. He also informed me that his company was evaluating adding a new system based on a hybrid PC/DCS/SCADA/Controller system. If this guy can really handle all of this, he needs a big raise! Try to select technology that fits most of your applications and stick with that if it works well for you. It may be a little under-gunned on some of your apps, a little overkill on some, and hopefully, a lot of good fits on others.

Additional Resources