Optimized Internet Protocol Network for Scada Systems

A. What is O-IP?

The basics of an O-IP system are to allow the use of Internet Protocol (IP) over narrow band systems with all the benefits of a licensed RF path. The data rates will be in the 4800 to 19200 bps range with a higher effective throughput. The O-IP product must be able to manage the Ethernet and IP packets such that only a minimum required amount of overheard information is sent through the air. The final O-IP product will manage both the amount of packet overhead sent over the air on the RF link and will also apply data compression algorithms to reduce the amount of user data sent.

 

B. Why an Optimized Internet Protocol Device?

Why is there a need for Optimized Internet Protocol (O-IP) communications? The Supervisory Control and Data Acquisition (SCADA) industry is moving toward the Internet Protocol (IP) enabled network in a very determined manner. There are several reasons: the need for network manageability; the movement of manufacturers to IP based products, the general movement away from serial connections and the fact that many SCADA systems and automation groups have been moved into existing Networking control groups or Information Technology (IT) organizations.

Greater distance Radio Frequency (RF) paths are achieved with narrow band Frequency Modulated (FM) licensed products. Since the frequencies are licensed and regulated, power amplifiers and specialized RF filtering products can be used to give system reliable spans measured in tens of miles, not just miles. It is not atypical for a narrow band Ultra High Frequency (UHF) SCADA system to cover 50 or 75 miles of territory with no repeaters or single systems. Some Very High Frequency (VHF) based systems reach in excess of 90 miles as a routine design requirement. The fact that the frequencies are assigned by a governing agency (Federal Communications Commission) and coordinated by local frequency coordinators also give a certain level of certainty that interference will be less likely and there is some recourse should it occur. This is not necessarily a feature of typical wide-band unlicensed products. The FCC Part 15 devices (spread spectrum) are required to "co-exist" with any interference and it is not uncommon that a move to a licensed frequency alleviates interference problems.

The movement away from RS-232 serial communications methods poses challenges. There is a significant installed base of serial-based Integra communications systems working on narrow band (25 kHz and 12.5 kHz channels). These systems are typically slow to mid-speed (1200-19200 bits per second (bps)) applications. It was not too long ago 9600 or 19200 bps was considered very fast in the SCADA business! There is also a large installed base of serial based Integra spread spectrum products. In either case, the wholesale replacement in terms of cost, downtime and staff time is appreciable and they make alternatives worth looking at.

 

C. How Will O-IP Work?

A typical Ethernet message consists of a lot of overhead information to make sure the data arrive at their intended destination. However, if the design of the network is known, a certain amount of that header information can be limited, lowering the on-air traffic.

Typical Ethernet User Datagram Protocol (UDP) or Transmission Control Protocol /IP (TCP/IP) Overhead:

In many cases the overhead can exceed the actual SCADA message, i.e., a 54-byte header to send a 6-byte SCADA message. This would not be an acceptable or efficient method of SCADA communications.

Dataradio's mobile VIS (Vehicular Information System) optimized IP product has been in service for sometime now. It has been deployed in many locations with strong success. Taking lessons from that product development, Dataradio Engineering developed a SCADA Optimized IP solution that focuses on the particular needs of the SCADA user for IP connectivity.

The requirement for duplicate packets generated by TCP/IP are significantly reduced. Customized Data Compression algorithms afford up to a 50% compression rate for data, dependent on the data type. Header reduction is a fixed reduction of 25%.

This type of network intelligence is designed into a small microprocessor board that will be available as an add-on enclosure (Phase One) and an integral (Phase Two) with Dataradio products. There will not be a need for a separate personal computer or server in the system. Set up will be via personal computer and a table file structure and/or command line/HTTP based interface.

When there is high bandwidth/short distance available, a Media Access Control (MAC) layer bridge with little or no filtering may work well. Inefficiencies in data transmission are compensated for with the higher speed of such a link. However, if a similar approach is taken over a narrow band FM RF link, performance will not be sufficient to allow acceptable operation. This is where the Optimized IP connection methodology is best utilized, allowing a reasonable connection in these cases.

Remote Terminal Unit (RTU) Test Set-up:

Figures 1 and 2 are diagrams that outline two test set-ups that were used to verify and test the operation of the O-IP device. Test set-ups were based on user feedback as to the type of possible networks. Other connections are likely however these two test scenarios represent how we would expect the product to be put into service on an initial basis. Additional addressing data is provided to indicate the set-up format.

 

Figure 1: Test RTU Network Setup:

 

Figure 2: IP Native RTU and Terminal Server Network

 

 

D. What Are Some System Design Considerations of an O-IP System?

System design criteria requires some up-front work, especially since there are not unlimited speed and bandwidth allocations. SCADA system design is not foreign to SCADA users, however, with Local Area Network (LAN) systems a larger amount of the system "design" is left to the equipment and less than optimal designs can be compensated for by the high throughput enjoyed in LAN type systems. Some design criteria are listed below:

1. These SCADA O-IP systems will not support web surfing. Email systems such as Outlook and Lotus Notes will not be efficient because of the half-duplex nature of the radio channel and full-duplex nature of TCP. The overhead is simply too large and the system responsiveness would likely not be acceptable. A simple text based email system would work if not overused. Drive sharing and other common network components will not function well.
2. Efficient data throughput is based on SCADA oriented messaging size. Structures of the SCADA messaging need to be understood and perhaps adjusted to fit the application. Throughput is based on application architecture; i.e., half-duplex or full-duplex, number of devices supported and message size. This is in effect no different than what is currently done for serial based systems.
3. Rockwell Automation offers the following advice: "The recommended Ethernet/IP network topology for control applications is an active star topology (10 MBPS and 100 MBPS Ethernet can be mixed) in which groups of devices are point to point connected to a switch. The switch is the heart of the network system." O-IP is closer to a WAN environment, an Ethernet switch (star topology) is used for deterministic networks and deterministic response times while a WAN tends to be designed for more flexible approach to data movement. The O-IP environment allows for the chance of a data collision unless a polling-based application is used - this is a more typical SCADA application. In this type of optimized system, the routing and gateway capabilities of O-IP are utilized to better manage on-air RF traffic and maintain system reliability - we need to work smarter not merely faster.
4. Dynamic Host Configuration Protocol (DHCP) will not be supported in the initial offering. Design requirements should limit any application protocol based on IP broadcasting. We recommend using multicasting instead. There has not been a strong requirement indicated for this feature which can create significant overhead. The system has to be laid out with as much determinism as possible. If elements are changed, then the tables get changed. Typically SCADA systems have minimal change so change control can be implemented and table up-dates managed. Simply stated, SCADA systems are typically static address based.
5. The O-IP product will function as a gateway and router intelligently limiting the amount of traffic it forwards on to the RF network. As a comparison, MAC layer bridging would forward all broadcast messages generated on local LAN; i.e., IP broadcast, Internet Packet exchange (IPX) broadcast would forward Address Resolution Protocol (ARP) requests over the RF channel.
6. There is no limit on the number of Remote Terminal Units (RTU)/Programmable Logic Controllers (PLC) but network latency is dependent upon the number of RTU/PLCs on the network. Most serial systems require some kind of traffic calculation/review to determine how many sites can be polled and respond within a given time frame. Most network administrators and vendors have tools that assist in calculating the system latency, throughput and scan rates. Dataradio provides at least two types for general rule-of-thumb use. System designers may need to work with system programmers to understand data structures and required throughput rates for the application. This may also involve the process control/system engineers to understand what overall system performance criteria are. It has been the experience of Dataradio Technical Services that when these items are not addressed, system performance is not optimal either serial communications or LAN. There are networking tools available to assist in system performance evaluation and some allow for system performance extrapolation. Parameters such as tuning of TCP/IP parameters (Maximum Transmission Unit (MTU) size, MSS size etc.) will need to be set correctly. Dataradio will publish starting benchmarks for these parameters as work progresses with more systems and products.
7. How will the SCADA network be linked to any other corporate networks - through hubs or switches? How will the demands for non-SCADA information be handled? Tight control needs to be exercised or random data requests could easily impact the basic system performance. Requests addressed to RTUs/PLCs/Intelligent Electrical Devices (IED) will be passed on but if those requests come from a non-SCADA application (Engineering, Accounting, and Maintenance) the amount of traffic can impact system performance. Understanding how broadcast messages move through the system is important. O-IP will have the capability to enable or disable broadcast IP messages in the O-IP set-up. Limiting the number of broadcasts will keep traffic levels down as well.
8. System addressing needs to be thought out in advance to avoid duplicate addresses and use of illegal addresses. If the SCADA networks are kept isolated from other networks private IP addresses can be used for RTU/PLCs.
9. What types of devices will be on the network? RTUs, PLCs, IEDs, terminal servers, meters and other process control devices (virtually any device that uses IP as a network layer) can be used with O-IP. Each type of device has a communication profile that needs to be taken into account as far as messaging size, latency control, reply message size and ad-hoc messaging. Network dynamic control is a part of future Dataradio O-IP work.
   If the system is a class C network, up to 254 devices could be on the segment. But having a device count capability is not the same as having the throughput capability. If all the messaging is small and short, 254 devices could easily be supported. What it really gets down to is this: The more points there are to monitor, the longer it will take the system to poll them. Network latencies will impose longer scan times on data collection routines.
10. What protocols can be used with an O-IP system? Protocols such as UDP, TCP, Internet Control Message Protocol (ICMP), ARP, Modbus/IP (IP and a Modbus header), Modbus/TCP, ASCII over IP, Distributed Network Protocol (DNP) 3.0 are supported (timing constraint issues have come up with DNP 3.0 in any number of applications- not just O-IP. Review of the application and latencies is necessary.
    A.  Items that should be reviewed are:
    1. What is a typical data request size?
    2. What is the typical data reply payload size?
    3. What latencies are allowed by the PLC/IED/RTU?
    4. Will LAN system latencies work with RF system latencies? (The longest latency will govern the system performance).
    5. A review of timing requirements for the SCADA host program needs to include timing for message turn-around, message reply timer, total message timer, and other system timers.
    6. Does the design of the network and other network devices allow for longer latencies inherent in an RF system? Some devices internally buffer data to avoid latency time issues; others allow a longer latency.
       
11. Once network design issues are addressed, full system design can be completed and implementation can go forward. Progressive system testing should be performed so that issues can be addressed and resolved in smaller groups as opposed to turning the entire system on and then trying to "whittle down" issue areas.
12. Most end users tend to use a few protocols, devices and designs. Once this effort is done for the first system, a lot of the information will be able to be transferable for use in other systems. These elements are also part of any design effort for maximized system operation. These efforts are often the difference between a marginally operating and a truly efficient system.

 

E. Conclusion:

O-IP has a place in the RF market, especially supporting the narrow band FM sector. It represents a significant step forward allowing a greater connectivity option for those users who are distance constrained and want to use their legacy Integra installations. It also provides a migration path that will minimize the cost of conversion to a more manageable level.

Used in conjunction with the Integra wireless modem, the full feature set of the Integra system is available to the user. This includes online, offline and remote diagnostics, plus Dataradio infrastructure products, base stations, repeaters, rack mounting, power supplies, power amplifiers, antenna kits, National Electrical Manufacturers Association (NEMA) enclosures and High Availability (redundant bases and repeaters) options. The High Availability option allows for a "no single point of failure" system-back up capability for those critical links that need guaranteed uptime.

The product will be available initially as an add-on product, allowing for maximum up-grade flexibility. However, the end user will need to do some up-front work to take as full advantage of the capabilities. In many cases this information should (generally) be available as normal system design or maintenance information. The end user has the responsibility of managing the network for maximum performance, understanding that O-IP is not a panacea for all IP network needs but a targeted answer for certain needs.

 

Notes

1. All respective trade names trademark, copyrights, and service marks are property of their respective owners.
2. The use of a trade name or product name does not necessarily constitute an endorsement of that product, device, or software.

This article was written and provided by Harry Ebbeson, Manager of Technical Services at Dataradio COR Ltd. Dataradio is a leading designer and manufacturer of advanced wireless data products and systems for mission critical applications.

Source:-http://www.automation.com/library/articles-white-papers/hmi-and-scada-software-technologies/optimized-internet-protocol-network-for-scada-systems