How to organize Ethernet connections in industrial areas?
The first and seemingly obvious option that comes to mind is to install Ethernet switches, connect them, if possible, with optics, and stretch a twisted pair cable from them to the connected devices.
This is where the first difficulty and the first difference between industrial networks and office ones arises: most of the switches are installed not in server rooms, but in assembly cabinets scattered throughout the workshops or on the territory of the facility. It is expensive and difficult to pull two optical lines from each enclosure along different paths, which makes it more difficult to organize the familiar and reliable star topology. I have to use rings. Anyone who has immersed themselves in building Ethernet networks remembers that switch rings are evil. In the case of Spanning-Tree, this is indeed the case. This protocol in a ring topology can run for up to 30 seconds, which is often unacceptable for networks serving industrial processes. Worse, the rate of STP convergence decreases as the number of hops from the root switch to the outermost switches increases, and as a rule,it is recommended not to exceed a distance of 7 hops. This means that there should be no more than 14 switches in the ring.
What can you do about it? In the case of Cisco switches, the simplest solution is to use the Resilient Ethernet Protocol - REP instead of STP and its modification REP Fast. This protocol is specially designed for ring topologies and provides network convergence in a maximum of 50-100ms for all types of faults and ring sizes up to 50 switches. Moreover, 50 switches in a ring is not the limit for such a protocol. The convergence time with the growth of the ring will certainly increase, but the same Spanning-Tree on rings of this size will never converge at all. REP is supported not only in industrial switches, but also in office ones, in particular in the Catalyst 9000 series, which can serve as aggregation switches.
The protocol is very easy to set up, here's an example:
For more complex cases, the PRP and HSR protocols are available. They assume full duplication of traffic along two paths in the network. If one of the paths fails, there is no loss in data transmission at all. However, the cost of implementing such stability is higher - the protocol is supported only in older models and only in industrial Ethernet switches (IE3400, IE4000, IE4010, IE5000). However, the requirements for the reliability and convergence of the industrial network, as a rule, are rigidly determined by the nature of the production process that this network will serve. One downtime, even 50 milliseconds, can sometimes cost more than good network equipment.
How to ensure the required operational reliability?
As a rule, the requirements for reliability and fault tolerance for the industrial network segment are higher than for the office one. Industrial networking equipment is made and tested to meet these requirements. In the case of Cisco switches, industrial design means:
• mounting in compact cabinets on a DIN rail, DC powered;
• protection of microcircuits and ports from electrostatic discharges up to 4000 kV;
• no fans - highly efficient convection cooling despite the small size of the units;
• the ability to withstand power surges in accordance with the requirements of the certifications IEC 61000-4-11, IEC 61850 - Cisco switches continue to operate in the event of a power outage for a period of up to 50 ms, and send a “Dying Gasp” signal when disconnected;
• high-precision internal clock;
• the ability to quickly replace a faulty switch by simply moving the SD card to a new one (the new switch will rise not only with the same config as the old one, but also with the same IOS image);
• fast loading (within 80 seconds);
• rigorous testing for compliance with industry certifications.
Figure 1. Simulation of the switch cooling process by convection
Figure 2. Testing the switch for immunity to electromagnetic influences
Figure 3. Tests with exposure to water on a switch with IP67 protection level
It often happens that industrial switches are connected to ordinary office switches, as to aggregation switches, and if the latter fails, the industrial segment stops working. To prevent this, it is necessary to ensure the aggregation of industrial switches located in workshops or on the territory of an industrial enterprise on special industrial aggregation switches. Cisco offers the Cisco IE5000 series switches for this purpose:
They are mounted in a 19-inch cabinet, can be powered by both DC and AC power, provide high port density, high performance, but at the same time industrial reliability and support for industrial protocols such as PTP, PRP, HSR, PROFINET MRP. Like other switches in the industrial line, Cisco IE5000 is able to receive alarms from automation devices, for example, climate control devices or a door opening sensor, and send them - for example, to turn on a siren and light notification, in addition, of course, SNMP and Syslog- switch status messages sent to monitoring systems. In addition, these switches support stacking over 10G ports. Here is an example of building a network using REP-rings and IE5000 switches as aggregators:
Industrial protocol support
Industrial Ethernet networks use Ethernet versions of various industrial protocols: PROFINET, CCLINK, CIP, etc. At the same time, as a rule, network equipment is required to support such protocols in one form or another. For example, when using PROFINET, it is required to control using this protocol not only controllers, sensors or actuators, but also the switches themselves that form the network. For this, in the models of Cisco industrial switches starting from IE3000, support for working over PROFINET as an input-output device is implemented. In addition, some Cisco switch models can be managed using the Siemens TIA portal.
Another example of an industry standard that is often required to support is Time-Sensitive Networking (TSN). It is a set of Ethernet standards that allows Ethernet frames to be delivered with predictable, time-constant latency. Familiar Ethernet, let me remind you, works asynchronously and frames in it reach the addressee "as quickly as possible." TSN protocol functionality is supported in Cisco IE3400, IE4000, IE4010 and IE5000 switches.
How to protect an industrial network?
Many standards and recommendations for building industrial networks provide for the implementation of a DMZ demilitarized zone between office and industrial networks. In this area, workstations can be located for remote access to industrial components. There are a number of recommendations for building such a DMZ, for example:
• traffic should not pass through between the office and industrial networks;
• any protocol allowed between the DMZ and the industrial segment must be explicitly prohibited between the DMZ and the office segment;
• the industrial segment of the network should not have access from the Internet, even through a firewall;
• And of course no "Any" in the fields for IP addresses, ports and protocols in firewall policies.
Ideally, the DMZ should be arranged so that if it is physically disconnected from the network, the industrial segment will continue to work.
The industrial, office and DMZ segments are separated from each other by firewalls. Here Cisco has a clear advantage - you can build a secure network inside and out on its products.
Cisco firewalls are able to recognize not only industrial network protocols:
but also industrial devices connected to the network:
And also detect, allow or prohibit specific actions within these protocols:
This allows you to build firewall policies not only based on IP addresses and TCP / UDP ports, but also on the basis of the names of specific device models and communication protocols between them. For most situations, the firewall has predefined rules that you can use with your own parameters. Such rules can protect not only from deliberate attacks, but also "from the fool" - commands mistakenly sent to industrial devices.
Firewalls provide the so-called macro-segmentation of the network - division into large sections, traffic between which is either prohibited or filtered through firewall rules. Thus, not only DMZ, industrial and office network segments are separated from each other, but also, for example, different workshops and production lines in the industrial segment. For the latter task, the Cisco Industrial Security Appliance (ISA) firewall can be useful - a full-fledged industrial firewall.
How to properly provide remote access?
As a rule, in industrial networks, it is required to provide remote access for organizations servicing ICS equipment. At the same time, it is important to control well who and where such access is provided and to protect against unauthorized access. Remote access is carried out through the DMZ described above.
Here, in addition to Cisco Firepower firewalls, the Cisco Identity Services Engine can be of great help. Firewalls provide connection using AnyConnect VPN or proxying remote desktop traffic, and ISE allows you to clearly identify both the person gaining access and the object on the network to which this access is provided, as well as define policies for such access in the form of a kind of matrix:
In addition, Cisco Firewall Intrusion Prevention System can detect and block industrial protocol attacks. We are talking here not only about a situation when a manufacturer's specialist carries out some malicious actions in the customer's network, but also about the fact that the manufacturer's specialist's computer and the remote access server that he uses may be infected with a virus that tries to open unauthorized access to third parties. - intruders.
How to comply with regulatory requirements?
International standards and Cisco Validated Design documents only provide "what is best" recommendations. But besides the recommendations, there are also the requirements of regulatory bodies that must be fulfilled when building industrial networks. In Russia, these include Order No. 239 of the FSTEC "On Approval of Requirements for Ensuring the Security of Significant Objects of Critical Information Infrastructure of the Russian Federation" It contains a list of architectural solutions for the functionality that must be implemented.
Some of the requirements of the order, such as the presence of a firewall and an updated IPS on the perimeter of the industrial segment, network segmentation, the organization of DMZ, are closed by the Cisco Firepower firewalls described above. Authentication and Authorization Requirements - Cisco ISE. Further, a whole set of requirements related to monitoring the industrial network is covered by the Cisco CyberVision solution.
The Cisco CyberVision solution can collect data from the industrial network in the form of SPAN traffic or from special sensors in network devices and present a picture of what is happening to the administrator, as well as send the necessary information to configuration management systems and SIEM. The network administrator can get a complete list of devices connected to the industrial network segment (not only Ethernet switches, but also industrial automation devices), check them for known vulnerabilities and track their behavior.
Figure 4. An example of displaying known vulnerabilities for industrial devices
We will discuss the topic of wired industrial networks in more detail at the webinar on June 22, if you want to sign up, you can register here