Clever Geek Handbook

Distributed Simulation Systems



To be able to combine individual simulators into a distributed simulation system, the following standards and technologies are currently used:



  • IEEE1516 (also replaces HLA and DIS)
  • OPC;
  • CAPE-OPEN and other "industry" standards.


Of greatest interest is the IEEE 1516 standard, since this standard is directly related to simulators, aimed at building distributed simulation systems (protocols, recommended control and feedback methods, system architecture, etc.).



The family of OPC (OLE for Process Control) software technologies that provide a single interface for managing automation objects and technological processes is also of considerable interest, but only if integration with automation objects and technological processes is required. The CAPE-OPEN standard is used for the interaction of simulators designed specifically for the chemical industry.



The Institute of Electrical and Electronic Engineers (IEEE) has made significant contributions to the standardization of modeling and simulation. Distributed modeling (simulation) is a technology for data exchange between simulators over local or global computer networks. This allows individual simulators to work together as one controlled modeling or simulation system. The distributed modeling concept is based on the use of a high-level architecture (HLA). In practice, the IEEE 1516 standard defines the architecture by using a single API (application programming interface). The starting postulates of the standard are:



  1. simple "monolithic" simulation models cannot meet the needs of professional users;
  2. all possible applications of simulation are unknown in advance;
  3. the possibility of arbitrary combination of individual simulators into complex simulation systems should be provided;
  4. The distributed modeling architecture should be as open as possible to future modeling and simulation technologies.


Currently, IEEE 1516 is the absolute standard for the interaction of simulators and simulators in military applications, due to stringent requirements for compatibility with simulators developed and used by the US Department of Defense and NATO. Currently, IEEE 1516 is increasingly used in the civilian sphere, in the development of simulators for training personnel of complex technical systems, in aviation, astronautics, transport, etc.



The OPC family of software technologies has been designed to reduce the cost of creating and maintaining industrial automation applications. In the early 90s, industrial software developers had a need for a universal tool for exchanging data with devices from different manufacturers or with different communication protocols. OPC provides industrial software developers with a universal fixed interface for data exchange with any device. At the same time, device developers provide a program that implements this interface.



To create complex simulation systems, you can combine the use of IEEE 1516 and OPC, getting the opportunity to use real equipment and SCADA systems (figure), which can be quite useful in many tasks.



Communication between the IEEE 1516 standards (which is the base for simulators) and OPC (used in SCADA systems) can be implemented both directly in the simulator and through an intermediary. The role of such an intermediary, for example, for me, is performed by the National Instruments LabView package. LabView can support mathematical models of any complexity, has built-in OPC support, can act as an OPC server, has effective support for interaction with various I / O cards, which allows you to use the necessary equipment directly, but, unfortunately, does not have means of interaction with IEEE 1516 , which requires writing the appropriate software components.



As a result of the use of IEEE 1516 and OPC, it is possible to create relatively complex distributed simulation systems, including many simulators, real equipment, SCADA systems, etc. The



issue of certification of a simulator or simulators with respect to supporting the IEEE 1516 standard deserves a separate consideration . federates in IEEE 1516 terminology) and software libraries that implement interaction. But the purpose of this certification is not to identify functional defects of the program (only certification of support for the IEEE 1516 standard).



Organizations capable of certification:



  • USA. The Department of Defense (DoD) Modeling and Simulation Coordination Office (M&S CO). Website: www.msco.mil
  • . ONERA. (Office National d’Etudes et Recherches Aérospatiales) is the French national aerospace research center. : www.onera.fr
  • . Pitch Technologies AB. : www.pitch.se














Let us consider the issues of building distributed simulation systems based on the IEEE 1516 standard. The basic terms used in information support correspond to the terminology of the IEEE 1516 distributed interactive simulation systems - these are federation, federation, object, attribute and interaction. The concept of an object is defined as a model of a separate phenomenon in the real world. Objects do not have methods, they only have states (only a data structure without functions for processing them). The state of objects is characterized by a fixed set of attributes - precise values ​​that can change over time. Each object at any time is characterized by its state, which is determined by a set of current values ​​of its attributes. Federates are mathematical descriptions of the behavior of objects - simulation models,software-defined (implemented in directive language) or represented by hardware sensor values. In fact, feds can be both imitators and real equipment or special software. The only requirement is to provide a uniform interface for communication. Federates can manipulate objects by changing (updating) or getting (displaying) the values ​​of their attributes. In particular, users of impersonators are also federates. The aggregate of all federates participating in the simulation forms a federation.The only requirement is to provide a uniform interface for communication. Federates can manipulate objects by changing (updating) or getting (displaying) the values ​​of their attributes. In particular, users of impersonators are also federates. The aggregate of all federates participating in the simulation forms a federation.The only requirement is to provide a uniform interface for communication. Federates can manipulate objects by changing (updating) or getting (displaying) the values ​​of their attributes. In particular, users of impersonators are also federates. The aggregate of all federates participating in the simulation forms a federation.



The term "interaction" is defined as an instant message (event), not tied to a specific instance of an object or federation, occurring at the federation level (ie, it is impossible to determine the sender). Interactions, in contrast to the states of objects, are not constantly maintained in the system, but are of an instantaneous nature. An example would be a one-way broadcast of a text message to all interested federation members. The hierarchical federation scheme (HLA / IEEE 1516) is shown in the figure



The interaction of federates is carried out using a general interaction mechanism (RTI), implemented as a subscription. A federate interested in obtaining certain attributes and interactions of another federation must subscribe to them through the RTI. The mechanism for requesting, providing and changing attribute values ​​is shown in the figure. The mechanism for organizing distributed simulation and collaboration is shown in the figure.





Picture. Hierarchical federation scheme



Objects in the simulator are, as a rule, 3D models, sound sources, respectively, the attributes of such objects are position and orientation in space, size, volume, etc. With regard to simulators, the actions of the user (federation), for example, the inclusion of a key, can be considered as interactions.





. (RTI)





. (RTI)





.



When creating distributed simulation systems that interact through RTI, it is necessary to take into account the following important features. All elements of federations and federations must be documented in certain files (FOM (federation object model) files are used to describe federation), federations are described in SOM files (Simulation Object Model). All data is stored only in federations, RTI does not store any data, but only transfers it. HLA allows only one federation to change the value of an attribute at any given time (there is a special rights management mechanism for transferring rights). Federates can manage local time, HLA uses various internal time management mechanisms (synchronization).



In general, the IEEE 1516 standard addresses a huge number of issues related to the creation of distributed simulation systems, such as maintaining the state of the federation, renewing the state, various time synchronization mechanisms, the areas of interaction of federates, etc. In connection with the significant volume of the standard itself and, moreover, the volume of program code for demonstrating all aspects described in the standard, only the fundamental implementation of the "basic" capabilities will be demonstrated below (figure).





Picture. IEEE 1516 Basic Capabilities Implementation Block Diagram



A more detailed presentation of the implementation is associated with the need to present a fairly large listing of the program, for this reason, the reader can independently use the examples of programs that are supplied with the software to support RTI. Simple examples with many comments are included in the The Portico Project library and are freely available at porticoproject.org . Almost all commercial implementations of the standard also contain many examples.



As an example, consider the following federation, which consists of two federates: a radio-controlled car and a control panel. Suppose that control is carried out by setting the speed of each of the 4 engines of the car and turning the front wheels. The machine is equipped with a sensor that determines the distance to the obstacle and transmits a signal to the control panel. For this it is necessary to define two object classes, cYpravlenie for the control panel and cDatchik for the distance sensor. The attributes of the cYpravlenie class are wheel1, wheel2, wheel3, wheel4, wheel_angle. The cDatchik class attribute is distance. The following is a federation description file, in HLA 1.3 format (interactions are given as an example). 



;;  —   (FED )   HLA 1.3

(Fed 
  (Federation Test) 
  (FedVersion v1.3) 
  (Federate "fed" "Public") 
  (Spaces 
	(Space "Geo" 
		(Dimension X) 
		(Dimension Y) 
	) 
  ) 

(Objects 
	(Class cYpravlenie 
		(Attribute wheel1 reliable timestamp) 
		(Attribute wheel2 reliable timestamp) 
		(Attribute wheel3 reliable timestamp) 
		(Attribute wheel4 reliable timestamp) 
		(Attribute wheel_angle reliable timestamp) 
	) 
	(Class cDatchik 
		(Attribute distance reliable timestamp) 
	) 
) 
	 
(Interactions 
	(Class reaction BEST_EFFORT RECEIVE 
	(Sec_Level "Public") 
		(Parameter dx) 
		(Parameter dy) 
		(Parameter dz) 
	) 
) 

)


Next, the simulator representing the control creates a federation and an object based on the cYpravlenie class. The simulator representing the car also creates a federation and an object based on the cDatchik class. The federates also subscribe to the changes they are interested in, i.e. The federated machine subscribes to receiving object data from the cYpravlenie class (i.e., the cYpravlenie class), and the federated control to the cDatchik class. Thus, the machine receives changes from the control panel, and the control panel receives data from the sensor in the car.



Building more sophisticated simulation systems requires some serious design. First, it is necessary to determine the principal composition of the federation in a first approximation, i.e. federates, federated objects and object attributes. When drawing up the federation scheme, it is necessary to take into account the hardware components of the distributed simulation systems, i.e. sensors and control hardware devices should also be represented in the form of federates, objects, and attributes. On the picture. shows the structure of the federation of the sucker rod pump installation simulator.





Picture. Federation structure



After the federation is composed, it is necessary to define links, that is, a reflection of which federates publish (that is, change) the attributes of objects, and which ones subscribe to changes of these attributes. As a rule, at the stage of defining links, a large number of "amendments" are established for the structure of the federation. After the required number of iterations of the "refinement" of the structure and relationships, the designers must establish the fact that the federation is "model correct". An example of defining links is shown in the figure (objects without links are hidden).





Picture. An example of the first stage of defining links



In the next stage, the required number of computers is determined and the corresponding distribution of federates. For example, federation "A" will function on computer "1", federates "B, C, D" will function on computer "2".





Picture. Distribution of federates by computers



As a rule, the distribution of federates is based on the economy of their mathematical model, if the mathematical models of federates do not require significant computing resources, then one computer can be used, if the mathematical models of federates require significant computing resources, it is necessary to determine the number of computers and the corresponding distribution of federations.



The next step is to create a federation description file (see example above) that reflects the approved "correct model" of the federation. You then create a software implementation of federates by writing the appropriate code to interact with the RTI and code to implement the mathematical model of the federation. At the final stage, it is necessary to test the distributed simulation system, identify errors, "overloads" of certain components in the system (based on statistics), correct synchronization, etc.



Statistics for each federation separately and for the federation as a whole shows the number and types of queries executed and allows you to identify possible problems during the operation of the system.



Example statistics for a federation: 



RTIA: Statistics (processed messages)
Joined federation as car_federate
Synchronization point announced: ReadyToRun
Achieved sync point: ReadyToRun, waiting for federation...
Federation Synchronized: ReadyToRun
Time Policy Enabled
Published and Subscribed
Add instance object: obj_datchik
Removing temporary file _RTIA_3033_ExampleFederation.fed on resign federation.
Resigned from Federation
Didn't destroy federation, federates still joined
List of federate initiated services 
--------------------------------------------------
1 Message::CLOSE_CONNEXION (MSG#1)
1 Message::DESTROY_FEDERATION_EXECUTION (MSG#3)
1 Message::JOIN_FEDERATION_EXECUTION (MSG#4)
1 Message::RESIGN_FEDERATION_EXECUTION (MSG#5)
1 Message::SYNCHRONIZATION_POINT_ACHIEVED (MSG#10)
1 Message::PUBLISH_OBJECT_CLASS (MSG#28)
1 Message::SUBSCRIBE_OBJECT_CLASS_ATTRIBUTES (MSG#32)
1 Message::SUBSCRIBE_INTERACTION_CLASS (MSG#34)
1 Message::REGISTER_OBJECT_INSTANCE (MSG#40)
1708 Message::UPDATE_ATTRIBUTE_VALUES (MSG#41)
855 Message::TIME_ADVANCE_REQUEST (MSG#91)
3 Message::GET_OBJECT_CLASS_HANDLE (MSG#112)
6 Message::GET_ATTRIBUTE_HANDLE (MSG#114)
1 Message::GET_INTERACTION_CLASS_HANDLE (MSG#116)
120516 Message::TICK_REQUEST (MSG#141)
2564 Message::TICK_REQUEST_NEXT (MSG#142)
List of RTI initiated services 
--------------------------------------------------
1 NetworkMessage::ANNOUNCE_SYNCHRONIZATION_POINT (MSG#13)
1 NetworkMessage::FEDERATION_SYNCHRONIZED (MSG#15)
1 NetworkMessage::DISCOVER_OBJECT (MSG#43)
1711 NetworkMessage::REFLECT_ATTRIBUTE_VALUES (MSG#45)
49 NetworkMessage::GET_FED_FILE (MSG#84)
Number of Federate messages : 125662
Number of RTIG messages : 1763
RTIA: Federate destroyed
TCP Socket 3 : total = 122015 Bytes sent 
TCP Socket 3 : total = 340249 Bytes received
UDP Socket 4 : total = 0 Bytes sent 
UDP Socket 4 : total = 0 Bytes received
	     :
CERTI RTIG up and running ... 
New federation: ExampleFederation 
Looking for FOM file... 
   Trying... open_test.fed --> cannot access. 
   Now trying.../usr/local/share/federations/open_test.fed... opened. 

 TCP Socket  7 : total =    340400 Bytes sent 
 TCP Socket  7 : total =    122015 Bytes received 
 UDP Socket  4 : total =         0 Bytes sent 
 UDP Socket  4 : total =         0 Bytes received 
 TCP Socket  6 : total =    258616 Bytes sent 
 TCP Socket  6 : total =    283044 Bytes received 
 UDP Socket  4 : total =         0 Bytes sent 
 UDP Socket  4 : total =         0 Bytes received


Time synchronization



As the practice of design and implementation of distributed simulation systems has shown, the most difficult issues are related to the control of the flow of time (time synchronization).



Typically, when you set how federated time is synchronized, two parameters are set, TimeRegulating and TimeConstrained. In practice, these modes affect the process of receiving messages from other federates and are directly related to the message ordering mechanism:

  • in order of receipt (messages are transmitted in the order in which they were received without time control);
  • priority (incoming messages are located in a priority queue, its timestamp is used to determine the priority of a message);
  • causal (ensures that messages are sent to the federates in an order consistent with the preceding and subsequent events represented by those messages);
  • by timestamps (when using this service, messages will be sent to federates in the order of their timestamps).


It is also worth noting that different federates can use different synchronization methods.



Software libraries for RTI implementation



A list of available HLA \ IEE1516 implementations is available at en.wikipedia.org/wiki/Run-Time_Infrastructure . Today, a fairly large number of implementations are available, both commercial and non-commercial. Most of the implementations are made in JAVA and C ++ (these are the languages ​​that are used in the standard), but there are also implementations for MatLab, Python (CERTI project), etc.



When choosing a library, special attention should be paid to "certification" to support IEEE 1516. As a rule , all commercial implementations have a "certificate", free ones do not (many of the free implementations are preparing for such certification).



Commercial Sales Table:





Non-commercial sales table:





I personally use the ONERA project CERTI (https://savannah.nongnu.org/projects/certi) to support the infrastructure of distributed applications.



Measurements of the speed of interaction of federates via RTI



Such tests are very important in the design of distributed simulation systems, especially if different federations are located on different computer networks, and even more important when interacting federates over the Internet.



To achieve the minimum time delays, you must choose the server with the lowest packet transit time delays (you can check using the ping command). As an example, let us consider the work of one of the distributed systems created at the NII EOR of TyumGNGU. A 100 megabit network is used (ping delays <0.231 ms), there is no time synchronization (to reduce delays inside RTI), 2 computers, the server (rtig) is running on one of the machines. Federation parameters - 2 objects contain 5 attributes (one object per federation / computer), interaction between federates is two-way. As a result of processing the measurements, the dependence of the number of interactions per second on the size of the transmitted data was obtained.



The results of such measurements allow us to draw many conclusions, for example, if the simulator should work in a given "real time", i.e. update, for example, at least 60 times per second, then for one interaction (for Fast Ethernet) no more than 300 kilobytes should be transmitted (if, for example, 5 attributes, then 60 kilobytes each). At the same time, 300 kilobytes of data transmitted 60 times per second also indicates the possibility of using RTI to transfer voice and video data between simulators, as well as to interact with VR devices.



When federated over the Internet, the minimum latency is largely determined by the packet transmission delays. For example, if the latency of a packet between the server and the simulator is 300 ms, then the maximum number of interactions per second will not exceed 3.



The use of faster solutions such as IpoIB (IP over InfiniBand, RFC 4392), 10G Ethernet, Myrinet-10G, etc. . will allow to increase throughput and significantly reduce latency (existing solutions allow producing 30 million interactions per second or more).



Interaction with real systems



Not only mathematical models of objects, that is, artificial systems, but also real systems and devices can act as federates. Examples include:



  • microphone providing audio data;
  • video camera providing video data;
  • various input / output devices such as joysticks (picture), printers, etc.
  • various sensors and control mechanisms connected to the computer via ADC / DAC boards;
  • real equipment and SCADA systems (Figure 2.10.1., Chapter 1.4.1.) via the industrial OPC interface;
  • Interfaces to the "desktop" of a real operating system running on a separate computer or virtual machine (figure);
  • VR devices (chapter 4.5.9.);
  • Database interface, etc.


Such possibilities are of considerable interest for simulators and simulation systems in general.


More articles:


All Articles