Clever Geek Handbook

Linux in real time



A real-time operating system is needed when time requirements are imposed on the processor or data flow. Thus, it often plays the role of a control unit in special devices. Scientific experiments, medical imaging applications, industrial control devices are real-time systems. Fuel injection mechanisms of automobile engines, controllers for household and military equipment are also real-time systems.



Moreover, different events have different time requirements. For example, the latency requirement for an anti-lock braking system can range from 3-5 milliseconds. That is, from the moment the wheel first detects that it is slipping, the system that controls the anti-lock brakes has 3-5 milliseconds to react and correct the situation.



Real-time kernel capabilities have been around for over a decade in the open source ecosystem. Red Hat Enterprise Linux (RHEL) support for the real-time kernel has been available for the same amount of time. However, many system administrators misinterpret its basic concepts and actual operating behavior. In this article, I will describe some of its main features, differences from the standard kernel, and installation steps.



Real-time CPU scheduler



For different classes of problems, one can designate soft real-time systems and hard real-time systems . The former does not guarantee the exact time when the critical process will be scheduled in real time. They only guarantee that the process will be favored over non-critical processes. The latter have more stringent requirements and the task is either completed within the specified time frame, or is considered not completed.



We call event delay the time that elapses from the moment the event occurs until the moment it is served. There are two types of delays that affect the performance of the real-time OS.



  • CPU . , . , interrupt service routine (ISR).





    . 1 .
  • , , . , . .





    . 2 .






The most important feature of a real-time operating system is to immediately respond to a critical process that requires access to CPU resources. As a result, the scheduler for the real-time operating system must support the preemptive interrupt algorithm. These algorithms assign priority to each process based on its degree of importance. If the scheduler also supports preemption, the current process on the CPU will be preempted on demand in favor of a higher priority process.





Figure: 3 Classification of planners.



There are several algorithms for the real-time scheduler.



  • Rate-Monotonic Scheduling โ€” . , . .





    n, ln2 โ‰ˆ 0.693147.
  • Earliest-deadline-first (EDF) Scheduling . , , . RMS, EDF , . , , .





    . 4 EDF.



    . 4 T1 T2 , T2. T3 T1, 23.
  • POSIX real-time-scheduling. POSIX.4 . , .



    • SCHED_FIFO โ€” , ยซ โ€” ยป (FIFO). 32 .
    • SCHED_RR โ€” SCHED_FIFO, ( ) . 32 .
    • SCHED_OTHER โ€” ; - .




Installing and using RHEL Real Time



First, you need to connect the Red Hat Enterprise Linux Real Time repository and install the RT package group. 



[root@server ~]# subscription-manager repos --enable rhel-8-for-x86_64-rt-rpms
[root@server ~]# yum groupinstall RT


RT includes these components:



  • kernel-rt - kernel with real-time functionality;
  • rt-setup - installation of the Red Hat Enterprise Linux Real Time environment;
  • rt-tests - RT function testing utilities;
  • rt-eval - to evaluate the possibility of using RT on a given system;


After installing RT and rebooting, make sure the kernel-rt is loaded. 



[root@server ~]# uname -a
Linux rt-server.example.com 4.18.0-80.rt9.138.el8.x86_64 โ€ฆ


Let's take a look at some of the differences between kernel-rt and the standard kernel.



  • At high load, the task priority is checked (1-99).
  • High priority (99) tasks are given preference when accessing CPU resources.
  • Does not enforce the Completely Fair Scheduling (CFS) policy .
  • Uses the SCHED_FIFO or SCHED_RR policy .




Figure: 5 Comparing kernet_rt to the standard kernel.





The graph shows a sample of the million-repetition response time for systems using the RHEL Linux 7 and RHEL Real Time kernels, respectively. The blue dots in this graph represent the response times (in microseconds) of systems with the standard RHEL 7 kernel, and the green dots represent the RHEL 7 Real Time. The graph shows that the feature of kernel-rt is much lower variance and, accordingly, more predictability of the response time of the system.



Setting up and testing



After RT is installed, additional tuning and tweaking may be required to achieve the most consistent system response times. Such requirements may be presented by companies in the financial or telecommunications sector. The setup itself is an iterative process and you need to be patient at the beginning of the process. It is unlikely that it will be possible to tweak a couple of variables and understand that the best possible result has been achieved.



The hwlatdetect utility from the rt-tests package will show the latency caused by the hardware and firmware by polling the clock source and looking for obscure gaps. 



[root@server ~]#  hwlatdetect --duration=60s
hwlatdetect:  test duration 60 seconds
	detector: tracer
	parameters:
		Latency threshold: 10us
		Sample window:     1000000us
		Sample width:      500000us
		Non-sampling period:  500000us
		Output File:       None

Starting test
test finished
Max Latency: Below threshold
Samples recorded: 0
Samples exceeding threshold: 0


In this example, parameters indicates the delay and detection method. The default latency threshold was set to 10 microseconds (10 ฮผs).



RT also has a utility called rteval for testing real-time system performance under load. The program places a heavy load on the system using the SCHED_OTHER scheduler and then measures the real-time response on each of the active CPUs. The goal is to keep various tasks running continuously, such as allocating / freeing memory, disk I / O, computation, copying memory, and others.



Each measurement thread takes a timestamp, is idle for a certain interval, and then takes another timestamp upon waking up. The measurement delay is equal to t1 - (t0 + i), where



  • t1 - actual measurement time;
  • t0 - theoretical wake-up time of the first timestamp;
  • i is the waiting interval.


The rteval utility report looks like this. 



System:
Statistics:
	Samples:           1440463955
	Mean:              4.40624790712us
	Median:            0.0us
	Mode:              4us
	Range:             54us
	Min:               2us
	Max:               56us
	Mean Absolute Dev: 1.0776661507us
	Std.dev:           1.81821060672us

CPU core 0       Priority: 95
Statistics:
	Samples:           36011847
	Mean:              5.46434910711us
	Median:            4us
	Mode:              4us
	Range:             38us
	Min:               2us
	Max:               40us
	Mean Absolute Dev: 2.13785341159us
	Std.dev:           3.50155558554us


Used materials











More articles:


All Articles