Nvidia DeepStream is a well-known tool for inference on neural networks and other high-performance processing of video streams in real time on Nvidia equipment. The most important components that ensure the effectiveness of the DeepStream application are:
built-in NVDEC hardware video decoding modules from different formats (H264, HEVC, AV1, MPEG-1, MPEG-2, VP8, VP9);
built-in NVENC video encoding modules in different formats (H264, HEVC, AV1, MPEG-1, MPEG-2, VP8, VP9).
Again, it is widely known in narrow circles of the tablet allows us to estimate that you can encode or decode at various GPU from Nvidia.
It is NVDEC and NVENC that provide an increase in the efficiency of inference due to the fact that, after decoding the compressed data using NVDEC, the raw video (in the optimal processing pipeline) does not leave the GPU memory until the end of processing, from where, if required, NVENC, on the contrary, takes the raw data and transmits they are already compressed (encoded) into the CPU memory.
The DeepStream pipeline itself runs on the Gstreamer engine with additional plugins from Nvidia that provide all sorts of tricky operations (for example, dewarp video from fisheye into several flat pictures) and interact with TensorRT through inference plugins. There are also plugins for tracking and other operations. A complete list is available at this link .
As a result, using DeepStream, you can get very high performance, incomparable with OpenCV / PyTorch, TensorFlow, etc. without plunging into the jungle of CUDA development.
DeepStream. , . , K8s keepalive , . , , .
, , . -, DeepStream NVDEC H264.
, DeepStream . -, , . - . DeepStream , , .
GPU Quadro RTX 4000, Geforce RTX 2080:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:02:00.0 Off | N/A |
| 30% 34C P5 17W / 215W | 434MiB / 7982MiB | 15% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
Gstreamer:
gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink Setting pipeline to PAUSED ... Pipeline is PREROLLING ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock
, . Turing ( Ampere , ).
CPU ( Gstreamer Intel VAAPI) :
gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! avdec_h264 ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.220984700
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Debug, , :
GST_DEBUG=5 gst-launch-1.0 filesrc location=/big-data/video/output.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! fakesink ... 0:00:00.614602755 19768 0x557836592cf0 DEBUG GST_MEMORY gstmemory.c:139:gst_memory_init: new memory 0x5578365c8470, maxsize:71 offset:0 size:64 0:00:00.614610302 19768 0x557836592cf0 DEBUG GST_PERFORMANCE gstallocator.c:465:_sysmem_copy: memcpy 64 memory 0x5578365c8390 -> 0x5578365c8470 0:00:00.614618503 19768 0x557836592cf0 DEBUG GST_BUFFER gstbuffer.c:2202:gst_buffer_add_meta: alloc metadata 0x7f0d6000daf8 (GstVideoMeta) of size 112 0:00:00.614625121 19768 0x557836592cf0 DEBUG videometa gstvideometa.c:86:gst_video_meta_transform: copy video metadata 0:00:00.614631365 19768 0x557836592cf0 DEBUG GST_MEMORY gstmemory.c:88:_gst_memory_free: free memory 0x5578365c8390 0:00:00.614640158 19768 0x557836592cf0 DEBUG GST_PADS gstpad.c:4072:gst_pad_query:<fakesink0:sink> sent query 0x7f0d6054f590 (allocation), result 0 0:00:00.614646925 19768 0x557836592cf0 DEBUG GST_PADS gstpad.c:4117:gst_pad_query:<fakesink0:sink> query failed 0:00:00.614653488 19768 0x557836592cf0 DEBUG GST_PADS gstpad.c:4238:gst_pad_peer_query:<nvv4l2decoder0:src> query failed 0:00:00.614660986 19768 0x557836592cf0 DEBUG v4l2 gstv4l2object.c:4389:gst_v4l2_object_stop:<nvv4l2decoder0:src> stopping
, Nvidia T4 :
BCM:: counter - 72 BCM:: counter - 73 BCM:: counter - 74 BCM:: counter - 75 BCM:: counter - 76 Redistribute latency... Redistribute latency... Redistribute latency... Redistribute latency... ERROR: from element /GstPipeline:pipeline0/GstQTDemux:qtdemux0: Internal data stream error. Additional debug info: qtdemux.c(6073): gst_qtdemux_loop (): /GstPipeline:pipeline0/GstQTDemux:qtdemux0: streaming stopped, reason not-negotiated (-4) Execution ended after 0:00:01.682024126 Setting pipeline to PAUSED ... Setting pipeline to READY ... Setting pipeline to NULL ... Freeing pipeline ...
, 76 , CPU , NVDEC .
In fact, a bug was opened on the Nvidia forum . So far, there is no practical advice from Nvidia. It is waiting for either a fix or at least instructions on how to make an "antivirus" by placing it in front of NVDEC in order to detect and discard malicious frames. As our partner's architect said: "The main thing is that the error is not in the chip."