(Almost) useless webcam streaming from the browser. Part 2. WebRTC

Once in one of the old and already abandoned articles, I wrote about how easily and naturally you can broadcast video from canvas via websockets. In that article, I briefly talked about how to capture video from a camera and sound from a microphone using the MediaStream API , how to encode the received stream and send it via websockets to the server. However, in reality, they do not do this, for broadcasts they use either special software that needs to be installed and configured: offhand it can be Open Broadcast Software , or they use WebRTC, which works right out of the box, that is, it does not require the installation of any plugins like a flash player, which already in December will be cut from the Chromium browser.

Today we'll talk about WebRTC.

Web Real-Time Communication (WebRTC) is not one protocol, it is a whole collection of standards, protocols and JavaScript APIs, which together provide peer-to-peer video-audio communication in real time, and can also be used to transfer any binary data ... Usually browsers act as peers, but it can also be a mobile application, for example. In order to organize p2p communication between clients, the browser needs support for various types of video and audio encoding, support for many network protocols, ensuring the interaction of hardware with the browser (through the OS layers): webcams, sound cards. This whole jumble of technologies is hidden behind a JavaScript API abstraction for the convenience of the developer.

It all comes down to three APIs:

• MediaStream API - we analyzed it last time, today I'll write a little more about it. Serves to receive video / audio streams from hardware

• RTCPeerConnection - provides communication between two clients (p2p)

• RTCDataChannel - serves to transfer arbitrary data between two clients

Preparing audio and video streams for transmission

"" . , : , , , . , -. , ( , ), . . 1:

, . . 2020 . , MediaStream API, . IE .

: , , , "" Media Stream <video> html. canvas , WebGL CSS3, , canvas , ( bigo live, twitch ). , , :

https://jeeliz.com/ - realtime CV Javascript. js- canvas: , , (, ) . , .

Canvas captureStream API - API canvas. Chrome, Opera Firefox

RTCPeerConnection

, ? RTCPeerConnection. , RTCPeerConnection:

const peerConnection = new RTCPeerConnection({
  iceServers: [{
    urls: 'stun:stun.l.google.com:19302'
  }]
});

iceServers - , , NAT'. : ip , NAT ? ICE , , ICE WebRTC, .

Usermedia :

navigator.mediaDevices.getUserMedia({ video: true, audio: true }).then(stream => {
  // Usermedia-,      
  const tracks = stream.getTracks();

   for (const track of tracks) {
     //     peerConnection
     peerConnection.addTrack(track);
   }
}).catch(console.error);

peerConnection onnegotiationneeded, offer ( SDP - Session Description Protocol) peerConnection setLocalDescription. SDP - offer answer - .

LocalDescription peerConnection, "" ice-, NAT. onicegatheringstatechange. onicegatheringstatechange webrtc-signaling- stream Session Description :

peerConnection.oniceconnectionstatechange = (event) => {
      console.log('Connection state: ', peerConnection.iceConnectionState);

      if (peerConnection.iceConnectionState === 'connected') {
        //    Start broadcast
        setBroadcasting(true);
        setBroadcastingBtnActive(true);
      }
    };

//   ,      peerConnection
peerConnection.onnegotiationneeded = (event) => {
      //    SDP offer
      peerConnection.createOffer().
        then((offer) => peerConnection.setLocalDescription(offer)).
        catch(console.error);
    };

//    ,   ICE 
peerConnection.onicegatheringstatechange = (ev) => {
      let connection = ev.target;

      // Now we can activate broadcast button
      if (connection.iceGatheringState === 'complete') {
        let delay = 50;
        let tries = 0;
        let maxTries = 3;

        let timerId = setTimeout(function allowStreaming() {
          if (isOnline) {
            setBroadcastingBtnActive(true);
            return;
          }

          if (tries < maxTries) {
            tries += 1;
            delay *= 2;
            timerId = setTimeout(allowStreaming, delay);
          } else {
            // TODO: show user notification
            console.error("Can't connect to server");

            alert("Can't connect to server");
          }
        }, delay);
      }
    };

webrtc-signaling- - , session description , websocket xhr- . : session description .

Session descriptions , , ontrack peerConnection, , <video> . .

:

https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection - RTCPeerConnection

https://github.com/pion/webrtc - WebRTC go

https://webrtcforthecurious.com/ - pion

https://hpbn.co/ - High Perfomance Browser Networking. web-. WebRTC. (2013), .

pion, HLS ffmpeg .

: react pion twitch ( ).