🔲 🏪 🙏🏽 About first graders, distance learning and asynchronous programming 👩‍🚀 🤭 👹

I'm a downshifter. It so happened that for the past three years my wife and I have been enjoying the rural landscape outside the window, fresh air and birdsong. Conveniences in the house, optical internet from a local provider, a powerful uninterruptible power supply, and a covid that suddenly appeared, made the idea of moving from a metropolis not so strange.

While I was enthusiastically engaged in web development, somewhere in the background, my wife periodically complained about the problems of choosing a school for my child. And then (suddenly) the child grew up and the school question stood upright. Okay, so the time has come. Let's figure it out together, what is wrong with the education system in the former 1/6 of the land, and what can we do about it?

I will leave the traditional face-to-face teaching methods outside the scope of this article. I will only say that ordinary schools have both indisputable advantages and serious disadvantages, to which, by the way, forced self-isolation has recently been added. Here we will look at the options for distance and family education, which, for a variety of reasons, have recently attracted more and more parents.

To be clear: distance learning implies classes in a regular school using "distance learning technologies" (DOT), and family education means voluntary leaving school and learning only by the family (in fact, this is a good old external study). However, in any case, the child must be assigned to any of the available schools, at least for passing intermediate attestations.

And now some observations from life. With the forced transfer to distance learning of children who have already studied in a regular school, everything is sad. Schoolchildren perceive this gift of fate as a kind of vacation, parents are not used to following discipline during classes, and as a result, overall academic performance inevitably falls.

With first-graders, especially in the case of the family form, the parents, perhaps, have a chance to put the child “on the rails” using natural interest and the effect of novelty. For me personally, achieving independence is the main task. Sitting and doing homework with a child, I consider the ~~height of stupidity~~not entirely reasonable. Of course, if you want your children to achieve something in life and not hang around your neck. I want, therefore, my goal is to teach a child to learn, ask questions correctly and, in general, think with his own head.

Get to the point. Choosing a public school

Perhaps I like family education more because of the opportunity to choose a program and a training schedule. And you can physically attend school less often. But you need to choose a public school, talk to the director about the child's placement and receive an order for admission to the first grade at the end of winter so that there are no surprises in September. Although, from a legal point of view, the law on education does not seem to require annual attestations, “deadlines”, in my experience, are very motivating, so let there be attestations. It is unlikely that any school will accept us with open arms, but I am sure we will be able to find a worthy option in the nearest city.

Choosing a curriculum

We choose exactly. Trying to compose a program on your own without having a specialized education is not reasonable. Although there are government educational resources such as the Russian Electronic School ( NES ) and the Moscow Electronic School ( MES ), which in theory would be sufficient, but ... Both options provide lesson plans, videos, tests and tutorials. What I could not find was the textbooks themselves, even for the compulsory curriculum.

And here the most important thing is missing: communication. Teaching a child by showing him endless videos and forcing him to tick off tests will not work. This means that you either need to conduct lessons completely independently, or choose one of the online schools.

Choosing an online school

We are almost back to where we started. Remote control? Okay, let's take a closer look at her. How can you organize the educational process remotely? This raises many questions, I will only raise the key ones:

* Live communication. What do schools offer? Skype, Tims at best. Skype lessons? Really? If I'm not mistaken, it's 2020. Open several windows with beautiful multi-colored buttons in front of the first grader and wait that he will not press them, but will obediently listen to a boring uncle or aunt for half a day? I have never seen such children. And you?

* Homework. More precisely, how does it get to the teacher for testing? In fact, this is a really difficult question, perhaps not even solved in principle. Existing options:

, . --, , , , - .
. , - .
. , .
. , , , , ? . , , .
. , , . , , , . , . . , , , , . .

* Estimates. Obviously, the grades given in the lesson and when checking homework should fall into an electronic diary available to parents. And they get there. But not right away. I asked the older children who graduated from one of the prestigious lyceums as a gold-domed (ironically, with an informational bias), why is this? The answer, to be honest, surprised me. It turns out that teachers write down grades on a piece of paper, and after lessons they drive them into this very electronic diary on the state portal. And this while Tesla Elon Musk plow the vastness of space ...

Okay, it's time to do a little technical research and check if there are objective reasons for this state of affairs?

Let's define the requirements for a hypothetical ideal learning platform. In fact, everything is simple: children should stay in the lesson, focusing on what the teacher says and showing, answering questions if necessary and raising their hand if desired. Basically, we want a full screen window with a stream from the teacher's camera, presentation, or whiteboard. The easiest way to achieve this is to use WebRTC technology (real-time communications, real-time communications). This thing works in any more or less modern browser, does not require the purchase of additional equipment and, moreover, provides a good quality connection. And yes, this standard requires asynchronous programming at least because the required JS method navigator.mediaDevices.getUserMedia () returns a promise . Everything seems to be clear, I'm starting to implement it.

Lyrical digression about choosing a framework

, «» JavaScript , . jQuery. , JS :

//  
element = $(selector);
element = document.querySelector(selector);

//    
element2 = element.find(selector2);
element2 = element.querySelector(selector2);

//  
element.hide();  //   display: none
element.classList.add('hidden');

, CSS «hidden», , opacity transition, fadeIn/fadeOut CSS. , JS !

//   onClick
element.click(e => { ... });
element.onclick = (e) => { ...  }

//  
element.toggleClass(class_name);
element.classList.toggle(class_name);

//  div
div = $("<div>");
div = document.createElement("div");

//   div  element
// (  ,   )
element.append(div);
element.append(div);

. .. , JS , . , , «» JS !

WebRTC is designed for direct communication between browsers, using point-to-point (p2p) technology. However, in order to establish this connection, browsers must inform each other of their intent to communicate. This requires an alarm server .

An example of a basic implementation of a simple video chat using the "full mesh" topology

'use strict';

(function () {
    const selfView = document.querySelector('#self-view'),
        remoteMaster = document.querySelector('#remote-master'),
        remoteSlaves = document.querySelector('#remote-slaves');

    let localStream,
        selfStream = null,
        socket = null,
        selfId = null,
        connections = {};

    // ***********************
    // UserMedia & DOM methods
    // ***********************

    const init = async () => {
        try {
            let stream = await navigator.mediaDevices.getUserMedia({
                audio: true, video: {
                    width: { max: 640 }, height: { max: 480 }
                }
            });
            localStream = stream;

            selfStream = new MediaStream();

            stream.getVideoTracks().forEach(track => {
                selfStream.addTrack(track, stream); // track.kind == 'video'
            });
            selfView.querySelector('video').srcObject = selfStream;

        } catch (e) {
            document.querySelector('#self-view').innerHTML =
                '<i>     </i>';
            console.error('Local stream not found: ', e);
        }
        wsInit();
    }

    const createRemoteView = (id, username) => {
        let iDiv = document.querySelector('#pc' + id);
        if (!iDiv) {
            iDiv = document.createElement('div');
            iDiv.className = 'remote-view';
            iDiv.id = 'pc' + id;

            let iVideo = document.createElement('video');
            iVideo.setAttribute('autoplay', 'true');
            iVideo.setAttribute('playsinline', 'true');

            let iLabel = document.createElement('span');

            iDiv.append(iVideo);
            iDiv.append(iLabel);

            if (!remoteMaster.querySelector('video')) {
                remoteMaster.append(iDiv);
                iLabel.textContent = '';
            } else {
                remoteSlaves.append(iDiv);
                iLabel.textContent = username;
            }
            remoteMaster.style.removeProperty('display');
        }
    }

    // *******************************
    // Signaling (Web Socket) methods
    // *******************************

    const wsInit = () => {
        socket = new WebSocket(SIGNALING_SERVER_URL);

        socket.onopen = function (e) {
            log('[socket open]  ');
        }

        socket.onmessage = function (event) {
            log('[socket message]    ', event);

            wsHandle(event.data);
        }

        socket.onclose = function (event) {
            if (event.wasClean) {
                log('[close]   , ' +
                    `=${event.code} =${event.reason}`);
            } else {
                log('[socket close]  ', event);
            }
            clearInterval(socket.timer);
        }

        socket.onerror = function (error) {
            logError('[socket error]', error);
        }

        socket.timer = setInterval(() => {
            socket.send('heartbeat');
        }, 10000);
    }

    const wsHandle = async (data) => {
        if (!data) {
            return;
        }
        try {
            data = JSON.parse(data);
        } catch (e) {
            return;
        }

        switch (data.type) {
            case 'handshake':
                selfId = data.uid;
                if (!Object.keys(data.users).length) {
                    createRemoteView(selfId, '');
                    remoteMaster.querySelector('video').srcObject =
                        selfStream;
                    selfView.remove();
                    break;
                } else {
                    selfView.style.removeProperty('display');
                }
                for (let id in data.users) {
                    await pcCreate(id, data.users[id]);
                }
                break;
            case 'offer':
                await wsHandleOffer(data);
                break;
            case 'answer':
                await wsHandleAnswer(data)
                break;
            case 'candidate':
                await wsHandleICECandidate(data);
                break;
            default:
                break;
        }
    }

    const wsHandleOffer = async (data) => {
        let pc = null;

        if (!connections[data.src]) {
            await pcCreate(data.src, data.username);
        }

        pc = connections[data.src].pc;

        // We need to set the remote description to the received SDP offer
        // so that our local WebRTC layer knows how to talk to the caller.
        let desc = new RTCSessionDescription(data.sdp);

        pc.setRemoteDescription(desc).catch(error => {
            logError('handleOffer', error);
        });

        await pc.setLocalDescription(await pc.createAnswer());

        wsSend({
            type: 'answer',
            target: data.src,
            sdp: pc.localDescription
        });

        connections[data.src].pc = pc; // ???
    }

    const wsHandleAnswer = async (data) => {
        log('*** Call recipient has accepted our call, answer:', data);

        let pc = connections[data.src].pc;

        // Configure the remote description,
        // which is the SDP payload in our 'answer' message.

        let desc = new RTCSessionDescription(data.sdp);
        await pc.setRemoteDescription(desc).catch((error) => {
            logError('handleAnswer', error);
        });
    }

    const wsHandleICECandidate = async (data) => {
        let pc = connections[data.src].pc;

        let candidate = new RTCIceCandidate(data.candidate);

        log('*** Adding received ICE candidate', candidate);

        pc.addIceCandidate(candidate).catch(error => {
            logError('handleICECandidate', error);
        });
    }

    const wsSend = (data) => {
        if (socket.readyState !== WebSocket.OPEN) {
            return;
        }
        socket.send(JSON.stringify(data));
    }

    // ***********************
    // Peer Connection methods
    // ***********************

    const pcCreate = async (id, username) => {
        if (connections[id]) {
            return;
        }
        try {
            let pc = new RTCPeerConnection(PC_CONFIG);

            pc.onicecandidate = (event) =>
                pcOnIceCandidate(event, id);
            pc.oniceconnectionstatechange = (event) =>
                pcOnIceConnectionStateChange(event, id);
            pc.onsignalingstatechange =  (event) =>
                pcOnSignalingStateChangeEvent(event, id);
            pc.onnegotiationneeded = (event) =>
                pcOnNegotiationNeeded(event, id);
            pc.ontrack = (event) =>
                pcOnTrack(event, id);

            connections[id] = {
                pc: pc,
                username: username
            }

            if (localStream) {
                try {
                    localStream.getTracks().forEach(
                        (track) => connections[id].pc.addTransceiver(track, {
                            streams: [localStream]
                        })
                    );
                } catch (err) {
                    logError(err);
                }
            } else {
                // Start negotiation to listen remote stream only
                pcOnNegotiationNeeded(null, id);
            }
            createRemoteView(id, username);
        } catch (error) {
            logError('Peer: Connection failed', error);
        }
    }

    const pcOnTrack = (event, id) => {
        let iVideo = document.querySelector('#pc' + id + ' video');
        iVideo.srcObject = event.streams[0];
    }

    const pcOnIceCandidate = (event, id) => {
        let pc = connections[id].pc;

        if (event.candidate && pc.remoteDescription) {
            log('*** Outgoing ICE candidate: ' + event.candidate);
            wsSend({
                type: 'candidate',
                target: id,
                candidate: event.candidate
            });
        }
    }

    const pcOnNegotiationNeeded = async (event, id) => {
        let pc = connections[id].pc;
        try {
            const offer = await pc.createOffer();

            // If the connection hasn't yet achieved the "stable" state,
            // return to the caller. Another negotiationneeded event
            // will be fired when the state stabilizes.
            if (pc.signalingState != 'stable') {
                return;
            }

            // Establish the offer as the local peer's current
            // description.
            await pc.setLocalDescription(offer);

            // Send the offer to the remote peer.
            wsSend({
                type: 'offer',
                target: id,
                sdp: pc.localDescription
            });
        } catch(err) {
            logError('*** The following error occurred while handling' +
                ' the negotiationneeded event:', err);
        };
    }

    const pcOnIceConnectionStateChange = (event, id) => {
        let pc = connections[id].pc;
        switch (pc.iceConnectionState) {
            case 'closed':
            case 'failed':
            case 'disconnected':
                pcClose(id);
                break;
        }
    }

    const pcOnSignalingStateChangeEvent = (event, id) => {
        let pc = connections[id].pc;

        log('*** WebRTC signaling state changed to: ' + pc.signalingState);

        switch (pc.signalingState) {
            case 'closed':
                pcClose(id);
                break;
        }
    }

    const pcClose = (id) => {
        let remoteView = document.querySelector('#pc' + id);

        if (connections[id]) {
            let pc = connections[id].pc;
            pc.close();
            delete connections[id];
        }
        if (remoteView) {
            remoteView.remove();
        }
    }

    // *******
    // Helpers
    // *******

    const log = (msg, data) => {
        if (!data) {
            data = ''
        }
        console.log(msg, data);
    }

    const logError = (msg, data) => {
        if (!data) {
            data = ''
        }
        console.error(msg, data);
    }

    init();
})();

The signaling server is made on the Python aiohttp framework and is a simple "view" that trivially proxies WebRTC requests. The connection to the server in this example is done over web sockets . Well, in addition, simple text chat data is transmitted through the signaling channel.

Signaling server implementation example

import json
from aiohttp.web import WebSocketResponse, Response
from aiohttp import WSMsgType
from uuid import uuid1
from lib.views import BaseView


class WebSocket(BaseView):
    """ Process WS connections """

    async def get(self):
        username = self.request['current_user'].firstname or ''

        room_id = self.request.match_info.get('room_id')

        if room_id != 'test_room' and
            self.request['current_user'].is_anonymous:
            self.raise_error('forbidden')  # @TODO: send 4000

        if (self.request.headers.get('connection', '').lower() != 'upgrade' or
            self.request.headers.get('upgrade', '').lower() != 'websocket'):
            return Response(text=self.request.path)  # ???

        self.ws = WebSocketResponse()
        await self.ws.prepare(self.request)

        self.uid = str(uuid1())

        if room_id not in self.request.app['web_sockets']:
            self.request.app['web_sockets'][room_id] = {}

        self.room = self.request.app['web_sockets'][room_id]

        users = {}
        for id, data in self.room.items():
            users[id] = data['name']

        ip = self.request.headers.get(
            'X-FORWARDED-FOR',
            self.request.headers.get('X-REAL-IP',
            self.request.remote))

        msg = {
            'type': 'handshake',
            'uid': str(self.uid),
            'users': users, 'ip': ip}
        await self.ws.send_str(json.dumps(msg, ensure_ascii=False))

        self.room[self.uid] = {'name': username, 'ws': self.ws}

        try:
            async for msg in self.ws:
                if msg.type == WSMsgType.TEXT:
                    if msg.data == 'heartbeat':
                        print('---heartbeat---')
                        continue

                    try:
                        msg_data = json.loads(msg.data)

                        if 'target' not in msg_data or
                            msg_data['target'] not in self.room:
                            continue

                        msg_data['src'] = self.uid

                        if 'type' in msg_data and 'target' in msg_data:
                            if msg_data['type'] == 'offer':
                                msg_data['username'] = username
                        else:
                            print('INVALID DATA:', msg_data)
                    except Exception as e:
                        print('INVALID JSON', e, msg)

                    try:
                        await self.room[msg_data['target']]['ws'].send_json(
                            msg_data);
                    except Exception as e:
                        if 'target' in msg_data:
                            self.room.pop(msg_data['target'])

        finally:
            self.room.pop(self.uid)

        return self.ws

WebRTC technology, in addition to video communication, allows you to give the browser permission to capture the contents of a display or a separate application, which can be indispensable when conducting online lessons, webinars or presentations. Great, let's use it.

I was so carried away by the modern possibilities of video communication that I almost forgot about the most important subject in the class - the interactive whiteboard. However, the basic implementation is so trivial that I will not overload this article with it. We just add the canvas , listen for onmousemove (ontouchmove for tablets) mouse move events, and send the resulting coordinates to all connected points through the same signaling server.

Testing your interactive whiteboard

Here you need a tablet, a digitizer and a live child. At the same time, we will check the possibility of digitizing handwriting input.

To begin with, I took an old Galaxy Tab tablet on Android 4.4, a homemade stylus and the first recipes I came across as a background for the canvas. I did not install additional programs. The result discouraged me: my tablet is absolutely not suitable for writing! That is, it is no problem to move your finger on it, but getting the stylus into the outline of a letter, even such a huge one as in the picture below, is already a problem. Plus, the gadget begins to dull in the process of drawing, as a result of which the lines become broken. Plus, I could not get the child not to rest his wrist on the screen, which leaves additional daub at hand, and the tablet itself starts to slow down even more. Bottom line: an ordinary tablet for writing on a whiteboard is not suitable. The maximum of its capabilities is to move your finger across the screen rather large figures. But it's too late to offer this to schoolchildren.

Okay, this is purely theoretical research, right? Then we take a digitizer (aka graphic tablet) Wacom Bamboo A8 format, and watch the child.

Note that my six-year-old test subject received a laptop with a graphic pen for the first time in his life. It took us about ten minutes to get the basic skills of using the pen, and already in the second lesson the child used the tablet quite confidently, independently erased from the board, drew faces, flowers, our dog, and even started poking buttons available in epsilon, simultaneously asking questions like "Why do they raise their hand at school?" But the result invariably left much to be desired. The fact is that designers and artists maximize a fragment of the image to draw an element, which makes the lines accurate. Here we should see the whole board, on a 1: 1 scale. Here and the adult will not fall into the line. Here's what we got:

Final verdict: no handwriting is out of the question. And if we want to "put a hand" to our children, we need to achieve this on our own, on paper, the school will not help in this.

I must say that the child took all my experiments with enthusiasm and, moreover, since then has been following me with a tail and asking me to "turn on the recipes." Already well, the acquired skill will be useful to him, only for completely different purposes.

Anyway, as a result of experiments, I actually got an MVP - a minimum viable product, almostsuitable for online lessons, with video / audio conferencing, shared screen, interactive whiteboard, simple text chat and raise hand button. This is in case the child suddenly does not have a microphone. Yes, this happens, especially among children who have not learned their lessons.

But in this barrel of honey, unfortunately, there are a couple of tar spoons.

Testing WebRTC

Spoon number 1. Since our video communication uses direct connections between clients, the first step is to test the scalability of such a solution. For the test, I took an old laptop with a dual-core i5-3230M on board, and began to connect clients with disabled web cameras to it, that is, emulating a one-to-many mode:

As you can see, the experimental laptop is able to more or less comfortably broadcast to five clients ( at CPU load within 60%). And this is provided the resolution of the outgoing video stream is reduced to 720p (640x480px) and the frame rate to 15 fps. In principle, not so bad, but when connecting a class of several dozen students, you will have to abandon the "full mesh" in favor of cascading, that is, each of the first five clients proxies the stream to the next five, and so on.

Spoon number 2.To create a direct interactive connection (ICE) between clients, they need to bypass firewalls and NATs. To do this, WebRTC uses a STUN server , which informs clients of external connection parameters. It is believed that in most cases this is sufficient. But I was almost immediately "lucky":

As you can see, the debugger complains about the impossibility of ICE connection and requires a TURN server to be connected, that is, a relay. And this is already EXPENSIVE. A simple signaling server is indispensable. Conclusion - you have to pass all streams through the media server.

Media Server

For testing, I used aiortc . An interesting development, allows you to connect the browser directly to the server via WebRTC. Separate signaling is not needed, you can use the data channel of the connection itself. This works, all my test points connected without issue. But the problem with performance. A simple echo of a video / audio stream with the same 720p and 15fps limits ate 50% of my virtual CPU on the test VDS. Moreover, if the load is increased to 100%, the video stream does not have time to be unloaded to clients and begins to clog up memory, which ultimately leads to a server stop. Obviously, the Python we love to use for I / O processing is not very CPU bound. We'll have to look for a more specialized solution, for example, Janusor Jitsy .

In any case, a complete solution will require dedicated servers, according to my estimates, at the rate of 1 core per room (class). This already costs money and goes beyond simple testing, so at this point I will consider the first phase of my research completed.

conclusions

1. To put it mildly, it's strange to see download instructions and links to register in the program of the former potential enemy number 1 (here about Microsoft Teams) on the official portal of the Russian Federation . And this is in the era of sanctions and import substitution.

No, personally I am for the friendship of peoples and, in general, all kinds of tolerance, but is it really only my hair that stands on end from such “integration”? Aren't there our developments?

2. Integration of MES / NES with schools. Actually, the MES developers are great, they even made integration with Yandex.tutor. What about real-time grading during lessons, when will there be an API? Or am I not aware of something?

3. When choosing a distance or family form of education, you need to be honest with yourself: you will not be able to shift responsibility for the education of your child to the school. All the work of conducting lessons (in the case of family education), maintaining discipline and self-organization (in any case) falls entirely on the parents. You need to be aware of this and find time for classes. However, in large families this should not be a problem.

4. I will not include links to selected online schools here, lest it be considered an advertisement. I will only say that we have chosen private schools of the middle price range. In any case, the final result will depend on the child and we will receive it no earlier than September.

Or does it make sense to bring the development started here to its logical conclusion and organize your own school? What do you think? Are there like-minded people with specialized knowledge and experience in the field of education?

Useful links:

Russian Electronic School

Moscow Electronic School

MES Library For

Developers

About first graders, distance learning and asynchronous programming