The customer wanted to evaluate the speed of all kinds of telecom operators in different geographic locations, in order to understand which mobile operator is the most optimal for him when installing equipment using an LTE connection, for example, for video broadcasts. At the same time, the task had to be solved as simply and cheaply as possible, without expensive equipment.
I must say right away that the task is not the simplest and most knowledge-intensive, I will tell you what problems I encountered and how I solved them. So let's go.
Note
Measuring the speed of an LTE connection is a very difficult matter: you need to choose the right equipment and measurement method, and also have a good idea of the topology and operation of the cellular network. Plus, the speed can be influenced by several factors: the number of subscribers per cell, weather conditions, even from cell to cell, the speed can be strikingly different due to the network topology. In general, this is a problem with a huge number of unknowns, and only a telecom operator can correctly solve it.
Initially, the customer just wanted to drive the courier with the operators' phones, take measurements directly on the phone and then write down the speed measurement results in a notebook. My solution for measuring the speed of lte networks, although not ideal, solves the problem.
Due to lack of time, I made decisions not in favor of convenience or practicality, but in favor of the speed of development. For example, for remote access, reverse ssh was raised, instead of the more practical vpn, in order to save time on setting up the server and each individual client.
Technical task
As stated in the article Without TK: why the client doesn't want it : Do not work without TK! Never, nowhere!
The technical assignment was quite simple, I will expand it a little to understand the end user. The choice of technical solutions and equipment was dictated by the customer. So, the TK itself, after all the approvals:
vim2 lte- Huawei e3372h — 153 ( n). GPS-, UART. www.speedtest.net :
csv. - 6 . , GPIO.
I described the TK in a free form, after many approvals. But the meaning of the task is already visible. The deadline for everything was given a week. But in reality it lasted for three weeks. This is in view of the fact that I did it only after the main work and on weekends.
Here I would like to draw your attention to the fact that the customer had previously agreed to use the speed measurement service and hardware, which greatly limited my capabilities. The budget was also limited, so nothing was bought in addition. So I had to play by these rules.
Architecture and development
The scheme is simple and straightforward. Therefore, I will leave it without any special comments.
I decided to implement the whole project in python, despite the fact that I had no experience of development in this language at all. I chose it because there were a bunch of ready-made examples and solutions that could speed up development. Therefore, I ask all professional programmers not to scold my first experience in python development, and I am always happy to hear constructive criticism to improve my skill.
Also in the process I discovered that python has two running versions 2 and 3, as a result, I stopped at the third.
Hardware nodes
Single board vim2
As the main machine, I was given a single-board vim2
An excellent, powerful media combine for a smart home and SMART-TV, but extremely unsuitable for this task, or, shall we say, poorly suited. For example, its main OS is Android, and Linux is a passing OS, and therefore no one guarantees the high-quality operation of all nodes and drivers under Linux. And I suppose that some of the problems were related to the USB drivers of this platform, so the modems did not work as expected on this board. He also has very poor and scattered documentation, so each operation took a lot of time digging in the docks. Even ordinary work with GPIO drank a lot of blood. For example, it took me several hours to set up the work with the LED. But, to be objective, it was fundamentally not important what kind of single-board device, the main thing is that it should work and have USB ports.
First, I need to install Linux on this board. In order not to scour the jungle of documentation for everyone, as well as for those who will deal with this single-board device, I am writing this chapter.
There are two options to install Linux: on an external SD card, or on an internal MMC. I struggled with the card in the evening, and did not know how to make it work, so I decided to install it on MMC, although without a doubt it would be much easier to work with an external card.
The firmware is crookedly described here . I am translating from strange into Russian. In order to flash the board, I need to connect the hardware UART. I connected it as follows.
- Tool Pin GND: <—> Pin17 of VIMs's GPIO
- Tool Pin TXD: <—> Pin18 of VIMs's GPIO (Linux_Rx)
- Tool Pin RXD: <—> Pin19 of VIMs's GPIO (Linux_Tx)
- Tool Pin VCC: <—> Pin20 of VIMs's GPIO
After that, I downloaded the firmware from here . The specific firmware version is VIM1_Ubuntu-server-bionic_Linux-4.9_arm64_EMMC_V20191231 .
In order to upload this firmware, I need certain utilities. Read more about this here . I haven’t tried to flash it under Windows, but I need to tell you a few words about the Linux firmware. First, I'll install the utilities according to the instructions.
git clone https://github.com/khadas/utils
cd /path/to/utils
sudo ./INSTALLIii ... Nothing works. I spent a couple of hours editing the installation scripts so that everything would be correctly installed for me. I don't remember what I did there, but also that circus with horses. So be careful. But without these utilities there is no point in torturing vim2 further. Better not to mess with him at all!
After seven circles of hell, script configuration and installation, I got a package of working utilities. I connected the board via USB to my Linux computer, and the UART is also connected according to the diagram above.
I am setting my favorite minicom terminal to 115200, no hardware or software error control. And let's get started.
When booting VIM2 in the UART terminal, I press any key, for example space bar, to stop booting. After the line appears
kvim2# I enter the command:
kvim2# run updateOn the host from where we download, I execute:
burn-tool -v aml -b VIM2 -i VIM2_Ubuntu-server-bionic_Linux-4.9_arm64_EMMC_V20191231.imgEveryone, phew. I asked, there is Linux on the board. Login / password khadas: khadas.
After that, small initial settings. For further work, I disable the password for sudo (yes, not secure, but convenient).
sudo visudoI edit the line to the view and save
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) NOPASSWD: ALLThen I change the current locale so that the time is in Moscow, otherwise it will be GMT.
sudo timedatectl set-timezone Europe/Moscowor
ln -s /usr/share/zoneinfo/Europe/Moscow /etc/localtimeIf you find it difficult, then do not use this board, Raspberry Pi is better. Fair.
Modem Huawei e3372h - 153
This modem drank my blood well, and, in fact, it became the bottleneck of the whole project. In general, the name “modem” for these devices does not at all reflect the essence of the work: this is a powerful combine, this piece of hardware has a composite device that pretends to be a CD-ROM in order to install drivers, and then goes into the network card mode.
Architecturally, from the point of view of a Linux user, after all the settings, it looks like this: after connecting the modem, I have the eth * network interface, which receives the ip address 192.168.8.100 via dhcp, and the default gateway 192.168.8.1.
And the most important point! This modem model does not know how to work in the modem mode, which is controlled by AT commands... Everything would be much easier, create ppp-connections for each modem and then operate with them. But in my case, "myself" (more precisely a Linux diver according to the udev rules), creates an eth-interface and assigns an ip-address to it via dhcp.
In order not to get confused further, I propose to forget the word "modem" and say a network card and a gateway, because in fact it is like connecting a new network card with a gateway.
When there is one modem, this does not cause any special problems, but when there is more than one modem, namely n-pieces, the following picture of the network appears.
That is, n network cards, with the same IP address, each with the same default gateway. But in fact, each of them is connected to its own operator.
Initially, I had a simple solution: using the ifconfig or ip command to extinguish all interfaces and just turn on one in turn and test it. The solution was good for everyone, except that during the moments of switching, I was not able to connect to the device. And since the switching is frequent and fast, in fact I had no opportunity to connect at all.
Therefore, I chose the way to change "manually" the ip-addresses of the modems and continue to drive traffic using the routing settings.
This was not the end of my problems with the modems: in case of power problems, they fell off, a good stable power supply to the USB hub was required. This problem was solved by hard soldering the power supply directly to the hub. Another problem that I faced and which ruined the whole project: after a reboot or a cold start of the device, not all modems were detected and not always, and why this happened and by what algorithm I could not install. But first things first.
For the modem to work correctly, I installed the usb-modeswitch package.
sudo apt update
sudo apt install -y usb-modeswitchAfter that, the modem after connection will be correctly detected and configured by the udev subsystem. I check by simply plugging in the modem and making sure the network is up.
Another problem that I could not solve: how to get the name of the operator we work with from this modem? The operator's name is contained in the modem's web interface at 192.168.8.1. This is a dynamic web page that receives data through ajax requests, so you can't just wget the page and parse the name. So I started looking at how to work out a web page, etc., and realized that I was doing some kind of nonsense. As a result, he spat, and began to receive the operator using the API of the Speedtest itself.
Many things would be easier if the modem could be accessed via AT commands. It would be possible to reconfigure it, create a ppp connection, assign an IP, get a carrier, etc. But alas, I work with what I gave.
GPS
The GPS receiver I was given had a UART interface and power. It was not the best solution, but nevertheless it was working and simple. The receiver was something like this.
To be honest, this was the first time I worked with a GPS receiver, but as expected, everything was invented for us long ago. So we just use ready-made solutions.
First, I turn on uart_AO_B (UART_RX_AO_B, UART_TX_AO_B) to connect the GPS.
khadas@Khadas:~$ sudo fdtput -t s /dtb.img /serial@c81004e0 status okayThen I check the success of the operation.
khadas@Khadas:~$ fdtget /dtb.img /serial@c81004e0 status
okayThis command, apparently, edits devtree on the fly, which is very convenient.
After the success of this operation, reboot and install the gps daemon.
khadas@Khadas:~$ sudo rebootInstalling the gps daemon. I install everything and chop it off right away for further configuration.
sudo apt install gpsd gpsd-clients -y
sudo killall gpsd
/* GPS daemon stop/disable */
sudo systemctl stop gpsd.socket
sudo systemctl disable gpsd.socketEditing the settings file.
sudo vim /etc/default/gpsdI install the UART on which the GPS will hang.
DEVICES="/dev/ttyS4"And then we turn on everything and start.
/* GPS daemon enable/start */
sudo systemctl enable gpsd.socket
sudo systemctl start gpsd.socketAfter that, I connect the GPS.
The GPS wire is in the hands, the UART wires of the debugger are visible under the fingers.
I reboot and check the GPS using the gpsmon program.
In this screenshot you cannot see the satellites, but you can see the communication with the GPS receiver, and this says that everything is fine.
In python I tried many options for working with this daemon, but I settled on the one that worked correctly with python 3.
Install the required library.
sudo -H pip3 install gps3 And I sculpt the code of work.
from gps3.agps3threaded import AGPS3mechanism
...
def getPositionData(agps_thread):
counter = 0;
while True:
longitude = agps_thread.data_stream.lon
latitude = agps_thread.data_stream.lat
if latitude != 'n/a' and longitude != 'n/a':
return '{}' .format(longitude), '{}' .format(latitude)
counter = counter + 1
print ("Wait gps counter = %d" % counter)
if counter == 10:
ErrorMessage(" GPS !!!")
return "NA", "NA"
time.sleep(1.0)
...
f __name__ == '__main__':
...
#gps
agps_thread = AGPS3mechanism() # Instantiate AGPS3 Mechanisms
agps_thread.stream_data() # From localhost (), or other hosts, by example, (host='gps.ddns.net')
agps_thread.run_thread() # Throttle time to sleep after an empty lookup, default '()' 0.2 two tenths of a second
If I need to get the coordinates, then this is done by the following call:
longitude, latitude = getPositionData(agps_thread)
And within 1-10 seconds I will either get the coordinate or not. Yes, I had ten attempts to get the coordinates. Not optimal, crooked and askew, but it works. I decided to do this, because GPS can catch poorly and not always receive data. If you wait for the data to be received, then if you work in a remote room, the program will freeze at this place. Therefore, I implemented such a not elegant option.
In principle, there would be more time, it would be possible to receive data from GPS directly via UART, parse them in a separate stream and work with them. But there was no time at all, hence the fierce ugly code. And yes, I'm not ashamed.
Light-emitting diode
Connecting the LED was all simple and complicated at the same time. The main difficulty is that the pin number in the system does not correspond to the pin number on the board and because the documentation is written with the left heel. To match the hardware pin number and the pin number in the OS, you need to run the command:
gpio readallA table of pin correspondence in the system and on the board will be displayed. After that, I can already operate the pin in the OS itself. In my case, the LED is connected to GPIOH_5 .
I transfer the GPIO pin to output mode.
gpio -g mode 421 outI write down zero.
gpio -g write 421 0I write down one.
gpio -g write 421 1
Everything is on, after recording "1"
#gpio subsistem
def gpio_init():
os.system("gpio -g mode 421 out")
os.system("gpio -g write 421 1")
def gpio_set(val):
os.system("gpio -g write 421 %d" % val)
def error_blink():
gpio_set(0)
time.sleep(0.1)
gpio_set(1)
time.sleep(0.1)
gpio_set(0)
time.sleep(0.1)
gpio_set(1)
time.sleep(0.1)
gpio_set(0)
time.sleep(1.0)
gpio_set(1)
def good_blink():
gpio_set(1)
Now, in case of errors, I call error_blink () and the LED blinks nicely for us.
Software nodes
Speedtest API
It's a great joy that the speedtest.net service has its own python-API, you can see it on Github .
The good news is that there are source codes that can also be viewed. How to work with this API (the simplest examples) can be found in the corresponding section .
Install the python library with the following command.
sudo -H pip3 install speedtest-cliFor example, you can even install the speed tester in Ubuntu directly from the rep. This is the same python application that you can then run directly from the console.
sudo apt install speedtest-cli -yAnd measure the speed of your Internet. As a result, as I did. I had to dig into the source codes of this speedtest in order to more fully implement them into my project. One of the most important tasks is to get the name of the telecom operator in order to substitute it in the plate.
speedtest-cli
Retrieving speedtest.net configuration...
Testing from B***** (*.*.*.*)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by MTS (Moscow) [0.12 km]: 11.8 ms
Testing download speed................................................................................
Download: 7.10 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.86 Mbit/s
import speedtest
from datetime import datetime
...
#
#6053) MaximaTelecom (Moscow, Russian Federation)
servers = ["6053"]
# If you want to use a single threaded test
threads = None
s = speedtest.Speedtest()
#
opos = '%(isp)s' % s.config['client']
s.get_servers(servers)
#
testserver = '%(sponsor)s (%(name)s) [%(d)0.2f km]: %(latency)s ms' % s.results.server
#
s.download(threads=threads)
#
s.upload(threads=threads)
#
s.results.share()
# csv-.
# GPS
longitude, latitude = getPositionData(agps_thread)
#
curdata = datetime.now().strftime('%d.%m.%Y')
curtime = datetime.now().strftime('%H:%M:%S')
delimiter = ';'
result_string = opos + delimiter + str(curpos) + delimiter + \
curdata + delimiter + curtime + delimiter + longitude + ', ' + latitude + delimiter + \
str(s.results.download/1000.0/1000.0) + delimiter + str(s.results.upload / 1000.0 / 1000.0) + \
delimiter + str(s.results.ping) + delimiter + testserver + "\n"
#
Here, too, everything turned out to be not so simple, although, it would seem, much easier. Initially, the servers parameter was equal to [] , they say, choose the best server. As a result, I had random servers, and as you might guess, floating speed. This is a rather complex topic, using a fixed server, if yes, then static or dynamic, requires research. But here is an example of graphs for measuring the speed of the Beeline operator with a dynamic selection of a test server and a statically fixed one.
The result of measuring the speed when choosing a dynamic server.
Speed test result, with one strictly selected one server.
"Wool" during testing is there and there, and it must be removed by mathematical methods. But with a fixed server it is slightly less and the amplitude is more stable.
In general, this is a place of great research. And I would measure the speed to my server using the iperf utility. But we stick to the TK.
Sending mail and errors
I tried several dozen different options for sending mail, but as a result I settled on the following. I registered a mailbox on yandex and then took this example of sending mail . I checked it and implemented it into the program. This example explores various options, including sending from gmail, etc. I didn't want to bother with raising my mail server and did not have time for this, but as it turned out later, it was also in vain.
The logs were sent according to the scheduler, if there was a connection , every 6 hours: at 00 hours, 06 am, 12 noon and 18 pm. I sent it as follows.
from send_email import *
...
message_log = " №1"
EmailForSend = ["dlinyj@trololo.ru", "pupkin@trololo.ru"]
files = ["/home/khadas/modems_speedtest/csv"]
...
def sendLogs():
global EmailForSend
curdata = datetime.now().strftime('%d.%m.%Y')
urtime = datetime.now().strftime('%H:%M:%S')
try:
for addr_to in EmailForSend:
send_email(addr_to, message_log, " " + curdata + " " + urtime, files)
except:
print("Network problem for send mail")
return False
return True
Errors were also initially submitted. To begin with, they accumulated in the list, and then sent them also using the scheduler, if there was a connection. However, then there were problems with the fact that yandex has a limit on the number of messages sent per day (this is pain, sadness and humiliation). Since there could be a huge number of errors even in a minute, they had to refuse to send errors by mail. So keep in mind when automatically sending through Yandex services about such a problem.
Feedback server
In order to have access to a remote piece of hardware and be able to tweak and reconfigure it, I needed an external server. In general, in all fairness, it would be correct to send all the data to the server and build all the beautiful graphs in the web interface. But not all at once.
I chose ruvds.com as the VPS . The simplest server could be taken. And in general, for my purposes, this would be enough for the eyes. But since I was not paying for the server out of pocket, I decided to take it with a small margin, so that it would be enough if we deploy a web interface, our own SMTP server, vpn, etc. Plus, to be able to set up a Telegram bot and not have problems with blocking it. Therefore, I chose Amsterdam and the following parameters.
As a way to communicate with a piece of hardware, vim2 chose a reverse ssh connection, and as practice has shown, it is not the best. If the connection is broken, the server holds the port and it is impossible to connect through it for some time. Therefore, it is still better to use other communication methods, for example vpn. In the future, I wanted to switch to vpn, but did not have time.
I will not go into the details of setting up a firewall, limiting rights, disabling ssh root connections and other common truths of VPS settings. I would like to believe that you already know everything. For a remote connection, I create a new user on the server.
adduser vimsshOn our piece of hardware, I generate ssh connection keys.
ssh-keygenAnd I copy them to our server.
ssh-copy-id vimssh@host.comOn our piece of hardware, I create an automatic connection of the reverse ssh at each boot. Pay attention to port 8083: it determines on which port I will connect via reverse ssh. Add to startup and start.
[Unit]
Description=Auto Reverse SSH
Requires=systemd-networkd-wait-online.service
After=systemd-networkd-wait-online.service
[Service]
User=khadas
ExecStart=/usr/bin/ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -CD 8080 -R 8083:localhost:22 vimssh@host.com
RestartSec=5
Restart=always
[Install]
WantedBy=multi-user.target
sudo systemctl enable autossh.service
sudo systemctl start autossh.serviceYou can even see the status:
sudo systemctl status autossh.serviceNow, on our VPS server, if you run:
ssh -p 8083 khadas@localhostThen I get to my test piece. And from the piece of iron I can also send logs and any data via ssh to my server, which is very convenient.
Putting it all together
Turning on, we start developing and debugging
Fuh, well, everything seems to have described all the nodes. Now it's time to put it all together. The code can be viewed here .
An important point with the code: This project like this "vlob" may not start, as it was sharpened on a specific task of a specific architecture. Although I give the source code, I will still analyze the most valuable here, right in the text, otherwise it is completely incomprehensible.
At the beginning, I have initialization of gps, gpio and launch of a separate scheduler thread.
#
pShedulerThread = threading.Thread(target=ShedulerThread, args=(1,))
pShedulerThread.start()The scheduler is quite simple: it looks to see if the time has come to send messages and what the error status is now. If there is an error flag, then blink the LED.
#sheduler
def ShedulerThread(name):
global ready_to_send
while True:
d = datetime.today()
time_x = d.strftime('%H:%M')
if time_x in time_send_csv:
ready_to_send = True
if error_status:
error_blink()
else:
good_blink()
time.sleep(1)The most difficult point in this project is to keep the reverse ssh connection on every test. In each test, the default gateway and dns server are re-configured. Since no one reads anyway, be aware that the train does not roll on wooden rails. Whoever finds an Easter egg will have a sweet tooth.
To do this, I create a separate routing table --set-mark 0x2 and a rule to redirect traffic.
def InitRouteForSSH():
cmd_run("sudo iptables -t mangle -A OUTPUT -p tcp -m tcp --dport 22 -j MARK --set-mark 0x2")
cmd_run("sudo ip rule add fwmark 0x2/0x2 lookup 102")You can read more about how this works in this article .
Then I go into an endless loop, where every time we get a list of connected modems (to find out if the network configuration has changed).
network_list = getNetworklist()Getting a list of network interfaces is fairly straightforward.
def getNetworklist():
full_networklist = os.listdir('/sys/class/net/')
network_list = [x for x in full_networklist if "eth" in x and x != "eth0"]
return network_listAfter receiving the list, I assign IP addresses to all interfaces, as I showed in the picture in the chapter about the modem.
SetIpAllNetwork(network_list)
def SetIpAllNetwork(network_list):
for iface in network_list:
lastip = "%d" % (3 + network_list.index(iface))
cmd_run ("sudo ifconfig " + iface + " 192.168.8." + lastip +" up")Then I just go through each interface in a loop. And I configure each interface.
for iface in network_list:
ConfigNetwork(iface)def ConfigNetwork(iface):
#
cmd_run("sudo ip route flush all")
#
cmd_run("sudo route add default gw 192.168.8.1 " + iface)
# dns- ( speedtest)
cmd_run ("sudo bash -c 'echo nameserver 8.8.8.8 > /etc/resolv.conf'")I check the interface for operability, if there is no network, then I generate errors. If there is a network, then it's time to act!
Here I set up ssh routing to this interface (if it was not done), send errors to the server, if the time has come, send logs and finally run a speedtest and save the logs to a csv file.
if not NetworkAvalible():
....
#
....
else: # , , !
# , ssh,
if (sshint == lastbanint or sshint =="free"):
print("********** Setup SSH ********************")
if sshint !="free":
md_run("sudo ip route del default via 192.168.8.1 dev " + sshint +" table 102")
SetupReverseSSH(iface)
sshint = iface
# , !!!
if ready_to_send:
print ("**** Ready to send!!!")
if sendLogs():
ready_to_send = False
if error_status:
SendErrors()
# . Except for the reverse ssh configuration function.
def SetupReverseSSH(iface):
cmd_run("sudo systemctl stop autossh.service")
cmd_run("sudo ip route add default via 192.168.8.1 dev " + iface +" table 102")
cmd_run("sudo systemctl start autossh.service")And of course, you need to add all this beauty to startup. To do this, I create a file:
sudo vim /etc/systemd/system/modems_speedtest.serviceAnd I write into it: I turn on autoload and start!
[Unit]
Description=Modem Speed Test
Requires=systemd-networkd-wait-online.service
After=systemd-networkd-wait-online.service
[Service]
User=khadas
ExecStart=/usr/bin/python3.6 /home/khadas/modems_speedtest/networks.py
RestartSec=5
Restart=always
[Install]
WantedBy=multi-user.target
sudo systemctl enable modems_speedtest.service
sudo systemctl start modems_speedtest.serviceNow I can watch the logs of what is happening using the command:
journalctl -u modems_speedtest.service --no-pager -fresults
Well, now the most important thing is what happened as a result? Here are some graphs that I managed to capture during the development and debugging process. The graphs were built using gnuplot with the following script.
#! /usr/bin/gnuplot -persist
set terminal postscript eps enhanced color solid
set output "Rostelecom.ps"
#set terminal png size 1024, 768
#set output "Rostelecom.png"
set datafile separator ';'
set grid xtics ytics
set xdata time
set ylabel "Speed Mb/s"
set xlabel 'Time'
set timefmt '%d.%m.%Y;%H:%M:%S'
set title "Rostelecom Speed"
plot "Rostelecom.csv" using 3:6 with lines title "Download", '' using 3:7 with lines title "Upload"
set title "Rostelecom 2 Ping"
set ylabel "Ping ms"
plot "Rostelecom.csv" using 3:8 with lines title "Ping"
The first experience was the operator Tele2, which I spent for several days.
I used a dynamic metering server here. The speed measurements work, but they float very much, however, some average value is still visible, and it can be obtained by filtering the data, for example, with a moving average.
Later I built a series of graphs for other telecom operators. In this case, there was already one testing server, and the results are also very interesting.
As you can see, the topic is very extensive for research and processing of this data, and clearly does not last a couple of weeks of work. But…
Outcome of work
The work was abruptly completed due to circumstances beyond my control. One of the weaknesses of this project, in my subjective opinion, was the modem, which did not really want to work simultaneously with other modems, and made such a trick with each boot. For these purposes, there are a huge number of other models of modems, usually they already have the Mini PCI-e format and are installed inside the device and are much easier to configure. But that's a completely different story. The project was interesting and was very glad that we managed to participate in it.
