🌲 🌠 🤸🏽 QEMU-based System Test Automation (Part 2/2) 🙈 ⚠️ 👩‍🚀

This is the second part of an article on VM-based system testing automation. The first part can be found here .

In this part of the article, we will use the skills learned in the first part to actually automate system tests. At the end of the article, we will receive a script that anyone can run on their computer and completely from scratch get a deployed stand of three machines, installed by the application under test, as well as passed real system tests (we will write three tests).

Disclaimer

, , , . " " , . , .

So, in the last part, we stocked up with an impressive arsenal of skills for working with virtual machines from the command line: we learned how to install virtual machines, roll out an OS on them (for example, Ubuntu Server 18.04), connect a virtual machine to a host over a network, and even organize a control channel via SSH. All of this will be useful for us in this article, but before moving on to practice, there are several issues to discuss.

What do we want to get?

The most important question to be answered is "What result do we want to get?" Yes, last time we talked a lot about automating the installation, deployment and configuration of virtual machines, but apart from the ultimate goal, this all does not make much sense.

For me personally, all-in-one system tests look like this: I download several small files from VCS (the script itself with the launch, plus, possibly, several auxiliary artifacts), put the program under test where I need it (in the form of an installer or a package, for example), I press one button and go to drink coffee. When I come back, I want to either see that all tests passed, or that certain tests have broken. I do not want to engage in any setup of the stand, I do not want to deploy virtual machines or set up something there. I want to download a script and use it.

, : , . , ( ).

, .

. -, Data Plane Development Kit (DPDK). DPDK — , , , DPDK. DPDK , , end-to-end .

DPDK

DPDK (Data Plane Development Kit) — , C. , . , . , . DPDK . ? . , , Linux, , , , . , Linux — , . , DPDK .

;
;
DROP — ;
ACCEPT — ;

, , :

;
, ;

, , , :

(client, middlebox, server), Ubuntu Server 18.04 ();
: client middlebox ( net_1) middlebox server ( net_2);
;
, ;
- middlebox;
.

. :

, SSH- , , , ;
SSH net_1 net_2, ( net_for_ssh). :

.. , , - ;
, - (, ), .

!

- :

, , :

run_tests.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   net_for_ssh
# =======================================
virsh net-define net_for_ssh.xml
virsh net-start net_for_ssh

# =======================================
#   net_1
# =======================================
virsh net-define net_1.xml
virsh net-start net_1

# =======================================
#   net_2
# =======================================
virsh net-define net_2.xml
virsh net-start net_2

# =======================================
#   client
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output client.qcow2 \
    --install wget \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_client.yaml:/etc/netplan/

virt-install \
    --import \
    --name client \
    --ram 1024 \
    --disk client.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,mac=52:54:56:11:00:00 \
    --noautoconsole

# =======================================
#   middlebox
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output middlebox.qcow2 \
    --install python,daemon,libnuma1 \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_middlebox.yaml:/etc/netplan/

virt-install \
    --import \
    --name middlebox \
    --vcpus=2,sockets=1,cores=2,threads=1 \
    --cpu host \
    --ram 2048 \
    --disk middlebox.qcow2 \
    --network network=net_for_ssh \
    --network network=net_1,model=e1000 \
    --network network=net_2,model=e1000 \
    --noautoconsole

# =======================================
#   server
# =======================================
virt-builder ubuntu-18.04 \
    --format qcow2 \
    --output server.qcow2 \
    --install nginx \
    --root-password password:1111 \
    --run-command "ssh-keygen -A" \
    --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
    --copy-in netcfg_server.yaml:/etc/netplan/

virt-install \
    --import \
    --name server \
    --ram 1024 \
    --disk server.qcow2 \
    --network network=net_for_ssh \
    --network network=net_2,mac=52:54:56:00:00:00 \
    --noautoconsole

# =======================================
# ,    
#     
# =======================================

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no"

while ! SSH_CMD root@192.168.100.2 "echo Hello world from client!" echo
do
    echo "Waiting for client VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.3 "echo Hello world from middlebox!" echo
do
    echo "Waiting for middlebox VM ..."
    sleep 1
done

while ! SSH_CMD root@192.168.100.4 "echo Hello world from server!" echo
do
    echo "Waiting for server VM ..."
    sleep 1
done

net_for_ssh.xml

<network>
    <name>net_for_ssh</name>
    <bridge name='net_for_ssh'/>
    <ip address='192.168.100.1' netmask='255.255.255.0'/>
</network>

net_1.xml

<network>
    <name>net_1</name>
    <bridge name='net_1'/>
    <ip address='192.168.101.1' netmask='255.255.255.0'/>
</network>

net_2.xml

<network>
    <name>net_2</name>
    <bridge name='net_2'/>
    <ip address='192.168.102.1' netmask='255.255.255.0'/>
</network>

netcfg_client.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.2/24
    ens4:
      addresses:
        - 192.168.101.2/24
      gateway4: 192.168.101.3

netcfg_middlebox.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.3/24

netcfg_server.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      addresses:
        - 192.168.100.4/24
    ens4:
      addresses:
        - 192.168.102.2/24
      gateway4: 192.168.102.3

, :

--install virt-builder, . --run-command "apt install ...". , , — virt-builder . client wget, server — nginx ( http- ). middlebox , DPDK-;
client server , - , . ;
middlebox ( --vcpus): CPU c hyperthreading. — , DPDK . --cpu host, , , . , - QEMU , SSE3 . , , DPDK .
middlebox , : e1000. , - DPDK.

run_tests.sh ( , ). , :

;
.

, run_tests.sh, , . :

run_clean.sh

#!/bin/bash
set -euo pipefail

# =======================================
#   client
# =======================================
if virsh list --all | grep -q " client "; then
    if virsh domstate client | grep -q "running"; then
        virsh destroy client
    fi
    virsh undefine client --snapshots-metadata --remove-all-storage
fi

# =======================================
#   middlebox
# =======================================
if virsh list --all | grep -q " middlebox "; then
    if virsh domstate middlebox | grep -q "running"; then
        virsh destroy middlebox
    fi
    virsh undefine middlebox --snapshots-metadata --remove-all-storage
fi

# =======================================
#   server
# =======================================
if virsh list --all | grep -q " server "; then
    if virsh domstate server | grep -q "running"; then
        virsh destroy server
    fi
    virsh undefine server --snapshots-metadata --remove-all-storage
fi

# =======================================
#   net_for_ssh
# =======================================
if virsh net-list --all | grep -q " net_for_ssh "; then
    if virsh net-list --all | grep " net_for_ssh " | grep -q " active "; then
        virsh net-destroy net_for_ssh
    fi
    virsh net-undefine net_for_ssh
fi

# =======================================
#   net_1
# =======================================
if virsh net-list --all | grep -q " net_1 "; then
    if virsh net-list --all | grep " net_1 " | grep -q " active "; then
        virsh net-destroy net_1
    fi
    virsh net-undefine net_1
fi

# =======================================
#   net_2
# =======================================
if virsh net-list --all | grep -q " net_2 "; then
    if virsh net-list --all | grep " net_2 " | grep -q " active "; then
        virsh net-destroy net_2
    fi
    virsh net-undefine net_2
fi

, run_tests.sh, . :

( virsh list --all, — virsh net-list -all);
/, , / , ;
( --snapshots-metadata) ( --remove-all-storage).

run_tests.sh run_clean.sh. run_tests.sh , run_clean.sh.

run_clean.sh . . , , , .

. SSH- — SSH ~/.ssh/known_hosts. , SSH — , , - . SSH_CMD:

SSH_CMD="sshpass -p 1111 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"

-o UserKnownHostsFile=/dev/null, .

, — . Ubuntu Server ( GUI) bash-, .

bash-, . , .

-, , :

EXEC_CLIENT="$SSH_CMD root@192.168.100.2"
EXEC_MIDDLEBOX="$SSH_CMD root@192.168.100.3"
EXEC_SERVER="$SSH_CMD root@192.168.100.4"

. . :

$EXEC_CLIENT echo hello from client
$EXEC_SERVER echo hello from server

-, ? bash- Heredoc. Heredoc :

$EXEC_CLIENT << EOF
    echo hello from client
    ls
    pwd
EOF

$EXEC_MIDDLEBOX << EOF
    echo hello from middlebox
    some_another_command
EOF

EOF — , . EOF , , .

, set -xeuo pipefail. :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    command2 | command3
EOF

, :

-x bash- ;
-e , ;
-u , ;
-o pipeline , - .

, - — .

, . bash- , - . :

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    command1
    ! command2
EOF

: command1 0, command2 , .

$SCP_CMD l3fwd-acl-1.0.0.deb root@192.168.100.3:~

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    dpkg -i l3fwd-acl-1.0.0.deb

    echo 256 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    mkdir -p /mnt/huge
    mount -t hugetlbfs nodev /mnt/huge

    modprobe uio_pci_generic
    dpdk-devbind --bind=uio_pci_generic ens4 ens5

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" > /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    echo "R0:0:0:0:0:0:0:0/0 0:0:0:0:0:0:0:0/0 0 : 65535 0 : 65535 0x0/0x0 0" > /etc/rule_ipv6.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

, ( ):

deb-;
256 2 (DPDK- - );
poll-mode uio_pci_generic ( Ubuntu Server). , DPDK ;
ens4 ( ) ens5 ( ) uio_pci_generic;
rule_ipv4.db IPv4 : 192.168.102.0/24 1 ( , ), 192.168.101.0/24 0 ( );
rule_ipv6.db, " 0". IPv6 , DPDK ;
l3fwd daemon. , l3fwd: https://doc.dpdk.org/guides/sample_app_ug/l3_forward.html

, DPDK :

.deb ;
;
;
;
, , uio_pci_generic;
.

: " ", - . , . .

: , DPDK- ( unit-, ), , : - :

$EXEC_CLIENT arp -s 192.168.101.3 52:54:56:00:00:00
$EXEC_SERVER arp -s 192.168.102.3 52:54:56:11:00:00

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    wget --timeout=5 --tries=1 http://192.168.102.2
EOF

ARP-

, , ARP-.

? , l3fwd,

DPDK, , ARP.

l3fwd ,

rule_ipv4.db rule_ipv6.db,

: , / , --.

:

, .

, middlebox

, MAC-

Ethernet- ( client server ).

:

destination MAC-.

ARP-.

# =======================================
#  ,  tcp 
# =======================================

$EXEC_MIDDLEBOX << EOF
    set -xeuo pipefail

    daemon --name l3fwd --stop

    #     
    echo "@0.0.0.0/0 0.0.0.0/0 0 : 65535 0 : 65535 0x06/0xff" > /etc/rule_ipv4.db

    echo "R0.0.0.0/0 192.168.102.0/24 0 : 65535 0 : 65535 0x0/0x0 1" >> /etc/rule_ipv4.db
    echo "R0.0.0.0/0 192.168.101.0/24 0 : 65535 0 : 65535 0x0/0x0 0" >> /etc/rule_ipv4.db

    daemon --name l3fwd --unsafe --output /var/log/l3fwd -- l3fwd-acl \
        -l 1 \
        -n 4 \
        -- \
        -p 0x3 \
        -P \
        --config="(0,0,1),(1,0,1)" \
        --rule_ipv4="/etc/rule_ipv4.db" \
        --rule_ipv6="/etc/rule_ipv6.db"
EOF

# =======================================
# ,  ping  ,
#  http  - 
# =======================================

$EXEC_CLIENT << EOF
    set -xeuo pipefail

    ping -c 5 192.168.102.2
    ! wget --timeout=5 --tries=1 http://192.168.102.2
EOF

, wget : , , wget .

`run_tests.sh`

, , run_clean.sh.

: : DPDK- middlebox . ? , ( ) DPDK- . , , . ?

. , ? , . : , init, .

# =======================================
#   client
# =======================================
if ! virsh list --all | grep -q " client "
then
    virt-builder ubuntu-18.04 \
        --format qcow2 \
        --output client.qcow2 \
        --hostname client \
        --install wget,net-tools \
        --root-password password:1111 \
        --run-command "ssh-keygen -A" \
        --run-command "sed -i \"s/.*PermitRootLogin.*/PermitRootLogin yes/g\" /etc/ssh/sshd_config" \
        --copy-in netcfg_client.yaml:/etc/netplan/

    virt-install \
        --import \
        --name client \
        --ram 1024 \
        --disk client.qcow2 \
        --network network=net_for_ssh \
        --network network=net_1,mac=52:54:56:11:00:00 \
        --noautoconsole

    virsh snapshot-create-as client --name init
else
    virsh snapshot-revert client --snapshotname init
fi

, init.

, , : , .deb- DPDK, . .

, , :

# =======================================
#   net_1
# =======================================
if ! virsh net-list --all | grep -q " net_1 "
then
    virsh net-define net_1.xml
    virsh net-start net_1
fi

, , , . : !

, , , , . : , , , (end-to-end) .

, , : DPDK- ( Ubuntu Server 18.04) ( ping wget). , .deb . , , ( , ). .

: (run_tests.sh run_clean.sh), xml- yaml-. VCS. .

, , " — ". , . , " ", .

"", "" "" , , , , . Proof Of Concept . - … . , , , , . ...

→

QEMU-based System Test Automation (Part 2/2)

What do we want to get?

!

run_tests.sh

More articles:

`run_tests.sh`