Clever Geek Handbook

Multitasking in shell scripts

Sometimes, when writing a shell script, you want to perform some actions in several threads. Suitable situations might be, for example, compressing a large number of large files on a multiprocessor host, and transferring files over a wide bandwidth, where the speed of an individual connection is limited.



All examples are written in bash, but (with minimal changes) will work in ksh. Csh also has a background process management facility, so a similar approach can be used.



JOB CONTROL



This is the name of the section in man bash where the details go, in case you like reading man. We use the following simple features:



command & - runs a command in the background

jobs - prints a list of background commands



A simple example that does no useful action. Numbers are read from the test.txt file, and 3 processes are launched in parallel, which sleep for the corresponding number of seconds. Every three seconds, the number of running processes is checked, and if there are less than three, a new one is started. The launch of the background process has been moved to a separate function mytask, but you can start it directly in a loop.



test.sh 
#!/bin/bash 

NJOBS=3 ; export NJOBS

function mytask () {
echo sleeping for $1
 sleep $1
}

for i in $( cat test.txt )
do
    while [  $(jobs | wc -l ) -ge $NJOBS ]
        do 
            sleep 3
        done
    echo executing task for $i
    mytask $i &
done

echo waiting for $( jobs | wc -l ) jobs to complete
wait




Input data:



test.txt
60

50

30

21

12

13



Pay attention to the wait after the loop, the command waits for the processes running in the background to finish. Without it, the script will be terminated immediately after the end of the loop and all background processes will be interrupted. Perhaps this wait is mentioned in the famous meme "oh, wait !!!".



Terminating background processes



If you interrupt the script with Ctrl-C, it will be killed with all background processes. all processes running in the terminal receive signals from the keyboard (for example, SIGINT). If the script is killed from another terminal with the kill command, then background processes will remain running until termination and you need to remember this.



Spoiler header
user@somehost ~/tmp2 $ ps -ef | grep -E «test|sleep»

user 1363 775 0 12:31 pts/5 00:00:00 ./test.sh

user 1368 1363 0 12:31 pts/5 00:00:00 ./test.sh

user 1370 1368 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60

user 1373 1363 0 12:31 pts/5 00:00:00 ./test.sh

user 1375 1373 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 50

user 1378 1363 0 12:31 pts/5 00:00:00 ./test.sh

user 1382 1378 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30

user 1387 1363 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 3

user 1389 556 0 12:31 pts/2 00:00:00 grep --colour=auto -E test|sleep

user@somehost ~/tmp2 $ kill 1363

user@somehost ~/tmp2 $ ps -ef | grep -E «test|sleep»

user 1368 1 0 12:31 pts/5 00:00:00 ./test.sh

user 1370 1368 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 60

user 1373 1 0 12:31 pts/5 00:00:00 ./test.sh

user 1375 1373 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 50

user 1378 1 0 12:31 pts/5 00:00:00 ./test.sh

user 1382 1378 0 12:31 pts/5 00:00:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 30

user 1399 556 0 12:32 pts/2 00:00:00 grep --colour=auto -E test|sleep



This situation can be handled by intercepting the necessary signals, for which we add a handler at the beginning of the script:



trap 
function pids_recursive() {
    cpids=`pgrep -P $1|xargs`
    echo $cpids
    for cpid in $cpids;
       do
          pids_recursive $cpid
       done
}

function kill_me () {
    kill -9 $( pids_recursive $$ | xargs )
    exit 1
}

# 
#trap 'echo trap SIGINT; kill_me ' SIGINT
trap 'echo trap SIGTERM; kill_me' SIGTERM




kill -L displays a list of existing signals, if necessary, you can add handlers for the necessary ones.


More articles:


All Articles