The Power of tmux in Scripting

こんにちは、次世代システム研究室のN.M.です。

tmux is a terminal multiplexer that allows users to run multiple terminal sessions within a single terminal window. This can be incredibly useful for running multiple commands at the same time, especially when working with long-running commands.

One of the key features of tmux is the ability to split the terminal window into panes, each of which can run a separate command. This can be especially useful when working with scripts, as it allows you to see the output of multiple commands at the same time.

tmux can also be nested, allowing you to run multiple tmux sessions within a single window. This can be useful for running multiple long-running commands in parallel, as each command can be run in it’s own tmux session.

Here’s a simple example of how you can use tmux in a script:

#!/usr/bin/env bash

user_id=$1

now=$(date '+%Y%m%d_%H%M')
tmux_session="session_${now}"
echo "running in tmux session: $tmux_session"
tmux new -d -s $tmux_session
tmux send-keys -t "${tmux_session}" "./get-todos.sh $user_id" Enter
tmux attach -t $tmux_session

This is a Bash script that accepts a user ID as an argument, creates a new tmux session with a unique name based on the current date and time, and then runs the ./get-todos.sh script with the user ID as an argument in the new session.

Let’s have a look at ./get-todos.sh

#!/usr/bin/env bash

curl -s https://jsonplaceholder.typicode.com/todos | jq ".[] | select(.userId == $1)"

This is a Bash script that sends an HTTP GET request to the https://jsonplaceholder.typicode.com/todos endpoint, pipes the response to jq, and then uses the select filter to select only the objects in the array that have a userId property equal to the first argument passed to the script.

So the first script invokes the second inside a tmux session and attaches to the session so you can see the output.

If invoked like so: ./run-get-todos.sh 1

The output will look something like this:


Here we see that all of the output of the script is inside a tmux session named session_20..., where the session name includes the current date-time, but is shortened for display in the tmux status bar.

As far as our toy example goes, this is not particularly interesting, but imagine that our script runs on a remote server and the command executed with session_20... was a command that took hours to complete and is important for us to see the output, check if the command was successful etc. Then we can begin to see the merits of execution within a tmux session. If our SSH connection to the remote server fails, we can always reconnect and re-attach to the tmux session to check our command.

Now let’s imagine that we want to run the script locally, but have multiple jobs we wish to execute on multiple remote servers, and be notified when all remote commands have finished.

First let’s change our initial tmux creator script to invoke a new script ./get-todos-parallel.sh:

#!/usr/bin/env bash

user_ids=$1

now=$(date '+%Y%m%d_%H%M')
tmux_session="session_${now}"
echo "running in tmux session: $tmux_session"
tmux new -d -s $tmux_session
tmux send-keys -t "${tmux_session}" "./get-todos-parallel.sh $user_ids" Enter
tmux attach -t $tmux_session

Let’s look at ./get-todos-parallel.sh.

#!/usr/bin/env bash

# Loop through the list of user_ids
for user_id in "$@"; do
  echo "user_id: $user_id"
  tmux split-window -v "./run-remote-get-todos.sh $user_id"
done

target_pane_ids=( $(tmux list-panes -F '#D' | tail -n +2) )
wait_time=2
i=0
line_matcher='FINISHED'

while (( i < $# ))
do
    while true
    do
        echo -n .
        captured=$(tmux capture-pane -p -t "${target_pane_ids[$i]}" | grep -E '[^[:space:]]')
        grep -q "${line_matcher}" <<< "${captured}" && break
        sleep "${wait_time}"
    done
    (( ++i ))
done

echo -e "\nALL FINISHED"

This script runs a loop that iterates through a list of user IDs passed as arguments to the script. For each user ID, it creates a new tmux pane with the tmux split-window -v "./run-remote-get-todos.sh $user_id" command and runs the ./run-remote-get-todos.sh script in the new pane with the current user ID as an argument.

The script then captures the output of the tmux panes with the tmux capture-pane -p -t "${target_pane_ids[$i]}" command and waits until the output contains the line “FINISHED” using the grep -q "${line_matcher}" command. When the “FINISHED” line is found, the script breaks out of the inner loop and increments the $i variable. The outer loop continues until $i is equal to the number of user IDs passed as arguments to the script.

Finally, the script prints the message “ALL FINISHED” when all of the tmux panes have finished running.

Here, we see how we can synchronized parallel execution of multiple commands, based on the output in each command’s tmux pane.

Let’s now look at ./run-remote-get-todos.sh.

#!/usr/bin/env bash

export TERM=xterm
vagrant ssh -- -t "/vagrant/run-remote-get-todos-tmux.sh ${1}"

This script connects to a Vagrant virtual machine over SSH and runs the /vagrant/run-remote-get-todos-tmux.sh script on the VM with the first argument passed to this script as an argument.

The export TERM=xterm command sets the TERM environment variable to the value “xterm”, which specifies the terminal type. This is used to set the correct terminal type for terminal emulators that do not set the TERM variable automatically.

The vagrant ssh -- -t "/vagrant/run-remote-get-todos-tmux.sh ${1}" command connects to the Vagrant VM over SSH and runs the /vagrant/run-remote-get-todos-tmux.sh script with the first argument passed to this script as an argument. The -t option forces the allocation of a pseudo-tty, which allows the /vagrant/run-remote-get-todos-tmux.sh script to run in a terminal emulator on the VM.

The /vagrant/run-remote-get-todos-tmux.sh script is run within a tmux session on the virtual machine, as indicated by the name of the script. The details of this script and how it functions are shown below:

#!/usr/bin/env bash

user_id=$1

tmux_session="session_${user_id}"
echo "running in tmux session: $tmux_session"
tmux new -d -s $tmux_session
tmux send-keys -t "${tmux_session}" "/vagrant/get-todos.sh $user_id; echo -------- FINISHED --------" Enter
tmux attach -t $tmux_session

This script creates a new tmux session with the name “session_USER_ID”, where “USER_ID” is the value of the $user_id variable. It then runs the /vagrant/get-todos.sh script in the new session with the $user_id variable as an argument and echoes the message “——– FINISHED ——–” after the script finishes running. Finally, the script attaches to the new tmux session so that you can see the output of the /vagrant/get-todos.sh script.

The tmux new -d -s $tmux_session command creates a new tmux session in detached mode (-d) with the name specified by the $tmux_session variable.
The tmux send-keys command sends the keys /vagrant/get-todos.sh $user_id; echo -------- FINISHED -------- followed by the Enter key to the tmux session specified by $tmux_session. The tmux attach -t $tmux_session command attaches to the tmux session specified by $tmux_session, allowing you to see the output of the /vagrant/get-todos.sh script.

Finally get-todos.sh is similar to the script shown at the beginning of this blog, except that it waits for a short random number of seconds after the curl command, to simulate real-world execution:

#!/usr/bin/env bash

curl -s https://jsonplaceholder.typicode.com/todos | jq ".[] | select(.userId == $1)"
sleep $(( (RANDOM % 10) + 1 ))

 

After invoking the scripts above with a command such as ./run-get-todos-parallel.sh "1 2 3", the output will be similar to the image below.


You can see that each curl GET request to the API was executed within a separate tmux session, where the session name is session_USER_ID. If you have  long-running remote jobs and the network connection is broken for some reason, you can just reconnect to the server and re-attach to the tmux sessions to manually check the progress of each job individually.

The top pane shows that it waited until all jobs were complete and then notified us that all were completed. In this way we are able to synchronize when all jobs are complete and if necessary perform some task such as notification, logging, or clean-up.

If you got this far congratulations! I hope you were able to see some of the possibilities of using tmux within your scripts.

 

次世代システム研究室では、グループ全体のインテグレーションを支援してくれるアーキテクトを募集しています。インフラ設計、構築経験者の方、次世代システム研究室にご興味を持って頂ける方がいらっしゃいましたら、ぜひ募集職種一覧からご応募をお願いします。