Using strace to spy on a terminal in real time

I administrate linux servers from time to time, and a common need in those situation is monitoring what a different terminal is doing. Either because of a lost connection to an SSH terminal which runs a command, or wanting to know what an unrecognized terminal is doing. To that end I use a very useful tool called strace.

Strace is a system utility on linux systems that can trace all systems calls and signals that a process uses. To monitor a terminal we can make strace print all occurrences of the read and write system calls that the terminal process is using.

Note: This tutorial assumes you have root privileges otherwise you won't be able to use strace to connect and monitor a running process.

To see who’s connected to the machine we can list /dev/pts. The directory contains special character files for each connected pts.

To use strace for the task at hand, we first need to identify the terminal we want to connect to. We can use ps which lists all the running processes like so:

alpha@milkyway:~$ ps aux | grep pts
alpha      943  0.0  0.0  44472  3344 pts/4    R+   09:23   0:00 ps aux
alpha      944  0.0  0.0  21536  1048 pts/4    S+   09:23   0:00 grep --color=auto pts
alpha     7978  0.0  0.0  29812  5172 pts/3    Ss+  09:16 0:00 bash
alpha    22659  0.0  0.0  29812  5192 pts/0    Ss   00:13   0:00 bash
root     22877  0.0  0.0  72716  4304 pts/0    S    00:24   0:00 sudo su alchemist
root     22878  0.0  0.0  72248  3884 pts/0    S    00:24   0:00 su alchemist
alchemi+ 22891  0.0  0.0  29808  5100 pts/0    S+   00:24   0:00 bash
alpha    23022  0.0  0.0  29812  5104 pts/1    Ss   00:28   0:00 bash
root     23034  0.0  0.0  72956  4424 pts/1    S    00:28   0:00 sudo su
root     23035  0.0  0.0  72248  3780 pts/1    S    00:28   0:00 su
root     23049  0.0  0.0  28712  3924 pts/1    S+   00:28   0:00 bash
alpha    24095  0.0  0.0  29812  5144 pts/2    Ss+  01:15   0:00 bash

To see all the input and output of the pts we’re interested in we can call strace with the following arguments:

strace -f -s999 -e trace=write -p 7978

-f: This option tells strace to follow the system calls for the child processes as well (commands run in the terminal).

-s: by default strace truncates strings, we can set the truncate limit higher so we can see more of the output.

-e trace=write: the system calls we want to monitor. For our task we only want to see the output in the terminal. If you want to log what the user types you can add the read call as well. The format is trace=syscalls1,syscall2,..

-p: The pid of the process we’re monitoring.

The output is useful, but it would be more useful if the output was more human suitable. You’re in luck, I wrote a simple wrapper to make the output appear like a normal terminal. You can download it from here.