Processes, signals and systemd services

What is a process?

A program is a file on disk. A process is that program while it is running, with its own memory and a unique number called the PID (process ID). Running the same program twice makes two processes with two PIDs. Every process has a parent; the first process (PID 1) on most modern Linux systems is systemd.

ps aux                     # every process: user, PID, %CPU, %MEM, command
ps aux | grep nginx        # just the nginx processes
pgrep -a node              # PIDs and command lines of node processes
top                        # live view, sorted by CPU (press M to sort by memory, q to quit)
htop                       # a friendlier top, if installed

Signals: asking a process to stop

A signal is a short message the kernel delivers to a process. kill sends signals, despite its name.

kill 4321          # send SIGTERM (15) to PID 4321
kill -9 4321       # send SIGKILL, only if SIGTERM did not work
pkill -f "node server.js"   # signal processes by command line

Exit codes

When a process ends, it returns an exit code (exit status). 0 means success; anything from 1 to 255 means some kind of failure. CI pipelines rely on this: if any command exits with non-zero, the step fails. Some codes are worth remembering: 127 means "command not found", and 128 plus a signal number means "killed by that signal", so 137 is 128 + 9 (SIGKILL, often the out-of-memory killer in containers) and 130 is 128 + 2 (Ctrl+C).

ls /nope
echo $?            # prints 2: ls failed
true; echo $?      # prints 0

Foreground and background

A command normally runs in the foreground: the shell waits for it. Add & to run it in the background and get the prompt back. jobs lists background jobs, fg brings one back, and Ctrl+Z pauses the current job (bg resumes it in the background). If you log out, background jobs usually die. nohup keeps a command running after logout, but for real services use systemd instead.

nohup ./import-data.sh > import.log 2>&1 &
jobs
fg %1

systemd services

systemd is the service manager on almost all modern distributions. A unit file describes a service: what to run, as which user, and what to do if it crashes. systemd starts it at boot, restarts it on failure and collects its logs.

# /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
After=network.target

[Service]
User=deploy
WorkingDirectory=/opt/orders-api
ExecStart=/usr/bin/node server.js
Environment=NODE_ENV=production
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload            # after creating or editing a unit file
sudo systemctl enable --now orders-api  # start now AND at every boot
sudo systemctl status orders-api        # running? since when? last log lines
sudo systemctl restart orders-api       # after a deploy
sudo systemctl stop orders-api
journalctl -u orders-api -f             # follow this service's logs
journalctl -u orders-api --since "1 hour ago"

Checking resources

free -h        # memory: total, used, available
df -h          # disk space per filesystem
du -sh /var/log/*   # size of each item in /var/log
uptime         # time since boot and load average
nproc          # number of CPU cores

uptime shows three load averages, for the last 1, 5 and 15 minutes. Load is roughly the number of processes running or waiting to run. Compare it with the number of cores from nproc: a load of 4 on a 4-core server means the CPUs are fully busy; a load of 12 on 4 cores means work is queuing up.