--- title: "Linux Processes" description: "A reference for managing Linux processes and services. Covers ps and top for inspection, kill and pkill for signals, nice and renice for priority, cgroups and limits, and systemctl and journalctl for managing and debugging systemd units." author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/cheatsheets/linux-processes --- # Linux Processes Most of what you need day to day is inspecting what is running, sending it the right signal, and reading why it stopped. The reference below covers those in order, but two ideas are worth internalising because they change your diagnosis rather than just your command. The first is the state column. A process at 100% CPU in `R` state and one stuck in `D` state look similar in a monitoring dashboard and require completely different responses. `R` is a code problem. `D` is almost always a storage problem, and no amount of `kill -9` will help, because a process blocked in an uninterruptible syscall cannot receive signals at all. _Process states and the signals that move between them_ The second is that on any modern distribution you are usually not managing processes, you are managing systemd units. Killing a supervised process by PID looks like a crash to systemd, which dutifully restarts it, and you conclude the kill did not work. `systemctl stop` and `systemctl kill` exist so that the supervisor knows what you meant. That distinction gets sharper on a fleet, where you rarely have SSH access to the host that is misbehaving. _Diagnosing a runaway process across an EC2 fleet_ The commands in the reference work the same whether you reached the host through SSH, Session Manager, or `kubectl exec`. What changes is how you get there and how much of it you can do to a hundred hosts at once. ## Inspecting Processes Finding out what is running, what it is doing, and what it costs. ### ps, the reliable one A snapshot of the process table. Scriptable, unlike top. **Keywords:** ps, aux, -eo, pgrep #### The two portable invocations ```bash # BSD style: everything, with user and terminal ps aux # UNIX style: everything, full command line ps -ef ``` Both list all processes. `aux` shows %CPU and %MEM, `-ef` shows the parent PID (PPID), which is what you want when tracing who spawned what. - Note there is no dash in `ps aux`. Adding one changes the meaning. #### Custom columns, sorted, the triage command ```bash # Top CPU consumers with the state column ps -eo pid,ppid,stat,pcpu,pmem,rss,etime,comm --sort=-pcpu | head -15 # Top memory consumers, RSS in human terms ps -eo pid,rss,comm --sort=-rss | head -15 | awk 'NR==1{print;next}{printf "%-8s %8.1f MB %s\n", $1, $2/1024, $3}' ``` _exec_ ```bash ps -eo pid,stat,pcpu,rss,comm --sort=-pcpu | head -5 ``` _output_ ```text PID STAT %CPU RSS COMMAND 1843 Rl 182.0 2451328 java 9021 S 4.1 198432 postgres 712 Ssl 1.8 84120 containerd 1122 S 0.9 42188 sshd ``` `-eo` gives you exactly the columns you need in a stable order, which makes the output safe to parse. Sorting by `-pcpu` puts the problem first. - RSS is in kilobytes and counts shared pages once per process, so summing it across processes over-counts. - `etime` is elapsed wall time, useful for spotting a process that should have exited hours ago. #### Finding a PID without grepping ps ```bash pgrep -f 'java.*api-server' # match the full command line pgrep -u deploy nginx # only this user's nginx pgrep -l -f celery # list PID and name pgrep -c sshd # just count them ``` `pgrep` avoids the classic bug where `ps aux | grep foo` also matches the grep process itself. - `-f` matches against the whole command line, not just the executable name. Without it, `pgrep java.*api` never matches. **Best practices:** - Use `ps -eo` with explicit columns in scripts; the default column set differs between distributions. - Reach for `pgrep`/`pkill` instead of parsing `ps` output. **Common errors:** - **`ps aux | grep nginx` returns a match even when nginx is not running.**: The grep process itself matches. Use `pgrep nginx`, or `ps aux | grep [n]ginx` to break the self-match. - **RSS values sum to more than the machine has RAM.**: RSS counts shared memory in every process that maps it. Use `ps_mem` or read `/proc/PID/smaps_rollup` for a proportional figure (PSS). ### Reading the STAT column The single most useful field, and the one most often skipped. **Keywords:** STAT, zombie, uninterruptible #### What each state means ```text R running or runnable (on CPU, or waiting for one) S interruptible sleep (waiting for an event, can be signalled) D uninterruptible sleep (in a syscall, usually blocked on I/O) T stopped (by SIGSTOP/SIGTSTP, or by a debugger) Z zombie (exited, exit status not yet collected by the parent) I idle kernel thread suffixes: < high priority N low priority (nice > 0) s session leader l multi-threaded + in the foreground process group ``` D and Z are the two that change your diagnosis completely: a D-state process is blocked in the kernel and cannot be killed, and a zombie is already dead. #### Hunting D-state (blocked) processes ```bash # Anything stuck uninterruptible ps -eo pid,stat,wchan:25,comm | awk '$2 ~ /^D/' # What kernel function is it waiting in? sudo cat /proc/1843/stack # Which file is it blocked on? sudo ls -l /proc/1843/fd | tail ``` `wchan` names the kernel function the process is sleeping in. A cluster of D-state processes almost always means a storage problem: a failing disk, a hung NFS mount, or a saturated volume. - You cannot kill a D-state process. Not with -9, not with anything. Fix the I/O or reboot. #### Zombies and who is responsible ```bash # List zombies with their parent ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/' # The parent is the bug. Ask it to reap: kill -CHLD # If the parent is broken, kill the parent. init adopts and reaps. kill ``` A zombie holds only a PID slot, no memory or CPU. It exists because the parent never called `wait()`. Killing the zombie is meaningless; it has already exited. - A handful of zombies is harmless. Thousands will exhaust the PID space and stop new processes from starting. ### top and friends, live view Interactive inspection when you need to watch behaviour over seconds. **Keywords:** top, htop, pidstat #### top, the keys worth knowing ```bash top # 1 toggle per-CPU lines H show individual threads # M sort by memory P sort by CPU # c full command line k kill a PID # e cycle memory units W save your config to ~/.toprc # o filter, e.g. COMMAND=java ``` Press `1` first on a multi-core box. A process showing 400% CPU is using four cores, which is fine, or four cores spinning, which is not. #### Watching one process, not the whole system ```bash top -H -p 1843 # per-thread view of one PID pidstat -p 1843 1 5 # CPU breakdown, 1s interval, 5 samples pidstat -d -p 1843 1 # its disk I/O ``` `top -H` splits a process into threads, which is how you find the one runaway thread inside an otherwise healthy JVM. - `pidstat` comes from the `sysstat` package and separates user time from system time, which `top` blends together. **Advanced notes:** - **Load average is not CPU usage:** The three load numbers count processes that are running *or* waiting on uninterruptible I/O. A load of 40 on a 4-core box with 2% CPU means you are I/O bound, not CPU bound. Compare load against core count (`nproc`), and always read it alongside `%iowait`. ## Signals and Termination Asking a process to stop, and the difference between asking and forcing. ### kill, pkill, killall Sending signals to one process or many. **Keywords:** kill, pkill, SIGTERM, SIGKILL #### The signals you will actually use ```bash kill 1843 # SIGTERM (15), the default. Polite, catchable. kill -TERM 1843 # same thing, explicit kill -HUP 1843 # SIGHUP (1), conventionally "reload config" kill -QUIT 1843 # SIGQUIT (3), quit + core dump (JVM: thread dump) kill -KILL 1843 # SIGKILL (9), last resort. Cannot be caught. kill -STOP 1843 # freeze kill -CONT 1843 # resume kill -USR1 1843 # app-defined (nginx: reopen logs) ``` SIGTERM lets the process run its shutdown handler, flush buffers, release locks, and remove its pidfile. SIGKILL is delivered by the kernel and gives it no chance to do any of that. - `kill -l` lists every signal name and number on your system. #### The right escalation ```bash # Ask, wait, then insist. kill -TERM "$PID" for _ in $(seq 1 15); do kill -0 "$PID" 2>/dev/null || { echo "exited cleanly"; exit 0; } sleep 1 done echo "still alive after 15s, forcing" kill -KILL "$PID" ``` `kill -0` sends no signal; it just tests whether you could. That makes it the correct way to poll for "has it exited yet". - Straight to `-9` is the habit worth breaking. It is why you find stale lock files and half-written data. #### Killing by name and pattern ```bash pkill nginx # by exact process name pkill -f 'python.*worker.py' # by full command line pkill -u deploy # everything owned by a user pkill -e -f celery # -e echoes what it killed killall -TERM nginx # by name; different arg style killall -w nginx # wait for them to actually die ``` `pkill -f` matches against the whole command line, which is the only way to distinguish two `python` processes running different scripts. - Always test the pattern with `pgrep -f` before running `pkill -f`. `pkill -f python` will happily kill your own tooling. **Common errors:** - **kill -9 does nothing; the process stays in the list.**: Check the STAT column. `Z` means it is already dead and waiting to be reaped. `D` means it is blocked in the kernel and unkillable until the I/O completes. - **The service dies and immediately comes back.**: systemd is restarting it. Use `systemctl stop unit` instead of `kill`, or the `Restart=` policy will undo you every time. - **`Operation not permitted` when killing your own process.**: It is likely a setuid process, or you are inside a container targeting a PID on the host. Check `/proc/PID/status` for the real UID. ### Trapping signals in scripts Making your own scripts shut down cleanly. #### Clean up on exit, however it happens ```bash #!/usr/bin/env bash set -euo pipefail workdir=$(mktemp -d) # EXIT fires on normal exit AND after INT/TERM handlers run, # so one trap covers every path out of the script. trap 'rm -rf "$workdir"' EXIT trap 'echo "interrupted, shutting down"; exit 130' INT TERM do_work_in "$workdir" ``` A single `EXIT` trap for cleanup plus an `INT TERM` trap for the message is the pattern that covers every exit path, including `set -e` failures. - You cannot trap SIGKILL or SIGSTOP. If your cleanup must survive that, it belongs in a separate reaper, not a trap. ## Priority and Limits Controlling how much CPU, memory, and I/O a process may take. ### nice and renice CPU scheduling priority, from -20 (greedy) to 19 (polite). **Keywords:** nice, renice, ionice #### Starting and changing priority ```bash nice -n 19 ./batch-job.sh # start it as low priority renice -n 10 -p 1843 # lower an existing process sudo renice -n -5 -p 1843 # raise it (needs root) renice -n 15 -u backup # every process of a user ionice -c 3 -p 1843 # idle I/O class: only when disk is free ionice -c 2 -n 7 ./rsync-job # best-effort, lowest priority ``` Positive nice values are polite and any user can set them. Negative values are greedy and require root. `ionice` is the disk equivalent and is often the one that actually helps, since batch jobs are usually I/O bound rather than CPU bound. - Unprivileged users can only ever *increase* niceness. You cannot undo your own `nice -n 19`. - nice affects CPU only. A niced process can still saturate your disk; that is what `ionice` is for. ### ulimit and cgroups Hard ceilings, per shell or per service. #### Per-shell limits with ulimit ```bash ulimit -a # everything, current shell ulimit -n # max open file descriptors ulimit -u # max user processes ulimit -n 65535 # raise the soft limit (up to the hard limit) ulimit -Hn # show the hard limit ``` `ulimit` applies to the current shell and anything it spawns. It does not retroactively affect a running service. - For services, `ulimit` in a shell profile is ignored. Set `LimitNOFILE=` in the systemd unit instead. #### Real ceilings with cgroups v2, via systemd ```bash # Cap a running service without editing files sudo systemctl set-property api.service MemoryMax=2G CPUQuota=150% # Run an ad-hoc command inside a limited scope sudo systemd-run --scope -p MemoryMax=512M -p CPUQuota=50% ./import.sh # What is this unit actually consuming? systemctl status api.service | grep -E 'Memory|CPU|Tasks' cat /sys/fs/cgroup/system.slice/api.service/memory.current ``` cgroups are the only enforcement that actually holds. `CPUQuota=150%` means one and a half cores, and `MemoryMax` triggers the OOM killer inside the cgroup rather than taking down the host. - `MemoryHigh=` throttles under pressure; `MemoryMax=` is a hard wall that kills. Prefer High plus a generous Max. **Advanced notes:** - **Finding out who the OOM killer chose:** When memory runs out, the kernel kills the process with the worst `oom_score`. Read the verdict with `journalctl -k | grep -i "killed process"` or `dmesg -T | grep -i oom`. To protect a process, write a negative value to `/proc/PID/oom_score_adj` (or set `OOMScoreAdjust=` in its unit). Protecting the wrong process just moves the kill to something else, so prefer fixing the memory ceiling. ## systemd Services On any modern distribution, most long-running processes are units, and you manage them through systemd rather than by PID. ### systemctl Start, stop, inspect, and enable units. **Keywords:** systemctl, unit, enable, mask #### Daily commands ```bash systemctl status api.service # state, PID, recent logs systemctl start|stop|restart api # .service is implied systemctl reload api # re-read config, no restart systemctl reload-or-restart api # reload if supported, else restart systemctl enable api # start at boot systemctl enable --now api # enable AND start right now systemctl disable api # do not start at boot systemctl mask api # make it impossible to start systemctl unmask api ``` `enable` and `start` are independent. Forgetting `enable` is why a service that works today is missing after a reboot. - `mask` symlinks the unit to /dev/null. It is the only way to stop something another unit keeps pulling in. #### Finding and understanding units ```bash systemctl list-units --type=service --state=running systemctl list-units --failed # what is broken systemctl list-unit-files --state=enabled # what starts at boot systemctl cat api.service # the effective unit file systemctl show api.service -p Restart -p ExecStart systemctl list-dependencies api.service ``` `systemctl cat` shows the unit plus every drop-in override in one view, which is what you want before wondering why a setting is not taking effect. - Never edit files in `/lib/systemd/system` directly. Use `systemctl edit api` to create a drop-in under `/etc/systemd/system/api.service.d/`. #### Signals through systemd ```bash systemctl kill api.service # SIGTERM to the main process systemctl kill --signal=SIGQUIT api.service # a specific signal systemctl kill --kill-whom=all --signal=SIGKILL api.service ``` Prefer this over `kill` for anything systemd supervises, because a plain `kill` looks like a crash and triggers the `Restart=` policy. **Common errors:** - **Changed the unit file, and nothing happened.**: Run `systemctl daemon-reload` first, then restart the unit. systemd caches parsed units. - **Unit is `activating (auto-restart)` in a loop.**: It crashes on startup and `Restart=always` keeps retrying. Read `journalctl -u unit -n 50`, then `systemctl stop` it to break the loop while you fix the cause. ### journalctl Reading the logs, which is where the actual reason lives. **Keywords:** journalctl, -u, -p, since #### The queries that answer real questions ```bash journalctl -u api.service -n 100 # last 100 lines for one unit journalctl -u api.service -f # follow, like tail -f journalctl -u api.service --since "10 min ago" journalctl -u api.service -p err # errors and worse only journalctl -u api.service -b -1 # from the PREVIOUS boot journalctl -k # kernel messages only journalctl _PID=1843 # by PID journalctl -u api --since today -o json-pretty | jq .MESSAGE ``` `-p err` and `--since` together cut a wall of logs down to the few lines that matter. `-b -1` is how you read what happened before an unexpected reboot. - Priorities run 0 (emerg) to 7 (debug). `-p warning` includes warning, err, crit, alert and emerg. #### Keeping the journal from eating the disk ```bash journalctl --disk-usage sudo journalctl --vacuum-size=500M # trim to a size sudo journalctl --vacuum-time=14d # trim by age # Permanent, in /etc/systemd/journald.conf: # SystemMaxUse=1G # MaxRetentionSec=1month ``` A journal with no cap will happily grow until the root filesystem is full, which then breaks everything else in confusing ways. **Best practices:** - Reach for `journalctl -u -p err --since` before anything else when a service misbehaves. - Set `SystemMaxUse=` on every host you build, before you need it.