Jump to content

Fixing command not found in cron jobs

From Pulsed Media Wiki


On a Pulsed Media seedbox, your SSH shell looks for commands in at least six system directories plus your own ~/.bin, ~/bin and ~/.local/bin. A job in your crontab looks in two: /usr/bin and /bin. A program installed in your home directory runs fine when you type it, and the same line in cron fails with not found and exit status 127.

The directories each one searches for a command. Anything installed only in your home directory is invisible to a cron job.

Why the command works in your shell but not in cron

When you log in over SSH, bash reads ~/.bashrc, and that file builds your PATH. Cron reads none of it. It runs each crontab line with sh (dash, on Debian), sets HOME and LOGNAME from your account, and hands the job a fixed default PATH. On Debian that default is /usr/bin:/bin.

So the line in your crontab is not run the way you typed it at the prompt. If the program lives anywhere other than those two directories, the shell writes a line ending in mytool: not found to the job's output and the job exits with status 127. Programs from system packages usually live in /usr/bin and keep working, which is why only some of your jobs break.

Why it worked for months and then stopped

The usual trigger is a program that moves. A tool you once installed system-wide switches to a per-user installer, or updates itself into ~/.local/bin. Your shell still finds it, because ~/.local/bin is in your interactive PATH, so nothing looks wrong when you test it by hand.

A long-running process can hide the change for a long time. A process started from your SSH session keeps that session's PATH, and every program it launches inherits it. While it stays up, everything it starts finds the tool. The failure appears the first time cron has to start that process again, after a reboot, a crash or an out-of-memory kill. By then the change that caused it may be days or weeks old, which is what makes it look random.

Check what your cron job actually sees

Do not guess the environment. Let cron write it down. Add this line with crontab -e:

* * * * * env > "$HOME/cron-env.txt" 2>&1

Wait a minute, then look at the result and remove the line again:

grep '^PATH=' ~/cron-env.txt

Then find out where the failing program actually is, from your normal SSH shell:

command -v mytool

If that prints a path in your home directory, such as /home/USERNAME/.local/bin/mytool, and that directory is missing from the PATH in cron-env.txt, you have found the cause. mytool stands for the command your job runs.

Three fixes

1. Use the full path in the crontab line

0 * * * * /home/USERNAME/.local/bin/mytool --sync /home/USERNAME/data

This is the simplest fix for a one-line job. It breaks if the program moves again, so prefer one of the other two when the program updates itself.

2. Set PATH at the top of the crontab

PATH=/home/USERNAME/.local/bin:/home/USERNAME/bin:/usr/bin:/bin

Put this line at the top of the crontab, above your jobs, and they run with that PATH. Write each directory out in full. Cron does not expand variables or ~ in these lines, so PATH=$HOME/bin:$PATH sets your PATH to that literal text, and every command without a full path stops being found.

3. Set PATH inside the script

If the job runs a script, set the PATH where the script starts:

#!/bin/bash
export PATH="$HOME/.local/bin:$HOME/bin:$PATH"
mytool --sync "$HOME/data"

Here the variables do expand, because the script's own shell reads them, not cron. The script now behaves the same whether you run it by hand or from cron. For anything longer than one line, this is the fix to choose: the fix lives in the file that needs it, and it still works if the crontab is rewritten.

Do not fix it by loading ~/.bashrc from the job. That file is written for interactive logins, and the job then depends on everything else in it.

A check that proves nothing: tmux and PATH

If your job starts a program inside tmux, for example with tmux new-session -d -s myapp "myapp --serve", the same problem moves one step further away, and tmux makes it look solved.

It seems natural to put the right PATH in ~/.tmux.conf:

set-environment -g PATH "/home/USERNAME/.local/bin:/usr/bin:/bin"

and to confirm it with tmux show-environment -g PATH, which prints the value back exactly. On tmux 3.5a, the version we tested, the value never reaches the program. A new pane gets the PATH of the tmux command that created it. When cron runs tmux new-session, that is cron's /usr/bin:/bin, whatever the config file says. Other variables set the same way do reach the pane. PATH is the exception.

The check passes because it reads the place where tmux stores the value, not the process that is supposed to use it. Read the process instead:

tmux list-panes -a -F '#{session_name} #{pane_pid}'
tr '\0' '\n' < /proc/PID/environ | grep '^PATH='

Replace PID with the number from the first command. That is the PATH the program in that pane really has. The same rule applies well beyond tmux: a setting is only in effect once the program that should use it shows it.

To fix a cron-started tmux session, use fix 3 for the program it runs, or give the tmux command itself a PATH with fix 2.

At Pulsed Media

Pulsed Media seedboxes run PMSS, which is open source, so how it sets up your shell can be read in its code instead of guessed. You manage your own schedule with crontab -e, and platform updates do not overwrite your crontab. The ~/.bashrc that PMSS manages puts the system directories first and then adds ~/.bin, ~/bin and ~/.local/bin if they exist. That only happens in an interactive shell. The media stack installer adds ~/.bin and its .NET runtime directory to PATH the same way, through ~/.bashrc.custom, which cron does not read either. Shell customization on PMSS explains which of those files are yours to edit.

Pulsed Media's autonomous AI sysadmin, Väinämöinen, once traced a fourteen-hour restart loop to exactly this: a program had moved into ~/.local/bin ten days earlier, and a long-running process had hidden it until cron had to start it again.

Pulsed Media seedboxes come with SSH access and your own crontab, on a platform whose code is public. See seedbox plans.

See also