A surprising number of "the cluster is broken" tickets are job scripts that never had a chance: wrong directory, stdout path that does not exist,
mpirun that ignored the allocation, walltime copied from a colleague's 48-hour production run into a compile test.The five lines that matter#
#SBATCH --output=pointing at a directory that exists.- An explicit
cdto scratch. module purgethen a known-good stack.srun(or the site launcher), not a nakedmpirun.- A walltime you measured, plus a margin, not a round number from folklore.
cwd is not the submission directory you remember#
If you
sbatch ../scripts/run.slurm from a random folder, relative paths inside the script follow Slurm's policy, not your intuition. Either cd "$SLURM_SUBMIT_DIR" or cd to an absolute scratch path. Pick one and stop mixing.Launchers#
mpirun -np 32 ./a.out inside an sbatch allocation sometimes works. On a multi-rail InfiniBand machine with cgroups, it sometimes spreads ranks onto cores you do not own, or onto the login image. srun ./a.out asks the daemon that owns the job to start the ranks. That is the difference between a lucky workstation habit and a cluster program. The flags themselves are in the sbatch man page.Arrays instead of a shell for-loop#
for i in $(seq 0 999); do sbatch run.sh $i; done makes the scheduler, the filesystem, and the support team sad. Job arrays exist. Use %20 if you must cap concurrency.After it dies#
sacct -j JOBID before you change the code. TIMEOUT, OOM, and a Python traceback in the log are different jobs. Fix the matching resource, not all of them at once.