Troubleshooting, checkpoints and Slurm
The inputs on this page are educational templates that have not been executed for this course. Use an authorized installation, verify version-specific options and establish your own convergence.
Diagnose the layer before changing chemistry
Parsing failures require checks of spelling, revision support, blank lines, charge/multiplicity and extra blocks. SCF issues require geometry/state and iteration history; a scheduler kill is not an algorithm failure. Optimization needs force/displacement and trajectory evidence for oscillation, dissociation or flat torsions; a step limit is not success. Resource failures require scheduler reason, wall time, memory, quota, permissions and scratch. Do not change chemistry to repair infrastructure.
For postprocessing, retain the original checkpoint, convert with a compatible utility and verify the reader. Renaming extensions does not convert data. Keep failed inputs/logs so a later workaround has a traceable reason.
Checkpoint lineage and restart
Preserve old input, log and checkpoint under distinct names, with source and termination status. Reading geometry/guess into a new job differs from restoring every internal state. %OldChk plus a new %Chk makes lineage explicit. True optimization restart needs revision-matched Opt=Restart instructions; other task types may need other scratch files. A checkpoint is not universally sufficient. Opt; CONFLEX Link 0.
For compatible files:
Verify conversion and downstream readability, retaining original and utility version. Checkpoints can contain sensitive project structures/wavefunctions; share appropriately. Gaussian utilities.
A single-node adaptation template
This unexecuted script assumes Slurm, Bash, authorized g16 on PATH, approved writable TMPDIR and direct single-node shared-memory execution. Confirm wrapper policy; some sites set defaults and discourage duplicate Link 0 controls. For this example, prepare water_batch.gjf with %NProcShared=4 and %Mem=4GB. Slurm reserves 6 GB for illustrative headroom, not a universal guarantee. Use the small water case before scaling.
#!/usr/bin/env bash
#SBATCH --job-name=water-g16
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=6G
#SBATCH --time=00:30:00
#SBATCH --output=slurm-%j.out
set -euo pipefail
# Add the authorized site-specific environment setup before submitting.
command -v g16 >/dev/null
cd "${SLURM_SUBMIT_DIR:?Run through the supported Slurm workflow}"
input="water_batch.gjf"
log="water_batch.log"
test -r "$input"
: "${TMPDIR:?Ask the site for an approved job-local scratch directory}"
test -d "$TMPDIR"
test -w "$TMPDIR"
GAUSS_SCRDIR=$(mktemp -d "$TMPDIR/gaussian.${SLURM_JOB_ID}.XXXXXX")
export GAUSS_SCRDIR
printf 'Gaussian scratch: %s\n' "$GAUSS_SCRDIR"
# Keep %Chk in the submission directory. Retain scratch for diagnosis.
set +e
g16 < "$input" > "$log" 2>&1
status=$?
set -e
printf 'Gaussian process exit status: %s\n' "$status"
printf 'Inspect %s and scheduler accounting before accepting results.\n' "$log"
exit "$status"
Replace or satisfy the environment comment with site instructions. If a wrapper is required, change the execution line as documented. Do not blindly add mpirun or launch four independent Gaussian processes. One task with four CPUs is the intended shared-memory model. The input's checkpoint stays in the submission directory.
Scratch is deliberately retained for diagnosis, but node-local storage may disappear after a job regardless of script behavior. Archive necessary evidence before cleaning only the unique job directory under site policy. Production restarts need deliberate staging/retention with the administrator; this short example is not a long-job backup policy.
Final audit
Explain the failed layer, selected remedy, parent checkpoints, resource agreement and whether the actual log confirms success. Preserve input, raw output, revision, convergence and stationary-point evidence alongside units and limitations. No input template implies a calculated numerical result. NIH Biowulf illustrates wrapper precedence; GWDG Gaussian shows another site's conventions.