Skip to content

A failing case.setup inside Case() raises only "Error running ./case.setup." and the caseroot with CaseStatus is then deleted, so the reason is lost #58

Open

Description

@manishvenu

Found by the CrocoDash 1.0.x audit (2026-11-06), against CrocoDash main b8991c89 and the copies it pins (regional-mom6 3ca8b9b, visualCaseGen 0eb6fd6, mom6_forge d74c851). Each finding was reproduced by a separate verifier; repro scripts are in the CROC monorepo under dev/crocodash_1_0_x/audit_repros/.

What's wrong. run_case_setup runs ./case.setup [--non-local] with capture_output=True. On a nonzero exit it raises RuntimeError(f"Error running {cmd}.") and drops both runout.stdout and runout.stderr, where CIME writes its "ERROR: ..." line. When this runs inside CrocoDash's Case(), the RuntimeError reaches case.py's except BaseException, and _remove_failed_attempt deletes the new caseroot and inputdir. That also removes CaseStatus, the only other place CIME recorded the error. The user sees "Error running ./case.setup --non-local." with no cause and no case directory left to inspect. The create_newcase call next to it prints runout.stderr, and xmlchange puts stderr in its exception message; only case.setup loses it.

Who hits it. Any nonzero exit of ./case.setup during Case() construction, for example a PE-layout or machine-environment problem, a permissions problem in the caseroot, or a setup problem specific to non-local cases. The repro uses a read-only LockedFiles/ dir, which makes the real CIME case.setup fail at its lock-file copy step.

Repro.

source dev/crocodash_1_0_x/audit_repros/env.sh; D=dev/crocodash_1_0_x/audit_repros/verify/vcg-a6; cd $D && timeout 900 $PY -u repro_setup_fail.py $D (script: $D/repro_setup_fail.py, log: $D/run1.log). Key lines: subprocess.run is wrapped so that, for a str cmd starting with "./case.setup", it first runs os.chmod(Path(kw['cwd'])/'LockedFiles', 0o555), then calls the REAL subprocess.run and records rc/stdout/stderr/CaseStatus. Then a tiny Case(): Grid(lenx=2, leny=2, resolution=0.5, xstart=-60, ystart=30) with flat 1000 m topo, VGrid.uniform(nk=5), compset 1850_DATM%NYF_SLND_SICE_MOM6%REGIONAL_SROF_SGLC_SWAV, atm_grid_name="T62", machine="derecho", override=True, case name audit_vcga6_1. The script prints the exception, the captured case.setup stderr, and caseroot.exists().

  • Expected: The exception raised from Case() includes case.setup's error output, here "ERROR: Command: '/usr/bin/cp -f -r .../env_mach_pes.xml .../LockedFiles/env_mach_pes.xml' failed with error '... Permission denied'". Alternatively, the CaseStatus record survives somewhere the user can read it.
  • Observed: What the user gets: "RuntimeError: RuntimeError('Error running ./case.setup --non-local.')", "cause: None", preceded by "Removing files from the failed attempt: .../input/audit_vcga6_1" and "Removing files from the failed attempt: .../cases/audit_vcga6_1". Finally "caseroot exists now: False". The wrapper shows that case.setup itself exited rc=1 with stderr "ERROR: Command: '/usr/bin/cp -f -r .../cases/audit_vcga6_1/env_mach_pes.xml .../LockedFiles/env_mach_pes.xml' failed with error '/usr/bin/cp: cannot create regular file ...: Permission denied'". The same text was in CaseStatus ("case.setup error / ERROR: Command: ..."), and both were discarded or deleted.

Root cause. case_tools.run_case_setup captures case.setup's stdout and stderr but never prints them and never puts them in the RuntimeError (and uses no text=True). CrocoDash's Case() treats any exception as a failed attempt and rmtree's the caseroot, deliberately so that a retry is not refused. That removes CaseStatus, the only on-disk copy of the CIME error. Together they leave no trace of why setup failed.

Smallest fix. In visualCaseGen case_tools.run_case_setup, run with capture_output=True, text=True. On a nonzero exit, raise RuntimeError(f"Error running {cmd} in {caseroot}:\n{runout.stderr}"), optionally with the tail of runout.stdout. This matches xmlchange in the same file and the case.setup --reset call in CROCODILE-CESM/CrocoDash#357. No change is needed in CrocoDash's cleanup once the message carries the reason.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions