Repository navigation
Description
Found by the CrocoDash 1.0.x audit (2026-11-06), against CrocoDash main b8991c89 and the copies it pins (regional-mom6 3ca8b9b, visualCaseGen 0eb6fd6, mom6_forge d74c851). Each finding was reproduced by a separate verifier; repro scripts are in the CROC monorepo under dev/crocodash_1_0_x/audit_repros/.
- Repo: visualCaseGen
- Severity: misleading error
- Evidence: confirmed by repro (audit id
vcg-a6) - Confidence: high
- Location: CrocoDash/visualCaseGen/visualCaseGen/custom_widget_types/case_tools.py:48 (subprocess.run with capture_output=True), CrocoDash/visualCaseGen/visualCaseGen/custom_widget_types/case_tools.py:50 (raise without output). Reached from CrocoDash/visualCaseGen/visualCaseGen/custom_widget_types/case_creator.py:166 <- CrocoDash/case.py:483 (cc.create_case) <- CrocoDash/case.py:220. The deletion is at CrocoDash/case.py:238-239 -> CrocoDash/case.py:41-49 (_remove_failed_attempt).
- Already fixed on main: No. The pin 0eb6fd6 equals ESMCI/visualCaseGen main (checked with gh api: main sha 0eb6fd6, same run_case_setup code), and no open PR in ESMCI/visualCaseGen fixes it. Reset case.setup after setting ntasks_ocn CROCODILE-CESM/CrocoDash#357 adds a separate case.setup --reset that does include stderr but leaves run_case_setup unchanged.
- Related: visualCaseGen Create New Case capture output? #25 (create_newcase output capture; that path now prints stderr). visualCaseGen Confusing error messages in VCG Case Creation #21 / TRIAGE item 12 (double misleading error from a failed CaseCreator; different bug). Reset case.setup after setting ntasks_ocn CROCODILE-CESM/CrocoDash#357 / Case(ntasks_ocn=...) leaves a case that won't build until case.setup --reset CROCODILE-CESM/CrocoDash#354 / TRIAGE item 2 (adds its own case.setup --reset with stderr in the message; the pattern to copy). visualCaseGen Miscellaneous pre-existing issues found while reviewing #45 #47 (other small case_creator defects).
What's wrong. run_case_setup runs ./case.setup [--non-local] with capture_output=True. On a nonzero exit it raises RuntimeError(f"Error running {cmd}.") and drops both runout.stdout and runout.stderr, where CIME writes its "ERROR: ..." line. When this runs inside CrocoDash's Case(), the RuntimeError reaches case.py's except BaseException, and _remove_failed_attempt deletes the new caseroot and inputdir. That also removes CaseStatus, the only other place CIME recorded the error. The user sees "Error running ./case.setup --non-local." with no cause and no case directory left to inspect. The create_newcase call next to it prints runout.stderr, and xmlchange puts stderr in its exception message; only case.setup loses it.
Who hits it. Any nonzero exit of ./case.setup during Case() construction, for example a PE-layout or machine-environment problem, a permissions problem in the caseroot, or a setup problem specific to non-local cases. The repro uses a read-only LockedFiles/ dir, which makes the real CIME case.setup fail at its lock-file copy step.
Repro.
source dev/crocodash_1_0_x/audit_repros/env.sh; D=dev/crocodash_1_0_x/audit_repros/verify/vcg-a6; cd $D && timeout 900 $PY -u repro_setup_fail.py $D (script: $D/repro_setup_fail.py, log: $D/run1.log). Key lines: subprocess.run is wrapped so that, for a str cmd starting with "./case.setup", it first runs os.chmod(Path(kw['cwd'])/'LockedFiles', 0o555), then calls the REAL subprocess.run and records rc/stdout/stderr/CaseStatus. Then a tiny Case(): Grid(lenx=2, leny=2, resolution=0.5, xstart=-60, ystart=30) with flat 1000 m topo, VGrid.uniform(nk=5), compset 1850_DATM%NYF_SLND_SICE_MOM6%REGIONAL_SROF_SGLC_SWAV, atm_grid_name="T62", machine="derecho", override=True, case name audit_vcga6_1. The script prints the exception, the captured case.setup stderr, and caseroot.exists().
- Expected: The exception raised from Case() includes case.setup's error output, here "ERROR: Command: '/usr/bin/cp -f -r .../env_mach_pes.xml .../LockedFiles/env_mach_pes.xml' failed with error '... Permission denied'". Alternatively, the CaseStatus record survives somewhere the user can read it.
- Observed: What the user gets: "RuntimeError: RuntimeError('Error running ./case.setup --non-local.')", "cause: None", preceded by "Removing files from the failed attempt: .../input/audit_vcga6_1" and "Removing files from the failed attempt: .../cases/audit_vcga6_1". Finally "caseroot exists now: False". The wrapper shows that case.setup itself exited rc=1 with stderr "ERROR: Command: '/usr/bin/cp -f -r .../cases/audit_vcga6_1/env_mach_pes.xml .../LockedFiles/env_mach_pes.xml' failed with error '/usr/bin/cp: cannot create regular file ...: Permission denied'". The same text was in CaseStatus ("case.setup error / ERROR: Command: ..."), and both were discarded or deleted.
Root cause. case_tools.run_case_setup captures case.setup's stdout and stderr but never prints them and never puts them in the RuntimeError (and uses no text=True). CrocoDash's Case() treats any exception as a failed attempt and rmtree's the caseroot, deliberately so that a retry is not refused. That removes CaseStatus, the only on-disk copy of the CIME error. Together they leave no trace of why setup failed.
Smallest fix. In visualCaseGen case_tools.run_case_setup, run with capture_output=True, text=True. On a nonzero exit, raise RuntimeError(f"Error running {cmd} in {caseroot}:\n{runout.stderr}"), optionally with the tail of runout.stdout. This matches xmlchange in the same file and the case.setup --reset call in CROCODILE-CESM/CrocoDash#357. No change is needed in CrocoDash's cleanup once the message carries the reason.
Activity