Repository navigation
Fix gemara yaml output - #15306
Open
ggbecker wants to merge 4 commits into
Open
Fix gemara yaml output#15306ggbecker wants to merge 4 commits into
ggbecker wants to merge 4 commits into
Conversation
Commit 8b37e9d replaced per-script ruamel.yaml configuration with a shared configure_yaml() helper imported from gemara.yaml_output, but the new module was never added to git. A fresh checkout, and therefore the export-to-gemara workflow, fails with: ModuleNotFoundError: No module named 'gemara.yaml_output' Add the module so export_to_gemara.py, generate_policies_pr.py, generate_complyctl_bundle.py and test_gemara_export.py can import it.
The workflow reads --oscal-catalog and --data-dir files that are gitignored (utils/nist_sync/.gitignore) and produced by download_oscal.py. The sibling cis-nist-sync workflow runs that script; this one never did, so the export silently skipped guidance_catalog.yaml and still exited 0. Install requests (already listed in utils/nist_sync/requirements.txt) and run the download step. Also import every nist_sync script before the export, so a module that was never committed fails the job with a clear message instead of a traceback from inside export_to_gemara.py (see ComplianceAsCode#15115, where gemara/yaml_output.py was missing).
cue rejects an absolute package path, so cue_validate runs cue with cwd=schema_dir. The artifact path was passed through unchanged, so a relative --output-dir (as used by the export-to-gemara workflow) was resolved inside the schema clone and every validation failed with "stat <schema_dir>/build/gemara/...: no such file or directory". Resolve the path first; CUE validation then passes for the control catalogs and the guidance catalog.
The export-to-gemara workflow only runs on push to master and on a schedule, so a broken nist_sync script (like the missing gemara/yaml_output.py in ComplianceAsCode#15115) is invisible until after merge. Add a PR-triggered gate that, on changes under utils/nist_sync/ or the NIST 800-53 control files: - imports every script (--help) so a missing module fails fast, - runs the full export with --validate and fails on any [WARN]/[ERROR]/ FAIL marker (the export itself only warns and exits 0), - runs the export test suite, - runs generate_complyctl_bundle.py and generate_policies_pr.py. The gate uses the in-Python structural validation rather than CUE, so it needs no external schema/binary pins; authoritative CUE validation stays in export-to-gemara.yml.
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
ggbecker
marked this pull request as ready for review
November 9, 2026 14:48
ggbecker
requested review from
Mab879,
jan-cerny,
marcusburghardt and
vojtapolasek
as code owners
November 9, 2026 14:49
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Mab879
requested changes
Nov 9, 2026
Mab879
left a comment
Mab879
left a comment
Member
There was a problem hiding this comment.
Please fix the yaml lint errors.
| - name: Generate a complyctl bundle | ||
| run: | | ||
| python3 utils/nist_sync/generate_complyctl_bundle.py \ | ||
| --product rhel9 \ |
Collaborator
There was a problem hiding this comment.
Why is there rhel9 hardcoded here?
| from ruamel.yaml import YAML | ||
|
|
||
|
|
||
| # The shared ComplyTime MegaLinter configuration extends yamllint's default |
Collaborator
There was a problem hiding this comment.
I think this comment should be moved inside the function.
vojtapolasek
requested changes
Nov 10, 2026
vojtapolasek
left a comment
vojtapolasek
left a comment
Collaborator
There was a problem hiding this comment.
Please fix the yamllint failures and see my two small comments.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
utils/nist_sync/gemara/yaml_output.py, which was imported but never committed in nist_sync: split ControlCatalogs by NIST family #15115, breaking thenist_syncscripts on a fresh checkout.pull_requestgate (gate_nist_sync.yml) that runs the NIST 800-53 → Gemara scripts end to end, so a broken script is caught before merge.Rationale:
ModuleNotFoundError: No module named 'gemara.yaml_output', and since it only runs onmaster/schedule, nothing caught it on the PR. Introduced by nist_sync: split ControlCatalogs by NIST family #15115.Review Hints:
cue vetpath fix, the new gate).--validate(no CUE pins) and hard-fails on any[WARN]/[ERROR]/FAIL, since the export itself only warns and exits 0.