Skip to content

About

Utility to parse data dictionaries and generate summary reports for tabular datasets

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

70 Commits

Folders and files

Repository files navigation

dd-parser-cleaner

One-line summary
dd-parser-cleaner inspects incoming datasets, emits validated manifests describing structure and modalities, runs deterministic integrity checks, and writes a handshake file that downstream featurizers must read before transforming data.

Purpose

This package provides discovery and validation for enterprise datasets. It detects dataset type (cross-sectional, event-log, panel, homogeneous/bipartite/heterogeneous graph), tags attributes with roles and modalities, validates keys and joins, and produces actionable diagnostics and remediation hints. The canonical outputs are dataset manifest, attribute manifest, and handshake.json.

dd-parser-cleaner schematic

Quick start (workflow)

  1. Initialize the workspace
init-workspace .
  1. Optionally verify file placement
location-helper .
  1. Bootstrap dataset metadata
dataset-bootstrap .

This writes bootstrap_metadata.yaml and captures dataset type, subject metadata, and optional use-case answers.

  • Supports tabular datasets and homogeneous graphs learnable from tabular data.
  • Other graph types (bipartite/heterogeneous graphs) are not supported in this version and are explicitly marked out of scope during bootstrapping.
  • Run bootstrap-config --output config.yaml . next so the bootstrap answers are propagated into config.yaml for parser, cleaner, and notebook metadata flows.
  1. Generate runtime config
bootstrap-config --output config.yaml .

This consumes bootstrap_metadata.yaml, discovers data and dictionary files, and writes config.yaml.

  1. Run the parser
classify-entities --config config.yaml

This produces parser artifacts such as:

  • documents/dd_analysis_results/<dataset_id>_analysis_results.csv
  • documents/dd_analysis_results/<dataset_id>_dataset_manifest.json
  • documents/dd_analysis_results/<dataset_id>_attribute_manifest.json
  • documents/dd_cleaner/<dataset_id>_parser_cleaner_handshake.md
  1. Run the cleaner
clean-dataset --config config.yaml --action full

This validates the manifests, produces diagnostics, and exports the synchronized dataset to:

  • data/dd_cleaner/<dataset_id>_clean.csv
  1. Featurizer must read the generated handshake file and proceed only if status == "ready".

Notebook examples

The repository includes example notebooks under tests/notebooks/, including a notebook that demonstrates notebook API usage for metadata bootstrap and dataset metadata exposure:

  • tests/notebooks/metadata_bootstrap_example.ipynb

Key capabilities

  • Dataset discovery: auto-detects dataset_type and primary/time keys.
  • Attribute tagging: emits role, time_dependency, granularity, modality, suggested_checks, generated_key_flag.
  • Graph support: homogeneous, bipartite, heterogeneous graphs with entity/relationship maps.
  • Longitudinal support: event-log vs panel; static vs dynamic attributes.
  • Manifest emission: canonical JSON manifests for downstream deterministic featurization.
  • Cleaner validations: monotonicity, lag consistency, cycle detection, relation consistency, URL/geo sanity checks.
  • Handshake contract: handshake.json with status (ready | blocked | warnings).
  • Config driven: behavior controlled by config.yaml flags.

Example artifacts

Example dataset manifest (snippet)

{
  "dataset_id": "orders_2026",
  "dataset_type": "event_log",
  "primary_key_spec": ["order_id"],
  "time_key_spec": "event_time",
  "entity_files": [],
  "relation_files": [],
  "panel_variable_map": null,
  "notes": "Order events from e-commerce pipeline",
  "validation_errors": []
}

Example attribute manifest entry

{
  "attribute_name": "order_id",
  "role": "subject_key",
  "time_dependency": "none",
  "granularity": null,
  "modality": "categorical",
  "suggested_checks": ["null_profile"],
  "generated_key_flag": false
}

Example handshake.json

{
  "status": "ready",
  "manifest_path": "manifests/orders_2026.json",
  "blocking_reasons": []
}

Where to find schemas and examples

  • JSON Schema files (manifest validation): schemas/dataset_manifest.json, schemas/attribute_manifest.json, schemas/handshake.json
  • Workspace questionnaire config: documents/config/dataset_questions.json
  • Sample manifests and fixtures: tests/fixtures/manifests/ and tests/fixtures/csvs/
  • Regression coverage: tests/test_sba_end_to_end.py, tests/test_mn_traffic_end_to_end.py, and tests/test_itsm_end_to_end.py
  • Docs and design: USER_GUIDE.md, documents/, and docs/manifest.md

Important config flags (defaults)

Add or review these in config.yaml under a manifest section:

manifest:
  require_manifest_before_featurize: true
  use_case_questions_enabled: false
  graph_entity_limit: 5
  generate_surrogate_keys: true
  url_sample_size: 10

Handshake contract (featurizer requirements)

  • Featurizer must read manifests/handshake.json before any transformation.
  • If status == "blocked", the featurizer must refuse to proceed.
  • If status == "warnings", the featurizer may proceed only after acknowledging and recording the warnings.
  • Handshake metadata now includes bootstrapped dataset context for downstream assistants.

Required handshake metadata fields:

  • dataset_type: dataset taxonomy from bootstrap.
  • subject: dataset subject, or Not applicable when missing.
  • subject_id_attribute: subject key name, or Not applicable for cross-sectional datasets.
  • wide_short_homogeneous: boolean signal for wide-short grouping.
  • wide_short_representative_column: representative column name, or Not applicable when not applicable.

Migration and compatibility

  • New manifest fields are additive and optional. Existing cross-sectional outputs remain unchanged during phased rollout.
  • Recommended phased rollout:
  1. Emit manifests and handshake while preserving legacy outputs.
  2. Enable cleaner validators and handshake enforcement behind config flags.
  3. Deprecate legacy outputs after one release cycle.

Troubleshooting (common validation failures)

  • Missing primary key: parser will generate a surrogate key and set generated_key_flag; prefer providing explicit keys.
  • Time key absent for longitudinal data: set time_key_spec or mark dataset as cross_sectional.
  • Relation file join mismatch: ensure entity_key_spec matches keys referenced in relation files.
  • Heterogeneous graph cycle detected: convert to acyclic tree or correct relationship files.
  • Invalid URLs or geo addresses: check modality tags and sample rows flagged in diagnostics.

Each validation error includes severity, remediation, and sample_rows in the cleaner report.

How clients and agents should use get_package_info()

Use get_package_info() to discover:

  • CLI commands and entry points
  • manifest_schema_paths for validation
  • handshake_spec and allowed status values
  • supported_dataset_types and important config_flags

Treat get_package_info() as the canonical programmatic discovery endpoint.

Support and contribution

  • Issue tracker: add issues at the repository issue tracker (link in get_package_info() output).
  • Contributing: follow repository CONTRIBUTING.md for tests, fixtures, and schema updates.
  • Contact: open an issue for integration questions or schema clarifications.

One-line blurb for top-level README

dd-parser-cleaner inspects datasets, emits validated manifests and a handshake file describing keys, time semantics, modalities, and graph structure, and provides deterministic diagnostics so downstream featurizers can safely and reproducibly transform data.

Existing quick links

  • USER_GUIDE.md for usage details
  • documents/ for methodology and internal design notes
  • tests/notebooks/ for example notebook workflows

About

Utility to parse data dictionaries and generate summary reports for tabular datasets

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages