One-line summary
dd-parser-cleaner inspects incoming datasets, emits validated manifests describing structure and modalities, runs deterministic integrity checks, and writes a handshake file that downstream featurizers must read before transforming data.
This package provides discovery and validation for enterprise datasets. It detects dataset type (cross-sectional, event-log, panel, homogeneous/bipartite/heterogeneous graph), tags attributes with roles and modalities, validates keys and joins, and produces actionable diagnostics and remediation hints. The canonical outputs are dataset manifest, attribute manifest, and handshake.json.
- Initialize the workspace
init-workspace .- Optionally verify file placement
location-helper .- Bootstrap dataset metadata
dataset-bootstrap .This writes bootstrap_metadata.yaml and captures dataset type, subject metadata, and optional use-case answers.
- Supports tabular datasets and homogeneous graphs learnable from tabular data.
- Other graph types (bipartite/heterogeneous graphs) are not supported in this version and are explicitly marked out of scope during bootstrapping.
- Run
bootstrap-config --output config.yaml .next so the bootstrap answers are propagated intoconfig.yamlfor parser, cleaner, and notebook metadata flows.
- Generate runtime config
bootstrap-config --output config.yaml .This consumes bootstrap_metadata.yaml, discovers data and dictionary files, and writes config.yaml.
- Run the parser
classify-entities --config config.yamlThis produces parser artifacts such as:
documents/dd_analysis_results/<dataset_id>_analysis_results.csvdocuments/dd_analysis_results/<dataset_id>_dataset_manifest.jsondocuments/dd_analysis_results/<dataset_id>_attribute_manifest.jsondocuments/dd_cleaner/<dataset_id>_parser_cleaner_handshake.md
- Run the cleaner
clean-dataset --config config.yaml --action fullThis validates the manifests, produces diagnostics, and exports the synchronized dataset to:
data/dd_cleaner/<dataset_id>_clean.csv
- Featurizer must read the generated handshake file and proceed only if
status == "ready".
The repository includes example notebooks under tests/notebooks/, including a notebook that demonstrates notebook API usage for metadata bootstrap and dataset metadata exposure:
tests/notebooks/metadata_bootstrap_example.ipynb
- Dataset discovery: auto-detects
dataset_typeand primary/time keys. - Attribute tagging: emits
role,time_dependency,granularity,modality,suggested_checks,generated_key_flag. - Graph support: homogeneous, bipartite, heterogeneous graphs with entity/relationship maps.
- Longitudinal support: event-log vs panel; static vs dynamic attributes.
- Manifest emission: canonical JSON manifests for downstream deterministic featurization.
- Cleaner validations: monotonicity, lag consistency, cycle detection, relation consistency, URL/geo sanity checks.
- Handshake contract:
handshake.jsonwithstatus(ready|blocked|warnings). - Config driven: behavior controlled by
config.yamlflags.
{
"dataset_id": "orders_2026",
"dataset_type": "event_log",
"primary_key_spec": ["order_id"],
"time_key_spec": "event_time",
"entity_files": [],
"relation_files": [],
"panel_variable_map": null,
"notes": "Order events from e-commerce pipeline",
"validation_errors": []
}{
"attribute_name": "order_id",
"role": "subject_key",
"time_dependency": "none",
"granularity": null,
"modality": "categorical",
"suggested_checks": ["null_profile"],
"generated_key_flag": false
}{
"status": "ready",
"manifest_path": "manifests/orders_2026.json",
"blocking_reasons": []
}- JSON Schema files (manifest validation):
schemas/dataset_manifest.json,schemas/attribute_manifest.json,schemas/handshake.json - Workspace questionnaire config:
documents/config/dataset_questions.json - Sample manifests and fixtures:
tests/fixtures/manifests/andtests/fixtures/csvs/ - Regression coverage:
tests/test_sba_end_to_end.py,tests/test_mn_traffic_end_to_end.py, andtests/test_itsm_end_to_end.py - Docs and design:
USER_GUIDE.md,documents/, anddocs/manifest.md
Add or review these in config.yaml under a manifest section:
manifest:
require_manifest_before_featurize: true
use_case_questions_enabled: false
graph_entity_limit: 5
generate_surrogate_keys: true
url_sample_size: 10- Featurizer must read
manifests/handshake.jsonbefore any transformation. - If
status == "blocked", the featurizer must refuse to proceed. - If
status == "warnings", the featurizer may proceed only after acknowledging and recording the warnings. - Handshake metadata now includes bootstrapped dataset context for downstream assistants.
Required handshake metadata fields:
dataset_type: dataset taxonomy from bootstrap.subject: dataset subject, orNot applicablewhen missing.subject_id_attribute: subject key name, orNot applicablefor cross-sectional datasets.wide_short_homogeneous: boolean signal for wide-short grouping.wide_short_representative_column: representative column name, orNot applicablewhen not applicable.
- New manifest fields are additive and optional. Existing cross-sectional outputs remain unchanged during phased rollout.
- Recommended phased rollout:
- Emit manifests and handshake while preserving legacy outputs.
- Enable cleaner validators and handshake enforcement behind config flags.
- Deprecate legacy outputs after one release cycle.
- Missing primary key: parser will generate a surrogate key and set
generated_key_flag; prefer providing explicit keys. - Time key absent for longitudinal data: set
time_key_specor mark dataset ascross_sectional. - Relation file join mismatch: ensure
entity_key_specmatches keys referenced in relation files. - Heterogeneous graph cycle detected: convert to acyclic tree or correct relationship files.
- Invalid URLs or geo addresses: check
modalitytags and sample rows flagged in diagnostics.
Each validation error includes severity, remediation, and sample_rows in the cleaner report.
Use get_package_info() to discover:
- CLI commands and entry points
manifest_schema_pathsfor validationhandshake_specand allowedstatusvaluessupported_dataset_typesand importantconfig_flags
Treat get_package_info() as the canonical programmatic discovery endpoint.
- Issue tracker: add issues at the repository issue tracker (link in
get_package_info()output). - Contributing: follow repository CONTRIBUTING.md for tests, fixtures, and schema updates.
- Contact: open an issue for integration questions or schema clarifications.
dd-parser-cleaner inspects datasets, emits validated manifests and a handshake file describing keys, time semantics, modalities, and graph structure, and provides deterministic diagnostics so downstream featurizers can safely and reproducibly transform data.
USER_GUIDE.mdfor usage detailsdocuments/for methodology and internal design notestests/notebooks/for example notebook workflows
