Skip to content

Add module introspection feature for automated algorithm discovery - #141

Open
jackyko1991 wants to merge 1 commit into
bilayer-containers:hackathonfrom
jackyko1991:module-introspection
Open

jackyko1991 wants to merge 1 commit into
bilayer-containers:hackathonfrom
jackyko1991:module-introspection

Conversation

@jackyko1991

Copy link
Copy Markdown

This PR implements a complete module introspection system that automatically discovers Python functions from libraries (scikit-image, scipy, etc.) and generates ready-to-run Docker containers.

Key Features

  • Automated Discovery: Introspect Python modules in isolated Docker containers
  • Smart Type Inference: Parse docstrings and use heuristics to identify data inputs vs parameters
  • General-Purpose I/O: Support for TIFF, PNG, JPG, and OME-TIFF formats
  • Streaming Output: Real-time execution logs in both console and web UI
  • Containerized Workflow: Complete Docker-based pipeline from discovery to deployment

Components Added

  • src/bilayers/introspection/: Core introspection modules
    • discovery.py: Function metadata extraction
    • resolver.py: Package source resolution (PyPI, Git, local)
    • scaffold.py: Algorithm folder and wrapper generation
    • containerize.py: Docker image building
  • docs/tool_user/module_introspection.md: Complete user documentation
  • CLI commands: discover, scaffold, build-algorithm
  • Tests for discovery, resolver, and scaffolding

Workflow Example

# Discover functions from scikit-image
bilayers_cli discover skimage.filters --source "pypi:scikit-image" --filter "gaussian"

# Generate complete algorithm folder
bilayers_cli scaffold gaussian --source "pypi:scikit-image"

# Build and serve via Docker
./scripts/build_docker.sh --algorithm gaussian --interface gradio 
docker run -it --rm -p 7878:7878 bilayer/gaussian:1.0.0-gradio

Gradio Screenshot

scikit-image gaussian function served via bilayer in gradio interface:
image

Technical Improvements

  • Enhanced Gradio template with streaming subprocess output
  • Smart parameter classification (inputs vs parameters vs outputs)
  • Type mapping aligned with Bilayers template expectations
  • Progress logging throughout execution pipeline
  • Hidden args support for output directory management

This PR implements a complete module introspection system that automatically discovers Python functions from libraries (scikit-image, scipy, etc.) and generates ready-to-run Docker containers.

## Key Features

- **Automated Discovery**: Introspect Python modules in isolated Docker containers
- **Smart Type Inference**: Parse docstrings and use heuristics to identify data inputs vs parameters
- **General-Purpose I/O**: Support for TIFF, PNG, JPG, and OME-TIFF formats
- **Streaming Output**: Real-time execution logs in both console and web UI
- **Containerized Workflow**: Complete Docker-based pipeline from discovery to deployment

## Components Added

- `src/bilayers/introspection/`: Core introspection modules
  - `discovery.py`: Function metadata extraction
  - `resolver.py`: Package source resolution (PyPI, Git, local)
  - `scaffold.py`: Algorithm folder and wrapper generation
  - `containerize.py`: Docker image building
- `docs/tool_user/module_introspection.md`: Complete user documentation
- CLI commands: `discover`, `scaffold`, `build-algorithm`
- Tests for discovery, resolver, and scaffolding

## Workflow Example

\`\`\`bash
# Discover functions from scikit-image
bilayers_cli discover skimage.filters --source "pypi:scikit-image" --filter "gaussian"

# Generate complete algorithm folder
bilayers_cli scaffold gaussian --source "pypi:scikit-image"

# Build and serve via Docker
./scripts/build_docker.sh --algorithm gaussian --interface gradio
docker run -it --rm -p 7878:7878 bilayer/gaussian:1.0.0-gradio
\`\`\`

## Technical Improvements

- Enhanced Gradio template with streaming subprocess output
- Smart parameter classification (inputs vs parameters vs outputs)
- Type mapping aligned with Bilayers template expectations
- Progress logging throughout execution pipeline
- Hidden args support for output directory management

Co-authored-by: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant