Stage Tracker - Pipeline Execution Monitor
A lightweight, zero-dependency Python library for tracking multi-stage pipeline executions with UUID-based run identification and structured JSON output.
View on Github
View on PyPi
Problem & Solution
The Problem
Complex pipelines—whether AI/ML experiments, cybersecurity workflows, or ETL jobs—lack standardized tracking mechanisms. Debugging failures, reproducing results, or understanding execution flow becomes challenging without structured logging. Most solutions require heavy dependencies or framework lock-in, adding overhead to simple projects.
The Solution
Stage Tracker provides a minimal, framework-agnostic API for tracking pipeline executions. Each run gets a unique UUID, stages record precise ISO timestamps, and outputs serialize to structured JSON. With zero external dependencies, it integrates seamlessly into existing codebases—whether you're tracking reinforcement learning episodes, penetration testing phases, or data processing batches.
Technologies Used
Core Technology
- Pure Python 3.8+
Key Features
Zero-Dependency Architecture
Designed entirely using Python's standard library (uuid, datetime, json) with no external dependencies, ensuring compatibility across environments and eliminating version conflicts or installation friction.
Pip-Installable Package
Published to PyPI as a production-ready pip package with proper versioning, enabling seamless integration into existing projects with a single pip install stage-tracker command.
Precise Timing & Exception Tracking
Implemented microsecond-precision ISO 8601 timestamps for stage start/end times plus automatic exception recording with full tracebacks, enabling detailed performance analysis and failure debugging.
UUID-Based Run Correlation
Built unique UUID4 identification for each execution, allowing correlation across distributed systems, log aggregation, and tracing complex multi-step workflows across different services or machines.
Structured JSON Serialization
Designed comprehensive JSON schema capturing arguments, timestamps, stage sequences, and exceptions—ideal for parsing with analytics tools, databases, or visualization dashboards for experiment tracking.
Primary Use Cases
- AI/ML Experiment Tracking: Monitor training episodes, hyperparameter sweeps, and model evaluation stages with reproducible run IDs
- Cybersecurity Pipelines: Track penetration testing phases (reconnaissance → scanning → exploitation → reporting) with complete audit trails
- ETL & Data Processing: Monitor batch job stages (extract → transform → validate → load) with performance metrics and failure points
- Research Reproducibility: Document experimental methodology with precise timestamps and configuration parameters for academic papers