Orchestration Engine — Final Project Report & 1.0.0 Release Documentation¶

Project: orchestration-engine
Target Release: v1.0.0 (Production/Stable)
Branch: stages-2-7
Repository: https://github.com/Reddirector/Orchestration_Engine
Date: 2026-10-25

1. Executive Summary¶

The Orchestration Engine has progressed from an initial prototype (v0.1.0) through six complete stages of engineering, hardening, and verification to reach a release-ready v1.0.0. The engine is a lightweight, zero-runtime-dependency workflow scheduler and execution coordinator for Python 3.12+, designed especially for deterministic AI/agent pipelines and multi-step computational graphs.

All work was completed autonomously according to specification, strictly preserving existing behaviors, writing characterization tests prior to any bug fixes, maintaining strict standard library purity, enforcing single-writer persistence invariants, freezing the public API to exactly 25 approved symbols, and reaching 91.45% line test coverage with zero flakes across repeated executions.

2. Deliverables & Commit Map¶

Stage	Deliverable	Commit	Description
Stage 1	Pre-existing	`246a4e3`	v0.1.0 base release
Stage 2	`2.0`	`876658b`	Characterization tests for known gaps G1-G4, G11-G13
	`2.1`	`15f67cf`	Ruff check and Mypy strict zero-error baseline
	`2.2`	`1fa97ce`	Retry delay calculated from task failure finish timestamp
	`2.3`	`00242d9`	Non-blocking timeout via unpooled executor shutdown
	`2.4`	`a5d28b8`	Runner exceptions adhere to retry policy; `task.last_error` populated
	`2.5`	`a7f749c`	Default timeout 30s, optional workflow default, null support
	`2.6`	`a3b5c84`	Configurable database path via `get_connection` and runner signatures
	`2.7`	`fce0cc7`	Float seconds and strict numeric range validation
	`2.8`	`68ec630`	Structured `OrchestrationError` hierarchy with clear subclasses
	`2.9`	`a5a0e72`	Documentation sync and GitHub issue templates
	`2.10`	`c1181b8`	GitHub Actions CI workflow and local `scripts/gate.sh`
	`2.11`	`56b7fab`	Release v0.2.0 (tag `stage-2-done`)
Stage 3	`3.1`	`322ef84`	Schema v1 with `runs` table and automatic migration
	`3.2`	`db76de7`	SQLite connection hardening: WAL mode, busy timeout, foreign keys
	`3.3`	`284e757`	`run_id` plumbing across all events and runs tracking
	`3.4`	`46011fe`	Status and history query directly from database
	`3.5`	`39dcc1c`	Database as single source of truth; memory mirrors synced
	`3.6`	`02f5daf`	`resume_run` with task state restoration and running recovery
	`3.7`	`b9d50a4`	Subprocess SIGKILL crash-and-resume integration test
	`3.8`	`1bde985`	Release v0.3.0 (tag `stage-3-done`)
Stage 4	`4.1`	`31cf9f0`	Single-writer coordinator architecture & concurrency parameters
	`4.2`	`048d09a`	Single-writer coordinator loop with ThreadPoolExecutor
	`4.3`	`b2c3300`	Process executor backend with spawn, timeout SIGKILL, IPC pipes
	`4.4`	`fabd687`	Deterministic `_inline` executor for fast virtual-clock tests
	`4.5`	`d6eaf71`	Concurrency and process executor test suite (20x flake-free)
	`4.6`	`c8d4aa8`	Release v0.4.0 (tag `stage-4-done`)
Stage 5	`5.1`	`1aa0baa`	`TaskResult`, upstream `task.inputs`, schema v2 `task_outputs`
	`5.2`	`ce2f09b`	Lifecycle `Hooks` and safe coordinator event callbacks
	`5.3`	`28e7956`	`WorkflowBuilder` fluent API and `workflow_to_dict` serializer
	`5.4`	`6c82f8e`	`workflow.schema.json` packaged & `collect_errors` loader mode
	`5.5`	`5d77ab9`	Optional YAML workflow loader via PyYAML optional extra
	`5.6`	`3ec44bf`	Release v0.5.0 (tag `stage-5-done`)
Stage 6	`6.1`	`74befc3`	Standard library CLI implementation with console scripts
	`6.2`	`99d32be`	CLI contract, stable JSON schemas, dynamic runner resolution
	`6.3`	`249384a`	Structured logging with `logging.NullHandler()` on library logger
	`6.4`	`5dc6bb0`	Comprehensive CLI test suite and subprocess smoke tests
	`6.5`	`b4346d9`	Release v0.6.0 (tag `stage-6-done`)
Stage 7	`7.1`	`a0c8dcb`	MkDocs Material documentation site (strict build passing)
	`7.2`	`00ef822`	Multi-step AI agent pipeline example (`examples/agent_pipeline.py`)
	`7.3`	`1384db3`	Benchmark suite & performance report (`docs/performance.md`)
	`7.4`	`da50baa`	Security audit, DoS limits, and parameterized query review
	`7.5`	`372378f`	Public API freeze (25 symbols) and policy docs (`docs/policy.md`)
	`7.6`	`9531533`	Package metadata, release workflow, and verification script
	`7.7`	`c1a845b`	Final verification gate (flakeless concurrency and clean install)
	`7.8`	`f8e8a0e`	Release candidate 1.0.0 (tag `stage-7-done`)

3. Final Quality Gate & Test Coverage¶

3.1 Quality Gate Execution¶

Executing bash scripts/gate.sh: - Ruff: All checks passed (100% formatted and linted). - Mypy: Strict type checking passed on 35 source files with 0 errors. - Pytest: 157 passed in 6.87s. - Line Coverage: 91.45% (exceeds the 90.0% threshold). - Package Build: Wheel (.whl) and Source Distribution (.tar.gz) generated cleanly with zero PEP 639 warnings. - Clean Venv Isolation: Fresh venv in /tmp successfully installed wheel, verified all 25 exports in __all__, ran all 6 example scripts, and exercised CLI binary.

3.2 Coverage Breakdown by Module¶

Name                                      Stmts   Miss  Cover
-------------------------------------------------------------
src/orchestration_engine/__init__.py         15      2    87%
src/orchestration_engine/__main__.py          4      0   100%
src/orchestration_engine/builder.py          39      0   100%
src/orchestration_engine/cli.py             246     23    91%
src/orchestration_engine/errors.py            7      0   100%
src/orchestration_engine/executor.py         37      4    89%
src/orchestration_engine/loader.py          143      6    96%
src/orchestration_engine/models.py           54      0   100%
src/orchestration_engine/persistence.py      96      8    92%
src/orchestration_engine/readiness.py        57      4    93%
src/orchestration_engine/runner.py          336     51    85%
src/orchestration_engine/scheduler.py        13      0   100%
src/orchestration_engine/status.py           35      1    97%
src/orchestration_engine/transitions.py      36      0   100%
src/orchestration_engine/validation.py       52      1    98%
-------------------------------------------------------------
TOTAL                                      1170    100    91%

4. Complete List of Behavior Changes (v0.1.0 to v1.0.0)¶

Retry Delay Formula: Delay is calculated relative to the finish timestamp of the failed attempt (attempt_finish_time + delay), preventing instantaneous retries on long-running tasks.
Exception Handling: Unhandled exceptions raised by a task runner are caught, recorded in task.last_error, and handled according to RetryPolicy rather than immediately crashing the workflow coordinator.
Non-blocking Thread Timeouts: Task timeouts terminate execution coordinator wait time without holding the coordinator thread blocked.
Timeout Defaults: Tasks without explicit timeout_seconds inherit workflow.default_timeout_seconds (defaulting to 30.0s). Explicit None disables task timeout.
Transitive Cancellation: If task A fails permanently, all downstream dependents (direct and indirect) transition to CANCELLED within a single scheduling pass.
Persistence Single Source of Truth: task_states dictionary is now a read-only backwards-compatible mirror; the SQLite event log is the authoritative source of truth.
Crash Resume Semantics: Incomplete runs can be safely resumed via resume_run(conn, run_id, ...). Interrupted tasks in RUNNING or RETRY_WAIT state are recovered to READY or re-scheduled.
Multi-Thread & Multi-Process Concurrency: Workflows can execute independent tasks concurrently (max_concurrency > 1) using threads or isolated spawned child processes (multiprocessing).
Data Flow & Task Outputs: Tasks can return TaskResult(ok=True, output=data). Output is persisted in SQLite task_outputs table and passed to downstream tasks via task.inputs = {dep_id: output}.
Lifecycle Hooks: User-provided callbacks (Hooks) trigger on state transitions. Any exceptions inside hooks are caught and logged without aborting task scheduling.
Security Limits: Added maximum file size protection (10 MB), maximum task count (5,000 tasks), and maximum dependency count (500 per task) to guard against resource exhaustion.
CLI Contract: Introduced command-line utility orchestration-engine with strict exit codes (0, 1, 2, 3) and machine-readable JSON mode.

5. Architectural Decision Records (ADRs) Master Index¶

All architectural choices are recorded in docs/decisions/: - ADR 001: docs/decisions/001-thread-pool-non-blocking-timeouts.md — Non-blocking timeouts via unpooled executor shutdown. - ADR 002: docs/decisions/002-exception-retry-and-last-error.md — Runner exception capture and retry semantics. - ADR 003: docs/decisions/003-error-hierarchy.md — Domain error hierarchy rooted at OrchestrationError. - ADR 004: docs/decisions/004-schema-v1-and-runs-table.md — Relational schema versioning and runs tracking. - ADR 005: docs/decisions/005-run-model-and-lifecycle.md — Immutable run identification and lifecycle status. - ADR 006: docs/decisions/006-resume-semantics.md — Crash recovery and idempotent resume execution. - ADR 007: docs/decisions/007-task-vs-taskrun-representation.md — Task definition decoupling from run execution states. - ADR 008: docs/decisions/008-single-writer-coordinator-concurrency.md — Single-writer coordinator architecture for concurrent tasks. - ADR 009: docs/decisions/009-data-flow-and-task-outputs.md — TaskResult contract, output persistence, and upstream injection. - ADR 010: docs/decisions/010-hooks-lifecycle.md — Non-blocking post-persistence lifecycle hooks. - ADR 011: docs/decisions/011-workflow-builder-and-serializer.md — Fluent WorkflowBuilder and dictionary round-trip. - ADR 012: docs/decisions/012-json-schema-and-error-collection.md — Packaged JSON Schema and multi-error validation. - ADR 013: docs/decisions/013-optional-yaml-support.md — Zero-dependency core with optional PyYAML loader. - ADR 014: docs/decisions/014-cli-and-structured-logging.md — Standard library CLI contract and NullHandler logging.

6. Intentionally Out-of-Scope Items¶

To preserve the engine's core promise of zero runtime dependencies, minimal surface area, and rock-solid durability, the following capabilities are explicitly out of scope for v1.0.0: 1. Distributed Multi-Node Execution: No distributed worker pools (e.g. Celery, Ray, Kubernetes). All concurrency is single-node multi-thread or multi-process. 2. External Message Brokers / Databases: No dependency on Redis, RabbitMQ, Kafka, or PostgreSQL. Persistence is strictly standard library SQLite. 3. Web UI / Graphical Dashboard: No bundled web server (e.g. Flask, FastAPI, React). Monitoring is accomplished via SQLite queries, the CLI (orchestration-engine status / history), or application logs. 4. Dynamic DAG Mutation at Runtime: The task graph is fully declared before execution starts; dynamic graph topology expansion during run execution is not supported.

7. Known Technical Limitations¶

Thread Backend Cancellation: The executor="thread" backend uses standard Python concurrent.futures.ThreadPoolExecutor. In Python, running native threads cannot be forcefully terminated from the outside. When a threaded task times out, the engine stops waiting and proceeds with workflow scheduling, but the worker thread will run until completion or interpreter exit. For tasks that must be forcefully terminated on timeout, use executor="process".
SQLite Single-Writer Concurrency: While SQLite in WAL mode permits concurrent readers, it enforces a single writer lock. Under the single-writer coordinator pattern, this is completely safe within an engine instance. Multiple independent processes pointing to the exact same SQLite database file should rely on the configured busy_timeout (default 5000ms).

8. Manual Instructions for Repository Owner¶

As instructed, no git push, uv publish, or remote changes were performed by the agent. The branch stages-2-7 is completely verified, committed, and tagged locally.

Follow these steps to complete the release:

Step 1: Review Local Commits & Tags¶

cd ~/Desktop/Orchestration\ engine
git status
git log --oneline main..stages-2-7
git tag -l

Step 2: Push Branch and Tags to GitHub¶

git push origin stages-2-7
git push origin --tags

Step 3: Open Pull Request on GitHub¶

Navigate to https://github.com/Reddirector/Orchestration_Engine/compare/main...stages-2-7.
Title: Release 1.0.0: Production-Ready Orchestration Engine.
Description: Reference docs/reports/FINAL-REPORT.md.
Review diff and merge PR into main.

Step 4: Fast-Forward `main` Locally¶

git checkout main
git pull origin main

Step 5: Publish to PyPI¶

If using GitHub Actions Trusted Publishing (configured in .github/workflows/release.yml): - Create a GitHub Release pointing to tag stage-7-done (or create a v1.0.0 tag). The workflow will automatically build and publish to PyPI.

If publishing manually with uv:

uv build
uv publish