Skip to content

Orchestration Engine

PyPI version Python versions License GitHub Release

A fault-tolerant, dependency-aware workflow orchestration engine for Python. Define tasks, dependencies, retries, and timeouts declaratively; supply the logic for each task; the engine schedules, executes, retries, times out, cancels dependents of failures, and records every state transition.

Quick Installation

pip install orchestration-engine

For optional YAML workflow loading support: 

pip install "orchestration-engine[yaml]"

[!TIP] Modern Linux (PEP 668): If installing on Ubuntu 23.04+, Debian 12+, or Fedora without a global pip environment, activate a virtual environment first: 

python3 -m venv .venv && source .venv/bin/activate
pip install orchestration-engine

Mission

Multi-step automation pipelines—such as AI agent workflows, ETL transformations, and distributed tool executions—frequently encounter transient network drops, rate limits, and unhandled worker crashes.

Orchestration Engine is designed to provide robust, deterministic workflow management with: - Zero runtime dependencies: Pure Python standard library implementation. - Durable persistence: Append-only SQLite audit trail recording all state transitions. - Crash resumption: Seamless resumption of interrupted runs without re-executing already-succeeded tasks. - Transparent concurrency: Deterministic execution scaling via threads or spawned isolated processes. - Fail-safe isolation: Faults in user task runners and lifecycle hooks never corrupt internal engine state.

Core Features

  • Dependency-Aware Scheduling: Tasks are scheduled as a Directed Acyclic Graph (DAG) and advance as soon as all upstream dependencies succeed.
  • Automatic Retries & Exponential Backoff: Transient errors are retried according to a configurable RetryPolicy using exponential backoff: delay_seconds * 2**(attempt_count - 1).
  • Per-Task Timeouts: Asynchronously enforced deadlines abandon stalled threads or terminate spawned child worker processes.
  • Transitive Cancellation: When a task fails permanently, all downstream dependents (direct and indirect) are cancelled to a fixpoint, while independent branches continue executing to completion.
  • Data Flow: Tasks receive outputs from completed upstream dependencies via task.inputs, with automatic JSON serialization and persistence in SQLite Schema v2.
  • Lifecycle Hooks: Non-blocking callbacks (on_task_start, on_task_succeeded, on_task_failed, on_task_retry, on_workflow_done) for monitoring and logging.
  • Developer Ergonomics: Fluent WorkflowBuilder API, JSON and YAML workflow definitions, and a CLI tool (orchestration-engine).

Production Use Cases

See the full Use Cases Guide for architectural blueprints and complete implementations: - Multi-Step AI Agent Pipelines: LLM tool calling, retrieval, rate limit handling, and fan-in synthesis. - RAG Ingestion & Vector Indexing: Document chunking, concurrent embedding generation, and quality verification gates. - Automated Incident Triage & Remediation: Fan-out telemetry diagnostics, root-cause correlation, automated mitigation, and verification. - Resilient ETL Pipelines: Fluent DAG construction, type-safe data passing, and transactional SQLite persistence.

Design Philosophy

  1. Zero runtime dependencies: Everything is implemented using the Python standard library (sqlite3, multiprocessing, concurrent.futures, json, argparse).
  2. Correctness over raw throughput: Every state transition is written to SQLite before in-memory mutation occurs (write-ahead log).
  3. Small, cohesive internals: Each internal module has one job (scheduler, readiness, executor, transitions, runner, persistence, recovery, loader, builder).
  4. Minimal, frozen public API: Only explicitly approved symbols are exported in __all__.