Why They Left Databricks Jobs

• Vendor Lock-In Was a Strategic Risk

Databricks Jobs only orchestrates Databricks. Every pipeline, every trigger, every dependency had to live inside the Databricks ecosystem. As the firm expanded its stack — adding Snowflake for serving, dbt for transformation, and custom Python services for pricing data ingestion — Databricks Jobs could not orchestrate across those boundaries. The team was forced to maintain a patchwork of cron jobs and external scripts alongside Databricks Jobs, creating a fragmented and unauditable control plane. Leadership viewed the single-vendor dependency as an unacceptable strategic risk.

• DAG Complexity Had Outgrown the Scheduler

Databricks Jobs supports linear task dependencies within a single job. The firm’s pipelines were not linear. Risk models depended on multiple upstream data feeds completing successfully. Regulatory reports had conditional branches based on market hours and data availability. Restatement workflows required dynamic task generation based on the number of affected periods. None of this could be expressed cleanly in Databricks Jobs — teams were hacking around limitations with polling jobs and sentinel files, creating fragile, hard-to-maintain pipelines that regularly failed in non-obvious ways.

• Compliance and Auditability Requirements

Financial services firms operate under strict regulatory obligations around data lineage, pipeline traceability, and audit logging. Databricks Jobs provided limited metadata — run logs existed, but there was no structured way to query pipeline history, trace a data artifact back to its originating job run, or produce the kind of execution audit trail that regulators expect. Airflow’s metadata database, combined with structured logging and task-level provenance, gave the compliance team the auditability infrastructure they needed.

• SLA Monitoring Was Non-Existent

The firm’s most critical pipelines — end-of-day position feeds, overnight risk calculations, and morning reporting packages — had strict delivery SLAs. Databricks Jobs had no native SLA tracking. When a pipeline ran long, nobody knew until a downstream consumer complained. Airflow’s built-in SLA miss detection and alerting gave the operations team proactive visibility, allowing them to intervene before an SLA breach became a business incident.

• On-Demand Cluster Spin-Up Was Costing Them at Scale

Because Databricks Jobs spun up a cluster for every task — even lightweight coordination tasks that did nothing more than trigger a downstream job or check a file’s existence — the firm was paying DBU costs for work that consumed almost no compute. Airflow separates orchestration from execution: the scheduler is a lightweight service, and compute is only provisioned when actual work runs. For a portfolio of 300+ pipelines, this distinction had material cost implications.

• Talent and Ecosystem

Airflow is the de facto standard for data orchestration. It has a massive open-source ecosystem with 1,000+ providers covering every major data platform, cloud service, and API. Recruiting engineers familiar with Databricks Jobs is a narrow hiring target. Recruiting Airflow engineers is straightforward. The firm’s engineering leadership wanted their orchestration layer to be a skill their entire data team could own — not a proprietary scheduler that required Databricks-specific expertise to operate and debug.