Apache Airflow is an open-source workflow orchestration platform (originally developed at Airbnb, donated to the Apache Software Foundation in 2016) that allows data engineers to define, schedule, execute, and monitor data pipelines as Python code. Airflow represents each workflow as a DAG (Directed Acyclic Graph) — a Python script that defines the pipeline’s tasks, their execution order, and their dependencies. Tasks in an Airflow DAG are executed by Airflow workers (processes that receive task assignments from the Airflow scheduler) on a configured schedule or triggered by an external event. Airflow’s web UI provides a visual DAG representation (showing the task dependency graph), real-time task execution status, task logs for debugging, and historical run records — making it the de facto standard tool for monitoring and operating data pipelines in environments that have adopted the modern data stack (dbt, Snowflake, Spark).
Airflow Architecture
| Airflow Component | Function |
|---|---|
| DAG files | Python files that define the workflow — tasks, operators, dependencies, schedule |
| Scheduler | Monitors DAG schedules and triggers task execution based on dependencies and time |
| Executor | Determines how tasks are executed — LocalExecutor (single server), CeleryExecutor (distributed workers), KubernetesExecutor (containerised tasks) |
| Workers | Processes that execute the actual task code — SQL queries, Python functions, API calls, dbt runs |
| Metadata database | PostgreSQL or MySQL database storing DAG definitions, task states, and execution history |
| Web UI | Browser-based interface for monitoring DAG runs, viewing task logs, triggering manual runs, and managing connections |
| Connections | Stored, encrypted credentials for data sources (Oracle EBS, Snowflake, Azure Synapse, Power BI REST API) |
Airflow Operators for Finance Pipelines
Airflow tasks are implemented using Operators — pre-built task templates for common operations. Key Airflow operators for GCC finance data pipeline orchestration include: OracleOperator (execute SQL against Oracle EBS — trigger a custom extraction procedure); SnowflakeOperator (execute SQL in Snowflake — run a dbt transformation or load a staging table); PythonOperator (execute arbitrary Python code — call the Power BI REST API, call the ZATCA API, trigger an EPM Automate job); BashOperator (execute shell commands — run EPM Automate CLI commands for Oracle EPM Cloud operations); HttpOperator (call an HTTP API — trigger Oracle Integration Cloud flows, check FCCS job status via REST API); and EmailOperator (send email notifications — alert the finance operations team of close cycle completion or failure).
Managed Airflow in GCC Azure Environments
For GCC enterprises on the Microsoft Azure stack, Apache Airflow is available as Azure Managed Airflow (a component of Azure Data Factory’s managed Airflow offering) — eliminating the operational overhead of running and maintaining Airflow infrastructure while providing the full Airflow DAG authoring and monitoring capability. Managed Airflow in Azure handles Airflow scheduler, worker, and metadata database infrastructure; the data engineering team writes DAG Python files and deploys them to the managed environment without managing the underlying infrastructure. For GCC finance technology teams that value Airflow’s orchestration capabilities but do not have the DevOps capacity to manage Airflow infrastructure, managed Airflow is the practical deployment model.
What Goes Wrong in Practice
The most common Airflow operational failure in production finance pipelines is a DAG that catches and suppresses exceptions in its task code — logging the error internally but allowing the task to complete with a “success” status from Airflow’s perspective. A data transformation task that encounters a data quality error, handles it silently, and reports success to Airflow will cause all downstream tasks (including the Power BI refresh and management pack distribution) to execute against incomplete or incorrect data. Every Airflow task in a finance pipeline must propagate exceptions — any error that affects the output data must fail the Airflow task and halt the downstream pipeline for investigation.
How Loop Wise Solutions Uses Airflow
We use Apache Airflow (managed on Azure or self-hosted on AKS) as the orchestration layer for complex finance close cycle automation — particularly where the pipeline crosses multiple systems (Oracle EBS, Oracle EPM Cloud, Azure Synapse, Power BI) and requires dependency management, SLA monitoring, and failure alerting across the full pipeline. Simpler pipelines with only Azure services are orchestrated using Azure Data Factory pipelines; Airflow is selected when the pipeline complexity or cross-platform coordination requirements exceed what ADF pipelines handle naturally.
