REAL AWS CLOUD PROOF

Cloud Flight Fare Pipeline

Real AWS cloud data engineering proof project using EventBridge Scheduler, ECS/Fargate, S3, Redshift Serverless, dbt, and CloudWatch Logs to run a scheduled batch pipeline from flight API ingestion to analytics-ready marts.

Validated cloud scope: proven AWS batch execution with EventBridge Scheduler, ECS/Fargate, S3 Bronze landing, Redshift Serverless loading, dbt marts/tests, CloudWatch success logs, proof screenshots, and runbooks.

AWSECS/FargateEventBridgeS3RedshiftdbtDockerCloudWatch

GitHub Repo Architecture Execution Proof Docs

Cloud Flight Fare Pipeline project image

Local validation

Docker + Postgres

Proven AWS path

Scheduler → ECS/Fargate → S3 → Redshift

Orchestration

EventBridge + CloudWatch

Modeling

dbt marts + tests

Reviewer proof

Screenshots, logs, SQL, CI

AWS Proof Diagram

Current Proven AWS Path

Real AWS execution path showing how EventBridge Scheduler triggers an ECS/Fargate batch container to ingest flight data, land raw data in S3 Bronze, load Redshift Serverless, build dbt staging/marts/tests, and capture execution proof in CloudWatch Logs.

LOCAL VALIDATION PATH

Reproducible local review path

Run the project locally with Docker, Postgres, dbt, and proof queries before reviewing the AWS proof path.

Step 01

Start local services

docker compose up -d

Step 02

Load sample fares

python scripts/load_sample_to_postgres.py

Step 03

Build dbt models

dbt build --project-dir dbt/flight_fares --profiles-dir dbt

Step 04

Run proof queries

python scripts/run_analysis_queries.py

Execution paths

Two clear paths, one analytics-ready outcome

The project is easiest to review through two paths: a reproducible local validation path and a proven AWS execution path. Both produce dbt-modeled marts and analytics-ready SQL outputs.

Local validation path

Start Docker + Postgres locally
Load sample fare snapshots
Build dbt staging, marts, and tests
Run proof queries against analytics outputs

Local validation guide

Proven AWS execution path

EventBridge triggers the batch run
ECS/Fargate runs the container job
Flight data lands in S3 Bronze
Redshift Serverless receives loaded tables
dbt builds marts and runs tests
CloudWatch captures execution evidence

AWS proof runbook

Shared analytics handoff

Mart outputs include marts.fact_fares, marts.dim_route, and marts.dim_date.
Docs explain route, pricing, and timing analysis
SQL examples cover trends, movement, and lead-time review
CI validates linting, tests, loading, and dbt build steps

Mart usage docs

TECHNICAL ARCHITECTURE

Full Repository Architecture

Detailed repository architecture showing ingestion, raw/bronze landing, cleaned processing, dbt modeling, validation, and analytics outputs.

Open full diagram

Cloud Flight Fare Pipeline full repository architecture diagram showing source, ingestion, bronze raw, silver cleaned, dbt modeling, validation, and analytics output stages.

Analytics-ready outputs

Downstream analysis is visible, not implied

The project goes beyond ingestion and modeling by documenting marts, SQL query patterns, and downstream handoff artifacts reviewers can inspect after the pipeline runs.

SQL output examples

Route fare trends

Route-level SQL shows pricing trends by origin and destination.

Route trend SQL

Monthly fare movement

Example queries summarize fare movement over time.

Example queries

Lead-time analysis

Lead-time SQL supports booking-window and pricing review.

Lead-time SQL

Preview artifact

Downstream preview artifact

Static downstream handoff artifact for reviewer inspection, not a live hosted BI app.

Open preview artifact

Execution Proof

Real proof assets shown directly on the page

Reviewer evidence is surfaced here instead of hidden behind link lists. These proof assets show the AWS scheduler, ECS/Fargate execution, CloudWatch success logs, Redshift/dbt validation, S3 Bronze landing, and local validation support.

EventBridge Scheduler target configuration for the Cloud Flight Fare Pipeline Fargate job

AWS scheduler proof

Open

EventBridge Scheduler target

Scheduler configuration showing the AWS batch trigger path for the Fargate job.

ECS Fargate task run showing exit code 0 for the Cloud Flight Fare Pipeline

ECS/Fargate proof

Open

Manual Fargate task exit code 0

Containerized batch run completed successfully in ECS/Fargate.

CloudWatch log showing WEEK9_BATCH_SUCCESS for the scheduled AWS run

CloudWatch proof

Open

WEEK9_BATCH_SUCCESS log

CloudWatch log evidence showing the scheduled AWS run completed successfully.

dbt build success output for Redshift validation in the Cloud Flight Fare Pipeline

Redshift + dbt proof

Open

Redshift/dbt build success

Warehouse proof showing Redshift loading and dbt build validation.

Amazon S3 console showing bronze flight fare snapshots

AWS S3 proof

Open

Bronze ingestion landed in S3

S3 storage proof showing dated fare snapshots under the Bronze path.

Airflow graph view for the local flight fare pipeline validation DAG

Local DAG proof

Open

Runnable validation DAG graph

Local Airflow graph showing the load, dbt build, and proof-query chain.

EventBridge Proof ECS/Fargate Proof CloudWatch Logs Redshift/dbt Proof S3 Bronze Proof AWS Proof Runbook GitHub Actions CI

Reviewer path

Start with the Current Proven AWS Path diagram, then review the AWS proof assets, local validation path, and downstream outputs. This page separates cloud proof, local validation, and reviewer handoff without overclaiming a live production service.

Current AWS Path AWS Proof Runbook Proof Screenshots GitHub Repo