Lambda Function: Promote Silver Layer Data

Purpose

Automatically promotes validated Silver layer data after successful ETL completion by:

1. Finding the most recent completed run (with _SUCCESS marker)
2. Copying Parquet files to current/ prefix for stable Athena queries
3. Updating _LATEST.json to point to the promoted run

Architecture

Step Functions → Glue Job (ETL) → Lambda (Promotion) → S3 (current/ + _LATEST.json)

Trigger

Invoked by Step Functions state machine after successful ETL job completion.

Input (from Step Functions)

{
  "silver_bucket": "ohpen-silver",
  "silver_prefix": "silver/mortgages/transactions",
  "schema_version": "v1",
  "run_key": "execution-name-from-step-functions",
  "glue_job_run_id": "jr_...",
  "execution_arn": "arn:aws:states:...",
  "execution_start_time": "2026-01-29T12:00:00.000Z"
}

When run_key is set, the Lambda promotes that run only. glue_job_run_id and execution_arn are logged and written into _LATEST.json for traceability.

Output

{
  "status": "success",
  "run_id": "20260129T120000Z",
  "promoted_at": "2026-01-29T12:05:00.000Z",
  "output_prefix": "silver/mortgages/transactions/year=2024/month=01/schema_v=v1/run_id=20260129T120000Z",
  "current_prefix": "silver/mortgages/transactions/year=2024/month=01/schema_v=v1/current",
  "files_copied": 5,
  "latest_json": "silver/mortgages/transactions/year=2024/month=01/schema_v=v1/_LATEST.json"
}

S3 Operations

1. Discovery: Searches for _SUCCESS markers in silver_prefix/**/schema_v={version}/run_id=*/_SUCCESS
2. Selection: Picks the most recently modified _SUCCESS marker
3. Copy: Copies all .parquet files from run_id=*/ to current/
4. Metadata: Updates _LATEST.json with promotion metadata

Error Handling

• Returns error status if no completed runs found
• Returns error status if no Parquet files found
• Step Functions will catch errors and route to HandleFailure state

Deployment

Deployed automatically via Terraform:

• Function code packaged as ZIP from lambda/promote_silver/ directory
• IAM role and policies created automatically
• CloudWatch Log Group created automatically

Testing

Test locally with:

python -m pytest tests/test_lambda_promote.py

Test in AWS:

aws lambda invoke \
  --function-name ohpen-promote-silver \
  --payload '{"silver_bucket":"ohpen-silver","silver_prefix":"silver/mortgages/transactions","schema_version":"v1"}' \
  response.json