Skip to main content

© 2026 Stephen Adei. All rights reserved. All content on this site is the intellectual property of Stephen Adei. See License for terms of use and attribution.

ADR-006: run_id Isolation Pattern

Status​

Accepted

Context​

The system needs to support safe backfills, parallel runs, and idempotent reruns without data loss or corruption.

The following options were considered:

  1. run_id isolation (chosen)
  2. Overwrite pattern (rejected)
  3. Versioned tables (rejected - adds complexity)

Decision​

Use run_id isolation pattern where each ETL run writes to a unique S3 path determined by run_id (e.g., run_id=20260121T120000Z/). Promotion occurs by updating _LATEST.json pointer, not by overwriting data.

Rationale​

  1. Safe backfills: Reprocess historical data without overwriting existing runs
  2. Idempotent reruns: Failed runs can be safely rerun without data loss
  3. Parallel runs: Multiple runs can execute concurrently without conflicts
  4. Audit trail: All runs preserved for compliance and debugging
  5. Rollback safety: Can revert to previous run by updating _LATEST.json

Consequences​

Positive:

  • Safe backfills: No overwrites, complete audit trail
  • Idempotent reruns: Failed runs can be safely rerun
  • Parallel execution: Multiple runs do not conflict
  • Audit compliance: All runs preserved for audit
  • Rollback capability: Easy reversion to previous runs

Negative:

  • Storage cost: Multiple versions per partition retained
  • Manual cleanup: Old runs must be manually archived/deleted

Alternatives Considered​

Overwrite Pattern​

  • Why rejected: Data loss risk on failures, no audit trail, unsafe backfills.

Versioned Tables (Iceberg/Delta)​

Separate Run Table (DynamoDB)​

  • Why rejected: Adds operational complexity. Traceability and auditability are top priority; current design uses AWS-native identifiers (Step Functions execution ARN).

Implementation Evidence​

© 2026 Stephen Adei•CC BY 4.0