Skip to main content

Platform Governance & Operational Workflows

© 2026 Stephen Adei. All rights reserved. All content on this site is the intellectual property of Stephen Adei. See License for terms of use and attribution.

This section contains governance and ownership documentation:

  • Ownership models and responsibility matrices
  • Approval workflows and decision frameworks
  • Data quality and quarantine governance
  • Compliance and audit governance

For technical architecture details (folder structure, schema evolution, failure modes), see Data Lake Architecture Details.

For a high-level overview, see the main Data Lake Architecture document.

1. Ownership & Responsibility Matrix​

Layer Ownership Diagram​

Responsibility Matrix​

LayerOwnerStewardWrite AccessRead AccessChange Approval
BronzeData Platform TeamIngestion LeadPlatform Team onlyPlatform, Compliance (audit)Platform Team
SilverDomain TeamsDomain AnalystDomain Teams, PlatformDomain Teams, AnalystsDomain + Platform review
GoldBusiness (Finance)Finance ControllerPlatform (on approval)Business, Analysts, BIFinance Controller
Quarantine (Error Handling)Data Platform TeamETL LeadETL Pipeline, PlatformPlatform, Quality Team (review), Domain Teams (read-only)Platform Team (with Quality Team review)
Condemned (Error Handling)Data Platform TeamETL LeadETL Pipeline, PlatformPlatform, Quality Team (review), Compliance (audit)Platform Team (human approval required)

2. Error Handling Layers: Detailed Ownership​

This section provides detailed ownership and governance information for the Error Handling Layers (Quarantine and Condemned), which are separate from the medallion data layers (Bronze/Silver/Gold).

Error Handling Layers Ownership Table​

LayerOwnerStewardReviewerResponsibilityImplementation Status
Quarantine (Error Handling)Data Platform TeamETL LeadData Quality TeamError detection, routing, retry logic, audit trail maintenance, infrastructure managementImplemented
Condemned (Error Handling)Data Platform TeamETL LeadData Quality TeamExclusion management, perpetual retention (financial audit), compliance, infrastructure managementImplemented

Quarantine Layer Ownership​

Owner (Quarantine): Data Platform Team​

  • Primary Responsibilities:
    • Infrastructure Management: S3 bucket configuration, folder structure, access controls, encryption
    • Error Detection & Routing: ETL pipeline logic that identifies invalid rows and routes them to Quarantine
    • Retry Logic Implementation: Automated retry mechanisms (max 3 attempts), attempt tracking, retry history management
    • Audit Trail Maintenance: Metadata enrichment (row_hash, attempt_count, retry_history, validation_error), run_id tracking
    • Data Movement: Writing invalid rows to Quarantine, managing retry workflows, moving rows to Condemned after max attempts
    • Infrastructure Operations: Monitoring S3 storage, managing retention policies, ensuring data availability

Steward (Quarantine): ETL Lead (Platform Team)​

  • Day-to-day Operations: Oversees ETL pipeline operations, ensures retry logic functions correctly
  • Technical Decisions: Makes decisions about retry strategies, attempt limits, error routing rules
  • Incident Response: Responds to infrastructure issues, pipeline failures, storage problems

Reviewer (Quarantine): Data Quality Team​

  • Error Analysis: Reviews invalid rows, identifies error patterns, performs root cause analysis
  • Resolution Recommendations: Provides guidance on whether errors are fixable, recommends fixes (source provider fix vs ETL logic fix)
  • Quality Monitoring: Tracks quarantine rates, error type distributions, retry success rates
  • Decision Support: Advises Platform Team on retry vs condemn decisions for edge cases
  • Quality Reporting: Maintains dashboards and reports on data quality metrics

Access Model (Quarantine)​

  • Write Access: Platform Team only (ETL pipeline writes invalid rows)
  • Read Access: Platform Team (full access), Quality Team (full access for review), Domain Teams (read-only for their domain's data)
  • Modify Access: Platform Team only (for retry workflows, metadata updates)

Condemned Layer Ownership​

Owner (Condemned): Data Platform Team​

  • Primary Responsibilities:
    • Infrastructure Management: S3 bucket configuration, folder structure (quarantine/condemned/), access controls, encryption
    • Exclusion Management: Moving rows from Quarantine to Condemned after max attempts (attempt_count >= 3) or exact duplicate detection
    • Retention Policy Enforcement: Perpetual retention for financial audit; Glacier transition after 5 years; deletion only via approved process
    • Audit Trail Maintenance: Preserving all metadata (row_hash, attempt_count, retry_history, validation_error, condemnation_reason)
    • Infrastructure Operations: Monitoring storage, managing lifecycle policies, ensuring compliance

Steward (Condemned): ETL Lead (Platform Team)​

  • Day-to-day Operations: Oversees condemnation logic, ensures max attempts are enforced correctly
  • Technical Decisions: Makes decisions about condemnation criteria, retention policies, storage optimization
  • Compliance Management: Ensures perpetual retention for financial audit; manages deletion approval workflows

Reviewer (Condemned): Data Quality Team​

  • Condemned Data Review: Reviews condemned rows to identify systemic issues, patterns that require upstream fixes
  • Resolution Recommendations: Provides guidance on whether condemned data can be reprocessed (requires human approval)
  • Compliance Oversight: Ensures condemned data is properly retained for audit purposes
  • Quality Insights: Uses condemned data to identify upstream data quality issues, recommends preventive measures

Access Model (Condemned)​

  • Write Access: Platform Team only (ETL pipeline moves rows to Condemned)
  • Read Access: Platform Team (full access), Quality Team (full access for review), Compliance Team (read-only for audit)
  • Modify/Delete Access: Requires human approval workflow

Error Handling Workflow: Quarantine Resolution​

The ownership model enables a collaborative workflow for resolving quarantined data:

  1. Platform Team detects invalid data: ETL pipeline routes invalid rows to Quarantine Layer
  2. Platform Team enriches metadata: Adds error details, attempt_count, retry_history
  3. Quality Team reviews errors: Analyzes error patterns, identifies root causes
  4. Quality Team provides recommendations: Suggests fixes (source provider fix, ETL logic fix, or condemn)
  5. Platform Team implements fix: Updates ETL logic or coordinates with source provider
  6. Platform Team decides retry vs condemn: Based on attempt_count (max 3 attempts: attempt_count < 3 allows retry; attempt_count >= 3 condemned) and Quality Team recommendations
  7. Quality Team monitors results: Tracks retry success rates, validates resolution effectiveness

Governance Rules for Error Handling Layers​

  • Quarantine Layer:

    • Platform Team manages all write operations (ETL pipeline only)
    • Quality Team has full read access for review and analysis
    • Retry decisions are automated (max 3 attempts: attempt_count < 3 allows retry; attempt_count >= 3 condemned) but can be overridden with Quality Team approval
    • All retries preserve audit trail (retry_history, attempt_count increments)

  • Condemned Layer:

    • Platform Team manages all write operations (automatic condemnation after max attempts)
    • Quality Team has full read access for review and compliance oversight
    • Reprocessing condemned data requires human approval workflow (not automatic)
    • Perpetual retention for financial audit (no automatic deletion)
    • Deletion only via explicit, approved process

  • Cross-Layer Interactions:

    • Quarantine → Silver (retry success): Platform Team manages, Quality Team monitors
    • Quarantine → Condemned (max attempts): Platform Team manages automatically, Quality Team reviews
    • Condemned → Reprocessing: Requires Quality Team recommendation + human approval

2. Schema Change Governance Workflow​

Schema Evolution Process​

Schema Versioning Timeline​

3. Data Quality & Quarantine Governance​

Quarantine Review Workflow​

Data Quality Escalation Matrix​

4. Backfill & Reprocessing Governance​

Backfill Approval Workflow​

Backfill Decision Tree​

5. Access Control & Permissions​

IAM Permission Matrix​

Permission Summary Table​

RoleBronzeSilverGoldQuarantine
Platform TeamRead/WriteRead/WriteWrite (on approval)Read/Write
Domain Teams-Read/Write (domain scope)Read-
Business Users--Read-
Data Analysts-ReadRead-
ComplianceRead (audit)--Read (audit)

6. Human Approval Workflows​

Condemned Data Management​

Complete Governance Workflow​

Promotion Workflow (Gold Layer) with Approval Process:

7. Governance Decision Framework​

Change Request Classification​

Governance Escalation Path​

8. Operational Governance​

Daily Operations Workflow​

Monitoring & Alerting Governance​

Audit Trail Workflow​

Data Retention Policy​

10. Governance Summary​

Key Principles​

  1. Layer-Based Ownership: Each layer has clear ownership and stewardship
  2. Approval Workflows: All changes require appropriate approvals based on layer and impact
  3. Human-in-the-Loop: Critical decisions (condemned data, Gold promotion) require human approval
  4. Audit Trail: All operations are logged and retained for compliance
  5. Versioning: Schema changes use version tags for backward compatibility
  6. Safe Publishing: Write-then-publish pattern prevents partial data exposure

Governance Checklist​

  • Schema changes follow approval workflow
  • Backfills have appropriate approvals
  • Quarantine data is reviewed regularly
  • Condemned data requires human approval
  • Gold layer promotion follows validation process
  • Access permissions are reviewed quarterly
  • Audit logs retained per policy (financial audit: perpetual)
  • Compliance reports are generated monthly

See also​

Last Updated: January 2026
Owner: Data Platform Team

© 2026 Stephen Adei•CC BY 4.0