Skip to main content

Deployment Summary: Fintech Compliance Enhancements

Changes Implemented​

1. KMS Encryption for Sensitive Buckets​

What Changed:

  • Added KMS key (aws_kms_key.data_lake_sensitive) with automatic key rotation
  • Updated S3 encryption to use SSE-KMS for ohpen-gold and ohpen-quarantine buckets
  • Standard buckets (ohpen-bronze, ohpen-silver, ohpen-artifacts) continue using SSE-S3

Terraform Changes:

  • tasks/devops_cicd/infra/terraform/main.tf:
    • Lines 30-90: KMS key and alias resources
    • Lines 96-110: Updated S3 encryption configuration with conditional logic

Benefits:

  • Customer-managed keys (CMK) for compliance
  • Fine-grained access control via KMS key policies
  • Automatic key rotation enabled
  • CloudTrail logging for key usage

2. Selective CloudTrail Data Events​

What Changed:

  • Added selective data events for sensitive buckets only (ohpen-gold, ohpen-quarantine)
  • Management events remain enabled for all infrastructure changes
  • Bronze/Silver/Artifacts buckets excluded to reduce cost and noise

Terraform Changes:

  • tasks/devops_cicd/infra/terraform/main.tf:
    • Lines 810-830: Updated CloudTrail configuration with selective data events

Benefits:

  • Cost-aware approach (data events are $0.10 per 100K events)
  • Compliance coverage for sensitive financial data
  • Reduced noise from routine ETL operations

Fintech Positioning:

"CloudTrail org/account trail for management events is enabled; for high-risk buckets/prefixes S3 data events are selectively enabled and retain them per policy."

3. Documentation Created​

New Documentation:

  • docs/technical/TOOLING_AND_CONTROLS.md: Complete inventory of implemented vs recommended services with architectural rationale

Updated Documentation:

  • docs/technical/AWS_SERVICES_ANALYSIS.md: Enhanced with architectural reasoning for why certain tools aren't used

Deployment Steps​

Pre-Deployment Checklist​

  • Review Terraform changes in tasks/devops_cicd/infra/terraform/main.tf
  • Verify KMS key policy allows required services (Glue, Athena)
  • Confirm CloudTrail data events configuration matches compliance requirements
  • Test Terraform plan: terraform plan

Deployment Process​

  1. Terraform Plan Review

    cd tasks/devops_cicd/infra/terraform
    terraform init
    terraform plan

  2. Review Plan Output

    • Verify KMS key creation
    • Verify S3 encryption changes (gold, quarantine buckets)
    • Verify CloudTrail data events configuration
    • Check for any unexpected changes

  3. Apply Changes

    terraform apply

  4. Post-Deployment Verification

    • Verify KMS key exists: aws kms describe-key --key-id alias/ohpen-data-lake-sensitive
    • Verify S3 encryption: aws s3api get-bucket-encryption --bucket ohpen-gold
    • Verify CloudTrail data events: aws cloudtrail get-event-selectors --trail-name ohpen-data-lake-audit

Expected Changes​

New Resources:

  • aws_kms_key.data_lake_sensitive - KMS key for sensitive data encryption
  • aws_kms_alias.data_lake_sensitive - KMS alias for easier key management

Modified Resources:

  • aws_s3_bucket_server_side_encryption_configuration.data_lake_encryption["ohpen-gold"] - Changed from SSE-S3 to SSE-KMS
  • aws_s3_bucket_server_side_encryption_configuration.data_lake_encryption["ohpen-quarantine"] - Changed from SSE-S3 to SSE-KMS
  • aws_cloudtrail.main - Added selective data events for sensitive buckets

No Breaking Changes:

  • Existing data remains accessible (encryption is transparent to applications)
  • Glue jobs will automatically use KMS key (via IAM role permissions)
  • Athena queries will automatically decrypt (via IAM role permissions)

Cost Impact​

KMS Costs​

  • Key: $1/month per key
  • Requests: $0.03 per 10,000 requests
  • Estimated Monthly Cost: ~$1-5/month (depending on query volume)

CloudTrail Data Events Costs​

  • Data Events: $0.10 per 100,000 events
  • Estimated Monthly Cost: ~$5-20/month (depending on access patterns to gold/quarantine buckets)

Total Estimated Additional Cost: ~$6-25/month

Rollback Plan​

If issues occur:

  1. Revert S3 Encryption (if needed):

    # Temporarily revert to SSE-S3
    sse_algorithm = "AES256"
    kms_master_key_id = null

  2. Disable CloudTrail Data Events (if cost concerns):

    # Remove data_events block from CloudTrail

  3. Keep KMS Key: Even if not actively used, keeping the key does not incur significant cost ($1/month)

Compliance Benefits​

Fintech Audit Readiness​

  • Encryption: Customer-managed keys (CMK) for sensitive data.
  • Audit trail: CloudTrail data events for sensitive bucket access.
  • Key management: Automatic key rotation enabled.
  • Access control: Fine-grained KMS key policies.

Documentation​

All architectural decisions and tooling choices are now documented in:

  • docs/technical/TOOLING_AND_CONTROLS.md - Complete tooling inventory
  • docs/technical/AWS_SERVICES_ANALYSIS.md - Architectural rationale

Next Steps (Optional Enhancements)​

  1. Secrets Manager: Add if external API credentials needed
  2. Lambda: Add for pre/post-processing automation
  3. DynamoDB: Add for operational metadata (run tracking, quality metrics)
  4. Glue Data Quality: Add for standardized quality checks

See docs/technical/TOOLING_AND_CONTROLS.md for detailed recommendations.

© 2026 Stephen Adei•CC BY 4.0