Skip to main content

​
DynamoDB to Supabase Streaming - Technical Design

​
Data Flow Architecture

Real-time streaming pipeline that moves consultation data from AWS DynamoDB to Supabase for analytics and reporting.

​
Schema Mapping

​
Source: DynamoDB ConsultationsTable

  • userId (String) - Partition Key
  • consultationId (String) - Sort Key
  • title, question, status, response, charts
  • createdAt, updatedAt (ISO timestamps)

​
Destination: Supabase scenarios table

  • user_id - Extracted from DynamoDB record
  • consultation_id - Extracted from DynamoDB record
  • scenario_data (JSONB) - Complete DynamoDB record as JSON
  • dynamo_sequence_number - For idempotency (prevents duplicates)
  • dynamo_event_name - INSERT, MODIFY, REMOVE operations

​
Key Design Decisions

​
JSON Storage Strategy

The system stores the entire DynamoDB record in the scenario_data JSONB field rather than mapping individual fields. This provides:
  • Schema flexibility - handles any DynamoDB schema changes
  • Complete data preservation - no data loss during streaming
  • Queryable structure - JSONB allows efficient queries on nested data

​
Idempotency & Error Handling

  • Sequence Numbers: Uses DynamoDB sequence numbers as unique keys to prevent duplicate processing
  • Upsert Operations: ON CONFLICT ensures safe retries and reprocessing
  • Dead Letter Queue: Failed messages are captured for manual review and replay
  • VPC Static IP: NAT Gateway provides consistent IP for Supabase network whitelisting

​
Environment Isolation

  • Dev Environment: dev- prefixed resources in separate CloudFormation stack
  • Production Safety: Existing AgentStack remains completely untouched
  • Resource Separation: Independent VPCs, Lambda functions, and DynamoDB tables per environment

​
Infrastructure Components

  • Lambda Processor: Processes DynamoDB stream events, transforms data, writes to Supabase
  • VPC with NAT Gateway: Provides static IP for secure Supabase connectivity
  • DynamoDB Streams: Captures real-time changes (NEW_AND_OLD_IMAGES)
  • Secrets Manager: Stores Supabase credentials securely
  • SQS Dead Letter Queue: Handles processing failures with retry logic

​
Benefits

✅ Real-time sync - Data appears in Supabase within seconds
✅ Complete audit trail - All DynamoDB operations tracked
✅ Schema evolution - Handles future DynamoDB changes automatically
✅ Production safe - Zero impact on existing systems
✅ Scalable - Handles high-volume consultation creation