> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dynamodb supabase streaming technical design

# DynamoDB to Supabase Streaming - Technical Design

## Data Flow Architecture

**Real-time streaming pipeline that moves consultation data from AWS DynamoDB to Supabase for analytics and reporting.**

```
DynamoDB ConsultationsTable → DynamoDB Streams → Lambda Processor → VPC NAT Gateway → Supabase scenarios table
```

## Schema Mapping

### Source: DynamoDB ConsultationsTable

* `userId` (String) - Partition Key
* `consultationId` (String) - Sort Key
* `title`, `question`, `status`, `response`, `charts`
* `createdAt`, `updatedAt` (ISO timestamps)

### Destination: Supabase scenarios table

* `user_id` - Extracted from DynamoDB record
* `consultation_id` - Extracted from DynamoDB record
* `scenario_data` (JSONB) - **Complete DynamoDB record as JSON**
* `dynamo_sequence_number` - For idempotency (prevents duplicates)
* `dynamo_event_name` - INSERT, MODIFY, REMOVE operations

## Key Design Decisions

### JSON Storage Strategy

The system stores the **entire DynamoDB record** in the `scenario_data` JSONB field rather than mapping individual fields. This provides:

* **Schema flexibility** - handles any DynamoDB schema changes
* **Complete data preservation** - no data loss during streaming
* **Queryable structure** - JSONB allows efficient queries on nested data

### Idempotency & Error Handling

* **Sequence Numbers**: Uses DynamoDB sequence numbers as unique keys to prevent duplicate processing
* **Upsert Operations**: `ON CONFLICT` ensures safe retries and reprocessing
* **Dead Letter Queue**: Failed messages are captured for manual review and replay
* **VPC Static IP**: NAT Gateway provides consistent IP for Supabase network whitelisting

### Environment Isolation

* **Dev Environment**: `dev-` prefixed resources in separate CloudFormation stack
* **Production Safety**: Existing `AgentStack` remains completely untouched
* **Resource Separation**: Independent VPCs, Lambda functions, and DynamoDB tables per environment

## Infrastructure Components

* **Lambda Processor**: Processes DynamoDB stream events, transforms data, writes to Supabase
* **VPC with NAT Gateway**: Provides static IP for secure Supabase connectivity
* **DynamoDB Streams**: Captures real-time changes (NEW\_AND\_OLD\_IMAGES)
* **Secrets Manager**: Stores Supabase credentials securely
* **SQS Dead Letter Queue**: Handles processing failures with retry logic

## Benefits

✅ **Real-time sync** - Data appears in Supabase within seconds\
✅ **Complete audit trail** - All DynamoDB operations tracked\
✅ **Schema evolution** - Handles future DynamoDB changes automatically\
✅ **Production safe** - Zero impact on existing systems\
✅ **Scalable** - Handles high-volume consultation creation


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.