> ## Documentation Index > Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt > Use this file to discover all available pages before exploring further. # Scenarios feature engineering guide # Scenarios Feature - Engineering Documentation > ⚠️ **IMPORTANT NOTICE (as of 9/13/2025)**: This feature currently operates in **PRODUCTION ONLY**. There is no development or test environment available. All testing and development work must be done carefully in the production environment. Please coordinate with the team before making any infrastructure changes. ## Overview The **Scenarios** feature is an AI-powered financial analysis tool that allows users to explore "what-if" scenarios in financial markets. Users can ask questions about market conditions, portfolio impacts, and investment strategies, receiving detailed analysis with charts and insights. **Key Capabilities:** * AI-powered financial scenario analysis * Real-time chart generation * Code execution tracking * Asynchronous processing with status updates * Infinite scroll pagination * Analytics tracking *** ## System Architecture Overview ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ │ │ │ │ Frontend │ │ API Gateway │ │ Lambda API │ │ (Next.js) │───▶│ (HTTP API) │───▶│ (30s, 1GB) │ │ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │ 5s polling │ │ ▼ │ ┌─────────────────┐ │ │ │ │ │ SQS Queue │ │ │ (15min vis) │ │ │ │ │ └─────────────────┘ │ │ │ │ triggers │ ▼ │ ┌─────────────────┐ │ │ │ │ │ Lambda Processor│ │ │ (15min, 3GB) │ │ │ │ │ └─────────────────┘ │ │ │ │ updates │ ▼ │ ┌─────────────────┐ │ │ │ │ │ DynamoDB │ │ │ • Consultations │ │ │ • Task Status │ │ │ │ │ └─────────────────┘ │ │ displays charts │ │ │ stores ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ │ │ │ │ S3 Bucket │◀─────────────────────────│ AI Processing │ │ (Charts) │ uploads │ • Data Analysis│ │ │ │ • Chart Gen │ └─────────────────┘ │ • Code Exec │ │ │ └─────────────────┘ Data Flow: 1. User submits scenario → API saves to DynamoDB → Message sent to SQS 2. SQS triggers Processor Lambda → AI analysis (1-10 min) → Charts to S3 3. Frontend polls status every 5s → Displays results when complete ``` ## Architecture ### Frontend (Next.js) * **Location**: `app/scenarios/` * **Main Component**: `page.tsx` - React component with real-time updates * **Layout**: Uses `LayoutWithSidebar` for consistent navigation ### Backend (AWS) * **Location**: `aws/` * **Infrastructure**: AWS CDK (Cloud Development Kit) * **Stack**: `AgentStack` deployed via CDK * **API**: REST API hosted on AWS Lambda + API Gateway * **Queue**: SQS for asynchronous consultation processing * **Storage**: DynamoDB for data, S3 for charts * **Secrets**: AWS Secrets Manager for API keys *** ## Technical Stack ### Frontend Dependencies ```typescript theme={null} // Key libraries used - React 18+ with hooks - Next.js App Router - Tailwind CSS + Shadcn UI - PostHog (analytics) - React Markdown (response rendering) - Lucide React (icons) ``` ### Backend Infrastructure ```python theme={null} # AWS CDK Stack (aws/ai/stack.py) - AWS Lambda (2 functions: API + Processor) - API Gateway (HTTP API with CORS) - SQS (consultation processing queue + DLQ) - DynamoDB (consultations + task status tables) - S3 (chart storage with lifecycle rules) - Secrets Manager (Anthropic API key) - CloudWatch (logging) ``` *** ## AWS Infrastructure Details ### SQS Queue Architecture ```python theme={null} # Consultation processing queue consultation_queue = sqs.Queue( self, "ConsultationQueue", visibility_timeout=Duration.seconds(900), # 15 minutes dead_letter_queue=sqs.DeadLetterQueue( max_receive_count=3, queue=sqs.Queue(self, "ConsultationDLQ") ) ) ``` ### DynamoDB Tables ```python theme={null} # Main consultations table consultations_table = dynamodb.Table( partition_key="userId", # User isolation sort_key="consultationId", # Unique consultation ID removal_policy=RemovalPolicy.DESTROY ) # Task status tracking table task_status_table = dynamodb.Table( partition_key="consultationId", # Status per consultation time_to_live_attribute="ttl" # Auto-cleanup old statuses ) ``` ### S3 Chart Storage ```python theme={null} charts_bucket = s3.Bucket( bucket_name=f"silvia-charts-{account}-{region}", cors=[s3.CorsRule( allowed_methods=[s3.HttpMethods.GET, s3.HttpMethods.PUT], allowed_origins=["*"], allowed_headers=["*"] )], lifecycle_rules=[ s3.LifecycleRule( expiration=Duration.days(30), # Auto-delete old charts enabled=True ) ] ) ``` ### Lambda Functions ```python theme={null} # 1. Consultation API Lambda (30s timeout, 1GB memory) consultation_api_lambda = _lambda.DockerImageFunction( timeout=Duration.seconds(30), memory_size=1024, environment={ 'CONSULTATIONS_TABLE': consultations_table.table_name, 'TASK_STATUS_TABLE': task_status_table.table_name, 'CONSULTATION_QUEUE_URL': consultation_queue.queue_url, 'CHARTS_S3_BUCKET': charts_bucket.bucket_name } ) # 2. Agent Processor Lambda (15min timeout, 3GB memory) agent_processor_lambda = _lambda.DockerImageFunction( timeout=Duration.seconds(900), # Long-running AI processing memory_size=3008, # High memory for AI workloads environment={ 'ANTHROPIC_SECRET_NAME': anthropic_secret.secret_name, 'CONSULTATIONS_TABLE': consultations_table.table_name, 'TASK_STATUS_TABLE': task_status_table.table_name, 'CHARTS_S3_BUCKET': charts_bucket.bucket_name } ) ``` *** ## Processing Flow ### 1. Consultation Creation Flow ```mermaid theme={null} sequenceDiagram participant Frontend participant API_Lambda participant SQS participant DynamoDB participant Processor_Lambda Frontend->>API_Lambda: POST /consultations API_Lambda->>DynamoDB: Save consultation (status: pending) API_Lambda->>SQS: Send message to queue API_Lambda->>Frontend: Return consultation object SQS->>Processor_Lambda: Trigger processing Processor_Lambda->>DynamoDB: Update status (queued → processing) Processor_Lambda->>Processor_Lambda: AI Analysis (1-10 minutes) Processor_Lambda->>S3: Upload generated charts Processor_Lambda->>DynamoDB: Update final results (completed) ``` ### 2. Status Polling Flow ```mermaid theme={null} sequenceDiagram participant Frontend participant API_Lambda participant DynamoDB loop Every 5 seconds Frontend->>API_Lambda: GET /consultations/{id}/status API_Lambda->>DynamoDB: Query task_status_table API_Lambda->>Frontend: Return current status & progress end ``` *** ## Core Functionality ### 1. Scenario Creation ```typescript theme={null} const createConsultation = async () => { const response = await fetch(`${API_BASE_URL}/consultations`, { method: 'POST', headers: { 'Content-Type': 'application/json', 'X-User-Id': user.id }, body: JSON.stringify({ question: newQuestion.trim() }) }) } ``` ### 2. Real-time Status Polling ```typescript theme={null} const startPolling = (consultationId: string) => { pollingIntervals.current[consultationId] = setInterval(async () => { const response = await fetch(`${API_BASE_URL}/consultations/${consultationId}/status`) // Update status every 5 seconds }, 5000) } ``` ### 3. Infinite Scroll Loading ```typescript theme={null} const loadMoreConsultations = useCallback(async () => { const url = new URL(`${API_BASE_URL}/consultations`) url.searchParams.set('limit', '20') if (lastKey) { url.searchParams.set('lastKey', lastKey) } // Load more scenarios as user scrolls }, [hasMore, isLoadingMore, lastKey]) ``` *** ## API Endpoints ### Base URL ``` Production: https://u712iszkza.execute-api.us-east-1.amazonaws.com Environment Variable: NEXT_PUBLIC_ADVISORY_API_URL ``` ### Endpoints #### 1. Create Consultation ```http theme={null} POST /consultations Headers: X-User-Id: {userId} Body: { "question": "What if Apple drops 20%?" } Response: { "consultationId": "uuid", "title": "Generated title", "question": "User question", "status": "pending", "createdAt": "ISO timestamp" } ``` #### 2. Get Consultations (Paginated) ```http theme={null} GET /consultations?limit=20&lastKey={optional} Headers: X-User-Id: {userId} Response: { "consultations": [...], "hasMore": boolean, "lastKey": "pagination key" } ``` #### 3. Get Consultation Status ```http theme={null} GET /consultations/{consultationId}/status Response: { "consultationId": "uuid", "status": "processing|completed|failed", "progress": 0-100, "intermediateOutput": "Current step description", "updatedAt": "ISO timestamp" } ``` *** ## Data Models ### Consultation Object ```typescript theme={null} interface Consultation { consultationId: string title: string question: string status: 'pending' | 'queued' | 'processing' | 'completed' | 'failed' response?: string // AI-generated analysis charts?: ChartMetadata[] // Generated visualizations executedCode?: CodeExecution[] // Code execution history createdAt: string updatedAt: string } ``` ### Chart Metadata ```typescript theme={null} interface ChartMetadata { filename: string s3_key: string url: string // S3 presigned URL created_at: string } ``` ### Code Execution ```typescript theme={null} interface CodeExecution { timestamp: string method: string code: string filename?: string status: 'running' | 'completed' success?: boolean result?: string order: number // Execution sequence } ``` *** ## Key Features ### 1. Status Management * **Pending**: Consultation created, waiting to start * **Queued**: In processing queue * **Processing**: AI actively analyzing (1-10 minutes) * **Completed**: Analysis finished with results * **Failed**: Error occurred, retry available ### 2. Chart Display * S3-hosted chart images with presigned URLs * Modal expansion for detailed viewing * Lazy loading with loading states * Error handling for failed image loads ### 3. Response Rendering * Markdown support for formatted AI responses * Tabbed interface (Response/Charts/Code) * Collapsible sections for better UX ### 4. Analytics Tracking ```typescript theme={null} // PostHog events tracked: - scenario_creation_started - scenario_created - scenario_creation_failed - scenario_deleted - scenario_retried - scenario_toggled - scenario_tab_switched - scenario_suggested_prompt_used ``` ### 5. Chart Generation & Display ```typescript theme={null} // Chart metadata from S3 interface ChartMetadata { filename: string s3_key: string url: string // S3 presigned URL (expires in 1 hour) created_at: string } // Chart display with modal expansion const ChartsDisplay = ({ charts }: { charts: ChartMetadata[] }) => { const [expandedChart, setExpandedChart] = useState(null) return (
{charts.map((chart, index) => ( setExpandedChart(chart.url)} className="cursor-pointer hover:opacity-90" /> ))}
) } ``` ### 6. Code Execution Timeline ```typescript theme={null} // Code execution tracking interface CodeExecution { timestamp: string method: string // e.g., "data_analysis", "chart_generation" code: string // Python code executed filename?: string // Generated file name status: 'running' | 'completed' success?: boolean // Execution success/failure result?: string // Output or error message order: number // Execution sequence } // Timeline display with expandable code blocks const CodeTimeline = ({ executedCode }: { executedCode: CodeExecution[] }) => { const [expandedBlocks, setExpandedBlocks] = useState>(new Set()) const [filter, setFilter] = useState<'all' | 'success' | 'failed'>('all') return (
{filteredCode.map((block, index) => (
{expandedBlocks.has(index) && ( 
{block.code}
{block.result && ( 
{block.result}
)}
)}
))}
) } ``` *** ## Deployment ### AWS Infrastructure ```bash theme={null} # Deploy CDK stack cd aws/ python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt # Set up environment variables cp ai/.env.example ai/.env # Edit ai/.env with your ANTHROPIC_API_KEY # Deploy the stack cdk bootstrap # First time only cdk deploy AgentStack # Monitor deployment cdk diff # Check changes before deploy cdk destroy # Clean up resources ``` ### SQS Queue Monitoring ```bash theme={null} # Monitor SQS queue aws sqs get-queue-attributes \ --queue-url https://sqs.region.amazonaws.com/account/ConsultationQueue \ --attribute-names All # Check dead letter queue aws sqs receive-message \ --queue-url https://sqs.region.amazonaws.com/account/ConsultationDLQ # Purge queue (development only) aws sqs purge-queue \ --queue-url https://sqs.region.amazonaws.com/account/ConsultationQueue ``` ### Environment Variables ```env theme={null} # Frontend (.env.local) NEXT_PUBLIC_ADVISORY_API_URL=https://your-api-gateway-url # Backend (AWS Lambda) # Set via CDK stack configuration ``` *** ## Development Workflow ### 1. Local Development ```bash theme={null} # Frontend npm run dev # Backend (if running locally) cd aws/ cdk watch # Hot reload for CDK changes ``` ### 2. Testing Scenarios ```typescript theme={null} // Example test scenarios const testPrompts = [ "What if Apple drops 20% next quarter?", "Compare Tesla vs Ford for 2025", "How would rising rates affect tech stocks?", "Analyze the energy sector rotation trend" ] ``` ### 3. Monitoring * CloudWatch logs for Lambda functions * PostHog analytics for user behavior * Real-time status polling for processing updates *** ## Performance Considerations ### 1. Frontend Optimizations * Intersection Observer for infinite scroll * Debounced polling intervals * Optimistic UI updates * Image lazy loading ### 2. Backend Optimizations * **SQS Queue Processing**: Asynchronous consultation processing with dead letter queue * **Lambda Scaling**: Auto-scaling based on queue depth * **DynamoDB**: Optimized for user-scoped queries with TTL for status cleanup * **S3 Chart Storage**: Presigned URLs with 30-day lifecycle policy * **Memory Allocation**: 3GB for AI processor, 1GB for API functions * **Timeout Management**: 15-minute timeout for long-running AI analysis ### 3. Error Handling * Retry mechanism for failed scenarios * Graceful degradation for missing charts * Timeout handling for long-running processes *** ## Security ### Authentication * User ID passed via `X-User-Id` header * Integration with existing auth system * User-scoped data access ### Data Protection * S3 presigned URLs for secure chart access * User data isolation * No sensitive financial data in logs *** ## Future Enhancements ### Planned Features 1. **Real-time Collaboration**: Share scenarios between users 2. **Scenario Templates**: Pre-built scenario types 3. **Export Functionality**: PDF/Excel export of results 4. **Advanced Filtering**: Filter by status, date, topic 5. **Scenario Comparison**: Side-by-side analysis ### Technical Improvements 1. **WebSocket Integration**: Replace polling with real-time updates 2. **Caching Layer**: Redis for frequently accessed scenarios 3. **Batch Processing**: Handle multiple scenarios simultaneously 4. **Advanced Analytics**: More detailed user behavior tracking *** ## Troubleshooting ### Common Issues #### 1. Polling Not Working ```typescript theme={null} // Check if intervals are being cleared properly useEffect(() => { return () => { Object.values(pollingIntervals.current).forEach(clearInterval) } }, []) ``` #### 2. Charts Not Loading * Verify S3 presigned URL validity (expires after 1 hour) * Check CORS configuration in S3 bucket * Ensure proper error handling for failed image loads * Verify chart generation completed successfully in processor logs #### 3. SQS Processing Issues ```bash theme={null} # Check queue depth and processing aws sqs get-queue-attributes \ --queue-url $CONSULTATION_QUEUE_URL \ --attribute-names ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible # Check dead letter queue for failed messages aws sqs receive-message --queue-url $CONSULTATION_DLQ_URL # Redrive messages from DLQ back to main queue aws sqs start-message-move-task \ --source-arn $CONSULTATION_DLQ_ARN \ --destination-arn $CONSULTATION_QUEUE_ARN ``` #### 4. Lambda Function Issues ```bash theme={null} # Check processor lambda logs aws logs tail /aws/lambda/AgentStack-AgentProcessorLambda --follow # Check API lambda logs aws logs tail /aws/lambda/AgentStack-ConsultationApiLambda --follow # Monitor lambda metrics aws cloudwatch get-metric-statistics \ --namespace AWS/Lambda \ --metric-name Duration \ --dimensions Name=FunctionName,Value=AgentStack-AgentProcessorLambda ``` #### 5. DynamoDB Query Issues ```bash theme={null} # Check table status and metrics aws dynamodb describe-table --table-name ConsultationsTable aws dynamodb describe-table --table-name TaskStatusTable # Query consultations for a user aws dynamodb query \ --table-name ConsultationsTable \ --key-condition-expression "userId = :userId" \ --expression-attribute-values '{":userId":{"S":"user-123"}}' # Check task status aws dynamodb get-item \ --table-name TaskStatusTable \ --key '{"consultationId":{"S":"consultation-123"}}' ``` #### 6. Infinite Scroll Issues * Verify Intersection Observer setup * Check `hasMore` and `lastKey` state management * Ensure proper cleanup on unmount ### Debug Commands ```bash theme={null} # Check CDK stack status cdk list cdk diff # View Lambda logs aws logs tail /aws/lambda/your-function-name --follow # Test API endpoints curl -X POST https://your-api-url/consultations \ -H "X-User-Id: test-user" \ -H "Content-Type: application/json" \ -d '{"question": "Test scenario"}' ``` *** ## Contact & Support For technical questions or issues: * **Frontend Issues**: Check React DevTools and browser console * **Backend Issues**: Review CloudWatch logs * **Infrastructure**: Verify CDK stack deployment * **Analytics**: Check PostHog dashboard for event tracking This feature represents a complex integration of AI, real-time updates, and financial data visualization, requiring careful attention to both user experience and system performance. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.