​

Radar System Observability Guide

​

Overview

​

Logging Enhancements

​

Structured JSON Logging

{
  "timestamp": "2025-11-19T16:30:00.000Z",
  "level": "INFO",
  "message": "Orchestrator execution started",
  "execution_id": "abc123-def456-ghi789",
  "lambda_request_id": "aws-request-id-123",
  "event_source": "aws.events"
}

​

Key Log Fields

• timestamp - ISO 8601 timestamp
• level - Log level (INFO, WARNING, ERROR, DEBUG)
• message - Human-readable description
• execution_id - Unique ID for tracking across services
• lambda_request_id - AWS Lambda request ID

• target_date - Date for which schedules are being processed
• schedules_found - Number of schedules due
• queued_count - Number successfully queued
• failed_count - Number that failed to queue
• batch_number - SQS batch number
• user_id - User ID from schedule
• schedule_id - Unique schedule identifier
• radar_type - Type of radar
• enabled - Whether schedule is enabled

• record_index - Position in batch
• api_duration_ms - Time spent calling chat API
• model_used - AI model used
• input_tokens / output_tokens - Token usage
• tool_calls_count - Number of tool calls made
• processing_time_ms - Total processing time
• next_run - Calculated next run time

• request_id - Unique request ID
• http_method - HTTP method (GET, POST, etc.)
• path - API endpoint path
• operation - Specific operation performed
• operation_duration_ms - Operation duration

​

Viewing Logs in CloudWatch

​

CloudWatch Log Groups

• /aws/lambda/{env}-RadarOrchestrator
• /aws/lambda/{env}-RadarProcessor
• /aws/lambda/{env}-RadarAPI

​

Useful CloudWatch Insights Queries

​

1. Find Why Schedules Didn’t Run

fields @timestamp, message, schedule_id, user_id, enabled, next_run, radar_type
| filter message = "Schedule queued for processing"
| filter enabled = false
| sort @timestamp desc
| limit 100

​

2. Track Execution Performance

fields @timestamp, execution_id, execution_duration_ms, total_schedules, queued_count, failed_count
| filter message = "Orchestrator execution completed"
| stats avg(execution_duration_ms) as avg_duration, max(execution_duration_ms) as max_duration by bin(5m)

​

3. Find Processor Errors

fields @timestamp, execution_id, schedule_id, user_id, error, error_type, traceback
| filter level = "ERROR"
| filter message = "Error processing record"
| sort @timestamp desc
| limit 50

​

4. Track API Call Performance

fields @timestamp, schedule_id, api_duration_ms, model_used, input_tokens, output_tokens
| filter message = "Internal chat API call successful"
| stats avg(api_duration_ms) as avg_api_time, sum(input_tokens) as total_input, sum(output_tokens) as total_output by schedule_id

​

5. Monitor Token Usage

fields @timestamp, schedule_id, input_tokens, output_tokens
| filter message = "Internal chat API call successful"
| stats sum(input_tokens) as total_input, sum(output_tokens) as total_output by bin(1h)

​

6. Find All Errors with Full Context

fields @timestamp, execution_id, schedule_id, user_id, error, error_type
| filter level = "ERROR"
| sort @timestamp desc

​

7. Track Schedule Processing Rate

fields @timestamp, record_duration_ms, schedule_id
| filter message = "Radar processed successfully"
| stats avg(record_duration_ms) as avg_processing_time, count() as total_processed by bin(1h)

​

8. Monitor DynamoDB Operations

fields @timestamp, operation, operation_duration_ms, user_id
| filter operation_duration_ms > 1000
| sort operation_duration_ms desc
| limit 50

​

CloudWatch Alarms

​

1. Orchestrator Errors

• Alarm: {env}-radar-orchestrator-errors
• Threshold: 1 error in 5 minutes
• Action: Investigate why schedules aren’t being queued

​

2. Processor Errors

• Alarm: {env}-radar-processor-errors
• Threshold: 3 errors in 10 minutes (2 consecutive periods)
• Action: Check DLQ and investigate failed radar executions

​

3. Dead Letter Queue Messages

• Alarm: {env}-radar-dlq-messages
• Threshold: Any message in DLQ
• Action: Messages failed after 3 retries - manual investigation required

​

4. API Gateway Errors

• Alarm: {env}-radar-api-errors
• Threshold: 5 errors in 10 minutes (2 consecutive periods)
• Action: Check API logs for client or server errors

​

X-Ray Tracing

​

Viewing X-Ray Traces

1. Go to AWS X-Ray Console
2. Select “Service Map” to see system architecture
3. Select “Traces” to view individual executions
4. Filter by:
   • Service name: radar-orchestrator, radar-processor, radar-api
   • HTTP status code
   • Response time
   • Error status

​

X-Ray Segments

• Lambda invocation duration
• DynamoDB operations
• SQS operations
• External HTTP calls (chat API)
• Subsegment timing

​

Troubleshooting Common Issues

​

Issue: Radar Not Running

1. Check CloudWatch logs for orchestrator around 6am UTC
2. Search for: message = "Schedule queued for processing" AND schedule_id = "your-schedule-id"
3. Look for:
   • enabled = false - Schedule is disabled
   • next_run date mismatch - Schedule not due yet
   • Missing schedule - Not in query results

fields @timestamp, schedule_id, enabled, next_run, next_run_date
| filter user_id = "your-user-id"
| sort @timestamp desc

​

Issue: Radar Processing Failed

1. Check processor logs for errors
2. Look for the execution_id in traceback
3. Check if message ended up in DLQ

fields @timestamp, execution_id, schedule_id, error, traceback
| filter schedule_id = "your-schedule-id"
| filter level = "ERROR"
| sort @timestamp desc

​

Issue: Slow Performance

1. Check X-Ray traces for bottlenecks
2. Review API call durations
3. Check DynamoDB query performance

fields @timestamp, api_duration_ms, db_update_duration_ms, record_duration_ms
| filter schedule_id = "your-schedule-id"
| stats avg(api_duration_ms) as avg_api, avg(record_duration_ms) as avg_total

​

Issue: High Token Usage

fields @timestamp, schedule_id, input_tokens, output_tokens
| filter user_id = "your-user-id"
| stats sum(input_tokens) as total_input, sum(output_tokens) as total_output by schedule_id

​

Metrics Dashboard

1. Orchestrator Metrics:
   • Lambda invocations
   • Duration
   • Errors
   • Schedules queued per execution
2. Processor Metrics:
   • Lambda invocations (from SQS)
   • Duration
   • Errors
   • API call latency
   • Token usage
3. API Metrics:
   • Request count by endpoint
   • Latency by endpoint
   • 4xx/5xx errors
   • Request rate
4. Queue Metrics:
   • Messages in queue
   • Messages in DLQ
   • Age of oldest message

​

Best Practices

1. Regular Monitoring:
   • Check CloudWatch dashboards daily
   • Review alarms weekly
   • Analyze token usage monthly
2. Log Retention:
   • Logs retained for 2 weeks
   • Export important logs to S3 for long-term storage
   • Create metric filters for key events
3. Alarm Actions:
   • Set up SNS topics for alarm notifications
   • Create runbooks for each alarm type
   • Test alarm thresholds regularly
4. Performance Optimization:
   • Monitor API call durations
   • Track DynamoDB query performance
   • Review batch processing efficiency

​

Example: Complete Investigation Flow

1. Find the schedule:

fields @timestamp, schedule_id, enabled, next_run, radar_type
| filter user_id = "user_123"
| filter radar_type = "PORTFOLIO_CHECK"

2. Check if it was queued:

fields @timestamp, execution_id, schedule_id, enabled, next_run
| filter message = "Schedule queued for processing"
| filter schedule_id = "sched_xyz"
| sort @timestamp desc

3. Check processing status:

fields @timestamp, execution_id, status, error
| filter schedule_id = "sched_xyz"
| filter message = "Radar processed successfully" or message = "Error processing record"
| sort @timestamp desc

4. Review full execution trace:
   • Go to X-Ray Console
   • Filter by execution_id
   • Review timing and errors
5. Check DLQ:
   • View SQS DLQ in AWS Console
   • Check for failed messages
   • Review failure reasons

​

Support

• Review X-Ray service map for architecture overview
• Check CloudWatch metrics for system health
• Analyze structured logs for detailed execution flow
• Monitor alarms for proactive issue detection