​

Document Vault System Architecture

​

Overview

​

Architecture Components

​

Frontend Components

• Vault Page (/app/vault/page.tsx): Main entry point for the vault UI
• Folder List (/app/vault/components/folder-list.tsx): Displays and manages folders
• Document List (/app/vault/components/document-list.tsx): Displays and manages documents
• Document Viewer: Renders PDFs and other document types
• Search Interface: Allows semantic search across document contents

​

Backend API Routes

• Process Document (/app/api/vault/process-document/): Handles document uploads and initiates processing
• Poll Document Status (/app/api/vault/poll-document-status/): Checks document processing status and extracts chunks
• Fetch Document Chunks (/app/api/vault/fetch-document-chunks/): Retrieves chunks for a specific document

​

Search Implementation

• Document Search Tool (/app/api/chat/document-search-tool.ts): Provides a semantic search tool for the chat system
• Document Retriever (/app/api/chat/document-retriever.ts): Implements vector search using pgvector
• Vector Matching: Uses Supabase’s match_document_chunks RPC function for similarity search

​

External Services

• Reducto: AI document parsing service
  • Processes documents (PDFs, images, etc.)
  • Extracts text and metadata
  • Splits documents into semantic chunks
• OpenAI: Provides embeddings for semantic search
• Supabase: Database and storage
  • Stores document metadata and chunks
  • Handles vector search via pgvector

​

Document Processing Flow

1. Upload Phase
   • User uploads document through the UI
   • Upload dialog closes immediately for better UX
   • Toast notifications inform user of upload progress
   • File is uploaded to Supabase Storage
   • Document metadata is created in vault_documents table with status “pending”
   • Processing job is initiated via Reducto SDK in the background
2. Processing Phase (Separate Concerns)
   • Frontend polls status endpoint to check processing progress (every 5s)
   • poll-document-status endpoint only checks Reducto job status
   • Status progression: “pending” → “processing” → “completed”
   • Document is displayed with appropriate status badge in the UI
   • When Reducto job completes, frontend triggers chunk extraction
3. Chunk Extraction Phase
   • fetch-document-chunks endpoint is called exactly once per document
   • Endpoint retrieves chunks from Reducto API
   • Frontend maintains tracking of processed documents to prevent duplicate calls
   • Backend extracts text from various chunk formats (handles structural variations)
4. Embedding Generation Phase
   • Text from each chunk is processed in batches (10 chunks at a time)
   • OpenAI embeddings are generated for each chunk
   • Embeddings are stored in the vault_document_chunks table via pgvector
   • Document status is updated to “completed” when all chunks are processed
5. Search Phase
   • User enters a search query
   • Query is converted to an embedding
   • Vector similarity search is performed against chunk embeddings
   • Matching chunks are returned sorted by relevance
   • UI displays results with document references

​

Database Schema

​

vault_folders Table

• id: Primary key
• name: Folder name
• clerk_id: User ID
• created_at: Timestamp
• updated_at: Timestamp

​

vault_documents Table

• id: Primary key
• name: Document name
• folder_uuid: Foreign key to vault_folders
• clerk_id: User ID
• path: Storage path in Supabase storage
• processing_status: Processing status (pending, processing, completed, error)
• content_type: MIME type of the document
• file_size: File size in bytes
• source: Source of the document (e.g., “vault”)
• external_id: Reducto job ID for processing
• created_at: Timestamp
• updated_at: Timestamp
• chunk_count: Number of chunks successfully processed

​

vault_document_chunks Table

• id: Primary key
• document_id: Foreign key to vault_documents
• chunk_index: Sequential index within document
• content: Text content of chunk
• metadata: JSON containing additional chunk data
  • blocks: Layout blocks
  • page: Page number
  • document_type: Detected document type
  • title: Chunk title or heading
  • text: Raw text content (may duplicate content field)
  • value: Alternative text content field
  • year: Detected document year
• embedding: Vector embedding for semantic search (pgvector type)

​

User Experience Flow

1. User navigates to the Vault section
2. The first folder is auto-selected if available
3. User can:
   • Create/rename/delete folders
   • Upload documents to folders
   • View documents in the selected folder
   • Search across all documents
   • Delete documents
   • View document details and content

​

Implementation Details

​

Reducto Integration

// Initialize client
const reducto = getReductoClient();

// Process document (in process-document endpoint)
const parseJob = await reducto.parse({
  file: fileBuffer,
  filename: document.name
});

// Save job ID to document record
await supabase
  .from('vault_documents')
  .update({ external_id: parseJob.id, processing_status: 'processing' })
  .eq('id', documentId);

// Poll job status (in poll-document-status endpoint)
const status = await reducto.jobs.get(jobId);

// Extract chunks (in fetch-document-chunks endpoint)
const response = await reducto.jobs.results(jobId);
const chunks = response?.result?.chunks || response?.chunks || [];

​

Embedding Generation

// Extract text for embedding from different possible chunk structures
let chunkText = '';
if (typeof chunk === 'string') {
  chunkText = chunk;
} else if (chunk.content) {
  chunkText = chunk.content;
} else if (chunk.text) {
  chunkText = chunk.text;
} else if (chunk.value) {
  chunkText = chunk.value;
}

// Truncate text to avoid token limits
const truncatedText = chunkText.slice(0, 8000);

// Process chunks in batches to respect API limits
for (let i = 0; i < chunkRecords.length; i += batchSize) {
  const batch = chunkRecords.slice(i, i + batchSize);
  
  // Generate embeddings with OpenAI
  try {
    const embedding = await getEmbedding(truncatedText);
    // Add to chunk record
    enhancedBatch.push({
      ...chunkRecord,
      embedding
    });
    successCount++;
  } catch (error) {
    console.error(`Error generating embedding for chunk ${i}:`, error);
    // Continue with other chunks even if this one failed
  }
}

​

Chunk Processing Optimizations

1. Frontend tracking of already processed documents to prevent duplicate calls
2. Batch processing of embeddings (10 chunks at a time) to manage API rate limits
3. Robust extraction of text from various chunk formats (string, content, text, value)
4. Error tolerance - continues processing chunks even if some embedding generations fail

​

Best Practices

1. Document Types: The system is optimized for financial documents like:
   • Tax Forms
   • Insurance Policies
   • Trust & Estate Documents
   • Real Estate Documents
   • Private Investment Documents
   • Legal Documents
2. File Formats: The system supports:
   • PDF files (preferred)
   • Images (PNG, JPG)
   • Text documents
3. Processing Times: Large documents may take longer to process. The UI provides feedback during processing.
4. Search Tips: Semantic search works best with specific queries rather than general terms.

​

Troubleshooting

​

Common Issues and Solutions

​

Document Processing Issues

• Document stuck in “pending” status:
  • Check frontend logs for errors during document upload
  • Verify that the document was properly uploaded to Supabase storage
  • Ensure the Reducto API key is valid and has sufficient credits
  • Check if the API call to process-document endpoint completed successfully
• Document stuck in “processing” status:
  • Check if the Reducto job was properly created (check external_id in document record)
  • Check Reducto job status directly via the Reducto dashboard
  • Verify that poll-document-status endpoint is being called by the frontend
  • Look for errors in the fetch-document-chunks API call
  • Try manually calling the fetch-document-chunks endpoint with the document ID
• Chunks not being generated:
  • Check if frontend is calling fetch-document-chunks after job completion
  • Look for errors in the embedding generation process in server logs
  • Check that the Reducto job is actually returning chunks
  • Verify the structure of chunks in the Reducto response
• Multiple calls to fetch chunks:
  • Check the chunksAlreadyFetched Set in frontend tracking
  • Look for race conditions in the status polling logic

​

Search Issues

• Search not returning expected results:
  • Verify chunks were successfully generated with embeddings
  • Check chunk count in the document record matches expected document size
  • Try shorter, more specific search queries
  • Look for errors during embedding generation that might have skipped chunks

​

Performance Issues

• Slow document processing:
  • Check document size and complexity (large documents take longer)
  • Monitor batch sizes for embedding generation (default: 10 chunks)
  • Consider adjusting polling interval (currently 5 seconds)

​

Debugging Tools

• Frontend console logs for document status polling and chunk fetching
• Backend server logs for API endpoint calls and embedding generation
• Supabase database explorer for direct inspection of document records
• Reducto dashboard for job status and parsing results