> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Vault architecture

# Document Vault System Architecture

## Overview

The Document Vault system provides secure storage, processing, and AI-powered semantic search for financial documents. It allows users to upload, organize, and search documents like tax forms, insurance policies, and legal documents. The system uses Reducto for document parsing and OpenAI embeddings for semantic search.

## Architecture Components

### Frontend Components

* **Vault Page (`/app/vault/page.tsx`)**: Main entry point for the vault UI
* **Folder List (`/app/vault/components/folder-list.tsx`)**: Displays and manages folders
* **Document List (`/app/vault/components/document-list.tsx`)**: Displays and manages documents
* **Document Viewer**: Renders PDFs and other document types
* **Search Interface**: Allows semantic search across document contents

### Backend API Routes

* **Process Document (`/app/api/vault/process-document/`)**: Handles document uploads and initiates processing
* **Poll Document Status (`/app/api/vault/poll-document-status/`)**: Checks document processing status and extracts chunks
* **Fetch Document Chunks (`/app/api/vault/fetch-document-chunks/`)**: Retrieves chunks for a specific document

### Search Implementation

* **Document Search Tool (`/app/api/chat/document-search-tool.ts`)**: Provides a semantic search tool for the chat system
* **Document Retriever (`/app/api/chat/document-retriever.ts`)**: Implements vector search using pgvector
* **Vector Matching**: Uses Supabase's `match_document_chunks` RPC function for similarity search

### External Services

* **Reducto**: AI document parsing service
  * Processes documents (PDFs, images, etc.)
  * Extracts text and metadata
  * Splits documents into semantic chunks
* **OpenAI**: Provides embeddings for semantic search
* **Supabase**: Database and storage
  * Stores document metadata and chunks
  * Handles vector search via pgvector

## Document Processing Flow

1. **Upload Phase**
   * User uploads document through the UI
   * Upload dialog closes immediately for better UX
   * Toast notifications inform user of upload progress
   * File is uploaded to Supabase Storage
   * Document metadata is created in `vault_documents` table with status "pending"
   * Processing job is initiated via Reducto SDK in the background

2. **Processing Phase** (Separate Concerns)
   * Frontend polls status endpoint to check processing progress (every 5s)
   * `poll-document-status` endpoint only checks Reducto job status
   * Status progression: "pending" → "processing" → "completed"
   * Document is displayed with appropriate status badge in the UI
   * When Reducto job completes, frontend triggers chunk extraction

3. **Chunk Extraction Phase**
   * `fetch-document-chunks` endpoint is called exactly once per document
   * Endpoint retrieves chunks from Reducto API
   * Frontend maintains tracking of processed documents to prevent duplicate calls
   * Backend extracts text from various chunk formats (handles structural variations)

4. **Embedding Generation Phase**
   * Text from each chunk is processed in batches (10 chunks at a time)
   * OpenAI embeddings are generated for each chunk
   * Embeddings are stored in the `vault_document_chunks` table via pgvector
   * Document status is updated to "completed" when all chunks are processed

5. **Search Phase**
   * User enters a search query
   * Query is converted to an embedding
   * Vector similarity search is performed against chunk embeddings
   * Matching chunks are returned sorted by relevance
   * UI displays results with document references

## Database Schema

### `vault_folders` Table

* `id`: Primary key
* `name`: Folder name
* `clerk_id`: User ID
* `created_at`: Timestamp
* `updated_at`: Timestamp

### `vault_documents` Table

* `id`: Primary key
* `name`: Document name
* `folder_uuid`: Foreign key to vault\_folders
* `clerk_id`: User ID
* `path`: Storage path in Supabase storage
* `processing_status`: Processing status (pending, processing, completed, error)
* `content_type`: MIME type of the document
* `file_size`: File size in bytes
* `source`: Source of the document (e.g., "vault")
* `external_id`: Reducto job ID for processing
* `created_at`: Timestamp
* `updated_at`: Timestamp
* `chunk_count`: Number of chunks successfully processed

### `vault_document_chunks` Table

* `id`: Primary key
* `document_id`: Foreign key to vault\_documents
* `chunk_index`: Sequential index within document
* `content`: Text content of chunk
* `metadata`: JSON containing additional chunk data
  * `blocks`: Layout blocks
  * `page`: Page number
  * `document_type`: Detected document type
  * `title`: Chunk title or heading
  * `text`: Raw text content (may duplicate content field)
  * `value`: Alternative text content field
  * `year`: Detected document year
* `embedding`: Vector embedding for semantic search (pgvector type)

## User Experience Flow

1. User navigates to the Vault section
2. The first folder is auto-selected if available
3. User can:
   * Create/rename/delete folders
   * Upload documents to folders
   * View documents in the selected folder
   * Search across all documents
   * Delete documents
   * View document details and content

## Implementation Details

### Reducto Integration

The system uses the official Reducto SDK for document processing with separation of concerns:

```typescript theme={null}
// Initialize client
const reducto = getReductoClient();

// Process document (in process-document endpoint)
const parseJob = await reducto.parse({
  file: fileBuffer,
  filename: document.name
});

// Save job ID to document record
await supabase
  .from('vault_documents')
  .update({ external_id: parseJob.id, processing_status: 'processing' })
  .eq('id', documentId);

// Poll job status (in poll-document-status endpoint)
const status = await reducto.jobs.get(jobId);

// Extract chunks (in fetch-document-chunks endpoint)
const response = await reducto.jobs.results(jobId);
const chunks = response?.result?.chunks || response?.chunks || [];
```

### Embedding Generation

Embeddings are generated in batches using OpenAI with error handling and text preprocessing:

```typescript theme={null}
// Extract text for embedding from different possible chunk structures
let chunkText = '';
if (typeof chunk === 'string') {
  chunkText = chunk;
} else if (chunk.content) {
  chunkText = chunk.content;
} else if (chunk.text) {
  chunkText = chunk.text;
} else if (chunk.value) {
  chunkText = chunk.value;
}

// Truncate text to avoid token limits
const truncatedText = chunkText.slice(0, 8000);

// Process chunks in batches to respect API limits
for (let i = 0; i < chunkRecords.length; i += batchSize) {
  const batch = chunkRecords.slice(i, i + batchSize);
  
  // Generate embeddings with OpenAI
  try {
    const embedding = await getEmbedding(truncatedText);
    // Add to chunk record
    enhancedBatch.push({
      ...chunkRecord,
      embedding
    });
    successCount++;
  } catch (error) {
    console.error(`Error generating embedding for chunk ${i}:`, error);
    // Continue with other chunks even if this one failed
  }
}
```

### Chunk Processing Optimizations

The system optimizes chunk processing through:

1. Frontend tracking of already processed documents to prevent duplicate calls
2. Batch processing of embeddings (10 chunks at a time) to manage API rate limits
3. Robust extraction of text from various chunk formats (string, content, text, value)
4. Error tolerance - continues processing chunks even if some embedding generations fail

## Best Practices

1. **Document Types**: The system is optimized for financial documents like:
   * Tax Forms
   * Insurance Policies
   * Trust & Estate Documents
   * Real Estate Documents
   * Private Investment Documents
   * Legal Documents

2. **File Formats**: The system supports:
   * PDF files (preferred)
   * Images (PNG, JPG)
   * Text documents

3. **Processing Times**: Large documents may take longer to process. The UI provides feedback during processing.

4. **Search Tips**: Semantic search works best with specific queries rather than general terms.

## Troubleshooting

### Common Issues and Solutions

#### Document Processing Issues

* **Document stuck in "pending" status**:
  * Check frontend logs for errors during document upload
  * Verify that the document was properly uploaded to Supabase storage
  * Ensure the Reducto API key is valid and has sufficient credits
  * Check if the API call to `process-document` endpoint completed successfully

* **Document stuck in "processing" status**:
  * Check if the Reducto job was properly created (check `external_id` in document record)
  * Check Reducto job status directly via the Reducto dashboard
  * Verify that `poll-document-status` endpoint is being called by the frontend
  * Look for errors in the `fetch-document-chunks` API call
  * Try manually calling the `fetch-document-chunks` endpoint with the document ID

* **Chunks not being generated**:
  * Check if frontend is calling `fetch-document-chunks` after job completion
  * Look for errors in the embedding generation process in server logs
  * Check that the Reducto job is actually returning chunks
  * Verify the structure of chunks in the Reducto response

* **Multiple calls to fetch chunks**:
  * Check the `chunksAlreadyFetched` Set in frontend tracking
  * Look for race conditions in the status polling logic

#### Search Issues

* **Search not returning expected results**:
  * Verify chunks were successfully generated with embeddings
  * Check chunk count in the document record matches expected document size
  * Try shorter, more specific search queries
  * Look for errors during embedding generation that might have skipped chunks

#### Performance Issues

* **Slow document processing**:
  * Check document size and complexity (large documents take longer)
  * Monitor batch sizes for embedding generation (default: 10 chunks)
  * Consider adjusting polling interval (currently 5 seconds)

### Debugging Tools

* Frontend console logs for document status polling and chunk fetching
* Backend server logs for API endpoint calls and embedding generation
* Supabase database explorer for direct inspection of document records
* Reducto dashboard for job status and parsing results


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.