> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Vault document system

# Portfolio Tracker: Vault Document Embedding System Documentation

## Table of Contents

1. [Overview](#overview)
2. [System Architecture](#system-architecture)
3. [Database Schema](#database-schema)
4. [Document Processing Pipeline](#document-processing-pipeline)
5. [User Interface Components](#user-interface-components)
6. [API Endpoints](#api-endpoints)
7. [Vector Search Implementation](#vector-search-implementation)
8. [Chat Integration](#chat-integration)
9. [Security Considerations](#security-considerations)
10. [Performance Optimizations](#performance-optimizations)
11. [Troubleshooting](#troubleshooting)
12. [Future Improvements](#future-improvements)

## Overview

The Vault Document Embedding System is a comprehensive document management solution integrated into the Portfolio Tracker application. It enables users to upload, organize, and semantically search through their documents using advanced natural language processing techniques. The system processes various document types, extracts their text content, splits them into semantically meaningful chunks, generates high-quality vector embeddings, and stores them for efficient retrieval during chat interactions. All document content is securely encrypted at rest, ensuring the highest level of data privacy and security.

### Key Features

* **Document Management**: Upload, organize, rename, and delete documents
* **Folder Organization**: Create and manage folders to organize documents
* **Document Processing**: Extract text from various file formats (PDF, DOCX, TXT, CSV, HTML) with robust fallback mechanisms
* **Content Encryption**: All document content is encrypted using AES encryption with a dedicated vault document encryption key
* **Decryption on Demand**: Document content is only decrypted when needed for search results or display
* **TLS Encryption**: All data in transit is encrypted using TLS
* **Database Encryption**: Supabase provides encryption at rest for the database
* **Storage Encryption**: Uploaded documents are stored with encryption at rest
* **API Key Protection**: OpenAI API keys are stored server-side only, never exposed to the client
* **Secure File Naming**: Timestamps and sanitized filenames prevent information disclosure
* **Natural Language Search**: Find relevant document sections based on natural language queries with similarity threshold optimizations
* **Chat Integration**: Reference document content in chat conversations with proper citations and formatted displays
* **Real-time Status Updates**: Track document processing status with visual indicators
* **Enhanced Markdown Rendering**: Display document content with tables, code blocks, and other rich formatting

## System Architecture

The Vault Document Embedding System follows a layered architecture with a strong focus on security and performance:

1. **Presentation Layer**: React components in the Next.js frontend with optimized hooks for document operations
2. **API Layer**: Next.js API routes for handling document operations using hybrid processing approach
3. **Service Layer**: Document processing and embedding generation logic with content encryption
4. **Data Layer**: Supabase database with pgvector extension for vector storage and search

### Hybrid Processing Approach

The system implements a hybrid client-server architecture that balances performance and security:

1. **Client-side Operations**:
   * Direct Supabase client access for status updates and document management
   * Background processing status monitoring with polling
   * UI state management with optimistic updates

2. **Server-side Processing**:
   * Computationally intensive operations (text extraction, embedding generation)
   * Secure handling of sensitive content with encryption
   * Background processing for large documents

### Technology Stack

* **Frontend**: Next.js 14+, React, TanStack Query, Tailwind CSS
* **Backend**: Next.js API routes, LangChain document loaders
* **Database**: Supabase with pgvector extension for similarity search
* **Authentication**: Clerk
* **Storage**: Supabase Storage with secure path generation
* **Embeddings**: OpenAI text-embedding-3-small model (1536 dimensions)
* **Document Processing**: LangChain, pdf-parse, mammoth
* **Security**: AES encryption for document content
* **Markdown Rendering**: react-markdown with remark-gfm for tables and react-syntax-highlighter for code blocks

### Data Flow

1. User uploads a document through the UI
2. Document metadata is stored in the database and the file in Supabase Storage
3. Document processing is initiated in the background
4. Text is extracted, chunked, and embedded
5. Embeddings are stored in the database
6. UI reflects the processing status in real-time
7. Documents become searchable in the chat interface

## Database Schema

The system uses three primary tables in Supabase:

### vault\_folders

Stores information about document folders for organization.

| Column | Type | Description |
| - | - | - |
| id | bigint | Auto-incrementing primary key |
| uuid | uuid | Unique identifier for the folder |
| clerk\_id | text | User ID from Clerk authentication |
| name | text | Folder name |
| created\_at | timestamp | Creation timestamp |
| updated\_at | timestamp | Last update timestamp |

### vault\_documents

Stores metadata about uploaded documents.

| Column | Type | Description |
| - | - | - |
| id | uuid | Primary key |
| document\_id | uuid | Reference to vault\_documents.id |
| clerk\_id | text | User's Clerk ID for permissions |
| encrypted\_content | text | AES-encrypted chunk text content |
| metadata | jsonb | Document metadata (page numbers, etc) |
| embedding | vector | 1536-dimensional OpenAI embedding |
| chunk\_index | integer | Sequential index of chunk in document |
| file\_size | numeric | Size of the document in bytes |
| source | text | Source of the document (e.g., "upload") |
| processed\_for\_embeddings | boolean | Whether embeddings have been generated |
| processing\_status | text | Status: 'pending', 'processing', 'completed', 'failed' |
| created\_at | timestamp | Creation timestamp |
| updated\_at | timestamp | Last update timestamp |

### vault\_document\_chunks

Stores the text chunks and their embeddings for semantic search.

| Column | Type | Description |
| - | - | - |
| id | uuid | Unique identifier for the chunk |
| document\_id | uuid | Reference to the parent document |
| clerk\_id | text | User ID from Clerk authentication |
| chunk\_index | integer | Index of the chunk within the document |
| content | text | Text content of the chunk |
| metadata | jsonb | Additional metadata about the chunk |
| embedding | vector | Vector embedding of the chunk (1536 dimensions) |
| created\_at | timestamp | Creation timestamp |

## Document Processing Pipeline

The document processing pipeline is implemented in the central `/app/api/document-processor/route.ts` file and consists of several key stages:

### 1. Document Upload and Storage Path Architecture

When a user uploads a document, the `useUploadDocumentSync` hook in `/hooks/mutations/use-upload-document-sync.ts` handles the process with an enhanced security-focused storage path architecture:

```typescript theme={null}
// From /hooks/mutations/use-upload-document-sync.ts

// Generate a storage path using the folder UUID as the primary organization method
const timestamp = Date.now()

// Use 'vault-docs' as the root, followed by folder UUID (or 'unfiled' if none)
// This groups all files in the same folder together in storage
// Using folder UUIDs doesn't expose user IDs while maintaining organization
const folderPath = folderId || 'unfiled'

// Ensure filename uniqueness with timestamp prefix
const fileName = `${timestamp}_${name.replace(/[^a-zA-Z0-9]/g, '_')}`

// Final path: vault-docs/[folder-uuid]/[timestamp_filename]
const storagePath = `vault-docs/${folderPath}/${fileName}`

// Upload the file to storage
const { error: uploadError } = await supabase.storage
  .from('vault')
  .upload(storagePath, file, {
    cacheControl: '3600',
    upsert: false
  })
      folderId,
      name: values.name,
      file: selectedFile,
    })
    
    // Reset form and close dialog
    form.reset()
    setSelectedFile(null)
    onOpenChange(false) // Close the dialog
    
    // Refresh the document list after the document is uploaded
    setTimeout(() => {
      onDocumentUploaded() // Refresh the document list
    }, 500) // Small delay to ensure database consistency
    
  } catch (error) {
    console.error('Error uploading document:', error)
  }
}
```

The document is first uploaded to Supabase Storage, and its metadata is stored in the `vault_documents` table with initial `processing_status` set to 'pending'.

### 2. Processing Initiation

After upload, the document processing is initiated via the `/api/vault/process-document` endpoint:

```typescript theme={null}
// Lines 77-90 in /hooks/mutations/use-upload-document.ts
// Trigger document processing for embeddings
try {
  console.log(`Initiating document processing for document ID: ${data.id}`)
  
  const response = await fetch('/api/vault/process-document', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ documentId: data.id }),
  })

  // Error handling and status updates...
}
```

The processing endpoint (`/app/api/vault/process-document/route.ts`) initiates background processing:

```typescript theme={null}
// Lines 68-75 in /app/api/vault/process-document/route.ts
// Initiate document processing in the background
void (async () => {
  try {
    // Process the document (extract text, chunk, embed, and store)
    const success = await processDocument(documentId, userId)
    
    // Status updates and error handling...
  } catch (error) {
    // Error handling...
  }
})()
```

### 2. Fallback Text Search Mechanism

The system includes a robust fallback text search mechanism to ensure search functionality even when vector search fails:

```typescript theme={null}
/**
 * Fallback text search when vector search fails or returns no results
 * 
 * @param query - The search query text
 * @param clerkId - The user's Clerk ID for authentication
 * @param limit - Maximum number of results to return
 * @returns Promise containing an array of matching document chunks
 */
async function performTextSearch(query: string, clerkId: string, limit: number): Promise<any[]> {
  try {
    // Use text search (ilike) to find relevant documents
    const { data: textResults, error: textError } = await supabase
      .from('vault_document_chunks')
      .select(`
        id,
        document_id,
        content,
        metadata,
        vault_documents!inner(id, name, folder_uuid, vault_folders(uuid, name))
      `)
      .eq('clerk_id', clerkId)
      .ilike('content', `%${query}%`)
      .limit(limit)
    
    if (textError) {
      return []
    }
    
    return textResults || []
  } catch (error) {
    return []
  }
}
```

This fallback mechanism ensures that users can still find relevant documents even if there are issues with the vector search functionality, providing a more resilient search experience.

### 3. Text Extraction

The `extractTextFromDocument` function in `/lib/document-processor.ts` handles text extraction from various file types:

```typescript theme={null}
// From /lib/document-processor.ts
export async function extractTextFromDocument(
  documentId: string, 
  clerkId: string
): Promise<Document[] | null> {
  // Download file from storage
  // Determine file type
  // Extract text using appropriate method
  // Return LangChain Document objects with metadata
}
```

The system supports multiple file types with robust fallback mechanisms and production-ready error handling:

* **PDF**: Primary extraction using PDFLoader from LangChain, with pdf-parse as a fallback mechanism. The system includes comprehensive error handling and detailed metadata extraction:
  ```typescript theme={null}
  try {
    // First try with PDFLoader from LangChain
    const loader = new PDFLoader(new Blob([fileData]))
    const documents = await loader.load()
    // Process documents with enhanced metadata
  } catch (pdfError) {
    // Fallback to pdf-parse if PDFLoader fails
    const text = await extractTextFromPDF(fileData)
    // Extract folder and document names for proper citation
    const folderName = await getFolderName(folderId)
    const documentName = await getDocumentName(documentId)
    // Update document status with detailed error tracking
    // Process text with complete metadata
  }
  ```

* **DOCX**: Primary extraction using DocxLoader from LangChain, with mammoth as a fallback mechanism if DocxLoader fails. This dual approach ensures maximum compatibility with different Word document formats and structures:
  ```typescript theme={null}
  try {
    // First try with DocxLoader from LangChain
    const loader = new DocxLoader(new Blob([fileData]))
    const documents = await loader.load()
    // Process documents with enhanced metadata
  } catch (docxError) {
    // Fallback to mammoth if DocxLoader fails
    const text = await extractTextFromDOCX(fileData)
    // Extract folder and document names for proper citation
    const folderName = await getFolderName(folderId)
    const documentName = await getDocumentName(documentId)
    // Update document status with detailed error tracking
    // Process text with complete metadata
  }
  ```

* **TXT**: Direct text extraction with improved error handling

* **CSV**: Using CSVLoader from LangChain with direct text extraction as fallback

* **HTML**: Direct text extraction to preserve content

All extraction methods now include comprehensive error handling with the `updateDocumentWithError` and `updateDocumentStatus` functions to provide detailed status tracking and recovery mechanisms.

The system only allows users to upload file types that can be successfully processed, chunked, and embedded. This ensures that all documents in the vault can be properly searched and referenced in the chat interface.

### 4. Text Chunking

The `splitDocumentsIntoChunks` function divides documents into semantically meaningful chunks with comprehensive documentation and production-ready error handling:

```typescript theme={null}
// From /lib/document-processor.ts
/**
 * Split documents into chunks for embedding using LangChain's RecursiveCharacterTextSplitter
 * 
 * @param docs - Array of Document objects to split into chunks
 * @returns Promise containing an array of chunked Document objects
 * 
 * This function uses RecursiveCharacterTextSplitter with a chunk size of 1000 characters
 * and 200 character overlap to create semantically meaningful chunks that preserve context
 * between chunks. The separators are configured to split on paragraphs, sentences, and words
 * in that order of preference.
 */
export async function splitDocumentsIntoChunks(docs: Document[]): Promise<Document[]> {
  // Create a text splitter with appropriate settings
  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 1000,
    chunkOverlap: 200,
    separators: ["\n\n", "\n", ". ", " ", ""],
  })
  
  // Process each document and return chunks with enhanced metadata
}
```

Key features of the production-ready chunking process:

* **Semantic Chunking**: Uses LangChain's `RecursiveCharacterTextSplitter` to create chunks that preserve semantic meaning
* **Configurable Parameters**: Chunk size (1000 characters) and overlap (200 characters) are optimized for the embedding model
* **Hierarchical Separators**: Uses a hierarchy of separators (paragraphs, newlines, sentences, spaces) to preserve context
* **Enhanced Metadata**: Includes document name, folder information, page numbers, and source details for better citations
* **Comprehensive Documentation**: Detailed JSDoc comments explain the purpose and behavior of the function
* **Robust Error Handling**: Includes error recovery mechanisms and status tracking

### 5. Embedding Generation

The `generateEmbeddings` function creates vector embeddings using OpenAI's model:

```typescript theme={null}
// Lines 160-180 in /lib/document-processor.ts
export async function generateEmbeddings(
  text: string
): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
    encoding_format: "float",
  })
  
  return response.data[0].embedding
}
```

We use the `text-embedding-3-small` model because:

* It provides high-quality embeddings with 1536 dimensions
* Offers a good balance between quality and cost
* Has strong performance for semantic search applications

### 6. Storage in Database

The `storeDocumentChunks` function saves chunks and their embeddings to the database:

```typescript theme={null}
// Lines 190-220 in /lib/document-processor.ts
export async function storeDocumentChunks(
  documentId: string,
  clerkId: string,
  chunks: Document[]
): Promise<boolean> {
  // Prepare chunks with metadata
  // Generate embeddings for each chunk
  // Store in vault_document_chunks table
}
```

## User Interface Components

The Vault UI is composed of several React components with enhanced rendering capabilities:

### Vault Page (`/app/vault/page.tsx`)

The main container component that orchestrates the vault experience:

```typescript theme={null}
// Lines 50-70 in /app/vault/page.tsx
const {
  documents,
  isLoading: isDocumentsLoading,
  refetch: refetchDocuments,
} = useVaultDocuments(selectedFolderId)

// Handler functions for user interactions
const handleDocumentUploaded = () => {
  // Force refetch documents to ensure the UI is updated
  refetchDocuments()
  
  // Set up a secondary refetch after a short delay to ensure database consistency
  setTimeout(() => {
    refetchDocuments()
  }, 1000)
  
  setIsUploadOpen(false)
}
```

### Folder List (`/app/vault/components/folder-list.tsx`)

Displays and manages folders:

```typescript theme={null}
// Folder selection and management
const handleFolderSelect = (folder: VaultFolder) => {
  onFolderSelect(folder.uuid)
  setSelectedFolder(folder)
}
```

### Document List (`/app/vault/components/document-list.tsx`)

Displays documents with processing status indicators:

```typescript theme={null}
// Lines 165-210 in /app/vault/components/document-list.tsx
// Helper function to render processing status
const renderProcessingStatus = (document: VaultDocument) => {
  // If the document is processed successfully, show a checkmark
  if (document.processed_for_embeddings === true || document.processing_status === 'completed') {
    return (
      <div className='flex items-center gap-1 px-2 py-0.5 text-xs rounded-full bg-green-100 text-green-800'>
        <CheckCircle2 className='h-3 w-3' />
        <span>Ready</span>
      </div>
    )
  }
  
  // Other status indicators...
}
```

### Upload Document Dialog (`/app/vault/components/upload-document.tsx`)

Handles document uploads with drag-and-drop support for supported file types (PDF, DOCX, TXT, CSV, HTML):

```typescript theme={null}
// Lines 70-90 in /app/vault/components/upload-document.tsx
const onSubmit = async (values: FormValues) => {
  // Upload document and handle UI updates
}

// File type validation
const acceptedTypes = ['.pdf', '.doc', '.docx', '.txt', '.csv', '.html', '.htm']
const fileExtension = '.' + file.name.split('.').pop()?.toLowerCase()

if (!acceptedTypes.some(type => fileExtension === type)) {
  form.setError('file', { 
    type: 'manual', 
    message: 'Unsupported file type. Please upload PDF, Word, text, CSV, or HTML files.'
  })
  return
}
```

### Status Polling Mechanism

The document list implements a polling mechanism to update processing status:

```typescript theme={null}
// Lines 72-115 in /app/vault/components/document-list.tsx
// Poll for document processing status
useEffect(() => {
  if (!documents?.length) return
  
  // Find documents that are pending or processing
  const docsToCheck = documents.filter(doc => 
    doc.processing_status === 'pending' || 
    doc.processing_status === 'processing' ||
    (doc.processed_for_embeddings === false && doc.processing_status !== 'failed')
  )
  
  // Add these documents to the processing set
  setProcessingDocuments(prev => {
    const newSet = new Set(prev)
    docsToCheck.forEach(doc => newSet.add(doc.id))
    return newSet
  })
  
  // If no documents need checking, return
  if (!docsToCheck.length) return
  
  // Set up polling for these documents
  const interval = setInterval(async () => {
    // Check status and update UI
  }, 3000) // Check every 3 seconds for faster updates
  
  return () => clearInterval(interval)
}, [documents, onDocumentDeleted])
```

## API Endpoints and Client-Side Data Access

The system uses a hybrid approach for data access, combining API endpoints for complex operations and direct client-side Supabase access for simpler operations:

### API Endpoints

These operations require server-side processing or access to service role keys:

#### Document Management

* **Document Processor**: `/api/document-processor` - Central API for document processing with text extraction, chunking, and embedding generation
* **Delete Document**: `/api/vault/admin-delete` - Admin endpoint to securely remove documents from storage
* **Download Document**: `/api/vault/download-document` - Retrieves files from storage

#### Deletion Process

Document deletion follows a secure two-step process:

1. **Client-Side Preparation**:
   ```typescript theme={null}
   // From /hooks/mutations/use-delete-document.ts

   // First, get the document to retrieve its storage path
   const { data: document } = await supabase
     .from('vault_documents')
     .select('path, folder_uuid')
     .eq('id', documentId)
     .single()

   // Call the admin API to delete the file from storage
   const response = await fetch('/api/vault/admin-delete', {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
     body: JSON.stringify({ 
       path: document.path,
       documentId: documentId 
     }),
   })
   ```

2. **Server-Side Deletion**:
   ```typescript theme={null}
   // From /app/api/vault/admin-delete/route.ts

   // Using service role key for admin access
   const adminSupabase = createClient(
     process.env.NEXT_PUBLIC_SUPABASE_URL!,
     process.env.SUPABASE_SERVICE_ROLE_KEY!
   )

   // Delete the file with multiple fallback attempts
   let { error } = await adminSupabase.storage.from('vault').remove([path])

   // If the main deletion fails, try variations of the path
   if (error) {
     // Try without vault-docs prefix if present
     if (path.startsWith('vault-docs/')) {
       const pathWithoutPrefix = path.split('/').slice(1).join('/')
       await adminSupabase.storage.from('vault').remove([pathWithoutPrefix])
     }
   }

   // Also delete embeddings and document chunks
   await adminSupabase
     .from('vault_document_chunks')
     .delete()
     .eq('document_id', documentId)

   // Finally delete the document record
   await adminSupabase
     .from('vault_documents')
     .delete()
     .eq('id', documentId)
   ```

#### Folder Management

* **Create Folder**: `/api/vault/create-folder` - Creates new folders
* **Delete Folder**: `/api/vault/delete-folder` - Removes folders and contained documents

### Client-Side Data Access

The following operations use direct Supabase client access from React hooks, eliminating the need for API routes:

#### User Preferences

* **Mark Vault as Read**: Updates the `has_read_vault` flag in the user's profile via the `useUserInfo` hook

#### Search and Retrieval

* **Search Documents**: Uses the `useSearchDocuments` hook to perform vector similarity search directly from the client
  * Includes automatic fallback to text search when vector search fails
  * Provides loading states and error handling via React Query
  * Maintains the same search capabilities as the previous API-based implementation

#### Document Processing Status

* **Check Processing Status**: Uses the `useDocumentProcessingStatus` hook to poll for document processing status
  * Automatically handles polling with configurable intervals
  * Includes automatic fixing of inconsistent document states
  * Provides detailed status information including counts by status type
  * Stops polling automatically when all documents are processed

#### Document Management

* **Rename Document**: Updates document names directly via the `useRenameDocument` hook

#### Folder Management

* **Rename Folder**: Updates folder names directly via the `useRenameFolder` hook

### Process Document Endpoint

The `/api/vault/process-document/route.ts` endpoint is particularly important as it manages the document processing pipeline:

```typescript theme={null}
// Lines 16-140 in /app/api/vault/process-document/route.ts
export async function POST(request: NextRequest) {
  try {
    // Authentication and validation
    // Update document status to 'processing'
    // Initiate background processing
    // Return quick response to client
  } catch (error) {
    // Error handling
  }
}
```

Key design decisions:

* **Background Processing**: Processing happens asynchronously to avoid blocking the API response
* **Status Tracking**: Document status is updated at each stage of processing
* **Error Handling**: Comprehensive error handling with status updates

### Document Processing Status Hook

The `useDocumentProcessingStatus` hook provides client-side polling for document processing status:

```typescript theme={null}
// Example usage in document-list.tsx
const {
  documents: processedDocs,
  completed: allCompleted,
  statusCounts
} = useDocumentProcessingStatus(
  documentIds,
  documentIds.length > 0
)

// Hook implementation in use-document-processing-status.ts
const { data, error } = await supabase
  .from('vault_documents')
  .select('id, processed_for_embeddings, processing_status')
  .in('id', documentIds)
  .eq('clerk_id', session.user.id)

// Check if all documents are processed
const allCompleted = data.every(doc => 
  doc.processed_for_embeddings === true || 
  doc.processing_status === 'completed' ||
  doc.processing_status === 'failed'
)
```

## Vector Search Implementation

The system uses Supabase's pgvector extension for vector similarity search:

### SQL Function for Vector Search

A custom SQL function `match_documents` is implemented in Supabase:

```sql theme={null}
CREATE OR REPLACE FUNCTION match_documents(
  query_embedding VECTOR(1536),
  match_threshold FLOAT,
  match_count INT,
  p_clerk_id TEXT
)
RETURNS TABLE (
  id UUID,
  document_id TEXT,
  content TEXT,
  metadata JSONB,
  similarity FLOAT,
  document_name TEXT,
  folder_name TEXT
) 
LANGUAGE plpgsql
AS $$
BEGIN
  RETURN QUERY
  SELECT
    vdc.id,
    vdc.document_id::TEXT,
    vdc.content,
    vdc.metadata,
    (vdc.embedding <=> query_embedding) AS similarity,
    vd.name AS document_name,
    vf.name AS folder_name
  FROM
    vault_document_chunks vdc
  JOIN
    vault_documents vd ON vdc.document_id = vd.id
  JOIN
    vault_folders vf ON vd.folder_uuid = vf.uuid
  WHERE
    vdc.clerk_id = p_clerk_id
    AND (vdc.embedding <=> query_embedding) < match_threshold
  ORDER BY
    similarity ASC
  LIMIT
    match_count;
END;
$$;
```

This function:

* Takes a query embedding vector and search parameters
* Performs cosine similarity search using the `<=>` operator
* Joins with document and folder tables to include metadata
* Filters by user ID for security
* Returns the most relevant chunks sorted by similarity

### Document Search Implementation

The `searchDocuments` function in `/lib/document-processor.ts` has been optimized for production with improved error handling and fallback mechanisms:

```typescript theme={null}
// From /lib/document-processor.ts
/**
 * Search for similar document chunks based on a query
 * 
 * @param query - The search query text
 * @param clerkId - The user's Clerk ID for authentication
 * @param limit - Maximum number of results to return (default: 5)
 * @returns Promise containing an array of matching document chunks with metadata
 * 
 * This function performs a semantic search using vector embeddings to find
 * the most relevant document chunks based on the query. It first generates
 * an embedding for the query, then uses pgvector's similarity search to find
 * matching documents in the database.
 */
export async function searchDocuments(
  query: string,
  clerkId: string,
  limit: number = 5
): Promise<any[]> {
  try {
    // Generate embedding for the query using OpenAI API
    const embeddingResponse = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: query,
      dimensions: 1536,
    })
    
    const queryEmbedding = embeddingResponse.data[0].embedding
    
    // Perform vector similarity search using pgvector
    const { data, error } = await supabase.rpc('match_documents', {
      query_embedding: queryEmbedding,
      match_threshold: 0.5, // Minimum similarity threshold
      match_count: limit,   // Maximum number of matches to return
      p_clerk_id: clerkId   // User's clerk ID for filtering
    })
    
    if (error || !data || data.length === 0) {
      // Fall back to text search if vector search fails
      return await performTextSearch(query, clerkId, limit)
    }
    
    // Format results with enhanced metadata and sort by similarity
    // ...
  } catch (error) {
    // Handle errors gracefully
    return []
  }
}
```

## Chat Integration

The document search is integrated with the chat system to provide document-based answers with enhanced formatting:

### Chat API Route

The `/app/api/chat/route.ts` file integrates document search into the chat with improved formatting and citation:

```typescript theme={null}
// Document search tool definition in chat system
searchVaultDocuments: tool({
  description: 'Search through the user\'s personal vault documents',
  execute: async ({ query, limit = 5 }) => {
    // Search for relevant document chunks
    const results = await searchDocuments(query, clerkId, limit)
    
    // Format results with proper citations and markdown formatting
    if (results.length === 0) {
      return "No relevant documents found."
    }
    
    return results.map(result => {
      return `**Document**: ${result.document_name} (Folder: ${result.folder_name})

${result.content}

---`
    }).join('\n')
  }
})
```

When the AI assistant needs to reference documents, it calls the `searchVaultDocuments` function, which:

1. Searches for relevant document chunks using vector similarity
2. Formats the content with proper markdown for better readability
3. Includes document and folder names for proper citation
4. Returns the formatted results to the assistant

### System Prompt Enhancement

The chat system prompt includes enhanced guidelines for document citations and response formatting:

````typescript theme={null}
// Document citation guidelines in system prompt
DOCUMENT CITATIONS:
When you answer questions using information from the user's vault documents:
1. ALWAYS cite your sources by mentioning the document name in your response
2. Use the format "According to [Document Name]" or "Based on your [Document Name] in [Folder Name]"
3. If multiple documents are used, cite each one separately
4. If the document is in a specific folder, include that information too

Example citation formats:
- "According to your W-2 form, your income for 2024 was $75,000."
- "Based on your Tax Return (in Tax Documents folder), you claimed $12,000 in deductions."
- "Your Insurance Policy shows coverage of $500,000 for property damage."

FORMATTING GUIDELINES:
- Your responses will be rendered with full markdown support, so use markdown formatting to create scannable, well-organized content.
- Use **bold** for important numbers, percentages, and key insights.
- Use headings (## and ###) to organize longer responses into clear sections.
- Present financial data in tables when comparing multiple items or showing trends.
- Use bullet points for lists of related items.
- For code or calculations, use code blocks with ```.
- Keep paragraphs short (2-3 sentences) for better readability.
````

### Direct Response Style

The system prompt also instructs the AI to provide direct answers without acknowledgment phrases:

```typescript theme={null}
// Direct response style instructions
- DO NOT acknowledge what you're about to do (e.g., "I'll search through your documents" or "Let me check your data").
- SKIP all acknowledgment phrases and directly provide the answer or information.
- When using tools like document search, NEVER mention that you're searching - just present the information you find with proper citations.
- For financial data queries, directly present the data without saying "Let me gather that information".
```

## Security Considerations

Security is a paramount concern for the Vault Document System due to the sensitive nature of the stored documents. The system implements several advanced security measures:

### User-Based Access Control

* Document storage paths use folder UUIDs instead of user IDs for enhanced privacy
* Database RLS policies ensure users can only access their own documents
* Admin endpoints use service role keys to manage storage without exposing credentials to clients

```typescript theme={null}
// Example of user-based filtering in queries
.eq('clerk_id', userId)
```

This ensures users can only access their own documents.

### Server-Side Validation

API endpoints validate requests on the server side:

```typescript theme={null}
// Example of server-side validation
if (!userId) {
  return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
}

if (!documentId) {
  return NextResponse.json(
    { error: 'Document ID is required' },
    { status: 400 }
  )
}
```

### Secure Storage Access

Document storage uses Supabase Storage with proper access controls:

```typescript theme={null}
// Example of secure storage access
const { data: fileData, error: fileError } = await supabase
  .storage
  .from('vault')
  .download(document.path)
```

## Performance Optimizations

The system includes several optimizations to ensure responsive performance even with large document collections:

### Document Processing

* **Hybrid Processing Model**: Combines client-side operations for UI updates with server-side processing for computation-intensive tasks
* **Background Processing**: Documents are processed asynchronously to prevent UI blocking
* **Batched Embedding Generation**: Embeddings are generated in batches to avoid rate limits
* **Chunking Strategy**: Documents are split into optimal-sized chunks (1000 characters with 200 character overlap) for better semantic understanding
* **Status Polling**: Processing status is polled efficiently to provide real-time updates without overloading the server
* **Production-Ready Logging**: Console logs have been removed/commented for production environments

### Vector Search

* **pgvector Indexing**: Utilizes pgvector's HNSW indexing for faster search on large collections
* **Custom RPC Function**: Avoids multiple round trips by performing complex queries in a single database call
* **Increased Similarity Threshold**: Only returns results above a higher similarity threshold (0.6) for better precision
* **Removed Fallback Complexity**: Simplified fallback strategy for more reliable and predictable search behavior
* **Result Limiting**: Limits search results to the most relevant matches
* **Efficient Content Handling**: Only decrypts content for the specific chunks needed

## Troubleshooting

Common issues and their solutions:

### Document Processing Issues

If document processing gets stuck or fails:

1. Check the document's `processing_status` in the database
2. Verify file permissions in Supabase Storage
3. Check server logs for errors during text extraction
4. Ensure the document format is supported

### Fallback Mechanism for DOCX and PDF Processing

The system implements a fallback mechanism for DOCX and PDF processing:

* **DOCX Fallback**: If the primary DOCX processor (DocxLoader) fails, the system falls back to mammoth for text extraction.
* **PDF Fallback**: If the primary PDF processor (pdf-parse) fails, the system falls back to an alternative PDF parser.

This ensures that document processing is robust and can handle different document formats and processing errors.

### Status Inconsistencies

If the UI shows incorrect processing status:

1. The check-processing-status endpoint includes automatic fixing of inconsistent states:

```typescript theme={null}
// Lines 55-75 in /app/api/vault/check-processing-status/route.ts
// For each document that's completed but doesn't have processed_for_embeddings set,
// update it to ensure consistency
const documentsToUpdate = data.filter(doc => 
  doc.processing_status === 'completed' && doc.processed_for_embeddings !== true
)

if (documentsToUpdate.length > 0) {
  console.log(`Fixing inconsistent status for ${documentsToUpdate.length} documents`)
  
  // Update each document to ensure processed_for_embeddings is set to true
  await Promise.all(documentsToUpdate.map(async (doc) => {
    // Update document status
  }))
}
```

### File Download Issues

If file downloads fail:

1. The document processor includes fallback mechanisms for file paths:

```typescript theme={null}
// Lines 52-90 in /lib/document-processor.ts
// First try with the exact path
let fileData;
let fileError;

try {
  const result = await supabase.storage.from('vault').download(document.path)
  fileData = result.data
  fileError = result.error
  
  // If there's an error, try some variations of the path
  if (fileError || !fileData) {
    console.error('[DOCUMENT PROCESSOR] Error downloading with exact path:', fileError)
    console.log('[DOCUMENT PROCESSOR] Trying alternative paths...')
    
    // Try alternative paths
  }
}
```

### Document Extraction Issues

If text extraction from documents fails:

1. The system implements robust fallback mechanisms for different document types:

```typescript theme={null}
// For DOCX files - First try DocxLoader, then fall back to mammoth
try {
  const loader = new DocxLoader(new Blob([fileData]))
  const documents = await loader.load()
  // Process documents...
} catch (docxError) {
  // Fallback to mammoth if DocxLoader fails
  const text = await extractTextFromDOCX(fileData)
  // Process text...
}

// For PDF files - Similar fallback pattern with pdf-parse
```

2. Check if the document format is supported and properly formatted
3. For complex documents, try simplifying them before upload

## Future Improvements

Planned enhancements to the system include:

### Document Processing

* **OCR Integration**: Extract text from images and scanned documents
* **Better PDF Handling**: Improved extraction of structured content from PDFs
* **Advanced Document Preprocessing**: Text normalization and noise reduction techniques
* **Additional Metadata Extraction**: Enhanced extraction of document structure and metadata
* **Improved Processing Status UI**: More granular status reporting during processing

3. **Caching Mechanism**: Add caching for frequently accessed documents and search results
4. **Batch Processing**: Implement batch processing for multiple documents
5. **User Feedback Loop**: Allow users to rate search result relevance to improve future searches
6. **Document Versioning**: Track document versions and changes over time
7. **Enhanced Security**: Add document-level access controls and sharing capabilities

***

This documentation provides a comprehensive overview of the Vault Document Embedding System in the Portfolio Tracker application. It covers the architecture, implementation details, and design decisions to help developers understand and extend the system.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.