> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cfosilvia.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Document intelligence pipeline

# Document Intelligence Pipeline for Private Investments

## Overview

The Document Intelligence Pipeline adds automated document classification and structured data extraction to Silvia's private investment tracking. Users can upload fund statements (quarterly reports, capital calls, distribution notices) and have financial data automatically extracted, validated, and applied to their private investments — with human-in-the-loop confirmation before any data is saved.

This builds on the existing Vault/Reducto document infrastructure and adds two new AI layers: document classification and structured financial extraction.

## Architecture

```text theme={null}
┌─────────────────┐     ┌──────────┐     ┌──────────────┐     ┌───────────────┐     ┌─────────────┐
│  User uploads    │────▶│  Reducto │────▶│  Classifier  │────▶│   Extractor   │────▶│ Confirmation│
│  PDF/doc/image   │     │  (parse) │     │  (Claude)    │     │   (Claude)    │     │   Dialog    │
└─────────────────┘     └──────────┘     └──────────────┘     └───────────────┘     └─────────────┘
                                                                                          │
                                                                                          ▼
                                                                                    ┌─────────────┐
                                                                                    │  Database    │
                                                                                    │  (Supabase)  │
                                                                                    └─────────────┘
```

### Two Entry Points

1. **Import Statement (page-level)**: Creates a new private investment from a document. Button on the Private Investments page.
2. **Upload Fund Statement (dialog-level)**: Updates an existing private investment with data from a new document. Button inside the Add/Edit Investment dialog.

### Processing Flow

1. **File upload** → Written to temp file on server
2. **Reducto parse** → Synchronous text extraction (not async QStash job, since user is waiting)
3. **Document classification** → Claude determines document type (fund statement, capital call, K-1, etc.)
4. **Gate check** → If not a financial investment document, returns error with explanation
5. **Structured extraction** → Claude extracts all financial fields with source text attribution
6. **Validation** → Cross-field math checks + comparison against previous values
7. **User confirmation** → Editable dialog showing extracted data, source quotes, confidence scores, and warnings
8. **Save** → User confirms, data is written to `private_investments` table

## Files

### New Files

| File | Purpose |
| - | - |
| `lib/document-classifier.ts` | Claude-powered document type classification |
| `lib/extractors/fund-statement-extractor.ts` | Structured financial data extraction with source attribution |
| `app/api/private-investments/parse-statement/route.ts` | API endpoint orchestrating upload → parse → classify → extract |
| `hooks/mutations/use-parse-statement.ts` | React Query mutation hook for the parse-statement API |
| `components/extraction-confirmation-dialog.tsx` | UI for reviewing, editing, and confirming extracted data |
| `supabase/migrations/20260319_add_private_investment_statement_columns.sql` | Database migration for new columns |

### Modified Files

| File | Changes |
| - | - |
| `hooks/queries/use-private-investments.ts` | Added `PrivateInvestmentType` type, new columns to `PrivateInvestment` interface, updated `NewPrivateInvestment` and `AddPrivateInvestmentInput` types |
| `components/investment-dialog.tsx` | Added "Upload Fund Statement" button, statement upload handler, extraction confirmation flow |
| `app/(app)/private-investments/page.tsx` | Added "Import Statement" button, page-level upload handler, extraction confirmation flow |
| `app/api/qstash/check-document-completion/route.ts` | Added awaited document classification after chunks are stored (for all vault documents) |

## Database Schema Changes

### Migration: `20260319_add_private_investment_statement_columns.sql`

**Must be run before deploying.** Safe for production — adds columns with nullable/default values, then backfills `investment_type` to `'basic'` for existing rows and adds a `NOT NULL` constraint. The backfill UPDATE is fast (single column, single value) and the `NOT NULL` alter is instantaneous since all rows will already have a value.

#### New columns on `private_investments`:

| Column | Type | Default | Purpose |
| - | - | - | - |
| `investment_type` | `text` | `'basic'` | Categorizes the investment (CHECK constraint enforced) |
| `commitment_amount` | `numeric` | `null` | LP commitment to the fund |
| `capital_called` | `numeric` | `null` | Capital called to date |
| `distributions_received` | `numeric` | `null` | Distributions received to date |
| `unfunded_commitment` | `numeric` | `null` | Remaining uncalled capital |
| `irr` | `numeric` | `null` | Internal rate of return (decimal, e.g. 0.15 = 15%) |
| `moic` | `numeric` | `null` | Multiple on invested capital |
| `tvpi` | `numeric` | `null` | Total value to paid-in |
| `dpi` | `numeric` | `null` | Distributions to paid-in |
| `last_statement_date` | `date` | `null` | Date of last processed statement |
| `last_statement_document_id` | `bigint` | `null` | FK to `vault_documents(id)` |

#### New column on `vault_documents`:

| Column | Type | Purpose |
| - | - | - |
| `private_investment_id` | `uuid` | FK to `private_investments(uuid)` — links documents to investments |

#### New indexes:

* `idx_vault_documents_private_investment_id` — partial index where `private_investment_id IS NOT NULL`
* `idx_private_investments_last_statement_date` — partial index where `is_active = true`

### Investment Types

The `investment_type` column uses a CHECK constraint with these allowed values:

| Value | Label | Description |
| - | - | - |
| `basic` | Basic / Direct | Simple share purchase, default for manual entry |
| `pe_fund` | Private Equity Fund | Buyout, growth equity, secondaries |
| `vc_fund` | Venture Capital Fund | Early-stage, Series A/B/C |
| `hedge_fund` | Hedge Fund | NAV-based, managed accounts |
| `angel` | Angel / Seed | Angel investments, seed rounds |
| `spv_syndicate` | SPV / Syndicate | Single-target SPVs, syndicate deals |
| `private_debt` | Private Debt / Notes | Private credit, promissory notes |
| `fund_of_funds` | Fund of Funds | Funds investing in other funds |
| `other` | Other | Anything else |

When importing from a document, the extractor infers the type from document content. When adding manually, it defaults to `basic`.

## Document Classification

**File:** `lib/document-classifier.ts`

Classifies documents into types: `fund_quarterly_statement`, `capital_call`, `distribution_notice`, `k1`, `bank_statement`, `tax_return`, `insurance`, `deed`, `brokerage_statement`, `loan_document`, `other`.

Used in two places:

1. **Parse-statement route** — Synchronous classification to gate extraction (rejects non-financial docs)
2. **Check-document-completion route** — Awaited classification inside the QStash handler for all vault documents (populates `vault_documents.document_classification`)

The classifier reads the first 3 chunks (up to 12K chars), sends to Claude with temperature 0, and returns a typed classification with confidence score and entity extraction (fund name, period, account number).

## Structured Extraction

**File:** `lib/extractors/fund-statement-extractor.ts`

Extracts 15 financial fields from fund statements. Every extracted field includes:

* **value** — The extracted data point
* **sourceText** — Verbatim quote from the document (max 100 chars)
* **sourcePage** — Page number where found
* **confidence** — 0.0 to 1.0

### Validation Checks

Built-in validation runs automatically after extraction:

1. **NAV × shares vs total value** — Flags if `navPerShare × totalShares` doesn't match `totalValue` (2% tolerance)
2. **Commitment - called vs unfunded** — Flags if `commitment - capitalCalled` doesn't match `unfundedCommitment` (2% tolerance)
3. **IRR bounds** — Flags if IRR is below -100% or above 1000%
4. **MOIC bounds** — Flags if MOIC is below 0x or above 100x
5. **Negative total value** — Flags negative values for review
6. **Previous value comparison** — When updating an existing investment, flags if NAV or total value changed by more than 5x (possible extraction error)

## Extraction Confirmation Dialog

**File:** `components/extraction-confirmation-dialog.tsx`

The human-in-the-loop guardrail. Shows:

* Document classification badge
* Investment type selector (dropdown, pre-filled from extraction)
* All extracted fields as editable inputs
* Source text quotes (italicized, with page numbers) below each field
* Confidence badges (High/Medium/Low) for each field
* Validation warnings in amber, errors in red
* Cancel and "Confirm & Update" buttons

Fields with `null` value and `0` confidence are automatically hidden (not present in document).

## How It Ties Into the Rest of the App

### Net Worth and Financial Calculations

Only `total_value` feeds into net worth. The pipeline:

1. `total_value` is set from extraction (or computed as `navPerShare × totalShares`)
2. `usePrivateInvestments().totalValue` sums all investments' `total_value`
3. `useNetWorth()` includes `privateInvestmentsValue` in `totalAssets`
4. `financial-data-cache.ts` uses the same `total_value` for cached snapshots
5. `portfolio_daily_snapshots` stores `total_private_investments`

The new metrics columns (`irr`, `moic`, `tvpi`, `dpi`, etc.) are display-only — they don't affect any financial calculations.

### Existing Consumers Unaffected

These all continue working without changes:

* Dashboard cards and net worth breakdown
* Assets/liabilities page
* AI chat financial tools
* PDF export
* Tax estimate calculations
* Unrealized gains calculations
* CSV/data export
* Daily snapshot cron

## Running the Migration

Execute the SQL in `supabase/migrations/20260319_add_private_investment_statement_columns.sql` against your Supabase database:

**Option 1:** Supabase SQL Editor — paste and run the SQL directly

**Option 2:** CLI

```bash theme={null}
supabase db push
# or
supabase migration up
```

**Safety:** The migration adds columns with nullable/default values, then backfills `investment_type = 'basic'` for any existing rows and sets it `NOT NULL`. No existing queries break. No downtime required.

## Future Considerations

* **Investment type selector on manual add form** — Currently manual adds default to `basic`. Could add a type selector to the investment dialog for manual categorization.
* **Type-specific detail views** — Show different metric layouts per investment type (e.g., commitment/unfunded for PE, NAV/redemption for hedge funds).
* **Email integration** — Route emailed documents through the same classification → extraction pipeline.
* **Vault integration** — Trigger extraction when documents are uploaded to the vault and classified as fund statements.
* **Google Drive / Dropbox connectors** — Direct feeds into the classification pipeline.
* **History tracking** — Store extraction results over time for trend analysis (IRR trajectory, NAV history).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.