Document Intelligence Pipeline for Private Investments

Overview

The Document Intelligence Pipeline adds automated document classification and structured data extraction to Silvia’s private investment tracking. Users can upload fund statements (quarterly reports, capital calls, distribution notices) and have financial data automatically extracted, validated, and applied to their private investments — with human-in-the-loop confirmation before any data is saved. This builds on the existing Vault/Reducto document infrastructure and adds two new AI layers: document classification and structured financial extraction.

Architecture

┌─────────────────┐     ┌──────────┐     ┌──────────────┐     ┌───────────────┐     ┌─────────────┐
│  User uploads    │────▶│  Reducto │────▶│  Classifier  │────▶│   Extractor   │────▶│ Confirmation│
│  PDF/doc/image   │     │  (parse) │     │  (Claude)    │     │   (Claude)    │     │   Dialog    │
└─────────────────┘     └──────────┘     └──────────────┘     └───────────────┘     └─────────────┘
                                                                                          │
                                                                                          ▼
                                                                                    ┌─────────────┐
                                                                                    │  Database    │
                                                                                    │  (Supabase)  │
                                                                                    └─────────────┘

Two Entry Points

Import Statement (page-level): Creates a new private investment from a document. Button on the Private Investments page.
Upload Fund Statement (dialog-level): Updates an existing private investment with data from a new document. Button inside the Add/Edit Investment dialog.

Processing Flow

File upload → Written to temp file on server
Reducto parse → Synchronous text extraction (not async QStash job, since user is waiting)
Document classification → Claude determines document type (fund statement, capital call, K-1, etc.)
Gate check → If not a financial investment document, returns error with explanation
Structured extraction → Claude extracts all financial fields with source text attribution
Validation → Cross-field math checks + comparison against previous values
User confirmation → Editable dialog showing extracted data, source quotes, confidence scores, and warnings
Save → User confirms, data is written to private_investments table

Files

New Files

File	Purpose
`lib/document-classifier.ts`	Claude-powered document type classification
`lib/extractors/fund-statement-extractor.ts`	Structured financial data extraction with source attribution
`app/api/private-investments/parse-statement/route.ts`	API endpoint orchestrating upload → parse → classify → extract
`hooks/mutations/use-parse-statement.ts`	React Query mutation hook for the parse-statement API
`components/extraction-confirmation-dialog.tsx`	UI for reviewing, editing, and confirming extracted data
`supabase/migrations/20260319_add_private_investment_statement_columns.sql`	Database migration for new columns

Modified Files

File	Changes
`hooks/queries/use-private-investments.ts`	Added `PrivateInvestmentType` type, new columns to `PrivateInvestment` interface, updated `NewPrivateInvestment` and `AddPrivateInvestmentInput` types
`components/investment-dialog.tsx`	Added “Upload Fund Statement” button, statement upload handler, extraction confirmation flow
`app/(app)/private-investments/page.tsx`	Added “Import Statement” button, page-level upload handler, extraction confirmation flow
`app/api/qstash/check-document-completion/route.ts`	Added awaited document classification after chunks are stored (for all vault documents)

Database Schema Changes

Migration: `20260319_add_private_investment_statement_columns.sql`

Must be run before deploying. Safe for production — adds columns with nullable/default values, then backfills investment_type to 'basic' for existing rows and adds a NOT NULL constraint. The backfill UPDATE is fast (single column, single value) and the NOT NULL alter is instantaneous since all rows will already have a value.

New columns on `private_investments`:

Column	Type	Default	Purpose
`investment_type`	`text`	`'basic'`	Categorizes the investment (CHECK constraint enforced)
`commitment_amount`	`numeric`	`null`	LP commitment to the fund
`capital_called`	`numeric`	`null`	Capital called to date
`distributions_received`	`numeric`	`null`	Distributions received to date
`unfunded_commitment`	`numeric`	`null`	Remaining uncalled capital
`irr`	`numeric`	`null`	Internal rate of return (decimal, e.g. 0.15 = 15%)
`moic`	`numeric`	`null`	Multiple on invested capital
`tvpi`	`numeric`	`null`	Total value to paid-in
`dpi`	`numeric`	`null`	Distributions to paid-in
`last_statement_date`	`date`	`null`	Date of last processed statement
`last_statement_document_id`	`bigint`	`null`	FK to `vault_documents(id)`

New column on `vault_documents`:

Column	Type	Purpose
`private_investment_id`	`uuid`	FK to `private_investments(uuid)` — links documents to investments

New indexes:

idx_vault_documents_private_investment_id — partial index where private_investment_id IS NOT NULL
idx_private_investments_last_statement_date — partial index where is_active = true

Investment Types

The investment_type column uses a CHECK constraint with these allowed values:

Value	Label	Description
`basic`	Basic / Direct	Simple share purchase, default for manual entry
`pe_fund`	Private Equity Fund	Buyout, growth equity, secondaries
`vc_fund`	Venture Capital Fund	Early-stage, Series A/B/C
`hedge_fund`	Hedge Fund	NAV-based, managed accounts
`angel`	Angel / Seed	Angel investments, seed rounds
`spv_syndicate`	SPV / Syndicate	Single-target SPVs, syndicate deals
`private_debt`	Private Debt / Notes	Private credit, promissory notes
`fund_of_funds`	Fund of Funds	Funds investing in other funds
`other`	Other	Anything else

When importing from a document, the extractor infers the type from document content. When adding manually, it defaults to basic.

Document Classification

File: lib/document-classifier.ts Classifies documents into types: fund_quarterly_statement, capital_call, distribution_notice, k1, bank_statement, tax_return, insurance, deed, brokerage_statement, loan_document, other. Used in two places:

Parse-statement route — Synchronous classification to gate extraction (rejects non-financial docs)
Check-document-completion route — Awaited classification inside the QStash handler for all vault documents (populates vault_documents.document_classification)

The classifier reads the first 3 chunks (up to 12K chars), sends to Claude with temperature 0, and returns a typed classification with confidence score and entity extraction (fund name, period, account number).

Structured Extraction

File: lib/extractors/fund-statement-extractor.ts Extracts 15 financial fields from fund statements. Every extracted field includes:

value — The extracted data point
sourceText — Verbatim quote from the document (max 100 chars)
sourcePage — Page number where found
confidence — 0.0 to 1.0

Validation Checks

Built-in validation runs automatically after extraction:

NAV × shares vs total value — Flags if navPerShare × totalShares doesn’t match totalValue (2% tolerance)
Commitment - called vs unfunded — Flags if commitment - capitalCalled doesn’t match unfundedCommitment (2% tolerance)
IRR bounds — Flags if IRR is below -100% or above 1000%
MOIC bounds — Flags if MOIC is below 0x or above 100x
Negative total value — Flags negative values for review
Previous value comparison — When updating an existing investment, flags if NAV or total value changed by more than 5x (possible extraction error)

Extraction Confirmation Dialog

File: components/extraction-confirmation-dialog.tsx The human-in-the-loop guardrail. Shows:

Document classification badge
Investment type selector (dropdown, pre-filled from extraction)
All extracted fields as editable inputs
Source text quotes (italicized, with page numbers) below each field
Confidence badges (High/Medium/Low) for each field
Validation warnings in amber, errors in red
Cancel and “Confirm & Update” buttons

Fields with null value and 0 confidence are automatically hidden (not present in document).

How It Ties Into the Rest of the App

Net Worth and Financial Calculations

Only total_value feeds into net worth. The pipeline:

total_value is set from extraction (or computed as navPerShare × totalShares)
usePrivateInvestments().totalValue sums all investments’ total_value
useNetWorth() includes privateInvestmentsValue in totalAssets
financial-data-cache.ts uses the same total_value for cached snapshots
portfolio_daily_snapshots stores total_private_investments

The new metrics columns (irr, moic, tvpi, dpi, etc.) are display-only — they don’t affect any financial calculations.

Existing Consumers Unaffected

These all continue working without changes:

Dashboard cards and net worth breakdown
Assets/liabilities page
AI chat financial tools
PDF export
Tax estimate calculations
Unrealized gains calculations
CSV/data export
Daily snapshot cron

Running the Migration

Execute the SQL in supabase/migrations/20260319_add_private_investment_statement_columns.sql against your Supabase database: Option 1: Supabase SQL Editor — paste and run the SQL directly Option 2: CLI

supabase db push
# or
supabase migration up

Safety: The migration adds columns with nullable/default values, then backfills investment_type = 'basic' for any existing rows and sets it NOT NULL. No existing queries break. No downtime required.

Future Considerations

Investment type selector on manual add form — Currently manual adds default to basic. Could add a type selector to the investment dialog for manual categorization.
Type-specific detail views — Show different metric layouts per investment type (e.g., commitment/unfunded for PE, NAV/redemption for hedge funds).
Email integration — Route emailed documents through the same classification → extraction pipeline.
Vault integration — Trigger extraction when documents are uploaded to the vault and classified as fund statements.
Google Drive / Dropbox connectors — Direct feeds into the classification pipeline.
History tracking — Store extraction results over time for trend analysis (IRR trajectory, NAV history).

Product Guide

Account

Help

Document intelligence pipeline

Document Intelligence Pipeline for Private Investments

Overview

Architecture

Two Entry Points

Processing Flow

Files

New Files

Modified Files

Database Schema Changes

Migration: `20260319_add_private_investment_statement_columns.sql`

New columns on `private_investments`:

New column on `vault_documents`:

New indexes:

Investment Types

Document Classification

Structured Extraction

Validation Checks

Extraction Confirmation Dialog

How It Ties Into the Rest of the App

Net Worth and Financial Calculations

Existing Consumers Unaffected

Running the Migration

Future Considerations

​Document Intelligence Pipeline for Private Investments

​Overview

​Architecture

​Two Entry Points

​Processing Flow

​Files

​New Files

​Modified Files

​Database Schema Changes

​Migration: 20260319_add_private_investment_statement_columns.sql

​New columns on private_investments:

​New column on vault_documents:

​New indexes:

​Investment Types

​Document Classification

​Structured Extraction

​Validation Checks

​Extraction Confirmation Dialog

​How It Ties Into the Rest of the App

​Net Worth and Financial Calculations

​Existing Consumers Unaffected

​Running the Migration

​Future Considerations

Document Intelligence Pipeline for Private Investments

Overview

Architecture

Two Entry Points

Processing Flow

Files

New Files

Modified Files

Database Schema Changes

Migration: `20260319_add_private_investment_statement_columns.sql`

New columns on `private_investments`:

New column on `vault_documents`:

New indexes:

Investment Types

Document Classification

Structured Extraction

Validation Checks

Extraction Confirmation Dialog

How It Ties Into the Rest of the App

Net Worth and Financial Calculations

Existing Consumers Unaffected

Running the Migration

Future Considerations