Skip to main content

​
silvia-1-1 in chat-turn.ts: integration checklist

Goal: a chat turn whose effectiveModel is silvia-1-1 runs on our vLLM endpoint through the adapter in lib/inference/silvia-chat/, and the browser sees the same SSE stream it sees today. Every branch below keys on isSilviaChatModel(model) from lib/inference/silvia-chat/model.ts. Line numbers are for lib/claude/chat-turn.ts on branch silvia-orch/plumbing at commit c15186711d (6,849 lines). Re-check them after rebasing.

​
Already done outside chat-turn.ts (plumbing lane)

​
Checklist for chat-turn.ts

Each item: lines, what is there, what to do.
  1. Imports (top of file, near 71-97). Add isSilviaChatModel, SILVIA_CAPABILITIES from @/lib/inference/silvia-chat/model, the stream factory from @/lib/inference/silvia-chat/create-stream (stream lane), and modelContextTokenLimits from @/lib/router/model-router.
  2. Stream creation, createAndConsumeStreamWithFallback 1791-1797. Today: usedBedrock ? createBedrockStream(...) : createStream(streamParams, betaFlags). Add a first branch: !usedBedrock && isSilviaChatModel(currentModel) -> the silvia stream factory with streamParams (it drops betas, cache markers, thinking, context_management, speed, container itself). The returned SilviaMessageStream must be accepted by consumeStream (1431): widen its stream param type to ReturnType<typeof anthropic.messages.stream> | SilviaMessageStream, or cast at the call site. consumeStream itself needs no other change.
  3. Fallback model, createAndConsumeStreamWithFallback 1740-1766 and the call site 4964-4971. Today fallbackModel is DEFAULT_LLM_CONFIG.fallbackModel (claude-sonnet-5). For a silvia turn pass fallbackModel: DEFAULT_LLM_CONFIG.model (Opus 4.8), so the model_fallback path lands on Opus. streamParams were built for silvia (no thinking, no betas), which is a valid Opus request, so the fallback runs Opus with thinking off. That is acceptable for v1. If you want thinking on the fallback: add fallbackThinkingSupported to StreamWithFallbackConfig (290-307), use it in the re-clamp at 1752-1756, and when it is true set streamParams.thinking/output_config from getThinkingConfig(...) (the existing code at 1759 only rewrites output_config if it already exists, so it does nothing for a silvia turn).
  4. Retry count for silvia, 1729-1737 and 4964. Default maxRetries: 2 means three silvia attempts (1s, 2s backoff) before Opus, and this repeats on every loop because streamParams.model is rebuilt from effectiveModel at 4784 each iteration. Pass maxRetries: 0 (or 1) for silvia, and after a loop returns usedFallback === true with a silvia effectiveModel, set effectiveModel = DEFAULT_LLM_CONFIG.model for the rest of the turn (after 5030). Persist that through the checkpoint (6171) so resumed slices stay on Opus.
  5. Error classification, isRetryableError 386-443 and isTransientConnectionError 331-371. Only 429/500/529 (or the matching type) are retryable, and a connection error must have no status. SilviaStreamError with type: 'timeout_error' and a numeric status is neither, so the turn fails with no fallback. Either the stream lane emits timeouts with name: 'APIConnectionTimeoutError' and no status (then 1857 fails over to Bedrock Opus once), or add isSilviaChatModel(currentModel) && error?.type === 'timeout_error' as retryable here. Same question for invalid_request_error 400 (context overflow): today it throws immediately at 2000-2003. For a silvia turn, treat it as fallback-worthy, since Opus has a far bigger window.
  6. Token counting and truncation, 4432-4539. Replace the two constants at 4433 and 4435 with modelContextTokenLimits(effectiveModel). The countTokens calls at 4440-4453 and 4492-4494 send model: effectiveModel; Anthropic rejects silvia-1-1, the catch at 4530 logs and moves on, so truncation never runs for silvia. Use model: isSilviaChatModel(effectiveModel) ? DEFAULT_LLM_CONFIG.model : effectiveModel in tokenCountParams (Claude’s tokenizer is a proxy; the 10% margin in modelContextTokenLimits covers the gap, and context-budget.ts still does the final fit at the wire).
  7. Static turn config, deriveTurnStaticConfig 2404-2850.
    • 2810 cacheableSystem, 277-290 getCacheableTools, and the message markers at 4789-4791: leave as is, the adapter drops cache_control. Optional: skip applyMessageCacheControl when silvia to keep params clean.
    • 2813 isFastMode: force false when isSilviaChatModel(effectiveModel). The router never picks silvia for a fast-mode request (suppressed), but a resumed slice or a future pin could; also stops the fast_mode SSE at 4307-4318.
    • 2819-2827 thinking: nothing to do (item in the table above).
    • 2831-2834 skills: set skillsEnabled = false for silvia. The code_execution_20250825 server tool (app/api/claude/skills.ts) and the container (4889-4891) are Anthropic-hosted; silvia cannot run them. Check whether the system prompt then still describes skills.
  8. Per-loop request build, 4782-4896.
    • 4797-4846 context_management: wrap the whole block in if (!isSilviaChatModel(effectiveModel)). Compaction is already gated off; client-sent clear_tool_uses edits (4826-4841) are not, and 4869 still adds context-management-2025-06-27. The adapter drops them, so this is log hygiene, not correctness.
    • 4849-4896 betas: with items 7 and 8 done, betaFlags is empty for silvia.
    • 4874-4877 fast mode: covered by item 7.
  9. Fallback notice, 5040-5070. The model_fallback SSE and PostHog MODEL_FALLBACK already fire when silvia falls back; no change. The client ignores the event today.
  10. Routing, 3846-3899. No code change needed (see Router below). effectiveModel comes from decideChatModel, which now applies the internal-only gate.
  11. Usage and cost, 5473-5530 and 6456-6497. The adapter’s finalMessage().usage must fill input_tokens/output_tokens and zero the cache fields; no iterations. Cost is priced by finalModelUsed, already correct. cost_usd is nulled on any fallback turn (6476), as today.
  12. max_tokens, 4785 and 5584. DEFAULT_LLM_CONFIG.maxTokens is 32,768; silvia reserves 8,192 (SILVIA_MAX_OUTPUT_TOKENS). The adapter clamps. A clamped reply that ends with stop_reason: 'max_tokens' and no tool call throws max_tokens_cutoff at 5584, as for Claude.
  13. Bedrock, 547-615 and 1768-1783, 1857-1876, 1883-1912. No change. Bedrock is only reached after the Anthropic fallback, or on a connection error, and createBedrockStream overwrites model with the Bedrock Opus id.
  14. Images, 497-515 and 3809. No change. Downscaling to 2,000 px is fine for silvia (SILVIA_CAPABILITIES.images: true).

​
Router: is the env var enough?

Yes, with conditions. CHAT_ROUTER_MODEL_SHORT=silvia-1-1 routes the short class with no code change, but only when all of these hold (lib/router/model-router.ts::decideChatModel):
  • CHAT_ROUTER_MODE=live. In shadow (the code default) the model never changes. Confirm the prod value in Vercel before relying on it.
  • Confidence >= CHAT_ROUTER_MIN_CONFIDENCE (default 0.65).
  • Not suppressed: research mode, fast mode (body.speed === 'fast'), attachments or inlined files, video intent, or an explicitly requested model. Each keeps the baseline (Opus 4.8).
  • Demo mode never calls the router (chat-turn 3846).
Superseded (2026-10-14): gated by user, not by internal cohort. silvia-1-1 serves only the Clerk ids in silviaChatUsers() (lib/router/model-router.ts): Jacob’s account by default, replaced by SILVIA_CHAT_USERS (comma list; empty string turns silvia off). Those users run every turn on silvia-1-1 with no router: no classifier and no suppressions (research, fast mode, attachments, video and explicit models included). A silvia failure still falls back to Opus. A CHAT_ROUTER_MODEL_<CLASS>=silvia-1-1 override never reaches anyone outside the list, internal or not. No env change is needed for Jacob to use it.

​
Other readers of the model string (checked, no change needed)

  • lib/token-limit.ts / lib/token-limit-client.ts: the USD cap calls get_user_cost_in_window, which prices chat rows by COALESCE(router_effective_model, model) through normalize_model_key. The new migration covers it. Note: a silvia turn that fell back to Opus is still priced as silvia (router_effective_model wins). Same rate today, so no billing difference.
  • ai_token_usage_daily rollup (compute_ai_token_usage_daily): same pricing path, covered.
  • PostHog RESPONSE_COMPLETED (6453-6512): model_used, cost_usd, thinking_*, genui_* all correct with no change. normalize-genui-fences does not branch on the model; it will report silvia’s GenUI slips, which is the signal we want.
  • ai_chat_logs.thinking_supported / thinking (6420-6428): false/off.
  • lib/router/log-decision.ts: logs router_effective_model = silvia-1-1.
  • lib/fine-tuning-collector.ts: records model verbatim, no branching.
  • lib/judge: judges with its own model (quant-analyst 1.0); does not read the chat model.
  • lib/suggested-prompts.ts: no model branching.
  • app/api/claude/internal/route.ts 383/392: only checks Opus and Sonnet.

​
Risks

  1. Silent no-fallback on timeouts (item 5). Highest risk: a Modal cold start or a stalled vLLM request kills the turn instead of falling back.
  2. Retry cost per loop (item 4). With an unhealthy endpoint every loop pays three failed attempts plus backoff before Opus.
  3. Opus fallback runs without thinking (item 3) and without the 1M beta. Quality on fallback turns is below a normal Opus turn.
  4. Training on our own outputs. Fine-tuning traces from silvia turns land in fine_tuning_traces with model = 'silvia-1-1'. Exclude them (or tag them) in any training cut, or the next model trains on the current one.
  5. Judge self-preference. The chat judge is Silvia Analyst 1.0 (a Qwen fine-tune). Scores comparing silvia-1-1 against Opus may favor silvia. Compare within judge_model and spot-check with a Claude judge.
  6. Mixed history in one turn. If loop 1 falls back to Sonnet/Opus and loop 2 returns to silvia, the history contains thinking blocks and possibly server_tool_use. The request converter must drop those block types.
  7. Pricing row only in prod. model_pricing.quant-analyst was added by hand; the new migration inserts it only when missing. Without it, model_cost_usd returns NULL for analyst and silvia rows.
  8. Migration not applied. Until 20261013_02 runs, SQL meters silvia-1-1 as opus. Same rate, so the cap is unaffected, but the model key in any per-model report is wrong.