AI providers

Kept uses AI for three tasks (AI_TASKS in packages/shared/src/ai.ts): extraction (reading a capture photo), the assistant (chat with tool calls), and embeddings (semantic search). All three go through one door in the server, so pacing, spend caps and the call ledger apply to every call the same way. For the person-facing side, see AI providers.

Code comments cite the decisions behind it as D19, D121, D167, D202 and D206 (decision labels).

The SDK and the provider kinds

The server uses the AI SDK (ai 7.x) with one provider package per kind. PROVIDER_KINDS lists them:

Kind	Package	Factory
openai	@ai-sdk/openai	createOpenAI
anthropic	@ai-sdk/anthropic	createAnthropic
google	@ai-sdk/google	createGoogle
groq	@ai-sdk/groq	createGroq
openrouter	@openrouter/ai-sdk-provider	createOpenRouter
openai_compatible	@ai-sdk/openai-compatible	createOpenAICompatible (any base URL, such as Ollama)

apps/server/src/ai/providers.ts turns a resolved provider into a model object (modelFor, embeddingModelFor) and the call settings each kind needs (callSettingsFor). Two rules:

• Always a model object, never a string id. A plain string is routed by the ai package to a hosted gateway. ModelObject refuses strings.
• Every factory gets an explicit baseURL and apiKey. Left out, the packages read ANTHROPIC_API_KEY, OPENAI_BASE_URL and the like from the environment.

Structured output is json_schema with strict mode off everywhere. An openai_compatible server that has no structured outputs gets the schema written into the prompt instead (schemaInPrompt).

Where providers come from

Providers are rows in ai_providers (apps/server/src/db/schema/ai.ts), scoped instance, account or user; there is no location scope. Each row holds the kind, an optional base URL, a model per task ({vision, chat, embeddings}), a reasoning level, and the key sealed as an envelope (key_ciphertext). People manage them in AI settings, through the routes in apps/server/src/ai/api.ts (/api/v1/ai/providers, /api/v1/ai/caps, /api/v1/ai/usage, /api/v1/ai/calls, and the admin price routes).

Which key pays is a cascade, run inside the database by kept.ai_provider_for, the only way application code reaches a key; apps/server/src/ai/resolve.ts states the same order in TypeScript:

• a Personal location: its owner’s user key, then their account key, then the instance key;
• any other location: its owner account’s key, then the instance key;
• no location (a private thread): the asker’s own key, then their account’s, then the instance’s.

Per task, the first scope with a model for that task wins: Groq has no embeddings model, so embeddings fall through.

Model lists (ai/models.ts) are read live: GET /api/v1/ai/providers/:id/models asks the provider each time the picker opens and reuses the answer for ten minutes (?refresh=1 asks again). It also writes ai_providers.model_list, which the price prefill, the status line’s “model missing” and Test’s vision settlement read; the picker never does. OpenRouter’s embeddings models come from its second listing, GET /api/v1/embeddings/models, and :free models are marked free. Any model id may be saved for any kind; one the listing doesn’t name is kept in model_list with typed: true, so it is never flagged as no longer offered.

Token estimates (estimateTextTokens in packages/shared/src/ai.ts) count text the way a byte-pair tokenizer splits it: a word of up to six ASCII letters is a token, digits three to a token, punctuation three, other letters 2.6 characters a token. It was measured against OpenAI’s o200k tokenizer (gpt-oss and OpenAI’s current models) on the samples in apps/server/test/fixtures/tokens/o200k.json, made with tiktoken 0.12.0; ai/estimate.test.ts keeps it 0–20% high on each. Kinds whose tokenizer wasn’t measured (Anthropic, Google, OpenRouter, an OpenAI-compatible server) get the higher of that and one token per 2.4 characters.

One door: callModel

apps/server/src/ai/call.ts exports callModel, and no other file may call the SDK’s generation functions. ai/no-direct-calls.test.ts fails on any generateText, streamText, embed or embedMany call elsewhere under src/. Each call:

1. checks images: JPEG, PNG or WebP with no EXIF;
2. is admitted by the pacer: the circuit breaker, the provider’s own token window (from its rate-limit headers, ai/pacing.ts) and the key’s concurrency (1 for Groq, 2 otherwise);
3. reserves its estimated cost against every budget bucket (kept.ai_reserve); a cap or a pause refuses with one over_budget ledger row;
4. runs generateText with maxRetries: 0, an output-token cap and an 80 s timeout, with no database transaction open;
5. maps the result to an outcome (LEDGER_OUTCOMES: ok, refused, rate_limited, over_budget, provider_error, timeout, schema_invalid, truncated);
6. settles: works out the cost (ai/cost.ts), writes one llm_calls row and releases its leases.

The ledger and spend

llm_calls is append-only, partitioned by month, and written only by the kept.ai_* database doors: the application roles have no INSERT policy on it, so a row can’t be forged from application code. A row records the task (LEDGER_TASKS, such as extract_receipt, assistant_turn, embed_query), who paid, tokens, cost and outcome. It never holds a prompt, an image, a reply or a key. Caps and budgets live in ai_budgets; reaching 80% and 100% of a cap sends a notice, and 100% pauses work until the cap resets or is raised. Prices are versioned rows in ai_model_prices. The full table definitions are in apps/server/src/db/schema/ai.ts.

Extraction

A capture enqueues one extract job (apps/server/src/extraction/job.ts), which runs in the capturing person’s scope:

1. claim the extraction and mark it running, committed before any AI work;
2. prepare the parts (extraction/image.ts): GPS-free images re-encoded on the server, or a PDF receipt’s text;
3. call extract() (ai/extract.ts) with the mode’s prompt (extraction/prompts/: thing, receipt, label, reading, service-invoice) and wire schema (ai/wire.ts);
4. parse leniently against EXTRACTION_SCHEMAS: a field that doesn’t parse is dropped, not the whole answer;
5. check in code (extraction/checks.ts): receipt lines reconcile with the total, readings against their meter, no future dates, VIN checksums; then apply and look for duplicates.

A cap pauses the extraction (paused_budget); a provider limit makes it waiting_provider until the reset. Neither spends a retry. Fields below 0.6 confidence always wait for review.

The assistant and tool calling

apps/server/src/assistant/loop.ts runs the loop itself. Every model step is one callModel: one provider request, one reservation, one ledger row. The SDK is never allowed to loop: tools carry no execute, and the stop condition is one step.

The tools are the MCP tool definitions (TOOL_DEFS from @kept/mcp), handled by apps/server/src/tools/registry.ts. Read tools run at once as the person. Write tools never run: each becomes a proposal card the person confirms. See MCP.

A step does not send every tool: assistant/select.ts picks the core (finding things, add, update, move, mark seen) plus each topic the thread’s last three questions name by keyword in the five languages, plus any tool the thread called. With a provider’s per-minute limit known, the loop tries that set, then the latest question’s alone, then the core, before trimming the output allowance and old tool results (assistant/fit.ts). assistant/select.test.ts checks the first request with every module on stays under Groq’s free 8,000 tokens a minute, and that every tool the evaluation set’s scripts call is offered for its question.

Embeddings

The source is the instance setting embeddings_source, set at boot from KEPT_EMBEDDINGS (apps/server/src/embeddings/provider.ts):

• provider (the default): each location’s resolved embeddings model, through embedValues in callModel. A location with none is keyword-only. OpenAI and Google are asked for EMBED_DIMS (768); OpenRouter (POST /api/v1/embeddings) and OpenAI-compatible servers answer their model’s own length, up to 2,000. embedValues refuses a response with vectors of another length than asked, or of two lengths (vector_length), so a model key never holds vectors search can’t compare.
• off: no embedding jobs; keyword search only.
• local (an on-server model) is not built in 1.0: a trial of it failed on memory, so the server refuses KEPT_EMBEDDINGS=local.

Vectors are stored with pgvector and searched with an exact cosine scan inside a database door; an HNSW index measured worse here.

The mock provider

With KEPT_AI_MOCK=1 every call answers from apps/server/src/ai/mock.ts, built on the SDK’s MockLanguageModelV4 and MockEmbeddingModelV4. Tests, the e2e run and the evaluation runs in CI use it; the server refuses it when NODE_ENV=production. It answers:

• by the SHA-256 of the first image, from test/fixtures/eval/mock-answers.json;
• otherwise with a schema-valid answer for the mode;
• for the assistant, from a script per question (ai/mock-script.ts, test/fixtures/assistant/cases.json).

An answer can act out failures: a length stop, invalid JSON, a 429 with retry-after, a 401, a timeout. Real-provider tests in ai/real-providers.test.ts are skipped unless KEPT_AI_SMOKE=1.

Evaluations

Three harnesses under apps/server/eval/, run from the repo root:

pnpm eval:extraction --dir apps/server/test/fixtures/eval   # the mock, no network
pnpm eval:assistant
pnpm eval:search

Each uses the mock unless --provider and --model are given. A real run reads its key only from KEPT_EVAL_API_KEY, never prints it, and stops before --max-calls or --max-cost. Reports go to .tmp/evals/ (git-ignored) with numbers and case ids only. The harness README is apps/server/eval/README.md.

Adding a provider kind

1. Add the kind to PROVIDER_KINDS in packages/shared/src/ai.ts, with its key prefix (if it has one) in KEY_PREFIXES, its DEFAULT_MODELS entry and its DATA_USE_NOTE_KEYS entry.
2. ai_providers.kind is checked against PROVIDER_KINDS, so the change needs a migration for the check constraint.
3. In apps/server/src/ai/providers.ts: a default base URL, a case in modelFor, callSettingsFor and, if it has embeddings, embeddingModelFor.
4. Its model listing in ai/models.ts and its rate-limit headers in ai/pacing.ts (a provider without such headers is paced by Kept’s budgets and the breaker only).
5. The web labels in apps/web/src/components/ai/ and strings in every catalogue (i18n).
6. Mock-backed tests for the new paths; a real smoke run is the maintainer’s step.