Claude Haiku 5.5 as the chat backend

How to run ctx with Anthropic's Claude Haiku 5.5 for the chat roles (synthesis, translate, digest, dream) while embeddings stay on the local CPU sidecar, what the OpenAI-compatible layer does and does not give you, and what it costs over a year. Everything measured below comes from one installation (ctx v5.21.1, 2026-10-08 to 2026-10-09); the numbers are a starting point, not a promise.

Why a shim is needed

ctx speaks the OpenAI and Ollama wire protocols. Anthropic offers an OpenAI SDK compatibility layer at https://api.anthropic.com/v1/, which its documentation describes as "primarily intended to test and compare model capabilities, and ... not considered a long-term or production-ready solution for most use cases". It works, with three consequences that matter to ctx:

Sampling parameters. The compatibility table lists temperature as supported between 0 and 1, but on the Claude 5 models the live endpoint answered 400 with "temperature is deprecated for this model" on 2026-10-08. ctx always sends temperature (its chat request type has no omitempty on the field), so the first query failed on exactly that. Other fields are ignored rather than rejected: response_format, logprobs, presence_penalty, frequency_penalty, seed, reasoning_effort.
No prompt caching through the layer. ctx's system prompts are paid in full on every call.
Thinking is on by default on Claude 5 models, is not returned through this layer, and is billed as output tokens. ctx cannot turn it off over this wire. Short tasks think little.

The fix for the first point is a small pass-through proxy between ctx and Anthropic, here called the shim: one Python file on the standard library, running as a sidecar in the ctx compose project. It strips temperature, top_p, top_k, min_p, the two penalties, logprobs, response_format, reasoning and seed from the JSON body, forwards the request with its Authorization header to api.anthropic.com over TLS, and streams the answer back. It holds no secret and keeps no state; the sealed key still comes from ctx. It logs one line per request with the status and the stripped keys, never the body.

A native Anthropic protocol in ctx would remove the shim. It is not planned here.

Setup

Seal the key. ctx secrets set anthropic-chat reads the Anthropic API key from stdin and seals it under CTX_SECRETS_KEY. The key never lands in a file or in the backend table.

Seed the pool. The chat row points at the shim, not at Anthropic:

{
  "chat": {
    "name": "chat-primary",
    "host": "http://ctx-shim:8080",
    "protocol": "openai",
    "model": "claude-haiku-5-5",
    "api_key_ref": "anthropic-chat",
    "num_ctx": 100000,
    "trust": "full-trust"
  },
  "embed": {
    "name": "embed-primary",
    "host": "http://ctx-embed:8081",
    "protocol": "openai",
    "model": "Qwen3-Embedding-8B-Q8_0",
    "trust": "full-trust",
    "provider_class": "llamacpp"
  }
}

ctx backends seed --file seed.json writes both rows. num_ctx is the declared window: without one, promptguard refuses every prompt for an external row, and 100,000 keeps every prompt under the 100K threshold above which Haiku 5.5 costs more. trust must be full-trust for the rows to serve your own blocks, whose default sensitivity is credentials. think is left out: on the OpenAI wire it sends a reasoning field the layer does not document.

Dream without JSON mode. CTX_DREAM_JSON_MODE=off. The dream stages send response_format by default, the shim strips it, and ctx's parsers take the fenced or bare JSON that Haiku returns. Measured: every dream answer parsed (parse_format fenced-array and array in context_llm_log).
Expect one probe to look wrong. The reachability probe of a generic chat row sends a GET to the bare host. The shim answers 404 for that, so ctx backends test reports it; the one-token chat probe in the same test is the one that counts.
Leave rerank off unless you run a reranker sidecar. With rerank.enabled and no rerank row, every query would add an LLM-as-judge call on the chat model.

What ctx sends to Haiku

Measured on the installation above: 67 Haiku calls from 2026-10-08 21:30Z through the morning of 2026-10-09, 17 blocks in the corpus, Dream on since 01:36Z on the second day.

Pipeline	Calls	Prompt tokens, average	Completion tokens, average	When
`dream-eval`	21	3,391	583	per dream cycle, sometimes twice (a retry after a cap hit)
`dream-temporal`	17	1,215	695	per dream cycle
`dream-keywords`	17	774	74	per dream cycle
`dream-recurrence`	2	1,241	379	some dream cycles
`query-synthesize`	5	2,156	266	per query
`query-translate`	3	284	100	per non-English query
`query-temporal`	1	1,319	193	some queries
`dream-daily-synthesis`	1	401	1,057	once a day at 03:00

Completion tokens are what the layer reports in usage.completion_tokens; the thinking that Claude 5 models do by default is billed as output, and this page assumes it is inside that number. Embeddings and the guard cost nothing here: they run on the local sidecar and on PostgreSQL.

context_llm_log.cost_usd stays empty for Anthropic. ctx reads usage.cost from the response, which only OpenRouter sends. The token counts are logged; the money is read in the Anthropic console, and the workspace's spend limit is the brake.

Cost model

List prices for Claude Haiku 5.5, read 2026-10-06: 0.10 USD per million input tokens and 0.50 USD per million output tokens for prompts up to 100K tokens. Check the pricing page before relying on them.

Per unit of work, from the table above:

Unit	Tokens in / out	Cost
One query (synthesis, translation, a temporal pass in one of five)	about 2,700 / 400	0.00047 USD
One dream cycle (temporal, keywords, 1.75 evals, a recurrence pass in one of six)	about 8,100 / 1,850	0.0017 USD
The daily synthesis	401 / 1,057	0.00057 USD

How often a block dreams is set by the back-off: a fresh block re-dreams after 12 hours, each interval grows by a factor 1.6, and the cap is 45 days. A block that keeps producing links sees about 16 cycles in its first year and 8 a year after that. A block that produces nothing starts seven steps up the curve and sees about 10 in its first year. Take 13 cycles per block-year, 0.022 USD. The dream-eval prompt grows with the number of candidate blocks, up to 20 of them at 1,500 characters each, so at a large corpus a cycle costs up to twice the figure above.

Three scenarios for one year, with those assumptions:

Scenario	Queries per day	New blocks per year	Queries	Dream	Daily synthesis	Total per year
Small	10	200	1.7 USD	4.5 USD	0.2 USD	about 7 USD
Medium	50	1,000	8.6 USD	23 to 34 USD	0.2 USD	about 35 to 45 USD
Large	200	5,000	34 USD	110 to 230 USD	0.2 USD	about 150 to 260 USD

The two days of measurement cost 0.027 USD in list price. What moves the number is Dream over a growing corpus, not queries; what would move it faster is turning rerank on without a reranker, or a model that thinks longer.

What to watch

docker logs ctx 2>&1 | grep 'dream: cycle complete' for links_written; the first cycles here proposed links but wrote none, which is normal on a small corpus.
context_llm_log grouped by pipeline and model for the token counts behind this page.
The workspace spend in the Anthropic console, weekly.
metadata.parse_format on the dream-eval rows for parse errors, since JSON mode is off.

Evidence layer how-to, which uses the same installation.
docs/operations.md in this repository: the backend pool, ctx backends seed, and the Dream keys.
Anthropic's OpenAI SDK compatibility page, for the field-by-field support table.