GuideData pipeline & provenance

This page covers what happens to an item after you save it: which steps use an AI model and which are plain code, which provider each one goes to, and what record each step leaves behind. That record is how you check what happened to an item, as a user or as someone evaluating Open Context.

The short version: every step that writes AI output onto an item records which model wrote it, which prompt version, when, and a fingerprint of the content it read. Search can explain every ranking. Every request has an ID you can quote. You can turn AI processing down or off, and you can export your data with what you wrote kept apart from what AI wrote.

The pipeline

An item passes through four stages. Each one keeps its own output, so what you wrote is never mixed up with what was computed from it.

1. Stored as you supplied it. The content, file, source URL and the tags you set are stored as you gave them. No model rewrites them.

One exception: the title and notes fields. A metadata step writes notes for each item and, when the title is generic (a filename, a raw URL, “Selection from…”, or empty), a better title; a clear title is kept as given. It fills category and language only when they’re empty. It never touches the content itself or your own tags. Because the stored title doesn’t record whether you or the metadata step wrote it, the export flags these fields instead of guessing (see Export).

2. AI processing (optional). At the default processing level, two providers work on each item:

  • Anthropic writes a summary, key points, suggested tags and the people, organizations and other entities the item mentions, plus the title and category from stage 1. The model reads the item’s content (up to the first 400,000 characters, about 100,000 tokens; longer items are summarized from that excerpt, and the model is told so), title, tags, notes and source URL. Images and PDFs are sent as the file.
  • OpenAI turns the item into a vector for meaning-based search. The input is the title, AI summary, tags and category, followed by the start of the content (about 8,000 characters in all).

Audio and video items are stored but never processed. You can turn processing down or off; see Your controls.

3. Retrieval. Search runs two searches side by side: one by meaning (comparing vectors) and one by keyword (BM25 over title, content, tags and summary). The two ranked lists are merged by rank: each result’s combined score is the sum of 1 / (60 + rank) over the lists it appears in. Merging is plain code. No model decides what ranks first.

4. What an agent receives. A connected assistant gets results through the MCP tools. search_library returns each match’s ID, title, a preview of its content, tags, and its summary and key points. get_relevant_context returns summaries with item IDs. get_library_item returns one item in full, including its processing record.

Which steps use a model

StepRuns onProvider
Summary & tagsModelAnthropic
Title & categoryModelAnthropic
Insight Snapshot textModelAnthropic
Space summaryModelAnthropic
Tag cleanupModelAnthropic
Profile draftingModelAnthropic
Semantic search indexEmbedding modelOpenAI
Insight patternsPlain codeNone
Search rankingPlain codeNone

Insight patterns (emerging patterns, unexpected connections, knowledge gaps, most connected) are computed in code from your tags and the stored search vectors. No model is called. The one model call in Insight is the Knowledge Snapshot text. It needs at least five items and runs only at the Full processing level.

Profile drafting runs when you ask for it: optimizing a profile, or drafting one from text or a file you upload. Tag cleanup sends your tags, along with item titles and summaries, so it can merge near-duplicate tags into one vocabulary. Space summary adds each shared item into a running summary of the space, which get_space returns.

Which models, and how they change

You choose a quality preset, not a model. The preset maps to whichever model we currently assign it, and that model is always named in Settings and on each item’s record. The semantic search model is the same for everyone and can’t be changed per account, because vectors from different models can’t be compared.

Current mapping

StepStandardEnhanced (Pro)
Summary & tagsClaude Haiku 5.5 (claude-haiku-5-5)Claude Sonnet 5
Title & categoryClaude Haiku 5.5 (claude-haiku-5-5)Claude Sonnet 5
Insight Snapshot textClaude Haiku 5.5 (claude-haiku-5-5)Claude Sonnet 5
Space summaryClaude Haiku 5.5 (claude-haiku-5-5)—
Tag cleanupClaude Sonnet 5—
Profile draftingClaude Sonnet 5—
Semantic search indexOpenAI text-embedding-3-small (1536 dimensions)—

Steps with a dash in the Enhanced column use one model for everyone.

The live mapping is public at https://api.opnctx.ai/api/models (JSON; the API reference describes it). In the app it’s shown under Settings → Data processing. It needs no sign-in, contains no user data, and reports the models production actually runs, including any configuration overrides. If this table and the endpoint ever disagree, the endpoint is right.

Occasionally we compare a second model against the assigned one on a share of requests. A comparison stays within a preset: Standard is only ever compared with another Standard-tier model, and Enhanced with another Enhanced-tier model. The item’s record names the model that actually ran, and its variant says whether it was the assigned model or the one being compared.

How a model change works

  • Announced ahead and logged publicly, from October 10, 2026. Each change is listed at https://api.opnctx.ai/api/models/changes (JSON), upcoming changes first, and in the app under Settings → Data processing → Model changes → History. The changes that shipped with that release (the Enhanced preset, Sonnet 5 for tag cleanup and profile drafting, Haiku 5.5 for Standard and space summaries) are logged under its date; later changes are announced at least 14 days ahead.
  • New and edited items use the new model. Existing AI output is not re-run automatically. Re-running costs money and would change your content without asking. Existing items keep their records, which go on naming the earlier model. An item’s page shows “Made with an earlier model” when that’s the case.
  • You decide when to move forward. Re-run on an item moves it to the current model (every plan). Rebuild re-runs your whole library (Pro).
  • Email before a change. You get an email before a model change unless you’ve turned off Product emails on your Account page. There’s no separate setting for these notices. Every notice has a one-click unsubscribe, which turns Product emails off.
  • Nothing breaks when a provider retires a model. AI output is stored as text, so only new runs need a live model.

Change history

StepFromToStatus
Summary & tags, Title & category, Insight Snapshot text (Standard)Claude Haiku 4.5Claude Haiku 5.5Done. October 10, 2026
Space summaryClaude Haiku 4.5Claude Haiku 5.5Done. October 10, 2026
Summary & tags, Title & category, Insight Snapshot text (Enhanced, new)—Claude Sonnet 5Done. October 10, 2026: adds the Enhanced preset (Pro)
Tag cleanup, Profile draftingClaude Sonnet 4.6Claude Sonnet 5Done. October 10, 2026
Profile draftingClaude Sonnet 4Claude Sonnet 4.6Done. Configuration changed July 2026
Tag cleanupClaude Sonnet 4.5Claude Sonnet 4.6Done. Configuration changed May 2026

The October 10, 2026 changes shipped with that day’s release. The months for earlier changes are when our configuration changed, taken from our configuration history; we don’t have a separate record of when each reached production, so treat them as approximate. Before the July 2026 change, profile drafting ran on Claude Sonnet 4; when that began isn’t recorded. Claude Haiku 4.5 handled summaries and tags from February 2026 until October 10, 2026, and text-embedding-3-small has built the semantic search index since February 2026.

We also have an optional quality check that sends a sample of AI outputs, with the inputs they came from, to a separate model for grading. It’s off: its sample rate defaults to zero, and no sample is sent.

Receipts: how to check what happened

Per item

Each AI step that writes onto an item leaves a receipt in the same database write as its output. If the output exists, its receipt exists. A failed run leaves the previous receipt in place.

ReceiptFields
Summary & tags (enrichment)model: the exact model ID the provider reports it ran. preset: standard or enhanced. variant: control, or challenger during a comparison. promptVersion: an opaque label such as enrich-v1. enrichedAt. contentHash. attempts
Title & category (metadata)Same fields as enrichment
Semantic search index (embedding)model, dims, inputVersion (which recipe built the input text), embeddedAt, contentHash

contentHash is a SHA-256 fingerprint of the content the step read: the content type, the content, the uploaded file’s identity and the source URL. The item also reports its currentContentHash. When a receipt’s hash matches it, that output was made from the content you have now. Title, notes, category, language and tags are left out of the fingerprint on purpose. AI writes some of those fields, so including them would make every item look out of date the moment it was processed.

The receipts add up to one status per item:

StatusShown asMeans
currentUp to dateEvery receipt matches the current content
processingUpdatingA run is queued or running
staleOut of dateSome output was made from content that has changed since
failedFailedThe last run failed. The previous output and receipt are kept
unknownNo receiptProcessed before receipts were recorded (October 2026), so there’s no evidence what it was made from. Clears on the next run
skippedNot summarizedAudio or video, which is never processed
offOffSaved or edited while your processing level was below Full

When more than one could apply, the status is the first that matches in this order: skipped, off, processing, failed, unknown, stale, current.

Where to see receipts: the AI processing panel on an item’s page, which also shows the summary, key points and entities. Over the API, GET /api/library/:id returns derived (the receipts), derivedStatus and currentContentHash. Over MCP, get_library_item returns the same fields. Your library’s Needs attention filter lists items that are updating, out of date or failed.

Insight runs

Every Insight generation gets a run ID (ir_ followed by 10 hex characters), shown in the Knowledge Snapshot footer and in each card’s detail view. A run records when it ran, how many items it considered, the model, preset and prompt version behind the Snapshot text (empty when no text was generated), and, for each card, the items it was computed from.

  • Cards show Based on N items, and the detail view lists them under Evidence.
  • A knowledge gap is something missing, so it has no items of its own. Its record instead says what it was computed against: the topics compared, the neighbouring topic whose coverage picked the gap, and that topic’s items.
  • History: Free keeps the latest run per library or space. Pro keeps the last 10, viewable from the Knowledge Snapshot. GET /api/insight/runs returns the same records.

Search explanations

Ask for an explanation with explain: true on search_library or get_relevant_context, or on POST /api/library/search/hybrid. In the web app, it’s Why this result under each search result. Each result then carries:

FieldMeaning
legsWhich searches found it: vector (meaning), text (keyword), or both
vectorRank, vectorScoreIts position in the meaning search and the similarity score. null if the meaning search didn’t return it
textRank, textScoreIts position in the keyword search and the BM25 score. BM25 is only comparable within one query. null if the keyword search didn’t return it
rrfScoreThe combined score results are ordered by

The response also gets a retrieval block: mode (hybrid, vector-only or text-only), the embedding model that embedded your query (null if it wasn’t embedded), and the fusion constant.

degraded is reported even without explain. If the meaning or keyword search failed, as opposed to finding nothing, degraded lists it (vector, text). A result list with degraded is partial, not a complete answer. If both fail, the API returns 503 and the MCP tools fall back to plain keyword matching, which lists both. Two cases are deliberately not reported as degraded, because they aren’t outages: Store only, where your query is never embedded, and a deployment without embeddings configured.

Request IDs

Every API response carries an X-Request-Id header. Every MCP tool call gets its own ID, passed to the API with each request the call makes and returned in the result’s _meta.requestId. An MCP error ends with (Ref <id>). In the web app, error messages show Ref with the first 8 characters, and Copy ref copies the whole ID. Quote it when you report a problem: it identifies the exact request in our logs.

Your controls

Processing level

Settings → Data processing → Processing level. It applies to everything you save from then on, and to every step that would send your content anywhere.

LevelWhat’s sentWhat you get
Full (default)Content to Anthropic for summaries, tags, entities, titles, Insight text, tag cleanup and profile drafting. Content and search queries to OpenAI for the semantic search indexEverything on this page, plus keyword search
Semantic search onlyContent and search queries to OpenAI, and nothing to AnthropicSearch by meaning and keyword search. No AI summaries, AI tags or Insight Snapshot text
Store onlyNothing to any AI provider, including your search queriesKeyword search only

The level is checked when each call is made, not only when work is queued. A job queued while you were on Full that runs after you switch to Store only still sends nothing. Actions that exist only to call a model, like Re-run or drafting a profile, return an error explaining your level instead of running.

Changing the level doesn’t reprocess anything on its own. Items saved while AI was off show Off, and Settings offers to process them when you ask.

In shared spaces, your level follows your items. If you’re below Full, your items are left out of the Anthropic steps of every space you share into: the space summary and the space’s Insight Snapshot text. Insight cards in a space are plain code and may still count your items, since nothing is sent. Searching a space embeds only the searcher’s own query. If you switch from Full to a lower level, the summary of every space holding your items is emptied and rebuilt from the other members’ items, cached space Insight is cleared, and the Snapshot text of past space Insight runs is removed (their run IDs and card records stay).

Leaving a space. Any member except the owner can leave a space. Your items always go back to your own library. You choose whether copies stay behind:

  • Leave my items in the space (the default). Each item is copied into the space. The copy belongs to the space owner and is theirs from then on: it’s processed under the owner’s settings like any of their items, whatever your own level, and shows who originally added it. Copies keep the summary and receipts of the original, since they describe the same content. Leaving doesn’t process anything. Files are copied too, even if that takes the owner over their storage limit.
  • Take my items with me. Nothing is left behind. The space summary is emptied and rebuilt from what’s still in the space, cached space Insight is cleared, and the Snapshot text of past space Insight runs is removed.

Removing a member. The owner can remove a member. Their items go back to their library and leave the space, as with taking them along. Only the member can leave copies behind, by leaving on their own.

Storage after a member leaves files. Copies count toward the space owner’s storage. If they take the owner over a limit, nothing is deleted, but the owner can’t upload more until they free up space. The owner gets one email when that happens, and Account → Plan & billing shows what’s over and by how much.

Deleting a shared item works like taking it along: the space summary is rebuilt without it. Several deletes in a row lead to one rebuild. The owner can’t leave their own space. Ownership can’t be transferred yet, so to stop sharing a space, the owner deletes it.

Quality presets

Summary and tag quality and Insight narrative quality each offer Standard or Enhanced (Pro). The model behind each is shown next to the option. If a Pro plan ends, the preference is kept but processing goes back to Standard from the next run.

Re-run and Rebuild

  • Re-run on an item’s AI processing panel re-runs summary and metadata on that item with the current model. Every plan: 10 a day on Free, 100 on Pro (UTC days).
  • Rebuild (Settings → Data processing, Pro) re-runs your whole library at your current preset, through the provider’s batch service, with progress shown in Settings. Once a day. An item you edit while a rebuild is running keeps your edit, and its rebuild result is discarded.

Both require the Full level.

Export

Settings → Export builds one JSON file with your data in separate sections:

  • canonical: what you supplied: items, profiles, spaces you belong to, your own tags.
  • derived: what our steps computed, with each item’s receipts and status, the AI-written version of each profile, and your Insight runs.
  • embeddings (optional): the search vectors, each with the model and content hash it was made from.

The file includes its own field guide. Title, notes, category and language are under canonical, marked possiblyAiWritten, because the stored value doesn’t say whether you or the metadata step wrote it. As evidence, the export gives the time the metadata step last ran on that item. A value you edited after that time is yours. When that evidence is empty, the step may still have run before receipts were recorded.

Export is free on every plan, once an hour. You get an email when it’s ready, and the file can be downloaded for 7 days. Uploaded files aren’t bundled. They’re listed with their SHA-256 and a download link.

What we don’t record or expose

  • Prompt text. Receipts name a prompt version, never the prompt itself.
  • Items processed before October 2026 have no receipt and show No receipt (unknown) until their next run. We don’t backfill a receipt we can’t vouch for.
  • Profiles have no receipts yet. The export includes the AI-written version of each profile, but not which model wrote it.
  • Insight history is limited to the latest run on Free and the last 10 on Pro. Older runs are deleted.
  • Copies you leave behind in a space. If you leave your items in a space when you leave it, the copies belong to the space owner and follow the owner’s processing level, not yours. Deleting your originals afterwards doesn’t remove the copies.
  • Change dates before October 2026 are configuration dates, not recorded production release times (see Change history).