Adding metadata filtering fixes RAG's blind spot

A same-corpus test of three retrievers

A developer (writing as Rituraj on Dev Genius in late July) built a corpus of paired documents — current and deprecated versions of the same metric definitions, runbook entries and table schemas — and asked the same seven questions three ways. All three arms ran locally on Ollama (a tool for running models on your own machine) and ChromaDB (a vector database that stores text as numerical embeddings). No cloud calls.

The three arms:

• Vector RAG — ChromaDB with cosine similarity. Standard retrieval.
• OKF — a directory of markdown files, each with YAML frontmatter. The agent reads metadata first, filters, then traverses bodies.
• Hybrid — OKF metadata filter as a pre-step, then vector search on the surviving subset.

Google Cloud published the Open Knowledge Format on June 12, 2026 — a vendor-neutral spec for storing organisational knowledge as markdown files. One concept per file, linked into a traversable knowledge graph, with type as the only required field and everything else (status, timestamps, ownership) optional.

Where RAG fails on deprecated definitions

On questions where a deprecated and a current definition sat side-by-side in the corpus, vector RAG returned both.

The Medium piece walks through the example: the current definition of gross margin at one warehouse — “revenue minus cost of goods sold, where COGS includes shipping and fulfillment as of FY2026” — and the retired version that excluded shipping. Both came back from the retriever with cosine similarity around 0.91. Sometimes the deprecated one ranked first, because it phrased the formula more directly.

0.91cosine similarity between a current and a deprecated definition — close enough that ranking became a coin flip.

That is not a bug in the embedding model. It is the only signal a vector database has. Semantic similarity is what it measures, and the two definitions are semantically almost identical — that is precisely why one replaced the other. The retriever has no way to ask but is this still true?

How OKF avoids the problem

OKF’s structured frontmatter moves the decision upstream. Before the retriever runs, the agent can drop anything tagged status: deprecated, anything past its valid_until date, or anything superseded by a newer file. The deprecated document is never a candidate.

The AlphaMatch piece credits AI researcher Andrej Karpathy with the LLM Wiki pattern — having an AI incrementally build and maintain a persistent, living wiki rather than re-searching raw documents. AlphaMatch’s framing is that LLMs do not get bored, do not forget to update a cross-reference, and can touch fifteen files in one pass, which is why a curated wiki is more durable than a raw-document search.

The same AlphaMatch analysis is careful with the framing: OKF does not wholesale replace RAG. It replaces the need for RAG in the common case where an agent repeatedly needs the same curated facts — table schemas, metric definitions, runbooks. For millions of unstructured raw documents with no pre-existing structure, vector retrieval still wins.

What the hybrid arm showed

The hybrid — pre-filter on OKF metadata, then vector search on what survives — caught the most correct answers and missed the fewest deprecated ones. The combination uses OKF’s structure to retire obvious no-go documents, then uses embeddings for the genuinely fuzzy matches.

The cost: someone has to write and maintain the OKF wiki. The format does not auto-generate from raw documents. Keeping a living wiki current takes ongoing work, even when an AI helps.

Rituraj captures the point in one line: Your vector database cannot tell a deprecated metric from a current one. A YAML field can. That is the clearest statement of why structured metadata sits upstream of any embedding lookup.

What to do with this

For a UK small team already running a local agent — see our business assistant build for under £50 a month or the local Qwen browser game walkthrough — the practical lesson is not “tear out ChromaDB”. It is smaller and more useful: tag your knowledge before you embed it.

Three things to try this week:

• Audit your RAG corpus for duplicates and deprecations. Most team wikis have retired runbooks and superseded metrics. List them. Our LM Studio vs Ollama piece gives a starting point for getting visibility into what your index actually contains.
• Add validity metadata to your knowledge files. Even a simple status: current or status: deprecated is enough to filter on. You do not need Google’s full OKF spec — you need the discipline.
• Filter before you embed. A pre-filter step that drops deprecated entries saves your agent from surfacing them. The pattern is the same one we used for the incident triage build: garbage in, garbage out.

The bigger lesson is older than OKF: vector stores do exactly what embeddings can do, which is measure semantic similarity. They cannot, by construction, know what is true right now. Validity lives upstream — in how you write and tag knowledge in the first place. Treat that as part of the build, not an afterthought, and the rest of the agent stack gets easier.