RAG Was Never Semantic Search
RAG and semantic search fused into one word over the last three years. Pulling them apart surfaces the real question: where in your pipeline does anything actually think?
Somewhere in the last three years, two different things fused into one word. People say RAG and mean semantic search: a vector database, embeddings, top-k. The two became synonyms, and semantic search quietly became the only kind of retrieval anyone reaches for.
That fusion is worth undoing. It shapes how a lot of teams build, and it hides a better question underneath. Pull the two words apart and that question shows up on its own.
What RAG actually is
RAG stands for retrieval-augmented generation, and stripped of jargon it is simpler than the acronym suggests.
You have a question. You have a pile of documents. You do not want the model to answer from memory, because its memory is generic and possibly wrong. So you do two things. First you find the relevant parts of the pile. Then you hand those parts to the model and say, in effect: answer this, using only what I just gave you.
That is it. Retrieve, then generate. The model is not recalling. It is reading a packet you assembled and writing on top of it.
Notice what the definition does not say. It does not say how you find the relevant parts. That step is a slot. RAG is the shape of the pipeline, not the method you use to fill it.
And that is the whole confusion. One particular way of filling that slot became so dominant it swallowed the name.
The slot got filled with one thing
The popular fill is semantic search. Run every document through an embedding model to get a vector, a long list of numbers that captures roughly what the text is about. Do the same to the question. Pull the document vectors closest to it.
It works, it scales, and a large industry is built around it. If you learned about RAG in the last three years, you almost certainly learned it as this, which is how the two words came to mean the same thing.
But look at what semantic search does at the moment of retrieval. It measures similarity. Nothing else. It does not ask which document is authoritative, which is current, whether two results contradict each other, or whether the closest match is the right one. It returns what is near. Proximity is the entire judgment, and proximity is not judgment.
People think they have described an intelligent retrieval system. They have described a similarity sort.
Two other ways to fill the same slot
The alternatives are not exotic. They just fill the slot with something that involves thinking.
The first is letting the agent search. Give the model the same plain tools an engineer uses: list files, grep for a term, open a file, follow a reference to another file. Then let it hunt. It reads something, realizes it needs a neighboring file, opens that, adjusts, keeps going. It builds context one step at a time, deciding each step from what it just saw.
This is what Claude Code does inside a codebase. The team behind it started with embeddings and moved away. Boris Cherny, who built it, described trying a vector index and then landing on ordinary code search because it simply worked better. His words were that it was mostly a matter of feel, not a published benchmark, so I will not oversell it as proven. But the reasoning holds: a live search reads the code as it is right now, not as some index captured it last Tuesday. No index to maintain, drift, or leak. And the model chooses what to pull. The reflection here happens at run time.
The second alternative moves the thinking to the other end. A knowledge graph, or any curated store, front-loads the judgment. Someone decides what the entities are, how they relate, which fact is current, which is superseded. By the time a question arrives, contradictions are already resolved. The reflection here happens at creation time, expensive up front and cheap at every read after.
Two very different designs. Both share the thing semantic search lacks. Somewhere, while answering or while building, something reasoned about what is true. Semantic similarity never does that, at either end.
Where the thinking lives
Line the three up and the real axis is not the kind of search. It is where the thinking lives.
Semantic search puts none in the loop. It is fast and scales to millions of documents, which is why it wins when you cannot afford to reason over everything and the questions are fuzzy enough that closeness is a good proxy. A real, large set of problems. I am not dismissing it.
Agentic search puts the thinking at run time. Slower, more tokens, but you get freshness and precision on exact matches. On a codebase, where truth is exact and changes hourly, that trade is usually worth it. But run-time thinking has a catch, the same one I wrote about in my last piece on who decides which version of a document is true. If the reflection happens fresh at every question, it is redone from scratch at every question. The agent that greps its way to an answer today re-discovers the same ten specs tomorrow, and works out all over again which one is current. It pays that cost every time, and can get it wrong every time. When the corpus is self-consistent, like code, the lottery lands right. When it is messy, the normal state of a company, run-time rediscovery is not just expensive. It is a fresh chance to be confidently wrong on every ask.
The graph puts the thinking at creation time. Most work to build, least forgiving if you stop feeding it. In exchange, nobody re-adjudicates on the hot path, because the adjudication already happened.
There is a trap in how these choices get made. Hand the whole corpus to every method and they all look equal, because upfront judgment has nothing to resolve when everything is already in front of the reader. That is the shape of most benchmarks, and it is the abnormal case. Curation only pulls ahead when the truth is scattered, stale, or contradictory, which a clean demo never reproduces. The value of thinking in advance is invisible until the corpus is messy enough to need it, and by then you have usually already picked the method that does no thinking at all.
One reader gets forgotten in all of this: people. A curated store serves the new joiner, the junior who needs the current spec and not a tour of the nine stale ones. For an agent, run-time discovery is slow and risky. For a human, it is a non-starter. Nobody onboards by grepping twenty files and reasoning about which is authoritative, because that sorting is exactly the context they do not have yet. The curated answer is the only one that serves both readers with the same artifact.
The honest part
None of this makes semantic search bad. The most convincing hard numbers in this space come from Anthropic improving a semantic pipeline, not replacing it: blend keyword matching back in, rerank the results, and retrieval failures drop sharply. Same lesson as mine, from the other direction. Pure similarity was the weakest version. Adding something that acts like judgment made it better.
And you do not have to pick one. The three fit together well. Curate a knowledge graph so the truth is adjudicated once and stays legible to a human. Index it so it is fast to reach. Put an agent on top that reads the curated store first, falls back to semantic search when the question is fuzzy, and drops to grep for the exact current state of something the graph does not cover. The graph carries the judgment, the index the speed, the agent the adaptability. The mistake was never using semantic search. The mistake was letting it be the only thing in the slot.
So when someone tells me they are “doing RAG,” I no longer know what they built. I know the shape. I do not know whether anything in their system ever reasons about what is true.
That is the question I ask now. Not what database, not which embedding model. Just: where, in your pipeline, does something actually think, and who gets to read what it thought? If the answer to the first is nowhere, you did not build retrieval. You built a similarity sort with a language model stapled to the end. It might be enough. But you should know that is what it is.