Contextual Retrieval Engine, Search by Idea Not Keyword

I built an on device contextual retrieval engine. It lets a team search their documents by the ideas within them, rather than by keyword. They search proposals, internal briefs, press releases and speeches by the idea, the intent and the context. It runs entirely on their computer, so nothing sensitive goes to a cloud based AI.

The problem

My clients had a specific problem. Their teams created documents over weeks and months and even years, but they could not easily find these documents for potential re-use, or to use as a base for comparison, or pretty much to remember and revisit that document.

Today’s search tools do a good job at matching file names and words, but finding a document, such as an old press release or an old speech, means remembering what that document was called or some of the words in it. You could upload all your files to a cloud based AI model like Anthropic or OpenAI. Unfortunately, that is not cost effective and it also exposes your content to the cloud AI vendors’ policies. Imagine sending thousands of files, that would be costly from a token perspective.

What I built

I built a contextual retrieval engine that runs locally and lets my clients search files by the idea, or the intent, or the context of that document, rather than just the keywords. This helps them find documents faster, and find the one that is tied to that particular idea of what is in it. Teams can reuse or revisit their documents using AI without sacrificing privacy, because all of this information is local.

The way I solved this is to leverage a multi-layered approach, where each AI model has a specific role, to generate all of the information about the meaning of those documents. I use Qwen to convert the content of those documents into an embedding. I use Gemma to read all the content and enrich it. That gives more meaningful information about what the document is about, who wrote it, and the key aspects and attributes of what is inside. I use Llama to do the reranking. Once all the files are found, Gemma reviews all the information to make sure that the meaning you are asking for is the meaning that actually gets shown to you.

How it works

To ensure fit, I benchmarked each local model against real documents, the instructions it could run per second, and the memory it would need. For embedding, that meant speed to encode the content into vectors. For enrichment, that meant richness of analysis. For reranking, that meant accuracy. I quantized the embeddings from 32 bit floating point down to 8 bit integers, so they run smaller and faster and use less GPU.

Both the embedding and enrichment run offline, so the entire contents of a folder or drive are indexed. The engine creates vector encodings for the content, an enrichment layer adds more meaning, and that gets embedded too. The engine uses math to find documents with meaning (cosine similarity). That makes it faster, and provides semantic and contextual meaning to match the user query. The engine then runs the search results step by step across multiple judges who match the intent and ideas. I designed this system for all types of documents found in a small business, from messages, to text within images and screenshots, to audio and video. My clients can search across multiple content types.

The engine stores all this information and the contextual information in a database, so other agents and my user interface can search it.

I wanted a way for clients, and for me, to find something by intent instead of the exact keyword, because you never remember the name of the content, or keyword, or file name, but you remember the idea, or meaning, of that content.

See it

In practice

“Find the proposal where the client’s budget didn’t match what they were asking for, and see how we handled it.”For example, a proposal might say something like “given the range you mentioned, here’s what we’d build first, and what would come later.”

“Find the brief where we had to talk someone out of a bad idea, and see how we did it.”For example, a brief might say something like “the instinct is to add a filter here. We’d rather cut the number of steps instead.”

“Find the press release where we had to announce something disappointing, and see how we framed it.”For example, a press release might say something like “while later than we’d hoped, this gives the team the room to get it right.”

“Find the poems I’ve written that are motivational.”For example, a poem might read something like “you kept walking after the map ran out.”

Back to Portfolio