Multi-LLM Orchestration, Turn Tacit Knowledge Into a Working System

Portfolio

Multi-LLM Orchestration, Turn Tacit Knowledge Into a Working System

I built a multi-LLM workflow orchestration for a creative agency that captures the tacit knowledge four teams used to break down incoming work and route it to the right team. The agency ran on email as its database, and the system turns that email into assigned tasks in less than three minutes. It has increased the capacity of the agency by a person and a half every week for more than nine months.

The problem

Four creative teams, video, graphics, writing, and social media, worked the same clients and the same deliverables. Almost every day the same work came in. That work was contained in meeting summaries, Microsoft Teams and Zoom transcripts and strategy briefings captured in email threads. Multiple members of the teams had to read all of that work, figure out what each team needed, catch what was slipping, and get everyone moving. All that work of reviewing, sorting, and trying to figure out what was done and not done and who needed to do it was costing a person and a half every week.

What I built

I built a system that analyzes the long email thread, every activity within it, every summary, every Microsoft Teams and Zoom transcript, every strategic decision and strategy briefing. It translates that into something meaningful. It works out what is real work, what is just talk, what is an open ask, and what got missed. The system turns all that work into activities, tasks, and breakdowns across all the teams in less than three minutes, assigned inside the agency’s project management system, and an actionable summary sent back over email, not a recap or a task list from a transcript.

To capture the logic and knowledge that people instinctively had to interpret for the system to work, I sat with the teams, watched them work, and turned what they knew into structure a model can use. That meant decision trees for their gray areas, paired right and wrong examples for what a real task looks like for them, and a mapping of every person, client, and work type the system would run into. To prevent hallucinations, every output carries its own confidence score. The model has to commit to a decision and know when to abstain.

How it works

WHAT IT READS Email and Google Drive threads twenty levels deep, with multiple customers inside Customer detection scored against the live customer list AI, a small model. It sees names only Customer anonymization every identifier becomes one ephemeral 16 digit token no AI. Mechanical process THE REASONING LAYER HOW IT WAS TUNED The reasoning layer the team’s tacit knowledge encapsulated in a prompt Frontier model. Never knows the client name LLM as a judge a second model read every output against known good examples LLM-as-a-judge Client name reinserted a mechanical swap using the client and key pair Mapped to the team’s lifecycle entered in their system of record A report to every owner delivered to the owners and filed on disk the project management system they already run THE MODEL MIX, PER JOB Each model matched to its job. Smaller models for detection, grouping, filtering and formatting, a frontier model for reasoning, 65 to 74 percent cheaper
Click to enlarge

The system is built from a set of AI components, mechanical components, adaptors that integrated other services or tools, webhooks and mailhooks.

The reasoning layer captures the teams’ judgment, and that includes information such as customer types, product lifecycles, the archetypes, the decision trees, the conventions and overrides, and the definitions of every person, client, role and work type. It was domain tuned by running hundreds of documents, to create the necessary base for the system to know how to operate as an agency employee.

Every decision the system makes is instructed to carry a confidence level, based on defined criteria, what was inferred, and where the source came from. This allows the system to know when to provide an answer and when not to. This reduces hallucinations and improves accuracy on time sensitive parts of the output.

To enable faster domain tuning and prompt tuning, I used a second AI flow, an LLM as judge, that checks the output of the first system, evaluating its response against known samples and also new samples. This approach greatly reduced the time to optimize the reasoning layer and ensure it provided the right output.

In addition, to ensure confidentiality, the system uses a compartmentalized approach where I use a 16 digit ephemeral token for all client identifiers, so that the AI system never has the content and the name or information of the client at the same time. This reduces risk if the system is compromised.

The system was highly optimized and used different models for every job, frontier models for reasoning and smaller models for other work in the system. This reduced operating cost, as smaller models represent close to 70 percent price reduction per process execution compared to the frontier models. The workflow automation platform plus the AI integration cost of all the AI models was tuned so that each execution costs less than 30 cents.

So that the agency is able to maintain the system, I provided training, documentation, and a support prompt and structure they can use with ChatGPT or Claude, that contains the system design, system structure, documentation, and an FAQ that can be used as a knowledge base. I prompted a chatbot with a debugging mode and questions, so that the agency can use this as first line support in maintaining their system.

This system has been running for more than nine months at the agency with no issues. It has increased the agency’s capacity by a person and a half every week.

I wrote about the thinking behind this system in The Work Between the Work.

Back to Portfolio