Your AI agent can't count

An agent that searches your transcripts finds examples. It can't reliably answer a question you run the business on. For that it needs structured data, and designing that data is RevOps work.

AI assistants now read your calls, your CRM and your emails. Ask them "show me what customers say about pricing" and they are impressive. Ask them "how many deals, for what amount" and they get it wrong with the same confidence.

Here is where the gap comes from, what two recent studies measured, and a method to design and calibrate your first fields.

Same question, two answers

Ask an AI agent a question you ask every week: "How many open deals this quarter mention Competitor A as an alternative, and for what amount?"

Depending on how the agent reaches your data, you get two very different answers. Both sound equally sure. Only one is right.

The answer on the left is how semantic search normally works. That is also what makes it dangerous for forecasting and pipeline reviews: nothing in the answer tells you it is incomplete.

We ran the test on our own calls. Asked which deals had a competitor named on a call, one transcript search returning the 10 closest passages surfaced 5% of the deals a full count in a view found, and widening it to 50 passages still reached only 13% (Claap call data, Q3 2026).

Our help center describes the same gap on a coaching question. Pointed at the raw folder of recordings, the agent had to hand-pick a small sample of transcripts. Pointed at a structured view, it read every row and returned an average score per rep and per SPICED dimension (Claap help center).

The librarian and the catalog

Semantic search works like a very well-read librarian. You ask for "books about Competitor A" and they bring you the ten closest to your question. And that is where they stop.

Technically this is top-k retrieval: every transcript passage becomes a vector, and the agent pulls the k passages closest to the question. It reads those excerpts, then writes an answer. In our fictional example, if 23 deals are concerned and it sees 7, it answers 7.

Structured data works like the library catalog. Every book has a card with author, year and genre. To answer "how many novels came out after 2010", you filter the cards and open zero books.

Three kinds of questions break semantic search, and they are exactly the ones RevOps asks:

• Counting and adding. "How many deals", "what total amount". The LLM adds up what it sees, and it only sees a sample.
• Filtering on several conditions. "Owner = Marie, stage = closing, competitor named = Competitor A". Embeddings don't do Boolean logic.
• Excluding. "Deals where the budget was not confirmed". Negation is a known weak spot of embeddings: "budget confirmed" and "budget not confirmed" sit very close together.

What the numbers say

On aggregative questions, moving from vector search to a query on well-designed fields takes you from 33% to 90% correct answers, in the cleanest test of AI21 Labs' Structured RAG study.

TAG (UC Berkeley and Stanford, CIDR 2025). The researchers asked AI systems 80 realistic questions over real databases (Biswal et al., 2025).

• In the TAG study, no standard method gets above 20% correct answers: the best one, Text2SQL, tops out at 17%.
• Classic RAG gets none of the TAG study's questions right, because it leaves all the counting, math and filtering to the LLM.
• The approach that combines an LLM with database queries reaches 55% in the TAG study: 53% on questions that need world knowledge, 60% on questions that need reasoning.

Structured RAG (AI21 Labs, 2025). The team compares classic vector search with an approach that extracts fields at ingestion, then answers in SQL (Koshorek et al., 2025).

• The most telling test uses 350 made-up hotel profiles the model cannot know in advance: vector search gets 33% of the questions right, structured fields with a designed schema get 90% (AI21 Labs' Structured RAG study).
• Across the three datasets in the AI21 Labs study, the designed schema scores between 73% and 90%.
• With a basic pipeline the gap is even wider. In a demo on 22 Wikipedia pages about the World Cups, a starter RAG pipeline got 5% of aggregative questions right, and OpenAI's Responses API got 11% (AI21 Labs, AI Engineer World's Fair 2025).

Models already know a good part of World Cup history, which lifts vector search on that corpus (68% in the AI21 Labs study). That makes the made-up hotels the most reliable test. These benchmarks use public or synthetic data: they prove the principle, and the exact gain on your own calls is yours to measure.

The real factor: field quality

Having a database is not enough. In the AI21 Labs study, when the model infers the schema on its own, accuracy falls to between 23% and 77% depending on the dataset. On the FinanceBench financial questions it goes from 73% with a designed schema to 23% with an inferred one, according to the same AI21 Labs study: same architecture, same model, only the schema changes.

The authors' own example will sound familiar to anyone in RevOps: a "capital expenditure" field described in one line recorded two values as 1, one meaning millions and the other billions (AI21 Labs, 2025).

This is the part the AI agent talk leaves out. An agent is exactly as good as the fields you give it to read. A badly defined field produces wrong numbers that look precise.

And defining the right fields is already the RevOps job:

• knowing which decisions the business has to make (forecast, prioritization, coaching);
• turning those decisions into properties with closed values;
• keeping the data clean over time.

The one thing AI changes is who fills the fields. Before, it was the rep, when they remembered. In Claap call and HubSpot deal data from Q3 2026, 53% of the deals where a buyer named a competing call-recording tool on a call had no matching value in the CRM's competitor field. Now a model can fill them from what was said on the call. RevOps stays in the loop and becomes the center of it.

How Claap applies this model

A quick note: Claap is our product.

Claap turns every call into structured data your team and your agents can query. It comes down to four building blocks.

• AI fields you define. On meetings, deals and companies, you create the fields your pipeline reviews need: competitor named, budget confirmed, main objection, next step. The model fills them from the conversation (AI context layer).
• Views to filter and count. Fields combine into views, like in a CRM: every deal in closing that mentions Competitor A, sorted by amount. The answer is the full list.
• Sync with your CRM. Each field can write to the HubSpot or Salesforce property of your choice (CRM auto-complete). The CRM fills up from what was actually said on the call.
• A way back to the source. In a meeting view, every row is a call. If a number surprises you, open the call and check.

Here is what a Claap view looks like for the opening question (fictional data):

Fictional data for illustration: companies, amounts and competitor are made up.

That view is what your AI agent reads. Through the Claap MCP connector, an assistant like Claude or ChatGPT queries views and fields directly: it reads a computed answer. The MCP connector is included in every Claap plan, the free one too, and CRM enrichment needs the Business plan.

When semantic search is still the right answer

Structured fields only answer the questions you planned for. For everything else, searching the transcripts stays essential, and Claap does both.

The last row matters most. Search is how you discover what deserves tracking. A field is how you track it reliably.

Use fields and views when the answer ends up in a forecast, a dashboard or a prioritization call, because you need the same definition from one week to the next. Use transcript search when you are after a quote or a topic nobody has named yet. This advice is wrong if your team has three reps and twenty open deals: at that size, rereading the deals one by one is faster than designing a schema.

Design your first 5 fields

A good field starts from a decision. For each field, answer these five questions before you create it:

1. Which decision does it feed? Forecast, deal prioritization, rep coaching. No decision, no field.
2. Which type? Yes/no, closed list, number or date. Keep free text for things you will read and leave uncounted.
3. Which values, exactly? Write the definition of each value: "Budget confirmed = the decision maker confirmed an amount". A phrase like "the customer seems to have budget" leaves the model to interpret.
4. What if the call doesn't cover it? Add a "not mentioned" value, otherwise the model will guess.
5. How do you calibrate it? With the dual-extraction method below.

Start with these five fields, because they cover most pipeline needs: competitor named, budget confirmed, decision maker identified, main objection and dated next step.

Calibrate a field: dual extraction

The most effective method is to run two independent extractions and measure how far apart they are. Each disagreement almost always points to an ambiguous definition.

1. Build a sample of 50 meetings. Use a balanced sample, because if Competitor A shows up in only 5 calls, 95% agreement means nothing.
2. Run two extractions in parallel. On one side the Claap AI field in a view, on the other a frontier model with exactly the same definition. You can drive both through the Claap MCP.
3. Measure agreement field by field. Set the threshold before you start, for example 90%, and higher for a field that feeds the forecast. For rare values, Cohen's kappa is more reliable than raw agreement.
4. Have a human settle the disagreements. Also check about ten cases where both models agree: two models can be wrong the same way.
5. Adjust the definition and run it again until you hit the threshold.
6. Validate on 50 fresh meetings. Otherwise the definition may be tailored to the first sample.

Sources

• Biswal et al., Text2SQL is Not Enough: Unifying AI and Databases with TAG, CIDR 2025
• Koshorek et al. (AI21 Labs), Structured RAG for Answering Aggregative Questions, 2025
• AI21 Labs, RAG Evaluation Is Broken! Here's Why (And How to Fix It), AI Engineer World's Fair 2025