6 min read
Before you switch on the AI, measure it
I built a full RAG to value properties, measured it on 113 queries and the simplest option is the one switched on. What to measure before paying for AI.
AIRAGEvaluation
For a real estate agency in Buenos Aires I built Tasador: an application that puts together a property’s valuation report with the comparable listings that justify the price. Inside it has a full RAG, what many vendors sell today as the heart of “their AI”: listing chunking, local embeddings in pgvector, lexical search, ranking fusion and a reranker. Out of all that, only one piece is switched on: lexical search, the simplest.
I didn’t pick it out of taste or to save money. I picked it because it won a measurement whose rule I wrote before knowing the result. It applies to any business about to add AI: first define what counts as an improvement, then measure, and switch on whatever wins.
What a RAG is and why building it isn’t enough
RAG means first searching for the relevant information and only then asking the model to work with it. In Tasador, the search has to bring back, out of thousands of listings, the 60 most similar to the property being valued. If it brings listings that don’t compare, the price comes out worse, however good the model that comes next.
There are several ways to do that search. Lexical search looks for words, with Spanish word stems (“ambientes”, rooms, also matches the singular “ambiente”). Dense search turns each text into a list of numbers and compares meanings. Fusion merges the two rankings (I used Reciprocal Rank Fusion, which doesn’t require calibrating the scales of one against the other). The reranker is a second model that reorders the finalists by looking at the property and the listing together.
- Lexical searchBy words, with Spanish word stems.On
- Dense searchLocal embeddings in pgvector.Off
- Ranking fusionReciprocal Rank Fusion between dense and lexical.Off
- RerankerA second model that reorders the finalists.Off
On paper, every piece adds something. In practice, every piece has a cost: compute time, maintenance and one more point where something can break. The only way to know if it pays for itself is to measure it.
The yardstick, written before measuring
Before running a single comparison I wrote the rule for switching on a new piece. It had to meet three conditions: beat the baseline on the main metric outside the margin of error, lower the number of listings the system itself ends up discarding as not comparable, and not worsen the price error. If it didn’t meet them, it stayed built, measured and switched off.
Writing it first matters. After weeks of building a reranker, it’s easy to find a metric where it shines. With the rule written down, that discussion doesn’t exist.
Then I built the yardstick: 113 fixed queries, the same for every variant, and four metrics:
- nDCG@25: how good the first 25 results are and how high the best ones appear. 1 is perfect.
- Recall: what share of the relevant listings appears among the first 30.
- MRR: how high the first good result appears.
- bpref: a metric that tolerates not every listing having a relevance judgment.
What counted as relevant was decided by the system’s own curation step, applied to the union of what all the variants brought back. And so as not to mistake noise for improvement, every difference against the baseline carries a paired bootstrap interval: the queries are resampled to see whether the advantage holds or depends on a couple of cases.
What came out
The baseline (a filter by neighborhood, surface area and rooms, with the most recent listings first) scored 0.747 on nDCG@25. Lexical search on its own scored 0.821, outside the interval. It was the best at what the report uses: the order, the coverage and how many listings the curation ends up discarding, which went down from 29% to 22%. And the price error, far from getting worse, went down two points.
Every variant with embeddings came in below lexical. The hybrids, which fuse dense and lexical, only won on bpref. The reranker bought the first position (MRR of 0.935) in exchange for 30 more seconds per report, and nothing else: against the hybrid without a reranker, the difference in nDCG was noise.
Search quality · nDCG@25 · 113 queries
- Baseline: filter + most recent0.747
- Dense, truncated listing0.805
- Dense, by sentences0.760
- Lexical, full text in SpanishSwitched on0.821
- Hybrid dense + lexical0.781
- Hybrid + reranker30 more seconds per report0.790
The explanation makes sense once you see the data. Real estate listings repeat a very specific vocabulary: ambientes, cochera, balcón (rooms, garage, balcony). Searching by words takes advantage of that better than a small embedding model, which was the one that could run on a CPU with repeatable experiments.
I switched on lexical. Dense and the reranker are still built and measured, switched off behind a configuration option. If a better model shows up tomorrow, it gets measured again with the same yardstick and decided with data.
Measuring the checks too
The same logic applies to the checks that keep the AI from making things up. In Tasador the price isn’t calculated by a model: it’s calculated by an explicit formula. The model writes the explanation, and a check with no AI looks up every figure in the text in the report’s data. A single figure that doesn’t appear, and the text is rejected.
I audited that check as well. I injected made-up prices into it to see how many got through. With a tolerance rule that seemed reasonable, 50.5% got through: half. After three fixes, each one measured, 4.3% gets through. It isn’t zero, and I don’t present it as zero. It’s a known number, measured again every time something changes.
Made-up prices that get past the check
- With the initial tolerance rule50.5%
- After three fixes4.3%
A full report costs between 2 and 5 US cents. And the switched-off reranker is 30 seconds no report has to wait for.
What to ask an AI vendor before hiring them
Tasador is a full application, tested with real data and with its code published, so every number in this article can be checked. If you’re evaluating an AI project for your business, these questions sort things out quickly:
- What did you compare it against? An improvement without a baseline isn’t an improvement. Ask for the number of the simplest thing that could work.
- What was the criterion for switching it on, and did you write it before measuring? If the rule shows up after the result, the rule is the result.
- How much does each query cost and how long does it take? In dollars and in seconds. A piece that improves a little and adds 30 seconds may not be worth it.
- How often does an error get past the check, and how did you measure it? Be wary of “zero errors”. A serious vendor gives you a percentage and tells you how they got it.
If you want to apply this approach to the AI in your business, get in touch and we’ll look at it together.
Frequently asked questions
What is RAG in artificial intelligence?
RAG (Retrieval-Augmented Generation) means first searching for the relevant information in your own data, and only then asking the model to work with it. That way the AI answers about your information without training a model of its own. Quality depends above all on the search: if it brings back the wrong things, not even the best model can fix it.
Should I use embeddings or keyword search?
It depends on your data, and the way to find out is to measure with your own queries. In Tasador, with real estate listings that repeat a very specific vocabulary, keyword search beat every variant with embeddings, including the hybrids that combine both. Embeddings usually add more when people describe the same thing with very different words, and that is also something you check by measuring.
What is a reranker and is it worth using?
A reranker is a second model that reorders the top results of a search by looking at the query and each result together. It usually improves the first position, but adds compute time to every query. In Tasador it took MRR to 0.935 in exchange for 30 more seconds per report, without improving the overall ranking beyond noise, so it stayed built, measured and switched off.
What is chunking in RAG?
Chunking means splitting documents into fragments before indexing them, because embedding models read a limited amount of text at a time. How you split them changes the results, and that gets measured too: in Tasador, 28.1% of the listings exceeded 512 tokens. With a small embedding model, using the truncated listing performed better than splitting it into sentences (nDCG of 0.805 against 0.760), the opposite of what was expected.
How do you measure whether a RAG works?
With a fixed set of real queries, the same for every variant, and metrics that evaluate the order of the results, always comparing against the simplest thing that could work. In Tasador it was 113 queries, four metrics and a bootstrap interval to tell a real improvement apart from noise. It's also worth measuring the effect on the business result: in that case, the price error of the report.
What is nDCG?
nDCG (Normalized Discounted Cumulative Gain) is a metric that evaluates how good the first results of a search are and how high the best ones appear. It goes from 0 to 1, where 1 is the perfect order, and it rewards getting it right at the top more than at the bottom. In Tasador, lexical search scored 0.821 on the first 25 results, against 0.747 for the baseline.
Do you need a vector database to do RAG?
Not necessarily. In Tasador the vectors are stored in PostgreSQL with the pgvector extension, in the same database as the rest of the data, and the search runs over an already filtered group of hundreds of listings instead of the whole corpus. That way exact search takes milliseconds, and a dedicated vector database would be one more piece to maintain without adding results.
How much does each query to an AI system cost?
It depends on how many times the model is called and which model is used, and it's a number you can measure step by step. In Tasador, a full report costs between 2 and 5 US cents, with the embeddings and the reranker on local models with no per-query cost. Before hiring someone, ask for that number in dollars and in seconds.
How do I know whether my business needs AI or something simpler is enough?
By comparing against the simplest solution that could work. Sometimes AI wins and sometimes a well-written rule is enough: in a real estate CRM I built, the emails from the listing portals have a fixed format, and classifying them with rules gives the same result, for free and verifiable with real cases. AI is worth it where it really adds something, and you see that by measuring.
What happens to an AI system when a better model comes out?
It's evaluated with the same queries and the same criteria as the current system, and it's switched on only if it wins. That's why it pays for each piece to be switchable through configuration: in Tasador, the embedding search and the reranker are still built behind an option, ready to be measured again. That review is part of maintaining an AI system, which I also offer.