7 min read
AI agents that don't make up numbers: where the model ends and the code begins
How I design AI agents that don't make up prices: the model understands and writes, and totals and money go through tested code. Two real cases.
AIAgentsBest practices
A customer writes on WhatsApp: “how much for two large and a half?”. The assistant replies with a confident, tidy total. The one on the menu is different, and higher. The shop has two options: lose the difference or argue with someone who has the screenshot. That’s the reasonable fear any business owner has about AI: that it “hallucinates” a number and states it with complete confidence.
The usual answer is to write “don’t make up prices” in the instructions. It isn’t enough. A language model produces likely text, and a plausible price is very likely text. My rule is different: the model handles language and judgment; prices, totals and any decision involving money go through tested code. I don’t ask it not to make things up. I design the system so it can’t.
The dividing line
Before writing any code, I split the work into two columns. One holds what code doesn’t do well: understanding how people write, reading a listing written in free prose, writing an explanation. The other holds everything that has a correct answer: adding, quoting, applying a rule, deciding whether a piece of data is enough. The first belongs to the model. The second belongs to deterministic code, with tests.
Language and judgment
The model
- Understand how people write
- Read a listing in free prose
- Write an explanation
Correct answer, with tests
The code
- Add up and quote
- Apply a rule
- Decide if the data is enough
The hard part is holding that line, because the model slips into the second column as soon as you leave a door open. Two of my projects show how to close it.
TomaNota: the model picks, the system charges
TomaNota (tomanota.lat) is my own product: an assistant that answers the WhatsApp of pizzerias, takeout shops and butchers and takes orders the way people write them. The AI is the Vercel AI SDK over an OpenAI-compatible API, with tool calling. This is how the work is split:
- The model has 8 tools and nothing else: add a product, add a half or a half-and-half, add an extra, change a quantity, remove, set delivery or pickup, show the cart and signal that the order is ready.
- None of the tools accepts a price. The model picks products by catalog ID and sets quantities. A sold-out product isn’t even among the valid IDs. It sees the prices so it can answer “how much is the large one?”, but it has no way to write one into the order.
- The system builds the summary and the total from the cart. The text the customer confirms was never written by the model.
- Until the person taps Confirm, there is no order. The “ready” tool confirms nothing: it asks the system to show the summary with the Confirm, Modify or Cancel buttons.
- On confirmation, the database quotes everything again with that moment’s prices and availability. If something sold out in the meantime, it says so and doesn’t confirm.
- 01Writes the order however it comes outPerson"Half mozzarella and half fugazzeta, and half a dozen empanadas."
- 02Picks products by catalog IDAI modelWith 8 tools. None of them accepts a price.
- 03Builds the cart, the summary and the totalSystemThe text the customer confirms is not written by the model.
- 04Taps Confirm, Modify or CancelPerson
- 05The database quotes it againSystemWith that moment's prices and availability. If something sold out, it says so and doesn't confirm.
The online QR menu and counter orders go through that same database function. An order costs the same no matter where it comes in.
The model can still say too much, so I also measure what it says. A test bench runs conversations against the real model, and a rule flags any amount the bot states that doesn’t come from the system (prices, delivery, minimum, total). Another rule checks every closing sentence against the orders actually confirmed.
With a pizzeria’s real menu, the bot quoted a half pizza wrong, and on top of that the rule accused it of making things up when it stated the correct price. On that menu the half doesn’t cost half of the whole, and I found four places that divided by two when the data was missing, including the rule itself. I removed all four. Today, if the half has no price set, it isn’t sold as a half. Dividing would be making up a price.
Tasador: the AI reads, the formula sets the price
Tasador builds a property’s valuation report with the comparable listings that justify the price. I developed it for a real estate agency in Buenos Aires: it’s a full application tested with real data, with its code published at github.com/ricardobing/tasador-rag. Here the AI has more work, and so more checks.
The AI looks for comparable listings with RAG, reads them and extracts what moves the price: condition, orientation, floor, elevator, age. It also decides which ones aren’t fit for comparison. But the price is calculated by an explicit formula: the median value in dollars per square meter, with written adjustment coefficients. An architecture test prevents putting a model on that step.
Wherever a model does step in, there’s a check:
- Every fact comes with its citation. The AI points to the passage of the listing it took it from, and a function with no AI verifies that the text is in the original. A fact without a verified citation doesn’t move the price.
- Every discarded listing gets one of five fixed reasons and a mandatory citation.
- A critic reviews the text before it’s delivered. Its first phase uses no AI: it looks for every figure in the report in the calculated data, and a single one that isn’t there rejects the text. I audited it by injecting made-up prices: at first 50.5% got through. After three fixes, 4.3%.
- With fewer than 5 valid comparables, it says it doesn’t know. It doesn’t deliver a number.
- 01Reads each listing and extracts what moves the priceAI modelEvery fact comes with the citation of the passage it came from.
- 02Verifies the citation is in the originalSystemA fact without a verified citation doesn't move the price.
- 03Calculates the price with an explicit formulaSystemMedian dollars per square meter, with written coefficients.
- 04Writes the report's explanationAI model
- 05The critic looks for every figure in the dataSystemA single figure that isn't there rejects the text.
If the critic rejects the text three times, the report goes out without narrative, with the table of comparables and the range, which are correct because they came from the formula. It happened in a test with real listings: the writer said there were no active listings when there were 60, and the report went out anyway with the right numbers. I’d rather deliver less than deliver something false.
What this has to do with your business
The pattern works for any process where AI touches numbers: quotes, estimates, settlements, stock. Three questions are enough to evaluate a system:
- Who writes the number the customer sees? If it’s the model, there’s a problem, however good the instructions are.
- What happens if the model gets it wrong? There has to be a point where the code recalculates and a person confirms.
- How is it measured? “It works fine” isn’t a measurement. A rule that flags every number that doesn’t come from the system is.
Your takeaway
The useful question isn’t whether you can trust AI, but which part of the work you give it. Give it the language: understanding, reading, writing. Leave the money to the code, with tests. When a number matters, let it be calculated by something that always gives the same result.
If you have a process with numbers that you want to automate without checking every sum by hand, get in touch and we’ll take a look.
Frequently asked questions
Can AI make up prices?
Yes: a language model can state a price that doesn't exist, and say it with complete confidence, because it generates the most likely text and a plausible price is very likely. That's why, in the systems I build, the number the customer sees is never written by the model. The AI understands the order and writes; the price comes from the database and is calculated by tested code.
What is an AI hallucination?
It's when a language model states something that isn't in any data, such as a made-up price, product or fact, in the same tone as a correct answer. It happens because the model completes likely text instead of checking a source, unless the system forces it to. You fight it with design: the data comes from verifiable sources, and a check with no AI reviews what the model writes.
Is it enough to tell the AI "don't make up prices" in its instructions?
No. An instruction lowers how often the error happens, but the model can still write any number. What works is taking that possibility away: in TomaNota, none of the 8 tools the model uses accepts a price, so it has no way to put an amount into the order. The guarantee lives in the structure of the system, not in the wording of the instructions.
What is tool calling in artificial intelligence?
Tool calling is a language model's ability to ask the system to run a specific function, with defined parameters, instead of answering with text only. The model decides which tool to use, for example "add product" or "show cart", and the code runs it and returns the result. Designed well, it's what lets an agent act on real data without making it up: the model chooses and the system calculates.
Can language models do math?
They often get it right, but they don't guarantee it, and the same query can give two different results. To add up an order, quote a price or apply a discount you need deterministic code, which always gives the same result with the same data and can be tested. That's why I leave language to the model and the math to the code.
Should a person confirm what the AI does?
Yes, at every step that involves money or a commitment to the customer. In TomaNota the model can signal that the order is ready, but the order only exists once the person taps Confirm on a summary the system built. That's called having a human in the loop: the AI proposes and someone approves what matters.
What happens if the AI model fails or stops responding?
The system is ready to keep serving customers without the model. In TomaNota, if the model doesn't reply in time or fails, the customer gets a clear message that a person will take over, and the owner gets an alert with the specific reason. A conversation handed to a person is always better than a chat that goes silent.
How do you check that AI-written text has no made-up figures?
With a check that uses no AI: it extracts every number from the text and looks for it in the calculated data, and if a single one is missing, the text is rejected. That check is measured too, for example by deliberately injecting false figures to see how many slip through. In Tasador, that audit took the made-up prices getting through from 50.5% down to 4.3%.
Can you use AI to put together quotes?
Yes, as long as the work is split well. AI is very good at understanding an order written in free language, on WhatsApp or by email, and turning it into a list of products and quantities. The price, the discounts and the total are calculated by the system with your price list, and a person confirms before it goes out. The same approach works for settlements, stock or any process where a number matters.
Can an AI answer "I don't know"?
It can, if the system is designed for it, and that decision is best made by the code, not the model. In Tasador, when fewer than 5 valid comparables are left, the system doesn't deliver a price: it reports that it doesn't have enough data. A timely "I don't know" protects the business more than a well-written made-up number.