Why AI Gets Numbers Wrong: Make It Answer From Data, Not Memory

"I asked ChatGPT what my revenue was this month, and a confident-looking number came right back." Sound familiar?

The problem is that the number may not be real. Even when it hasn't seen a single one of your store's numbers, AI can build a plausible figure and state it with full confidence. And a wrong number, unlike a wrong sentence, doesn't look wrong on screen.

This guide covers three things: why AI builds numbers from memory, how harmless "generic advice" differs from a dangerous "confident-wrong" number, and how to make AI answer from your real data — written so beginners can follow.

TL;DR#

• AI gets numbers wrong not because it's weak, but because it hasn't seen your store's real data and builds a plausible number from what it learned.
• The danger isn't harmless generic advice — it's the confident-wrong number stated as a specific figure. It looks real, so you use it in decisions as-is.
• Copy and paste your numbers in and you can fix it once. But every time is heavy, and even if the pasted number is wrong, the AI can't tell.
• What decides the quality of a decision isn't how smart the AI is, but whether the numbers it sees are real. The key isn't better prompting — it's letting the AI read your real data on the spot.

1. Why AI answers numbers from memory#

The short answer: AI gets numbers wrong because, instead of seeing your store's real data, it builds a plausible figure from what it learned.

Ask ChatGPT "What's my revenue this month?" and a number comes back. But if you haven't handed over your real data, all the AI has is the patterns it saw during training. Within that range, it assembles the most likely-looking number and answers[1]. This is the phenomenon known as "hallucination" — AI confidently producing something that isn't true[2].

What makes it tricky is that a wrong number doesn't look wrong. You can catch a wrong sentence by reading it, but whether "revenue this month is 4.2M yen" is real or invented can't be told from the screen. Because it's lined up with a plausible number of digits and a unit, it looks all the more real.

The chart above lines up the AI's memory-only answer against the actual value, with actual set to 100. The memory numbers swing both ways and none of them match. By "your store's numbers," we mean first the five numbers every EC store should track, like revenue and revenue per session. Even these five, if you don't hand them over, the AI fills in by guessing. If you want to rebuild your grip on what the metrics mean, see the first three metrics to check in EC analytics.

2. Generic advice and a confident-wrong number are different things#

The short answer: AI has two kinds of weakness, and with numbers the dangerous one isn't "generic advice" — it's the "confident-wrong" answer.

Generic advice is the bland kind: "Strengthen social media," "Review your product pages." Every store gets the same thing, which ties into why AI only gives generic advice. It's unsatisfying, but it has the safety of being easy to spot as off.

The scary one is the other kind. "Revenue this month is 4.2M yen, up 12% from last month," stated as a specific figure. The finer the number, the more real it looks — but if you haven't handed over real data, it may be built from memory. The point to hold on to: how confident the AI's answer sounds has nothing to do with whether it's right[3]. It will get things wrong confidently, without a flinch.

Telling them apart is simple. As the figure shows, ask whether you can check the number's source and period. If you can't, suspect it may be built from memory. Judge by whether it can be verified, not by how confident it sounds. It's the same as how AI decides which store to recommend — the AI does its best with the material at hand. If that material isn't real data, the answer won't be real data either.

3. Pasting by hand fixes it, but the pasted number isn't guaranteed right#

The short answer: copy and paste your numbers in and, right then, you can make the AI read the real thing. For trying it once, that's plenty.

Copy the numbers from your dashboards into ChatGPT and the AI answers from real data. It's worth doing once. The catch shows up when you try to keep it up across every conversation, and in the holes that remain in the pasting itself.

• You have to decide what to paste, on the spot. Just revenue? Over what period? By channel too? The numbers you need change with each question, so you keep re-choosing them.
• You can't paste in the ranges, prior-period figures, and breakdowns. This month and last, by channel, mobile and desktop. Gathering and pasting every number you need to compare doesn't finish with a copy.
• The next question means pasting again. Every time the conversation moves a step, different numbers are needed, so you go back to the dashboard and re-paste.
• Even if the pasted number is wrong, the AI can't tell. This is the scariest part. Paste a figure with one digit off and the AI takes it as correct and answers confidently. The AI can't check the numbers you paste.

As the table shows, the gap between AI that answers from memory and AI that reads your data comes out in four points: can it name the source, can it scope the range exactly, is it current, can it catch a mistake. Pasting by hand only plugs these holes for a moment; the root stays. The idea is simple, but gathering the right numbers in the right range and re-pasting them every time is heavy and easy to get wrong. So pasting by hand stays in the "try it once" supporting role.

RevenueScope solution

Whether AI gets numbers wrong or hand-pasting leaves holes, the root cause is one: the "latest, real numbers for your store" handed to the AI can only be prepared by hand.

RevenueScope takes on that handing-over part. Connect it to ChatGPT or Claude and the AI reads your EC numbers directly to answer you[4]. No complicated setup, no SQL — just ask, "Revenue this month — which channel is driving it?" It's read-only, so there's no worry your data gets rewritten.

What the AI reads is five numbers — Revenue, AOV (average order value), RPS (revenue per session), CVR (purchase rate), and Sessions — plus their change versus the prior period, already shaped by channel and handed straight to the AI. So the AI doesn't build from memory; it answers with real data you can verify.

For example, an AI that has read your numbers answers like this (demo data):

Unlike the memory-based left, the right is drawn from your store's real data. What RevenueScope does isn't make the AI smarter — it keeps handing over the real, verifiable numbers the AI needs. You're freed from re-pasting every time, and you cut off the drifting-numbers accident at the root.

FAQ#

Frequently asked questions#

Q. Without handing over my numbers, can good prompting alone make the numbers accurate?

A. Prompting[1] can line up the direction of the answer to a degree. Numbers are a different matter. Since the AI doesn't hold your store's real data, however well you ask, the numbers it returns stay in the realm of guesswork. Reaching accurate numbers requires letting it read your real data.

Q. If I hand my numbers to the AI, could it rewrite them?

A. Keep the connection read-only and the AI only reads — it won't rewrite your numbers. You also decide which numbers to show and how far. Pasting by hand is the same: the AI won't change the numbers you paste.

Q. Where should I start?

A. Pick one number the AI just returned and check it against the real data in your dashboard. If it's off, that's a sign the AI is answering from memory. Even verifying one number with your own eyes changes how you work with the AI's answers.

Conclusion#

AI gets numbers wrong not because of its performance, but because it builds a plausible figure from memory instead of seeing your store's real data. The danger isn't bland generic advice — it's the "confident-wrong" number stated flatly as a specific figure. Because it looks real, using it as-is lets losses pile up quietly.

You can hand numbers over by pasting them in, but every time is heavy, and even if the pasted number is wrong, the AI can't tell. In the end, what decides the quality of a decision isn't how smart the AI is, but whether the numbers it sees are real. Rather than hunting for a better prompt, let the AI read your real data on the spot. Start by taking one number the AI returned and checking it against your real data. If you're wondering whether you're missing the link between search traffic and revenue, see our piece on the search-data-to-revenue blind spot.

References#

• [1] OpenAI "Prompt engineering" (2024)
• [2] Anthropic "Reduce hallucinations" (2024)
• [3] arXiv "A Survey on Hallucination in Large Language Models" (2023)
• [4] Model Context Protocol "Introduction" (2025)