Your RAG-Based AI Just Made Up a Payroll Answer. Here's Why That's Not Rare.
Your RAG-Based AI Just Made Up a Payroll Answer. Here's Why That's Not Rare.
A new benchmark puts a number on how often "grounded in your documents" isn't the same as correct — and why that matters for anyone pointing AI at award or super questions.
Your AI tool didn't lie to you. It retrieved the wrong document, and told you the answer with exactly the same confidence it would have used if it had retrieved the right one. That distinction matters a lot more than most finance teams realise, and a new benchmark released this year puts a number on how often it happens.
The technique is called retrieval-augmented generation, or RAG — it's how most "AI that answers questions from your documents" tools actually work. Instead of relying purely on what the model learned during training, the system searches your own documents at the moment you ask a question, pulls out what it thinks is the relevant passage, and generates an answer grounded in that passage. It's the architecture behind most enterprise AI assistants, including the kind finance teams are increasingly pointing at award interpretations, super guarantee rules, and internal policy documents.
|
67%
Hallucination rate the HaluBench 2.0 benchmark found in multi-hop RAG systems, where an answer requires pulling information from more than one document.
|
1st fine
A European banking consortium became the first organisation fined under the enforced EU AI Act, after its customer-facing RAG chatbot gave a materially wrong answer.
|
Why "Grounded in Your Documents" Isn't the Same as "Correct"
The whole selling point of RAG is that it reduces hallucination by anchoring the model's answer in real source material, rather than letting it generate purely from memory. That's true, as far as it goes. The problem is what happens at the retrieval step, before the model even starts generating an answer.
If the retrieval system pulls the wrong passage — an outdated award version, a super guarantee rate that was superseded, a policy document from before your last update — the model will confidently build an answer on top of that wrong passage. It has no way of knowing the retrieved document is stale or irrelevant. From the model's perspective, it did exactly what it was designed to do: answer based on what it was given.
The HaluBench 2.0 results this year make the scale of the problem concrete. When a question requires connecting information across more than one document — which describes most real payroll questions, since an answer usually depends on an award clause, a current rate, and an employee's specific circumstances all at once — accuracy drops sharply. Two-thirds of multi-hop answers in the benchmark contained a hallucinated or unsupported claim.
The EU Case Is a Preview, Not an Outlier
Europe isn't where Australian payroll compliance gets tested, but the mechanism behind that first EU AI Act fine is universal: a customer-facing RAG system gave a confident, wrong answer on something with real financial consequences, and nobody caught it before it reached the customer. The specific regulation differs by jurisdiction. The underlying failure mode — retrieval error masquerading as a correct answer — doesn't care what country you're in.
For Australian finance and payroll teams, the equivalent failure looks like this: an AI tool confidently tells a team member the super guarantee rate is 11.5% because that's what it retrieved from a document that hasn't been updated since the last rate change, or it applies an old award clause because the retrieval system ranked an outdated PDF above the current one. Nobody notices until a pay run is wrong, or worse, until an audit finds a pattern of it.
What Actually Reduces This Risk
The fix isn't "don't use AI for payroll questions." Done properly, retrieval-grounded AI is still far more reliable than relying on memory, and far faster than manually searching award documents for every query. The fix is being deliberate about three things: version control on the source documents the system retrieves from, so superseded rates and clauses are actually removed rather than left sitting alongside current ones; visibility into which specific passage the system used to generate its answer, so a human can spot-check the source rather than trusting the output blind; and treating any AI-generated payroll answer as a first draft that gets checked against the primary source before it's acted on, not a final answer.
None of that is exotic. It's closer to basic data hygiene than advanced AI engineering — which is exactly why it gets skipped under time pressure, and exactly why the failure mode above keeps recurring across industries.
Questions Worth Asking Before You Trust a RAG Tool
If your organisation is evaluating or already using a RAG-based tool for anything with financial or compliance consequences, there's a short list of questions worth putting to the vendor directly, rather than assuming the answer. Ask how frequently the underlying document set is refreshed, and what happens to a superseded document — is it removed, or does it sit alongside the current version where the retrieval system might still surface it. Ask whether the tool shows you which specific passage it used to generate an answer, or whether you only see the final output with no way to trace it back to a source.
It's also worth asking how the vendor tested accuracy on multi-hop questions specifically — ones that require combining information from more than one document — rather than only single-document lookups, since that's precisely where the HaluBench 2.0 results show the biggest gap between confidence and accuracy. A vendor who can answer that question specifically, with real numbers, is a meaningfully different proposition to one who answers with general reassurance about "advanced AI" and moves on.
None of this means avoiding RAG-based tools. It means treating the answer they produce the same way you'd treat a junior team member's first draft — useful, often right, but worth a second look before it goes anywhere that matters. That habit costs a few extra minutes per query. The alternative, as the EU banking case shows, can cost a great deal more.
AI Answers Are Only as Good as What They Retrieve
If your finance team is relying on AI tools for award, super, or compliance questions, the retrieval layer behind that tool matters more than the model powering it. PFL provides senior-level outsourced finance, management reporting, and AI automation for Australian NFP, NDIS, and SME organisations — including building AI tooling you can actually verify.
Talk to PFL →
Comments
Post a Comment