In Data Quality Isn't Data Correctness we argued that a test suite can stay green while a pipeline reports the wrong number for a hundred days. Since then we built the thing that article asked for, and then tried hard to break it.
This is that test, with every number in it.
The test bed
We built an accounts-payable datamart the way an Oracle Fusion shop would: a staging layer copied from the source, conformed dimensions with full supplier history (SCD2), fact tables loaded incrementally, and a reporting layer on top with spend by supplier and month, invoice counts and open payables.
The data was synthetic but behaved like Oracle Payables does. Invoices came in rupees, dollars and euros. Cancelled invoices kept their original lines and gained reversing ones. Domestic invoices left the ledger-currency amount empty, as Oracle does. There were prepayments, withholding-tax lines, five suppliers renamed mid-stream, and thirty invoices that arrived after their month had closed.
Before planting anything, we proved the reporting layer correct: all 156 supplier-month cells matched the truth computed directly from staging, to the rupee.
Nine bugs
Four are mechanical: the kind a careful engineer might catch by eye. Five break a business rule that the mapping sheet never writes down.
| Bug | Kind | What it does to the report |
|---|---|---|
| Supplier joined on every version, not one | Mechanical | Renamed suppliers counted twice: +26.6M |
| Unmatched supplier keys dropped | Mechanical | 3% of lines vanish: −13.9M |
| Incremental load skipped late rows | Mechanical | 70 supplier-months missing: −16.8M |
| Two current versions of a supplier | Mechanical | No change to totals; the dimension is wrong |
| A clean-up filter drops negative lines | Business rule | Reversals and credits vanish: +40.8M |
| Ledger amount summed without its fallback | Business rule | Every domestic invoice drops out: −11.0M |
| Prepayment invoices counted as spend | Business rule | Advances counted twice: +38.9M |
| Withholding tax netted into spend | Business rule | Small, and real: −29K |
| Supplier named by today's version | Business rule | History rewritten under new names |
Two ways to test it
Tests generated from the mapping. We took the reporting layer's source-to-target mapping literally: spend comes from the ledger amount, invoice count from the invoice id, open payables from the amount remaining, grouped as mapped. No questions, no extra rules. This is roughly what mapping-driven test generation produces.
Verum. The same mapping sheet, plus the scoping agent. It reads the mapping, looks at the data, and before asking about any rule it measures whether the plausible answers change the number. It asks only when they do. Each check is then locked and recomputed from source on every run, with no AI in the loop.
A simulated business owner answered Verum's questions from the written business rules. A check counts as a catch only if it passes on the correct pipeline and fails on the broken one.
The scorecard
| Tests from the mapping | Verum | |
|---|---|---|
| Mechanical bugs caught | 0 of 4 | 4 of 4 |
| Business-rule bugs caught | 0 of 5 | 4 of 5 |
| False alarms on the correct pipeline | 2 of 3 checks | 0 |
| Questions asked | none | 3, across 4 checks |
Why the mapping tests had no signal
They were not lazy tests. They checked exactly what the mapping says. The problem is what it doesn't say. The mapping maps spend to the ledger amount, and says nothing about the ledger amount being empty for domestic invoices, which line types count, or that prepayments are advances. So the spend and invoice checks were red on the correct pipeline, and red on almost every broken one.
A check that is red either way is not a check. In practice it gets muted, and then it is not even a red light.
The three questions
Everything else it either read from the mapping and the data, or measured, found it did not change the number, and recorded as an assumption instead of asking.
- Spend: which distribution lines count as AP spend? (It had already found the ledger-amount fallback by profiling the column.)
- Invoice count: should an invoice count if its lines were reversed?
- Open payables: should credit balances net off what is owed?
The one it did not catch
Renaming suppliers in history leaves every per-supplier total unchanged. Only the names move. Staging keeps current supplier names only, so no check computed from source can tell which name was right last quarter.
Verum did not pretend otherwise. Every check listed supplier names under "not tested by this check". We would rather show you a limit than a green light that means less than it says.
The one we did not plant
While scoping the open-payables check, Verum pointed out that the amount remaining was being summed across rupees, dollars and euros with no conversion. That was a real defect, in the reporting layer we had built ourselves.
What the test found in Verum
Running this end to end also found four problems in our own product, all fixed before the result above:
- Ledger-currency spend was inexpressible: a check could not say "the ledger amount, or the invoice amount when it is empty". It can now.
- Postgres returns a month from a date column with a time zone and from a timestamp without one, so month-by-month comparisons refused to line up.
- A NUMERIC supplier id did not match the same id stored as BIGINT.
- A 37-table staging layer pushed the reporting tables out of the agent's view, so it could not find the open-payables report.
Try it on your own pipeline
Bring a mapping sheet and a report you would like to trust. Book a demo, or start free.