Data Quality Isn't Data Correctness

A pipeline started double-counting revenue on March 3rd. It was found on June 11th, by a finance analyst, in a board deck.

For a hundred days, every automated check stayed green. The data was fresh. No nulls appeared. The primary key stayed unique. Row counts held steady. Nothing drifted more than two standard deviations. The dashboards loaded on time and the numbers looked plausible — revenue was up 8%, which everyone was pleased about.

The pipeline was wrong the entire time, and the entire test suite agreed it was fine.

This is not a story about a team that forgot to write tests. They had 340 of them. It is a story about what those tests are able to check — and it applies to every data quality tool on the market today.

The bug was a fan-out. A change to handle late-arriving order lines introduced a join that produced one row per order line instead of one per order. Orders with three line items were counted three times.

Here is what each layer of tooling saw:

Look closely at why the volume check was useless here. The fan-out happened before the daily aggregation, so the output still had exactly one row per day. Row counts could not possibly have moved. The check was blind by construction, not by misconfiguration.

And the drift check? An 8% month-over-month move in revenue is entirely normal. The bug hid inside the noise band of the thing designed to detect it.

Every one of those assertions is a statement about the output: its shape, its nullity, its cardinality, its statistical behaviour. Every one asks is the warehouse internally consistent? Not one asks is the warehouse correct?

Those are different questions, and the second one is the reason I started building Eternity. The shape of the answer is simple enough to draw before explaining any of it:

Software testing has a name for this. It is called the oracle problem: to verify that a program is correct, you need an independent way of knowing the right answer. And usually, obtaining that independent answer means writing the program a second time — at which point you have two programs that might both be wrong, and no way to tell which.

This is why most software testing settles for something weaker than correctness. We assert properties (this never returns null), invariants (the balance never goes negative), and regressions (yesterday's output equals today's). All useful. None of them is the answer.

Data quality tooling inherited exactly this compromise. not_null, unique, accepted_values, freshness, volume, drift — these are property assertions and regression checks. They are the data equivalent of "it didn't crash and the output looks like it usually does."

The industry has been extremely good at this for a decade. dbt tests, Great Expectations, Soda, Monte Carlo, Anomalo, Databricks DLT expectations — all excellent, all mature, and all of them answer a question that is one step to the left of the one that matters.

Here is the part that I think is under-appreciated, and it is the reason this is worth building.

In general software, the oracle problem is genuinely hard. If you want to verify a compiler, your only real option is another compiler. There is no shortcut to the right answer.

Data engineering has a structural asymmetry that general software does not:

For deterministic derivations, the expected value can be computed from the source by a path that is radically simpler than the pipeline that produced it.

Your pipeline might be fourteen dbt models with incremental materialisation, an SCD2 dimension join, late-arriving-fact handling, and a dedup CTE with a window function. That complexity is real, and most of it exists for performance, historisation, and incrementality — not because the business definition is complicated.

The business definition is usually one sentence: net revenue is the sum of settled order amounts, excluding internal test accounts.

One sentence. One SUM. Two filters. And you can run it directly against the source system.

If the one-line answer and the fourteen-model answer disagree, the pipeline has a bug — and you never had to read the pipeline to find out.

Concretely, for the double-count above. The expectation is computed against the source — the operational database where orders are actually written:

-- EXPECTED: computed from source, independent of the pipeline
SELECT
  date_trunc('day', created_at) AS day,
  SUM(amount)                  AS revenue
FROM orders
WHERE status = 'settled'
  AND is_test_account = false
GROUP BY 1

The actual is read from the target — the warehouse table the pipeline produces:

-- ACTUAL: read from the pipeline's output
SELECT day, revenue
FROM analytics.fct_daily_revenue

Then you compare them, per grain:

March 3rd. The exact day the fan-out shipped. Found in one run, by a check that knows nothing whatsoever about the fourteen models in between.

Notice what made this work, because it is the whole design constraint: we did not reverse-engineer the pipeline's SQL. That is deliberate and it is not laziness. If you derive your expected value by parsing and re-executing the developer's transformation logic, you faithfully reproduce their bug and the test passes. An oracle that reads the implementation is not an oracle. It is a mirror.

The expected value has to come from somewhere the bug cannot reach. That place is the source system, plus a human statement of what the number means.

This suggests an inversion that I think is the actual idea here, and it is worth stating plainly.

Today, the pipeline's output is the truth by default — not because anyone verified it, but because it is what the dashboards read. Everything downstream inherits its authority from the fact that it is downstream. Ask why a number is right and the honest answer is usually "because that's what the table says."

But a warehouse table is a derived artifact. It is a cache of a computation over the source. And we treat it as authoritative despite the fact that no one has ever checked it against the thing it was derived from.

Your warehouse is not the source of truth. It is a cache that has never been proven correct against its origin.

Flip the arrow and everything reorganises. A human declares what a number means, anchored to the source, once. The system computes it independently. And the pipeline stops being the arbiter of truth and becomes the thing under test — which is what it always should have been, because it is the part that has bugs in it.

The obvious question: if this is such a clear gap, why hasn't Databricks or Snowflake or Palantir shipped it?

They could. Any of them has the engineering capacity to build this in a quarter. I do not think capability is the constraint — incentives are, and that is a more durable thing to bet on than a head start.

The idea is not secret. Any senior data engineer, asked how they would verify a pipeline, will eventually say "well, you'd compute it independently from source and compare." Everyone knows. Almost nobody does it.

The reason is not that the execution is hard. The bottleneck is authoring.

Great Expectations did not fail to take over the world because its checks were bad. They were good. It struggled because writing the expectations was work that nobody had time for, and because the person who actually knows what revenue means — the analyst, the finance lead, the domain expert — cannot write Python. So the checks got written by whoever had capacity, which meant they got written shallowly, or not at all.

Go and look at your own dbt project. Count the not_null and unique tests. Now count the tests that assert a business quantity is correct. The ratio tells you everything: teams write the tests that are cheap to write, not the tests that catch the expensive bugs.

So the unlock is not a better assertion engine. It is making it possible for the person who holds the definition to state it, in their own words, and have that become an executable, reviewable, version-controlled artifact.

The unit is an Expectation: a declaration of what a number should be, anchored to source, independently executable, and versioned.

It is authored by drafting in natural language and then locking a deterministic form. The language model drafts the structure from a sentence; the human reviews and corrects the form; the form is what gets stored and re-run. This matters more than it sounds, and I will come back to it.

1. The target is queried in place. Never ingested.

A downloaded export is a stale snapshot, not a test. We push the aggregate down to the live target and pull back only the small result set. Same for the source. This is not only about freshness — it is what makes the thing approvable. "We will copy your production orders table into our system" is a conversation that ends with security saying no. "We will run a read-only aggregate and receive back forty rows" is a conversation that ends differently. It is also the difference between an affordable product and an unaffordable one.

2. The language model never computes your numbers.

The LLM drafts the expectation's structure from a sentence. That is the entire extent of its involvement. Once the human locks the form, execution is pure SQL — deterministic, inspectable, and identical every run. You can read the exact query. The model is in the authoring path, never in the arithmetic. This is the same principle the query engine already runs on: numbers are computed by SQL against your rows, never generated by a model.

3. Lineage is declared, not inferred.

An expectation already knows its source operands and its target column, because a human stated them. Combined with the database's own foreign key constraints, that is trustworthy lineage for free — and it needs no access to the transformation code.

4. A run is immutable.

Re-running an expectation writes a new run rather than updating the old one. Every run is a point-in-time snapshot with the exact SQL, the result, and a proof certificate you can hand to an auditor or attach to a migration sign-off. "It passes now" is a weaker claim than "it passed on March 3rd at 14:02 UTC, and here is the query and the numbers."

5. An expectation is a definition, not a materialised view.

We store the definition plus a cached result snapshot. Re-opening is instant and costs nothing. Re-running recomputes against current data. This is also, incidentally, a semantic layer — the definition of "net revenue" now exists in exactly one executable place instead of being re-derived slightly differently in a dbt model, a Looker explore, and three analysts' heads.

Here is the part that I think turns this from a useful tool into infrastructure.

Because an expectation is anchored to the source and expressed in business terms, it survives changes to everything in between.

Refactor your fourteen models into six. Migrate Redshift to Snowflake. Replace the batch job with streaming. Change your orchestration, your warehouse, your entire modelling philosophy. The expectation still holds, and it still checks, because it never referred to any of that.

Compare that to what we have now. A dbt test lives inside the model it tests, and dies when you refactor the model — the test and the thing under test are the same artifact, which is a bit like grading your own homework. A data diff compares two versions of the same table, which is genuinely useful in a pull request and completely useless the moment you re-platform, because there is no "same table" anymore.

An expectation outlives the pipeline it tests.

That is the asset. Not the test run — the accumulated, executable, source-anchored statement of what your numbers actually mean. Most organisations have never written that down anywhere it can be executed.

I would rather tell you the limits than have you discover them.

It does not work for everything. Sums, counts, ratios, allocations, reconciliations, balances — yes. Machine-learning features, attribution models, forecasts — no. For those there is no independent way to compute the truth; the model is the definition. Anyone claiming to verify a forecast's correctness against source is selling something. Deterministic derivations are the scope, and in our experience they are the overwhelming majority of what silently breaks.

The oracle can be wrong too. If the expectation is mis-declared, it fails when the pipeline is fine. Three false alarms and a human mutes it forever — alert fatigue is what killed the last generation of data quality tooling, and it will kill this one if we are careless. So an expectation has to be provable at authoring time: before it can be saved, you see the rows it matched and the number it produced, and confirm that is what you meant. And a failure has to arrive as a diff you can drill into, not a red X. A red X that you cannot investigate in thirty seconds is functionally the same as no alert at all.

Source access is the real friction, and it is not technical. Reading production source systems is exactly what security teams exist to prevent. Read-only credentials, pushdown-only aggregates, and no data leaving the boundary are the design answer, but this is a conversation to have honestly and early rather than a problem to engineer around.

Recomputing from source is not free. At scale you cannot diff row by row. Pushdown aggregation, grain selection, and bounded windows are how this stays affordable, and there are datasets where the honest answer is that a full independent recomputation is too expensive to run hourly.

Being precise about what exists, because "building in public" should not mean describing plans in the present tense.

Live today. Source-to-target reconciliation: row counts, schema and type drift, value mismatches on matched keys, rows present on only one side, aggregate deltas. Runs are persisted, you can re-open one and ask follow-up questions about it in chat, and you can export a self-contained proof certificate. Row-level lineage traces an answer back to the source rows it came from. The natural-language query engine computes every number by SQL and shows you the query.

Being built now. Query-mode connectors — aggregate pushdown against a live target instead of ingesting it. Then the expectation store itself: authoring, versioning, the expected-versus-actual run, and the verdict. That is the piece described in this post, and it is not finished.

First use case: migration validation. Every enterprise is migrating something, and today the validation is a data engineer writing four hundred ad-hoc comparison queries and tracking them in a spreadsheet. That is the same problem — prove the numbers did not change — and it is one I have done by hand more than once, on a SAP HANA to Snowflake migration and on a bank's trade data lake. It is the shortest distance between what we have and something a team would pay for on a Tuesday.

If you take one thing from this, make it this question, and ask it about your own stack.

Pick the number your company cares most about. Revenue, active users, whatever appears in the board deck. Now answer: how would you know if it had been wrong for the last hundred days?

Not "how would you know if the pipeline broke" — you would know that, you have monitoring. How would you know if it ran perfectly and produced the wrong number?

If the honest answer is "someone downstream would eventually notice something looked off," then you do not have a data quality problem. You have an oracle problem. And it is worth knowing the difference, whether or not you ever use anything we build.