Computing Batch Hash Totals for Billing Runs

You have a billing run of tens of thousands of records leaving one system and arriving in another, and you need a single, cheap, deterministic answer to one question: is the population that arrived exactly the population that left? A batch hash total — sometimes called a batch fingerprint — is that answer. It is a small triple, (record_count, amount_total, digest), that you can recompute independently on the receiving side and compare in one line. This page sits under hash-total validation controls, part of the broader Billing Reconciliation & Audit Logging subsystem, and it is narrowly about the function itself: how to compute a fingerprint that is genuinely order-independent, exact in money, and unambiguous about missing fields — and how to re-verify it on the far side so a mismatch tells you which component drifted.

Minimal Prerequisites

  • Python 3.11+ — for NamedTuple typing ergonomics and the standard-library hashlib and decimal modules used below.
  • decimal.Decimal for every amount — never float. A fingerprint is only reproducible if the amount total is computed in exact base-10 arithmetic and quantized to a fixed exponent.
  • A stable canonical field set — agree, once, on which fields define a record’s identity and value. The fingerprint is computed over exactly those and nothing else.

Data assumptions: a batch is a list of mappings (one per billing record), each carrying at least the canonical fields with amounts already as Decimal. The batch is a multiset — duplicate records are legitimate and must be preserved, not collapsed.

Annotated Implementation

The whole primitive fits in one module. The design turns on three deliberate choices: values are canonicalized to unambiguous strings before hashing (so None, empty, and zero can never collide); amounts are quantized to a fixed exponent before summing (so representation never changes the total); and rows are sorted before hashing (so record order never changes the digest).

# batch_fingerprint.py
from __future__ import annotations

import hashlib
from decimal import Decimal, ROUND_HALF_UP
from typing import Iterable, Mapping, NamedTuple, Any

MONEY = Decimal("0.01")           # fixed exponent every amount is quantized to
NULL = "\x00"                     # sentinel for None, distinct from any real value
SEP = "\x1f"                      # field separator (unit separator, unprintable)

# The only fields that define a record's identity and value. Everything else is
# presentation and is intentionally excluded from the fingerprint.
CANONICAL_FIELDS: tuple[str, ...] = ("bill_id", "account_id", "service", "amount")


class BatchFingerprint(NamedTuple):
    record_count: int
    amount_total: Decimal
    digest: str                   # SHA-256 hex over sorted canonical rows


def _canon_value(field: str, value: Any) -> str:
    """Render one field to an unambiguous string. None gets a reserved sentinel."""
    if value is None:
        return NULL
    if field == "amount":
        # Quantize here too: the digest must not depend on trailing-zero formatting.
        return format(Decimal(value).quantize(MONEY, rounding=ROUND_HALF_UP), "f")
    return str(value)


def _canon_row(rec: Mapping[str, Any]) -> str:
    """One record → one canonical line, field order fixed by CANONICAL_FIELDS."""
    return SEP.join(_canon_value(f, rec.get(f)) for f in CANONICAL_FIELDS)


def fingerprint(records: Iterable[Mapping[str, Any]]) -> BatchFingerprint:
    """Compute the order-independent fingerprint of a billing batch."""
    rows: list[str] = []
    total = Decimal("0")
    for rec in records:
        rows.append(_canon_row(rec))
        total += Decimal(rec["amount"]).quantize(MONEY, rounding=ROUND_HALF_UP)
    rows.sort()                                   # order-independence lives here
    h = hashlib.sha256()
    for row in rows:                              # feed rows with a separator so
        h.update(row.encode("utf-8"))            # concatenation can't hide a shift
        h.update(b"\n")
    return BatchFingerprint(len(rows), total, h.hexdigest())

Two details do the heavy lifting. Sorting rows before feeding the hash makes the digest a function of the set of records rather than their sequence, so a batch and any shuffle of it fingerprint identically. And separating fields and rows with reserved unprintable bytes (\x1f, \n) closes the concatenation ambiguity where "12" + "34" and "1" + "234" would otherwise hash the same — a separator that can never appear inside a value guarantees each row and field boundary is unambiguous.

The re-verify step is what makes the fingerprint useful across a hand-off. The receiver recomputes a fresh fingerprint from the records it actually holds and diffs it, component by component, against the one it was sent — so the result is not just pass/fail but a pointer to the dimension that drifted.

def reverify(records: Iterable[Mapping[str, Any]],
             expected: BatchFingerprint) -> dict[str, Any]:
    """Recompute on the receiving side and diff against the expected fingerprint."""
    got = fingerprint(records)
    diff = {
        "ok": got == expected,
        "count_delta": got.record_count - expected.record_count,
        "amount_delta": got.amount_total - expected.amount_total,
        "digest_match": got.digest == expected.digest,
    }
    return diff
Computing a batch fingerprint on the sender and re-verifying it on the receiver On the left, a billing batch on the sending side is passed to the fingerprint function, which quantizes amounts, canonicalizes each record to a row, sorts the rows, and emits a triple of record count, Decimal amount total, and a SHA-256 digest. The batch and its fingerprint travel together to the receiving side. There the reverify function recomputes a fresh fingerprint from the records actually received and compares it component by component to the fingerprint sent, producing a count delta, an amount delta, and a digest-match flag. If all three agree the batch is accepted; if any differ the diff points to the drifted component. Batch (sender) records, Decimal amounts fingerprint() quantize · canonicalize sort rows · SHA-256 (count, total, digest) Batch (receiver) records as delivered reverify() recompute + diff Accept all three agree Report drift count / amount / digest delta batch travels fingerprint travels match mismatch

Edge Cases

  • Order must not matter, but multiplicity must. Sorting the canonical rows makes the digest order-independent, yet duplicates are kept in the sorted list, so two identical legitimate line items both count and both feed the hash. A far-side stage that de-duplicates them changes the count and the digest — correctly flagged as an alteration rather than silently accepted.
  • Decimal canonicalization and quantize. Decimal("42.5"), Decimal("42.50"), and Decimal("42.500") are equal in value but format differently. Quantizing to MONEY with ROUND_HALF_UP before both the sum and the row rendering guarantees the same economic amount always produces the same total and the same digest, so trailing zeros or an upstream reformat never manufacture a false mismatch.
  • None versus empty versus zero. A missing service (None), an empty string, and a real value must never canonicalize to the same token. The reserved NULL sentinel keeps None distinct from "", and the separator bytes keep an empty field from merging into its neighbor — so a genuinely lost field changes the digest instead of hiding behind a coincidental collision.
  • Duplicate rows and stable sort. Because the sort is over full canonical strings, records that are identical across every canonical field sort adjacently and deterministically; the fingerprint is stable no matter what order they arrived in. This is why the fingerprint is safe to compute after the reordering that double-entry journal manifest generation performs.

Verification Snippet

Assert the two properties that make the fingerprint trustworthy: it is invariant under reordering, and it is sensitive to a dropped record and to a one-cent change.

from decimal import Decimal
import copy, random

def _batch() -> list[dict]:
    return [
        {"bill_id": "B-1", "account_id": "A-1", "service": "water",
         "amount": Decimal("42.50")},
        {"bill_id": "B-2", "account_id": "A-1", "service": "sewer",
         "amount": Decimal("18.00")},
        {"bill_id": "B-3", "account_id": "A-2", "service": "water",
         "amount": Decimal("42.5")},   # same value, different formatting
    ]

def test_order_independent_and_sensitive() -> None:
    fp = fingerprint(_batch())
    assert fp.record_count == 3
    assert fp.amount_total == Decimal("103.00")

    shuffled = random.sample(_batch(), k=3)
    assert reverify(shuffled, fp)["ok"]                    # reorder → still ok

    dropped = _batch()[:-1]
    assert reverify(dropped, fp)["count_delta"] == -1      # a lost record shows up

    altered = copy.deepcopy(_batch())
    altered[0]["amount"] = Decimal("42.51")               # one-cent change
    d = reverify(altered, fp)
    assert not d["ok"] and d["amount_delta"] == Decimal("0.01")
    assert d["digest_match"] is False

The formatting case (42.5 versus 42.50) is the one that proves canonicalization works: both must contribute Decimal("42.50") to the total and render to the same row, or the fingerprint would flag an intact batch.

FAQ

Why sort the rows before hashing instead of hashing them as they arrive?
Because billing records legitimately arrive in different orders on different runs — parallel workers, database planner order, and manifest grouping all reshuffle them. Hashing in arrival order would make every harmless reordering look like tampering. Sorting the canonical rows first makes the digest a function of the set of records, so a batch and any permutation of it fingerprint identically while a genuine drop, add, or alteration still changes the hash.
Why quantize amounts before both summing and hashing?
A fingerprint must be reproducible from the same economic batch regardless of how amounts were formatted upstream. Decimal("42.5"), Decimal("42.50"), and a value with extra trailing digits are equal in worth but render differently. Quantizing to a fixed exponent with ROUND_HALF_UP before the amount enters the total and the canonical row guarantees the same money always yields the same total and digest, so representation never manufactures a false mismatch.
How does the fingerprint handle None or missing fields?
Each field is rendered by _canon_value, which maps None to a reserved sentinel byte distinct from every real value, and fields are joined with a separator byte that cannot appear inside a value. That keeps None, an empty string, and a zero from ever collapsing to the same canonical row, so a genuinely lost field changes the digest instead of hiding behind a coincidental string collision.
What does reverify tell me beyond pass or fail?
It returns a component diff: a count delta, an amount delta, and a digest-match flag. That triage points straight at the failure mode — a nonzero count delta means a record was dropped or added, an amount delta with a zero count delta means a value was rewritten, and a digest mismatch with both counts and totals intact means a same-count, same-sum change such as two amounts transposed. The pass/fail flag alone would leave you guessing.

Up: Hash-Total Validation Controls · Billing Reconciliation & Audit Logging