The System of Record Forgets

Introduction

A multi-branch operator asks its CRM a simple question and gets a wrong answer, not because the data is dirty but because the question is about the past. How many accounts were active in that branch eighteen months ago, at what price, on which plan? The system knows what is true today, because it was built to keep exactly one correct answer to that and to overwrite whatever was true before. The record is not broken; it is doing its job. The trouble is that the questions that decide the business - retention by cohort, price and plan history, an audit of who changed what and when, a forecast trained on how the book actually moved - are questions about time, and a system of record keeps the present by forgetting the past. This essay is about that gap, and about the difference between a system that runs operations and one that can reason over them.

• A record is built to hold what is true now. A transactional CRM or field-service platform is engineered for correct, concurrent, current-state writes. It is not architected for the historical, cross-branch, point-in-time workload that decision-making needs.
• Overwriting the present erases the past.When a status flips or a price is changed in place, the prior value is gone. The record stays correct about now and loses the ability to answer “what was true back then.”
• The forgetting is measurable. In a seeded model on this page, a field that changes about four times over three years lets a current-state record answer only about one in four point-in-time questions; a six-field snapshot, about one in twelve. A layer that keeps history answers all of them.
• Retaining time is a data-model choice, not a purchase. History can be added to the record itself (an append-only log, a slowly changing dimension, system-versioned tables) or held in a separate layer. The concern has to be owned somewhere; who owns it is a separate decision.

Record, engagement, intelligence

The vocabulary here is worth getting right, because it is half marketing and half architecture. In a widely cited AIIM white paper commonly dated to 2011, Geoffrey Moore drew a line between the system of record - the transactional ledger that holds the authoritative current state of the business - and the system of engagement, the collaborative, front-line software layered on top of it. Years later the venture investor Jerry Chen added a third term, the system of intelligence, in a 2017 essay whose thesis is stated in its own subtitle: systems of intelligence are “the next defensible business model.” Chen’s definition is useful and worth quoting: what makes a system of intelligence valuable, he wrote, is that it “typically crosses multiple datasets and multiple systems of record.”

Two honesty notes travel with that term. First, it is a coinage, not a standard: it comes from an investor, a Greylock partner, arguing where the next software moats will be built, which is a commercial argument about where value accrues. Ardenus has the same interest, and this essay says so plainly below. Second, the useful part of the distinction for an operator is not the branding but the architecture underneath it: a system of record and the analytical, historical, cross-system work of a system of intelligence are different workloads, with different shapes, and the reason they are different is older and more concrete than any of these labels. That is where the argument actually lives.

What a record is built for

A transactional database is optimized for a specific job. Since Codd’s relational model in 1970, the operational store has been normalized so that each fact is written in one place and kept consistent, and since the principles Haerder and Reuter canonized as ACID in 1983, it has guaranteed that small, concurrent writes are atomic, consistent, isolated, and durable. That is the whole point of a system of record: many people booking appointments, taking payments, and updating statuses at once, and the database keeping one correct, current answer through all of it. This is online transaction processing, OLTP, and a FieldRoutes, a PestPac, or an Aspire is a very good one.

The analytical question has a different shape. Answering “how did revenue by service line, by branch, by month move over three years” means scanning wide and reaching back, not writing one row correctly right now. The database field has known this for decades: the analytical workload was given its own name, online analytical processing or OLAP, in a 1993 paper by Codd and colleagues, and it has long lived in a separately architected system. Whether that system is the top-down enterprise warehouse of Inmon or the dimensional, star-schema marts of Kimball, the two schools disagree about method but agree on the core point: analysis belongs in a store shaped for reading history, not in the one shaped for recording the present. The 1993 OLAP paper, it is worth noting up front, came with a commercial backstory this essay returns to below; the load-bearing architectural claim rests on the peer-reviewed work, not on it.

The overwrite that forgets

Here is the mechanism, in one sentence: a system of record keeps the present correct by overwriting the past. When an account is cancelled, the status field is updated from active to cancelled in place; the row now says cancelled, and the fact that it was active until last Tuesday is not stored anywhere unless something was built to store it. The same is true when a price is raised, a plan is switched, an account is reassigned to a different branch, or a technician is changed on a route. Each write keeps “what is true now” correct and quietly discards “what was true then.” The record is a snapshot that is always current, and historical only where something was built to keep it.

That is exactly the wrong substrate for a point-in-time, or as-of, question - a question about the state of the book at some past moment. Retention by cohort needs to know who was active at the start of each period. A price-realization analysis needs the price that was actually in effect, not today’s. An audit or a dispute needs to reconstruct what the record said on a given date. A forecast worth trusting is trained on how the book truly moved, not on today’s values projected backward as if they had always held. None of these is recoverable from a store that kept only the latest value, and no amount of cleaning fixes it, because the past was never wrong; it was overwritten.

How fast the record forgets

The chart at the top of this page puts a number on the forgetting. It is a self-contained, seeded synthetic model computed from public parameters, with no client data anywhere. Treat a single tracked field as changing at random over a three-year window, and ask what share of point-in-time questions a current-state record can still answer: a query about some past moment is answerable only if the field has not been overwritten since. Averaging over the window gives an exact curve, f(K) equals (1 minus e to the minus K) over K, where K is how many times the field changes in those three years, and a seeded Monte-Carlo simulation reproduces the curve to confirm the arithmetic.

Read off the curve and the shape is stark. In this model, a field that changes about four times over three years lets the record answer only about a quarter of point-in-time questions - roughly one in four - so a history layer, which keeps that field’s every version, answers roughly four times as many of them. A whole-record snapshot is worse: six fields that each change a couple of times leave the current-state record able to reconstruct the full past state only about one in twelve times, while the history layer still answers all of them. Every one of these numbers is reproduced and locked by the committed build script from a fixed seed, so the figure and the prose can never drift apart.

Be exact about what this is. It is an illustrative model that shows a mechanism - the more a field is overwritten, the less of its own past a current-state store can recover - and its specific figures are analytical implications of the stated assumptions, not a measurement of any operator’s book. The history layer’s hundred percent is true by construction, because a store that keeps every version can always answer as of any date; it is not an empirical win. The point the model makes is only this: the rate at which a record forgets is a function of how often it is overwritten, and for the fields that matter that rate is not small.

Retaining time

The fix is to stop throwing the past away, and it is a well-worn part of the data field rather than anything exotic. The most direct form is an append-only design, in which a change is recorded as a new event rather than an overwrite, so the current state becomes a replay of the log and every prior state is still there; event sourcing is the common name for it. The dimensional-modeling tradition has its own version, the slowly changing dimension, which closes off the old row and opens a new one with valid-from and valid-to dates instead of updating in place. The relational world has since standardized the idea: the SQL standard added system-versioned temporal tablesin SQL:2011, so a table can keep its own history and be queried “as of” any past instant. Underneath all of them is the temporal data model, which formalizes valid time and transaction time and what it means to reconstruct a record at a point in the past.

The through-line is simple: retaining time is a property of the data model. Once the history is kept, the point-in-time questions become answerable - not because a cleverer query was written, but because the answer was no longer thrown away when the present changed.

Not necessarily a separate system

It would be convenient, and this essay is careful not to claim it, to say that keeping history requires a separate system. It does not. The techniques above can be added to the operational store itself, and the once-firm wall between OLTP and OLAP has genuinely softened: hybrid transactional and analytical processing, or HTAP, describes engines built to serve both workloads at once, so physical separation is a design pressure, not an iron law. The honest reading of the classic argument for splitting them, Stonebraker and Cetintemel’s“one size fits all” paper, is that specialized engines beat one general engine for each job, which reinforces the separation of concerns rather than demanding two products; and it is worth noting that Stonebraker, like most people who argue database architecture, has his own database ventures. The concern is real and has to be owned somewhere. Whether an operator builds it into the record, buys a layer that provides it, or grows into it over time is a genuine choice, not a foregone conclusion.

Two disclosures belong right here, in the same breath, because they are the same kind of thing. Ardenus sells exactly the history-keeping layer this essay argues for, which makes the argument seller-convenient; a reader should weigh it knowing that. And the term OLAP itself arrived the same way: the 1993 paper that coined it, as the OLAP history records, was a short consulting deliverable Codd undertook for Arbor Software, its rules referenced Arbor’s product by name, and Computerworld retracted its coverage when it learned he had been paid. A respected authority publishing a vendor-convenient architectural mandate is exactly the pattern to be wary of, including when the vendor is us. The right posture is the same in both cases: use the idea because the peer-reviewed foundation under it holds, and surface the interest rather than hide it. The system of record is not the villain of this story. It is good at the job it was built for, and the layer that answers the historical questions sits on top of the systems an operator already runs, not in place of them.

What this means for the operator

For a multi-branch operator the practical test is quick. Take a question that matters and put a date on it: what did this branch’s book look like a year ago, what price was actually in effect last spring, which accounts were active at the start of the cohort you are measuring. If those answers can only be estimated from today’s values projected backward, the business is running on a system of record that has forgotten its past, and the fix is to start keeping time - in the record or beside it - before the next question about history is asked. This connects to the neighboring research: what a model needs in order to reason over that data at all is the subject of the data foundation beneath enterprise AI; whether the current record is even correct before you ask it about the past is the subject of the negative effects of dirty data; whether to migrate, consolidate, or layer over the systems a roll-up inherits is the subject of the post-acquisition system decision; and which kind of model to point at the history once you have kept it is the subject of the regimes of prediction.

The boundary a vendor has to state is plain. Software does not decide how much of the past is worth keeping or what to do with it; those are the operator’s calls. What a data and intelligence layer can do is keep the history the record overwrites and make the point-in-time questions answerable, whether that history is kept inside the systems of record or in a layer on top of them. Ardenus is built for that layer and sits on top of the systems an operator already runs, but no result, saving, or forecast is attributed to Ardenus here; the argument is the public record of database architecture, and no client or first-party data appears anywhere in this essay. You can read more of our research on the Ardenus articles hub, or see the platform itself on the technology page.