The Regimes of Prediction

Introduction

The newest version of an old pitch says the model has finally swallowed the method: give a large language model enough context and it will out-predict the classical machine-learning stack you spent years tuning, so you can retire the pipelines and just ask. It is a clean story, and for the tasks language models were built for it is even true. But prediction is not generation, and on the kind of prediction most operators actually run - a number or a probability drawn from structured records - the measured evidence runs the other way. Handed the same history, a gradient-boosted tree still beats a frontier LLM on the great majority of structured tasks, more cheaply and with better-calibrated answers. The language model earns its place at the edges of the problem, not at its numeric core, and the most durable enterprise pattern is not one model winning but the right model for each regime, chosen and fed by an ontology.

• Classical machine learning still wins structured prediction: on tabular and numeric records with real history, gradient-boosted trees remain the strong incumbent, and a frontier LLM given the same data usually loses.
• The LLM wins the edges: extreme cold-start, where there are almost no labels to learn from, and prediction drawn from unstructured text.
• It was never the language model:the strongest “beats the trees” results are purpose-built tabular and time-series foundation models, not chat models, and the general LLM used the same way underperforms plain statistics.
• The frontier is the hybrid inside an ontology: the language model supplies semantics and orchestration, a classical model does the numeric prediction, and the ontology decides which sees which slice of governed data.

Prediction is not generation

Most of the awe around large language models is earned by generation: drafting, summarizing, translating, writing code, answering in fluent prose. Prediction is a different job. It asks for a specific output - a risk score, a churn probability, a next-week demand number, a class label - computed from structured features, and it is judged not by how the answer reads but by whether it is right and whether its confidence can be trusted. Conflating the two is the root of the “the model ate the method” pitch. An assistant that writes a flawless service summary is not thereby a better estimator of which accounts will cancel next month.

“An LLM with context” is also not one thing. It can mean pasting a table’s rows into a prompt as text, retrieving passages and asking over them, or a foundation model doing in-context learning over the actual data. Those are different mechanisms with different results, and the honest question is narrow: on a given prediction task, handed the same information, does the language-model approach beat a model trained to predict it? For structured, numeric prediction the answer is usually no, and the rest of this essay is where that holds, where it flips, and what to build because of it.

Where machine learning still wins

Start with the incumbent, because the incumbent is strong. A peer-reviewed survey of the whole field concludes that gradient-boosting methods - the XGBoost, LightGBM, and CatBoost family - still outperform deep-learning methods on most tabular datasets, with the language-model advantage confined to a few narrow regimes. The underlying benchmark work is blunt about it: across 45 datasets and roughly 20,000 compute-hours of hyperparameter search per learner, tree-based models stayed state-of-the-art on medium-sized data of around ten thousand rows - the exact scale of most operational tables - and the gap held even after exhaustive tuning. This is the format that describes a company’s governed history: technicians, jobs, invoices, routes, outcomes, one row per event.

The cleanest test is a direct head-to-head, and it has been run on real records. In a 2025 study in the Journal of the American Medical Informatics Association, researchers gave a frontier model the same electronic-health-record data a classical model trained on, and measured both. A locally trained model predicted hospital discharge within 24 hours at 0.847 AUROC versus GPT-4’s 0.629 and GPT-3.5’s 0.537, and ICU transfer within 24 hours at 0.894 versus 0.602 and 0.517. That is roughly a quarter of a point of AUROC, an enormous margin in this setting, and the model’s probabilities were about three times better calibrated - a Brier score of 0.134 against GPT-4’s 0.251 on the first task - with every gap significant. The authors’ conclusion is that non-fine-tuned LLMs are simply less effective and less robust than locally trained machine learning for clinical prediction.

The pattern is not a quirk of medicine. On credit and loan-default prediction, the closest widely-benchmarked analogue to operational risk scoring, a plain LightGBM reached 0.73 ROC-AUC on a real loan book and beat every zero-shot LLM tested, while a LightGBM-plus-LLM ensemble added nothing over the tree alone. And on the shape of problem field-service operators know best - a rare event in an imbalanced table - the tree still leads: predicting patient no-shows at a positive rate of about one and a half percent, XGBoost reached an F1 of 0.62 against a fine-tuned clinical language model’s 0.51, with the language model losing exactly where the positives are scarce. This is the regime that matters for predicting a missed appointment, a churned account, or an equipment failure.

The LLM you can delete

Forecasting sharpens the point to something almost surgical. A line of “LLM-based” time-series forecasters wraps a language model in a numeric pipeline and reports strong numbers. At NeurIPS 2024, researchers asked the obvious control question: what does the language model actually contribute? They removed it, or replaced it with a trivial attention layer, and re-ran three popular methods. The forecasts did not get worse. They mostly got better. Across 13 datasets and two metrics, the ablated versions matched or beat the original in 26 of 26 cases for one method, 22 of 26 and 19 of 26 for the others, while the language model they deleted carried on the order of twelve thousand times more parameters. For numeric forecasting, the LLM was dead weight.

What does win forecasting is well-engineered classical learning. In the M5 competition - the largest public retail-demand contest, run on real Walmart sales - pure machine-learning methods beat every statistical benchmark for the first time in the competition’s history, with the winning gradient-boosted entry ahead by 22.4%. The value lived in the features and the structure of the sequence, not in a language model reading the numbers as text. If you are predicting next week’s route volume, the lesson is to spend on the pipeline and the features, not on wrapping a chatbot around the series.

The crossover

None of this means the language model never wins. It wins a real, bounded region, and naming it precisely is what keeps the argument honest. The clearest map comes from serialized-row prompting, where a model reads a table’s rows as text. In the study that introduced the technique, an LLM was on par with gradient-boosted tree ensembles up to about 256 labeled examples, and in the extreme cold-start case reached 0.84 AUC on an income-prediction task with zero training examples, where a four-shot XGBoost sat at chance, 0.50. But follow the curve and it crosses: as labels accumulate, the tree climbs past the flat language-model line and finishes ahead at full data, around 0.93. The hero of this essay is that crossover. Left of it, with almost no labeled history, the LLM’s built-in priors win; right of it, once you have real data, the trained model does.

The second genuine win is prediction drawn from unstructured text, and here the margin is decisive. Predicting postoperative risk from surgeons’ free-text notes, pretrained language models beat traditional word-embedding baselines by 38.3 points of AUROC. When the signal lives in language, use the model built for language. Both wins are edges of the data - the cold-start corner and the unstructured-text corner - not its structured, numeric center.

There is a further, uncomfortable wrinkle in even those cold-start wins: some of them are not real. Frontier models have memorized the canonical public tables - Titanic, Adult, Iris, Wine - verbatim, and on genuinely novel, post-cutoff data GPT-4 “does not outperform logistic regression on any dataset”, with few-shot sample efficiency that trails classical statistics. A striking zero-shot score on a famous dataset can be recall, not reasoning. Any LLM prediction result measured on a well-known benchmark has to be read with that discount in mind, which only narrows the honest winning region further.

It was never the language model

The most important clarification in this whole debate is that the results which do beat the trees are usually not chat models at all. The strongest recent challenge to gradient boosting on small tables is a tabular foundation model, TabPFN, which does in-context learning over the actual data rather than reading it as prose. It beat the strongest default baseline, CatBoost, by 0.187 in normalized ROC-AUC - 0.939 against 0.752 - and did it in seconds rather than hours. But it is not a language model, and its authors bound the result to datasets of up to ten thousand samples and five hundred features. It is a model built for tables winning at tables.

Forecasting tells the same story twice. A domain-pretrained time-series model, Chronos, scored 0.645 in aggregate relative error on 27 unseen datasets, beating AutoARIMA at 0.761- but the general language model used as a forecaster, LLMTime, scored 0.804 on the same benchmark, worse than plain AutoARIMA. A model pretrained on time series wins; a chat model reading the same series as text loses to a decades-old statistical method. The source of the gains is grounding in the structure of the domain, not the generality of language, which is the same argument we make about grounding an AI system in the right evidence: the model is only as good as what it is anchored to.

The hybrid within the ontology

Put the pieces together and the enterprise answer is not a contest between the language model and the classical one. It is a division of labor, arbitrated by an ontology - a model of the entities and relationships in the business - that decides which predictor sees which slice of governed data. The first job of that layer is to make the language model usable on structured data at all. Grounding a model in a knowledge-graph representation of a database, rather than pointing it at raw tables, is the difference between a wrong answer and a right one: on an independent enterprise benchmark, adding external grounding raised GPT-4’s execution accuracy from 34.88% to 54.89%, where unaided humans reach 92.96 percent and the model on its own does not come close. We measure the same lift from the semantic layer itself in our companion research on precision over capacity, where an ontology roughly triples a model’s accuracy on governed enterprise data.

The ontology is not only a scaffold for the language model; the same structure lifts the classical predictor too. Independent, peer-reviewed work outside any single vendor shows graph and knowledge-graph structure improving prediction directly: modeling the knowledge graph explicitly improved recommendation accuracy by 8.95% in recall over strong baselines, and in fraud detection an LLM-enhanced graph model improved the area under the curve by 6.97% over a graph-only model. The pattern that recurs across all of it is the same one: the language model contributes semantics, feature enrichment, and orchestration, while a classical model does the numeric prediction, and the ontology governs the handoff. This is the predictor that lives on top of the data foundation we argue enterprise AI actually stands or falls on.

Where the argument bends

An honest version of this has to mark its limits, because several of them are real. On small data the foundation model genuinely beats tuned trees, as TabPFN shows, so “trees always win” is too strong. The field is also moving: on a leakage-audited 2025 benchmark, well-tuned modern neural networks, combined through ensembling, now match or edge gradient-boosted trees, and the authors frame the whole trees-versus-deep-learning fight as a false dichotomy that ensembles across both settle. The right reading is not that one family always wins; it is that the winner is regime-specific, and the engineering around the model decides more than the model’s name - the argument we make about the value living in the system, not the leaderboard score.

The ontology has the sharpest caveat of all, and it is the one that makes curation a moat rather than a slogan. The grounding gains above assume the graph is complete and well-built. Build it carelessly and the advantage inverts: an automatically constructed, incomplete knowledge graph collapsed a retrieval pipeline to an F1 of 34.28, against 64.78 for plain vector retrieval, because only about two-thirds of the answer entities were even present. A governed, curated ontology captures the value; a sloppy auto-extracted one destroys it. That is why the discipline, not the diagram, is the product. It is also worth stating plainly that no published study measures any of this on field-service data specifically; the evidence here is drawn from the nearest rigorous analogues - clinical risk, credit default, retail demand, no-show prediction - and the transfer is an argument about architecture, never a promised number.

Two production realities close the case even where accuracy is a tie. First, calibration: a classical model’s probabilities can be wrapped in distribution-free conformal guarantees that a language model’s stated confidence cannot match, and alignment training measurably degrades an LLM’s calibration. Second, cost: for always-on scoring a gradient-boosted tree runs on a CPU and retrains nightly, while a peer-reviewed comparison found the classical model was both the cheapest and the most accurate on a text-classification task, with the largest language models using thousands of times more energy for equal or worse results. When the answers tie, the model you can afford to run and trust is the one that wins.

Build the system

The shape of the claim is now clear and clearly bounded. On structured, numeric prediction with real history - the numeric core of a field-service business - classical machine learning still wins the majority, more cheaply and with more trustworthy probabilities. Language models win the cold-start corner and the unstructured-text corner, and the celebrated exceptions on tables and time series are foundation models built for those data types, not chat models. And the durable enterprise pattern is neither model alone; it is both, arbitrated by an ontology that routes each prediction to the tool that should make it.

This is the discipline Ardenus brings to the operators who run the physical economy. Rather than point one large model at a company’s entire history and hope it predicts, Ardenus models that data into an ontology - the technicians, routes, invoices, chemicals, and customers, and the relationships between them - and lets that layer decide what each question needs: a calibrated classical model over governed history for the numeric prediction, a language model for the cold-start case, the free-text signal, and the orchestration, and grounded retrieval for the answer. We make the companion arguments - why the data foundation is the part that actually fails, why precision beats a bigger context window, and why grounding is what separates a real answer from a fluent one - in our companion research, and you can see the platform itself on the technology page. Do not buy the model that promises to replace the method. Build the system that knows which method to use.