When the Problem Is Simulation, Not Prediction

A forecast is treated as good if its average error is small. This sounds like a complete score. It isn't, once the cost sits in a handful of weeks the average cannot see.

There is a category of problem that gets a predictive model because a predictive model is what the field knows how to ship. The fit is good enough on the mean that the mismatch never becomes a ticket. You're right on the routine weeks and wrong on the weeks that move inventory, cash, or capacity. The target is emergent, so a stable input-output map is the wrong architecture.

What prediction and simulation are

A predictive model learns a function from inputs to outputs. Given enough (input, output) pairs, it fits a mapping and hopes the mapping holds on the next batch. The assumption underneath is that the relationship is stable, that it does not change in a way the model cannot see coming. Demand forecasting as usually practiced is this job: historical sales plus a few context signals in, a future sales number out.

Simulation is a different object. You specify components and the rules they follow, let them interact, and watch what the system produces. Aggregate demand is not estimated; it falls out of the run. You do not fit "demand." You run the agents whose collective behavior is demand.

The thing you are scoring is either a stable function of observables or an emergent property of components that condition on each other.

Where the mismatch shows up

Demand is the case I've worked in directly. It isn't produced by one consumer making an isolated choice. It is produced by thousands of consumers whose choices move with what peers are buying, what media is saying, what retailers are stocking, and what competitors are pricing. Those interactions produce panic buying, trend adoption, sentiment cascades. A map from last year's inputs to next week's units does not contain them.

Predictive models handle this with proxies: consumer sentiment indices, social media engagement metrics, news event flags. Those variables measure the aggregate state of the system. They don't model the system. When the social dynamics shift in a way that is not in the training distribution — a new kind of viral event, a policy change that creates unusual purchasing, a cascade from a novel combination — the proxies miss the shift and the model fails.

The same pattern shows up in financial market forecasting, social network behavior, epidemiological modeling, and supply chain disruption. In each case the target is an emergent property of interacting agents. Historical correlations hold while the system repeats itself. They fail when the system does something structurally new.

What the average hides

Predictive models have a specific failure mode here. They are right on average, which is enough to look like they are working, and they are wrong on the weeks that decide the inventory.

The model nails the routine weeks and misses the disruptions. MAPE looks acceptable at 4%, 5%, sometimes 3%, because most weeks are routine. The weeks that drive the buy are the disruption weeks. Those are the weeks where emergent social dynamics produce demand that is not in the training distribution.

The average hides the split. If 48 weeks are routine and 4 weeks are disruptions, and the model is fine on the 48 and bad on the 4, MAPE still looks fine — and the cost of those 4 weeks, excess inventory, missed sales, supply chain scrambles, is not in that average.

Those MAPE figures and the 48-versus-4 split are a shape, not a named run. A different portfolio will have a different split. The claim I'm making is the weighting, not a particular week's error.

What simulation adds

Simulation does not extrapolate the last pattern. It derives the next aggregate from components modeled one level down, individual agents rather than a fitted curve on the total.

Emergent behavior then falls out of the run. If you model thousands of consumers whose rules include "buying is influenced by what peers are buying," sentiment cascades appear without being written as a feature. If you inject a shock, a tariff announcement or a viral post, you can watch the system respond, because the response is in the rules, not in a mapping that had to have seen this shock before.

The tax is real. You have to specify component rules, and those rules are not sitting in a table. You also have to validate. A simulation is useful only to the extent that the aggregates it produces match the aggregates of the real system, which means comparing simulation output to observed aggregate behavior.

That check is stricter than plausibility. A useful simulation should reproduce known aggregate patterns, fail in interpretable ways when its assumptions are wrong, and respond to counterfactual shocks without collapsing. If a demand simulation cannot reproduce past panic-buying when given the conditions that produced it, it has not earned the next disruption. Plausibility is cheap. Calibration against observed system behavior is the expensive part.

A richer feature set, or a regime-switching predictor, can imitate some of this without standing up agents. If the new regime is already in the covariates, you do not need the tax. The substitution is only worth it when the object is the interaction, not a better encoding of the last aggregate.

Statistical failure and structural failure

Statistical failures such as noise, bias, or poor generalization to a subgroup are data, architecture, and regularization problems; they stay inside the predictive approach.

Structural failures come from interaction. The model fails because the system is doing something it has not done, and the cause is interacting components. More rows and a better architecture do not address that. You model the system that generates the observations, not the observations as a fixed map.

Most applied ML projects skip this diagnosis. They start with a predictor because that is what the pipeline ships; they treat the structural misses as a data problem. You get incremental MAPE movement, and you never touch the weeks that cost money.

The reverse mistake happens too. Some problems are prediction problems, and wrapping them in an agent-based simulation adds machinery without adding a mechanism. If the system is stable, the causal structure is not changing, and the failures are noise, bias, or sparse data, a simulation is an expensive way to get a worse forecast. Stable systems with those failures get worse when you wrap them in agents.

The disruptions in demand are predictable in kind; they happen when social dynamics dominate. I'd price the 4 weeks before you stand up agents. If those weeks are rare and cheap, keep the predictor. If they are the scenarios that drive the highest-value decisions, build the simulation.