Using data to extend expert judgement

People develop reliable judgement when they observe the same system many times and receive quick feedback. A daily commute provides both conditions. A regional healthcare system does not.

This difference matters when organisations use experience to make decisions about large systems. The experience may be sound, but each person can observe only part of the system. Data and models can extend that field of view.

Repetition and feedback build intuition

You can usually estimate how long a familiar journey will take. You account for roadworks, school holidays, local events and the behaviour of a particular junction. You have observed the journey many times, and you learn whether each estimate was correct on the same day.

This is a useful form of systems thinking. It develops through a small number of important variables, frequent repetition and prompt feedback.

Large systems limit direct observation

A regional healthcare system contains thousands of interacting variables across many organisations. No one observes the whole system directly. The effect of a policy decision may also take several years to appear, while other conditions continue to change.

These conditions make it difficult to learn from experience alone. A professional may have excellent judgement about their own service or department. They have less direct evidence about the way that service interacts with the wider system.

The same problem appears at smaller scales. A construction manager understands crane operations, and a traffic planner understands cyclist flows. Neither can continuously observe how a crane cycle and several hundred cyclists interact at one site. The relevant knowledge is divided between people and organisations.

Data should support expert judgement

Data does not remove judgement from a decision. Someone still chooses what to measure, how to define it and how to interpret the result. Domain experts provide this knowledge.

Data infrastructure extends what those experts can observe. It can show an epidemiologist patterns across many municipalities and years or help a site manager compare crane movements with traffic flows. The expert can then assess evidence that would be impossible to collect by direct observation alone.

This changes the purpose of a data product. It should present the part of the system that the user cannot already see, using terms and measures that match their work. A dashboard or model that produces an unexplained answer is less useful than one that shows the evidence and assumptions behind it.

Check interactions and gaps in observation

Before starting an analysis, identify whether the parts of the system respond to each other. If they do, an intervention may have effects beyond its immediate target. A model should include the interactions that could materially change the result.

Also identify who can observe each part of the system. If no person or organisation has a complete view, disagreement may come from different evidence rather than different goals. The analysis should combine those views and make the remaining gaps explicit.

Large systems already contain experienced people. They need measurements and models that allow their judgement to reach beyond the limits of direct observation.