How we build predictive scores: A Data Scientist’s methodology

What does it actually look like to build a predictive scoring model from the ground up?

The honest answer is that it looks different at every stage, and the work that happens before you write a single line of model code is often the most consequential. Having spent the better part of my career building and refining data science and machine learning models, I want to walk through how we approach this at Company Watch, because the methodology matters as much as the technology.

Step 1: Define the goal with precision

The first question on any scoring project is not “what data do we have?” and it is not “what model should we use?” It is: what problem are we actually trying to solve? Everything else follows from that; what you are trying to predict, and what counts as success, both fall out of a clear answer to that first question. 

This may sound obvious. In practice, it is the step most likely to be rushed, and the one that causes the most downstream problems when it is. A vague problem definition produces a vague model, and a vague model tends to perform poorly on the metrics that actually matter to the business.

At Company Watch, before any technical work begins, we spend significant time establishing a precise problem statement and the success metrics we will use to evaluate it. Those metrics become the guiding reference point for every subsequent decision in the project. When there is ambiguity later, which there always is, you return to the problem definition to resolve it.

The Company Watch data team.

Step 2: Assess the data honestly

Once the goal is clearly defined, the next step is a thorough assessment of the available data. This means looking at two dimensions: quality and variety.

Quality assessment involves understanding the completeness, consistency, and accuracy of each data source. Missing values, inconsistent formatting, and coverage gaps all affect model performance in ways that are difficult to compensate for at the modelling stage. It is considerably more efficient to understand these limitations early and address them through data engineering than to discover them once the model is already built.

Variety refers to the breadth of signals available. A richer feature set generally produces a more predictive model, provided the features are genuinely informative and not simply adding noise. The practical question is whether the data available is sufficient to address the problem as defined, and if not, whether additional data sources can be acquired or constructed.

This is an area where Company Watch has a structural advantage.

Step 3: Choose the right model architecture

Model selection involves a genuine trade-off between three competing constraints: predictive accuracy, interpretability, and runtime performance.

A highly expressive model architecture will generally produce better predictions than a simpler one, but it will also be harder to explain to end users and may be slower to run at scale. A simpler, more interpretable model is easier to explain and faster to deploy, but may leave predictive performance on the table. The right choice depends on the specific requirements of the score and the business context it will operate in.

For scores where interpretability is critical and the feature set is relatively small, a regression-based approach may be entirely appropriate. For scores that need to incorporate a large number of heterogeneous features and capture non-linear relationships between them, a more expressive framework such as gradient boosted decision trees will typically outperform. There is no universally correct answer, and treating model selection as a methodological decision rather than a default produces better outcomes.

Step 4: Build a proof of concept quickly

The fourth step, and one I consider underrated, is building a small proof of concept as early as possible. Not a production model, not a fully optimised pipeline, but a working prototype that can be evaluated against real data and shown to stakeholders.

The value of this is twofold. First, it surfaces assumptions that look reasonable on paper but do not hold up in practice. Second, it generates early evidence that the approach is viable, which is important for maintaining alignment and momentum on longer projects. A proof of concept built in a few weeks is a far more persuasive argument than a detailed plan.

Why this matters

This methodology, defining the goal clearly, assessing data honestly, selecting the model architecture deliberately, and validating early through proof of concept, is not specific to any single project. It is the foundation of how good scoring work gets done.

The scores that perform well over time are not the ones built with the most sophisticated technology. They are the ones built on the clearest thinking. The technology is important, but it is downstream of the methodology.

We have been applying this approach to a project that I think represents the most rigorous implementation of it we have done to date. More on that very soon.