Evaluation, drift and deployment

Choose your learning level

BeginnerIntermediateAdvanced

Start with plain-language explanations. Switch levels at any time.

Beginner explanation

A model needs testing on examples it did not learn from. After deployment, data can change and performance can decline. Keep a simple fallback and a person responsible for deciding when the model should stop being used.

Intermediate explanation

Use time-based splits when predicting the future, and group-based splits when related records should stay together. Monitor input changes, outcomes and errors. Data drift describes changing inputs; concept drift concerns a changing input–outcome relationship.

Advanced explanation

Avoid relying on drift statistics alone: a detectable input shift may not hurt performance, and harmful concept change may be missed. Delayed labels require interim monitoring plus later outcome audits. Compare a new model against a fixed baseline with an appropriate evaluation design.

PUT IT IN CONTEXT

A worked example

A demand model trained before a road closure overestimates sales afterward. A monitored error increase can trigger a fallback forecast. Merely retraining every night without checking data and outcomes may repeat the same problem.

TRY IT YOURSELF

Practice the idea

For a next-week sales forecast, explain why randomly mixing all dates into train and test can produce an unrealistic score.

Your working notes (not saved or sent)

Reveal the answer guide

Later records can inform patterns used to predict earlier ones, and overlapping windows may leak information. Train on earlier periods and evaluate later periods, respecting feature availability and any necessary gap.

Go further: advanced challenge

Define an operational alert using error, sample size and a review window. Explain how delayed outcomes affect the alert’s timing.

An open-ended discussion task. Explain your assumptions and evidence; there may be more than one defensible answer.

Continue the conversation

What part of this idea would you like to test or question? Compare your reasoning with the community.

Find related forum discussions ↗

Opens a topic search in the existing AstraCruz forum. Results depend on available posts; signing in may be required. No discussion or activity is fabricated.

Browse all forums