Training-Serving Skew

Training-serving skew is a subtle bug where the data a model receives in production differs from the data it was trained on — not because the world changed, but because the pipelines diverged. Maybe training used a cleaned, batch-computed feature while serving computes it live with slightly different code, or a value that was fresh in training arrives stale at inference. The model then performs far worse in production than your offline metrics promised. It is one of the classic MLOps failure modes precisely because it is silent: every test passes, yet real accuracy sags. The usual culprit is duplicated transformation logic — one copy for training, one for serving — that drifts apart over time. This is a big reason feature stores exist, since they let both paths share exactly one definition. Practical note: log the actual features seen at serving time and periodically diff their distribution against the training set; a mismatch is your early warning that skew has crept in.