Most CI/CD advice is written for application code, and a lot of it maps cleanly onto ML — but not all of it. Models have a testing problem regular software doesn't: correctness isn't binary.
Start with the stages that map directly from software CI: lint, unit test the data-processing code, and build the container image. None of that is ML-specific, and skipping it because 'it's just a model' is how bugs sneak into production.
The ML-specific stage comes next: evaluating the candidate model against a held-out validation set and comparing it against the currently deployed model's metrics. A pipeline that deploys any model that trains successfully, regardless of whether it's actually better, isn't a CI/CD pipeline — it's a liability.
Gate the deployment on that comparison. If the new model doesn't beat the current one on the metrics that matter, the pipeline should fail loudly, the same way a broken unit test would.
Finally, treat rollout the same way you would for any risky software deploy: canary a small percentage of traffic to the new model, watch real-world metrics, and only then promote it fully.
Written by Aryan
Aryan spent six years building and maintaining ML infrastructure for fintech and healthcare platforms before becoming a full-time AI trainer.
View Aryan's course