Taking Machine Learning Models to Production
Most machine learning models never make it to production. This guide addresses the engineering, organizational, and operational disciplines required to close the gap between promising model performance and business value at scale.
The Production Gap in Machine Learning
Industry surveys consistently find that 80 to 90 percent of machine learning models developed by enterprise data science teams never reach production. The reasons span technical, organizational, and cultural dimensions. On the technical side, models developed in Jupyter notebooks against cleaned, static datasets encounter a fundamentally different environment in production: data pipelines that introduce latency, schema drift, distribution shifts, and infrastructure that was designed for conventional software rather than stateful, probabilistic artifacts. Organizationally, the gap between data science teams and software engineering teams creates friction at the handoff. Data scientists optimize for model accuracy; engineers optimize for reliability, latency, and maintainability. These objectives conflict when a high-accuracy model requires ten seconds of inference time, depends on a fragile feature pipeline, or cannot be updated without retraining from scratch. Without explicit ownership of the production system, models atrophy: their accuracy degrades as data distributions shift while no one monitors or retrains them. The companies that consistently close the production gap share three characteristics: they treat ML model deployment as a software engineering problem with the same rigor applied to any production service; they invest in ML infrastructure before attempting to scale the model portfolio; and they establish clear ownership of model performance in production as an operational metric, not a research outcome.
MLOps: The Engineering Discipline Behind Production ML
MLOps — the application of DevOps principles to machine learning systems — has emerged as the operational framework for closing the production gap. At its core, MLOps encompasses four disciplines: data versioning and pipeline management, experiment tracking and model registry, model serving and inference infrastructure, and monitoring and observability. Data versioning tools like DVC and Delta Lake enable reproducibility by linking model versions to the exact training data and parameters used to produce them. This reproducibility is essential for debugging production issues, auditing model behavior for regulatory purposes, and retraining models when performance degrades. Without data versioning, teams waste weeks attempting to recreate conditions that produced a model artifact, often finding it is impossible. Model registries — available natively in MLflow, Amazon SageMaker, and Databricks — provide a catalog of model versions with associated metadata: training metrics, validation performance, serving infrastructure requirements, and promotion status. The registry becomes the handoff mechanism between data science and engineering teams, establishing a formal interface that replaces informal Slack messages and shared Google Drive folders as the method for moving models toward production.
Inference Architecture and Serving Patterns
Model serving architecture must be designed around the specific latency, throughput, and availability requirements of the application. Real-time inference for a fraud detection model processing payment transactions must respond in under 50 milliseconds with five-nines availability. Batch inference for a next-day demand forecasting model can tolerate hour-level latency and operate as a scheduled job. These requirements demand fundamentally different infrastructure patterns. For real-time inference, managed serving platforms — AWS SageMaker, Google Vertex AI, Azure ML, and open-source alternatives like Seldon Core and BentoML — provide auto-scaling inference endpoints with model version management. Containerizing models with Docker and deploying them behind API gateways provides portability and allows inference infrastructure to be managed with the same Kubernetes-based tooling used for conventional microservices. Feature stores — Feast, Tecton, Hopsworks — address the critical problem of feature consistency between training and serving environments, which is responsible for a disproportionate share of production model failures. Large language model inference introduces additional complexity. LLM serving requires GPU infrastructure, careful attention to model quantization trade-offs between cost and quality, and prompt caching strategies to manage inference costs at scale. Companies serving LLM-powered features to thousands of concurrent users should evaluate vLLM, TensorRT-LLM, or managed inference APIs from Anthropic, OpenAI, and Google before building custom serving infrastructure.
Model Monitoring and the Drift Problem
A model that performs well at launch will degrade over time as the real-world distribution of inputs diverges from the training distribution. This phenomenon — data drift or concept drift — is the most insidious production ML problem because it is invisible without explicit monitoring. A fraud detection model trained on pre-pandemic transaction patterns will underperform when consumer behavior shifts. A churn prediction model trained on a homogeneous customer cohort will produce systematically biased predictions when the customer base expands into new segments. Production ML monitoring must track three categories of signals: infrastructure metrics (latency, error rates, throughput), data quality metrics (schema violations, null rates, value distribution shifts), and model performance metrics (prediction accuracy, calibration, fairness across demographic segments). Infrastructure metrics are monitored with conventional observability tools. Data and model metrics require ML-specific tooling like Evidently AI, WhyLabs, or Arize AI, which provide statistical drift detection and alert when model behavior deviates from baseline. Retraining strategies should be defined before a model reaches production, not as a response to observed degradation. Scheduled retraining — weekly, monthly — provides a floor of freshness for models with slowly shifting distributions. Triggered retraining — initiated when drift monitors breach defined thresholds — is appropriate for models in rapidly changing environments. Champion-challenger frameworks allow candidate models to handle a percentage of production traffic before full promotion, reducing the risk of retraining introducing regressions.
Organizational Structures That Enable Production ML
The organizational structure around ML engineering matters as much as the technology. The most effective model is an embedded ML engineer within each product team, supported by a central ML platform team that owns shared infrastructure: feature store, experiment tracking, model registry, and serving infrastructure. This avoids the failure mode of a centralized data science team that produces models but cannot deploy them because engineering resources are controlled by product teams with competing priorities. ML product ownership — assigning a product manager or technical program manager accountable for model outcomes in production — is underutilized but highly effective. When model accuracy and business outcome metrics are owned by a named individual who attends sprint reviews and can escalate infrastructure investments, production quality improves measurably. Data science teams without this ownership structure tend to optimize for model performance metrics that are disconnected from the business outcomes they are meant to predict. For CTOs evaluating whether to build ML infrastructure in-house or leverage managed cloud platforms, the calculus has shifted decisively toward managed platforms for most mid-market companies. The engineering cost of maintaining a custom ML platform is substantial, and cloud platform capabilities have matured to the point where the flexibility of custom infrastructure rarely justifies the overhead. The exception is companies with very large model portfolios, strict data residency requirements, or unique serving requirements that managed platforms cannot accommodate.
Frequently Asked Questions
What is the most common reason ML models fail in production?
Feature pipeline failures — where the data transformations applied during serving diverge from those applied during training — are the single most common cause of ML model failures in production. Feature stores address this by providing a consistent feature computation layer used by both training and serving systems.
How do we know when to retrain a model?
Retrain when monitored performance metrics — accuracy, precision, recall, or business outcome metrics like conversion rate — degrade beyond a defined threshold. Statistical drift detection tools can identify input distribution shifts that typically precede performance degradation, enabling proactive retraining before customers are affected.
Should a startup build its own ML platform?
No. Startups should leverage managed platforms — SageMaker, Vertex AI, Databricks, or Weights and Biases — rather than building custom ML infrastructure. Custom platform development consumes engineering resources that create more business value applied to product features and model development. Revisit this decision at significant scale.
What team structure supports production ML?
Embedding ML engineers within product teams, supported by a central platform team that owns shared infrastructure, produces the best outcomes. This structure ensures ML capabilities are integrated into product development rather than managed as a separate research function with limited production impact.
Related Articles
What a Fractional CTO Actually Does
Most companies hire a fractional CTO expecting a part-time employee. What they get — when they get the right person — is an operating partner who reshapes how technology creates value across the enterprise.
Read →
Technology Due Diligence: A PE Firm's Guide
Technology due diligence has evolved from a box-checking exercise into a value-creation lever. PE firms that treat it as the former consistently overpay for assets and underperform on returns.
Read →
AI Strategy for Mid-Market Companies
Mid-market companies face a distinctive AI challenge: enough scale to benefit materially from AI adoption, but insufficient resources to build the infrastructure that makes large-enterprise AI initiatives possible. The answer is not a scaled-down enterprise strategy — it is a fundamentally different one.
Read →
The Crimson Bench · Est. 2002 · Founded in New York City
Deploy an Executive in 48 Hours
Verified corporate accounts only. Ivy League-educated. Flat-rate pricing. 14-day no-cause cancellation.
25,000+ Ivy League Executives · 150,000+ Global Consultants · 48-Hour Deployment