Summary: Monitoring

Table of Contents
1. Fundamentals of Model Monitoring
2. Types of Drift
3. Performance Monitoring Metrics
4. Statistical Monitoring Methods
5. Data Quality Monitoring
6. Operational Monitoring
7. Alerting and Thresholds
8. Monitoring Implementation
9. Ground Truth and Feedback Loops
10. Monitoring Dashboards and Visualization
11. Retraining Triggers and Model Updates
12. Monitoring Tools and Technologies
13. Business Impact Monitoring
14. Monitoring Documentation and Governance
View more

1. Fundamentals of Model Monitoring

• Model Monitoring: continuous tracking of model performance and behavior in production to detect issues and ensure reliability.
• Model Degradation: drop in model performance over time due to changing data, relationships, or business context.
• Production Environment: live system where models make real-time predictions that affect business decisions.
• Key objectives: detect degradation, find data quality issues, track concept/data drift, ensure prediction accuracy, monitor system health, and validate compliance.

2. Types of Drift

• Data Drift (covariate shift): changes in input feature distributions P(X); includes Feature Drift (per-feature statistical changes).
• Concept Drift: changes in P(Y|X) requiring retraining; forms: sudden, gradual, incremental, recurring.
• Prediction Drift: shifts in distribution of model predictions P(Ŷ); related: Label Drift (changes in P(Y)).

3. Performance Monitoring Metrics

• Classification Metrics: Accuracy = (TP + TN) / (TP + TN + FP + FN); Precision = TP / (TP + FP); Recall (Sensitivity) = TP / (TP + FN); F1 = 2 × (Precision × Recall) / (Precision + Recall); ROC-AUC measures discrimination; Log Loss = -Σ(y × log(ŷ) + (1-y) × log(1-ŷ)).
• Regression Metrics: MAE = Σ|yi - ŷi| / n; MSE = Σ(yi - ŷi)² / n; RMSE = √(Σ(yi - ŷi)² / n); R² = 1 - (Σ(yi - ŷi)²) / (Σ(yi - ȳ)²); MAPE = (Σ|yi - ŷi| / |yi|) / n × 100%.

4. Statistical Monitoring Methods

• Distribution tests: Kolmogorov-Smirnov (KS) for continuous distributions, Chi-Square for categorical, PSI for distribution shift (Σ(actual% - expected%) × ln(actual% / expected%)), Jensen-Shannon divergence, Wasserstein distance.
• PSI interpretation: a category for no significant change; 0.1 - 0.25 indicates moderate change (investigate); > 0.25 indicates significant change (action required).
• Statistical process control: control charts with UCL = mean + 3 × sd and LCL = mean - 3 × sd; points outside limits show special cause variation; sequential patterns suggest systematic drift.

5. Data Quality Monitoring

• Data quality dimensions: Completeness (non-null rate), Validity (formats/ranges), Consistency (agreement across fields), Timeliness (freshness), Accuracy (correctness vs ground truth).
• Checks: null value rate, out-of-range values, cardinality changes, schema validation, referential integrity, duplicate detection.
• Feature statistics to monitor: mean/median, standard deviation, min/max, percentiles (25th, 75th), skewness, kurtosis.

6. Operational Monitoring

• System metrics: Latency (request→response time), Throughput (predictions per time), Error Rate (failed requests %), Resource Utilization (CPU, memory, disk), Availability = uptime / (uptime + downtime).
• SLIs: P50, P95, P99 latencies; request success rate; service availability over a time window.
• Model versioning: active version ID, deployment timestamp and rollback history, lineage and training data version, config/hyperparameters, A/B comparison results.

7. Alerting and Thresholds

• Alert types: Threshold (metric exceeds static value), Anomaly (statistical deviation), Trend (sustained directional change), Composite (multiple conditions).
• Threshold strategies: fixed, dynamic, percentile-based, moving average, seasonal baselines.
• Prioritization: Critical (immediate business impact), High (action within hours), Medium (investigate within day), Low (minor anomaly; monitor).

8. Monitoring Implementation

• Architecture components: Data Collection Layer (predictions, features, actuals, metadata), Storage Layer (time-series DB or warehouse), Computation Layer (metrics, stats, drift measures), Visualization Layer (dashboards), Alerting Layer (notifications).
• Logging best practices: log inputs/outputs/timestamps, model version and config, feature values at prediction, ground truth when available, unique request IDs, system errors with stack traces.
• Monitoring frequency: real-time performance - continuous or sub-minute; data quality - hourly/daily; distribution drift - daily/weekly; model performance - weekly/monthly if labels delayed.

9. Ground Truth and Feedback Loops

• Ground truth collection: direct observation, delayed labels, human annotation, implicit feedback, proxy metrics.
• Feedback challenges: label delay, label bias, sampling bias, feedback loops (predictions influence future data), missing labels.
• Handling delayed feedback: use proxy metrics, monitor prediction confidence, track relative prediction changes, sliding window evaluation as labels arrive, set baseline expectations.

10. Monitoring Dashboards and Visualization

• Essential components: performance trend charts, distribution comparisons, feature statistics tables, alert status panel, prediction volume chart, error rate graph.
• Best practices: use consistent time ranges, include threshold/reference lines, color-code by severity, provide drill-down, show confidence intervals, display absolute and relative changes.

11. Retraining Triggers and Model Updates

• Retraining triggers: performance degradation below thresholds, significant data drift by tests, concept drift, scheduled retraining, sufficient new labeled data.
• Strategies: periodic retraining, performance-based, drift-based, online learning (continuous incremental), hybrid (scheduled + event-driven).
• Model update workflow: detect trigger → collect/validate new data → train candidate → validate on holdout → compare with current model → deploy if improved → monitor initial deployment → retain rollback capability.

12. Monitoring Tools and Technologies

• Open source: Prometheus (time-series metrics & alerting), Grafana (visualization/dashboards), Evidently (drift & performance monitoring), Alibi Detect (outlier & drift detection), Great Expectations (data quality validation).
• Cloud services: AWS (SageMaker Model Monitor, CloudWatch), Google Cloud (Vertex AI Model Monitoring, Cloud Monitoring), Azure (Azure ML Model Monitoring, Application Insights).
• MLOps platforms: MLflow (tracking & registry with monitoring integration), Weights & Biases (performance tracking & visualization), Neptune.ai (metadata & monitoring), Kubeflow (Kubernetes ML workflows), DataRobot (automated monitoring & drift detection).

13. Business Impact Monitoring

• Business metrics: revenue impact, cost savings, conversion rate, customer satisfaction, false positive cost, false negative cost.
• ROI monitoring: compare outcomes with/without model, track cost per prediction and maintenance, measure KPI impact, calculate expected value of decisions, monitor customer lifetime value changes.
• Fairness metrics: Demographic Parity (equal prediction rates across groups), Equal Opportunity (equal true positive rates), Disparate Impact (ratio of positive rates), Group Calibration (prediction probabilities match outcomes within groups).

14. Monitoring Documentation and Governance

• Documentation: model card (purpose, performance, limits, ethics), monitoring plan (metrics, thresholds, alerts), baseline statistics, SLA definitions, incident response procedures, retraining protocols.
• Audit trail: prediction logs (inputs, predictions, timestamps, model versions), performance history (metrics over time), deployment records (versions, rollbacks, approvals), data lineage (sources, transforms, versions), alert history (triggers, responses, resolution).
• Compliance: track regulatory requirements, data privacy/security validation, model explainability checks, bias/fairness reports, access control audits, data retention/deletion policy compliance.