☂️Introduction

Monitor what matters, find what is broken, and fix it.

What is NannyML Cloud?

NannyML Cloud is a machine learning monitoring platform to monitor, analyze, and improve ML models in production.

To monitor today’s and tomorrow’s ML models properly, we need solutions that frame the monitoring problem better, solutions that have full coverage and that monitor the model, not just its inputs. Because of this, NannyML Cloud focuses on model performance estimation and takes a performance-centric monitoring approach. Our monitoring approach goes as follows.

The NannyML way

1. Monitor what matters

Focus on one single metric to know how your model is performing and get alerted when the performance drops.

Estimate model performance: Know the performance of your ML models 24/7. NannyML estimates the performance of your ML models even if the ground truth is delayed or absent.
Measure the business impact of your models: Tie the performance of your model to monetary or business-oriented outcomes. So that you always know what your ML brings to the table.
Avoid alert fatigue: Traditional ML monitoring tends to overwhelm teams with many false alarms. By focusing on what matters, NannyML alerts are always meaningful.

2. Find what is broken

If something is broken, understand the underlying causes by correlating performance issues with data drift alerts and concept shift variations. This approach provides an actionable path for solving model performance issues.

Measure the impact of Concept Drift on model performance: Our concept shift algorithms use the latest ground-truth data to validate if the performance change has been due to a change of concept.
Uncover the most subtle changes in the data structure: Leverage our multivariate drift detection method to uncover changes that univariate approaches cannot detect.
Go on a granular investigation: Apply univariate drift detection methods allow to perform a granular investigation across the model's features. And easily find which ones correlate with the performance changes.
Intelligent alert ranking: NannyML links back data drift alerts to changes in the model performance. So you can easily detect which features are causing the performance issues.

3. Fix it

Once you understand what went wrong, it becomes possible to tell whether a model is worth retraining, refactoring or if another issue resolution strategy is needed.

Retrain only when it is necessary: Leverage NannyML Cloud SDK to automate monitoring data ingestion and the webhooks to trigger retraining pipelines when the estimated performance drops.

How to set up NannyML Cloud?

NannyML Cloud is available on both the Azure and AWS marketplaces, and you can deploy it in two different ways, depending on your needs.

Managed Application: With the Managed application, no data will leave your environment. This option will provision the NannyML Cloud components and the required infrastructure within your own Azure or AWS subscription. To learn more about this, check out the docs on how to set up NannyML Cloud on Azure and AWS.
Software-as-a-Service (SaaS): With the SaaS option, you send us the monitoring data, and the monitoring happens in our own infrastructure. To learn more about this, check out the docs on how to set up NannyML Cloud on Azure and AWS.

NannyML OSS vs NannyML Cloud

NannyML Cloud, built on our open-source NannyML library, adds new features and algorithms into an all-in-one monitoring platform.

Feature	NannyML OSS	NannyML Cloud
Performance estimation - CBPE and DLE	✅	✅
M-CBPE - 10% better performance estimations than CBPE	❌	✅
Concept shift detection	❌	✅
Covariate shift detection	✅	✅
Data quality checks	✅	✅
Intelligent alerting	✅	✅
Interactive visualizations	✅	✅
Slack and email notifications	❌	✅
Customizable dashboards	❌	✅
Programmatic data collection	❌	✅
Metric storage	❌	✅
Scheduling monitoring runs	❌	✅

Feature

NannyML OSS

NannyML Cloud

Performance estimation - CBPE and DLE

✅

M-CBPE - 10% better performance estimations than CBPE

❌

✅

Concept shift detection

❌

✅

Covariate shift detection