Adding a model

Two methods are available to add a model to NannyML Cloud. The optimal way is to do it programmatically, which can be found on the NannyML SDK Cloud page. Alternatively, you can manually add the model using the NannyML Cloud UI, which we'll explain here.
If you prefer a video walkthrough and you upload your data from Azure, here's our YouTube guide:
Azure
AWS

1. Press "Add model" in the navigation bar

Add model button in the navigation bar.

2. Provide model information

New model setup page.
You need to provide four pieces of information about your model:
  1. 1.
    The machine learning problem type The type of problem the machine learning model is dealing with. The different options are binary classification, multiclass classification, and regression. This has lots of implications. It impacts what type of model output and target data NannyML is expecting and which metrics NannyML can calculate. This cannot be changed later.
  2. 2.
    The main performance metric Depending on which problem type you select, the available metrics will change. These metrics can always be changed later, and you can monitor multiple metrics simultaneously. Currently, we support the following metrics:
Binary classification
Multi-class classification
Regression
  • ROC-AUC
  • F1
  • Precision
  • Recall
  • Specificity
  • Accuracy
  • Business value
  • Confusion matrix elements
  • ROC-AUC
  • F1
  • Precision
  • Recall
  • Specificity
  • Accuracy
  • Business value
  • MAE
  • MAPE
  • MSE
  • RMSE
  • RMSLE
  • MSLE
  1. 3.
    How the data has to be chunked The time interval over which the metrics have to be aggregated, i.e., the granularity of the monitoring analysis. The options are daily, weekly, monthly, quarterly, and yearly. The chunk period can always be changed later in the model settings.
Currently, we only support time-based chunking; if you need support for size or number-based chunking, contact us.

3. Configure the reference dataset

The reference dataset is the dataset NannyML will use as a baseline for monitoring your model. This dataset ideally represents a time when the model worked as expected. The ideal candidate for this is the test set. You need to point NannyML to where this dataset is located and provide some basic information about the dataset schema.
  1. 1.
    Point nannyML to the reference dataset location
Reference dataset setup page.
Pick one of the following upload options:
Public URL
Azure blob storage
AWS S3
Local file
Database connection
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
Reference dataset configuration page for public URL upload.
To try out NannyML, use one of our public datasets on GitHub. Here is a link to the synthetic car price prediction - reference dataset:
Provide Azure blob storage location
There are six fields on the configuration page:
Reference dataset configuration with Azure.
The first three fields are mandatory and related to the location of the dataset:
  1. 1.
    Azure Account Name
  2. 2.
    Blob storage container
  3. 3.
    File path
The easiest way to obtain the right values for the respective fields is by going to the Azure storage browser via the Azure portal:
The values for the first three fields can be derived as follows:
The last three fields provide ways of accessing/authenticating the blob storage. Only one of them has to be provided:
  • If "Is public*" is enabled, NannyML will try to connect without credentials (only possible if the account is configured to allow for public access)
  • The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. Link to the Microsoft docs.
  • The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to be specifically created when doing the onboarding. Link to the Microsoft docs.
Provide AWS S3 storage location
In this configuration, there are just two fields: one for the URI to your S3 bucket and another for the authentication mode, with options including anonymous, integrated, and access key.
Reference dataset configuration with AWS.
Upload via local file system
If you have your dataset downloaded to your computer and it is smaller than 100 MB, you can upload it directly to NannyML Cloud.
Local file configuration page.
Not currently available in the product. For custom support, please contact us.
  1. 2.
    Provide reference dataset information
NannyML requires schema information about the reference dataset. While it automatically gets column details, it's always wise to double-check. The most critical columns to define are listed on the left. The columns you need to specify depend on the type of machine-learning problem you chose at the beginning of this workflow. All other columns are automatically treated as features. Additionally, NannyML automatically detects the data types of these feature columns.
Regression
Binary classification
Multi-class classification
Reference dataset configuration for the regression problem.
The following columns have to be specified:
  • Timestamp This provides NannyML with the date and time that the prediction was made.
  • Prediction The model output that the model predicted for its target outcome.
  • Target The ground truth or actual outcome of what the model is predicting.
  • Identifier A unique identifier for each row in the dataset. NannyML will use this column to join analysis and target data sources.
The mapping of the columns can be changed when scrolling horizontally. It is possible to ignore specific columns or flag columns that should be used for joining predictions and targets later.
Reference dataset configuration for the binary classification problem.
The following columns have to be specified:
  • Timestamp This provides NannyML with the date and time that the prediction was made.
  • Prediction The model output that the model predicts for its target outcome.
  • Prediction score The model output scores or probabilities that the model predicts for its target outcome.
  • Target The ground truth or actual outcome of what the model is predicting.
  • Identifier A unique identifier for each row in the dataset. NannyML will use this column to join analysis and target data sources.
The mapping of the columns can be changed when scrolling horizontally. It is possible to ignore specific columns or flag columns that should be used for joining predictions and targets later.
Reference dataset configuration for the multi-class classification problem.
The following columns have to be specified:
  • Timestamp This provides NannyML with the date and time that the prediction was made.
  • Prediction The model output that the model predicts for its target outcome.
  • Target The ground truth or actual outcome of what the model is predicting.
  • Identifier A unique identifier for each row in the dataset. NannyML will use this column to join analysis and target data sources.
The mapping of the columns can be changed when scrolling horizontally. It is possible to ignore specific columns or flag columns that should be used for joining predictions and targets later.
Since the problem is multiclass, we also need to flag the prediction score column of each class as a prediction score.
After the columns that were flagged as "prediction score," you need to map the classes that those scores belong to:
Prediction classes name definition for the multi-class classification problem.

4. Configure the analysis dataset

The analysis dataset is what NannyML uses to analyze the performance of the monitored model. Typically, it will consist of the latest production data up to a desired point in the past, which should be after the reference dataset ends.
Note: NannyML assumes that the schema of the analysis dataset is the same as the reference dataset.
  1. 1.
    Point nannyML to the analysis dataset location
Analysis dataset setup page.
Pick one of the following options:
Public URL
Azure Blob Storage
AWS S3
Local file
Database connection
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
Analysis dataset setup with public URL.
To try out NannyML, you can use one of our public datasets on GitHub. Here is a link to the synthetic car price prediction-analysis dataset:
Provide Azure blob storage location
There are six fields on the configuration page. If you have also used Azure blob storage for the reference dataset, the relevant fields will already be filled in, and only the file path has to be provided, assuming the analysis dataset is stored in the same Blob storage container:
Analysis dataset setup with Azure Blob.
The first three fields are mandatory and related to the location of the dataset:
  1. 1.
    Azure Account Name
  2. 2.
    Blob storage container
  3. 3.
    File path
The easiest way to obtain the right values for the respective fields is by going to the Azure storage browser via the Azure portal:
The values for the first three fields can be derived as follows:
The last three fields provide ways of accessing/authenticating the blob storage. Only one of them has to be provided:
  • If "Is public*" is enabled, NannyML will try to connect without credentials (only possible if the account is configured to allow for public access)
  • The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. Link to the Microsoft docs.
  • The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to be specifically created when doing the onboarding. Link to the Microsoft docs.
Provide AWS S3 storage location
In this configuration, there are just two fields: one for the URI to your S3 bucket and another for the authentication mode, with options including anonymous, integrated, and access key.
Analysis dataset setup with Amazon S3 Storage.
Upload via local file system
If you have your dataset downloaded to your computer and it is smaller than 100 MB, you can upload it directly to nannyML Cloud.
Analysis dataset setup with local dataset file.
Not currently available in the product. For custom support, please contact us.

5. Configure the target dataset

This step is only necessary when targets are not part of the analysis dataset and when they are available.
Note: NannyML expects the identifier and target column to be present in the target dataset.
  1. 1.
    Point nannyML to the target dataset location
Pick one of the following options:
Public URL
Azure Blob Storage
AWS S3
Local file
Database connection
Provide a public URL
If the dataset is accessible via a public URL, you can provide that link here:
Target dataset setup with public URL.
To try out NannyML, you can use one of our public datasets on Git Hub. Here is a link to the synthetic car price prediction-analysis target dataset:
Provide Azure blob storage location
There are six fields on the configuration page. If you have also used Azure blob storage before as part of reference or analysis configuration, the relevant fields will already be filled in, and only the file path has to be provided, assuming the target dataset is stored in the same Blob storage container:
Target dataset setup with Azure Blob.
The first three fields are mandatory and related to the location of the dataset:
  1. 1.
    Azure Account Name
  2. 2.
    Blob storage container
  3. 3.
    File path
The easiest way to obtain the right values for the respective fields is by going to the Azure storage browser via the Azure portal:
The values for the first three fields can be derived as follows:
The last three fields provide ways of accessing/authenticating the blob storage. Only one of them has to be provided:
  • If "Is public*" is enabled, NannyML will try to connect without credentials (only possible if the account is configured to allow for public access)
  • The Account key is a secret key that gives access to all the files in the storage account. It can be found through the Azure portal. Link to the Microsoft docs.
  • The Sas Token is a temporary token that allows NannyML to impersonate the user. It has to specifically be created when doing the onboarding. Link to the Microsoft docs.
Provide AWS S3 storage location
Target dataset setup with Amazon S3 Bucket.
Upload via local file system If you have your dataset downloaded on your computer and it is smaller than 100 MB, you can upload it directly to nannyML Cloud.
Target dataset setup with local file.
Not currently available in the product. For custom support, please contact us.

6. Review model settings

Review your model settings and start monitoring! 🚀
Review model page.