Incremental ML Model Training

Beginner | 10 Minutes

Artificial Intelligence Data Fabric

Overview

In Machine Learning, initial model training usually requires large data sets to train on to ensure reliable predictive models. Training is naturally intense on resource consumption and can be very time-consuming. In addition, a model's life cycle does not conclude with its initial creation -- it must be regularly retrained with the latest data to stay accurate and responsive to evolving environments. Incremental training allows users to efficiently update existing machine learning models with new data, enhancing performance without retraining from scratch -- significantly reducing computational resources and time required to retrain.

Tutorial

1

Save a custom algorithm

A saved custom algorithm is a requirement for incremental training. In an already existing quest, go to the "Train Model with Custom Algorithm" block, and make sure you have checked "Save Model." Once the quest has been executed, it will be stored in the Models Library.

2

Create the incremental dataset

Create a dataset in the Data Collection > Datasets section of the platform containing only the new and incremental data. For this activity, you are likely creating a new dataset from the data lake.

3

Update your quest data source

Go back to your quest and change the source of the data to the new set of incremental data.

4

Use the Input Model activity

Add an Input Model block to the Quest. In the sidebar, select the saved model to be used, and feed the activity into the train custom algorithm block.

5

Update the saved model (or don't)

You may or may not want to save the new model after the retraining. Return to the "Train Model with Custom Algorithm" block and verify the "Save Model" box is checked if you'd like the new model saved or not checked if you do not wish to save new versions of the model.

6

Run the training quest

Save and run the quest. Once the execution is finished, your machine-learning model will be updated to include the new data!

1

Save a custom algorithm

A saved custom algorithm is a requirement for incremental training. In an already existing quest, go to the "Train Model with Custom Algorithm" block, and make sure you have checked "Save Model." Once the quest has been executed, it will be stored in the Models Library.

2

Create the incremental dataset

Create a dataset in the Data Collection > Datasets section of the platform containing only the new and incremental data. For this activity, you are likely creating a new dataset from the data lake.

3

Update your quest data source

Go back to your quest and change the source of the data to the new set of incremental data.

4

Use the Input Model activity

Add an Input Model block to the Quest. In the sidebar, select the saved model to be used, and feed the activity into the train custom algorithm block.

5

Update the saved model (or don't)

You may or may not want to save the new model after the retraining. Return to the "Train Model with Custom Algorithm" block and verify the "Save Model" box is checked if you'd like the new model saved or not checked if you do not wish to save new versions of the model.

6

Run the training quest

Save and run the quest. Once the execution is finished, your machine-learning model will be updated to include the new data!