Viewing Experiments

The upper-right corner of the Driverless AI UI includes an Experiments link.

Show Experiments link

Click this link to open the Experiments page. From this page, you can rename an experiment, view previous experiments, begin a new experiment, rerun an experiment, and delete an experiment.

View experiments

Checkpointing and Rerunning

In Driverless AI, you can retry an experiment from the last checkpoint, or you can run a new experiment using an existing experiments settings.

Rerun experiment

Checkpointing Experiments

In real-world scenarios, data can change. For example, you may have a model currently in production that was built using 1 million records. At a later date, you may receive several hundred thousand more records. Rather than building a new model from scratch, Driverless AI includes H2O.ai Brain, which enables caching and smart re-use of prior models to generate features for new models.

You can configure one of the following Brain levels in the experiment’s Expert Settings.

  • Level 0: Don’t use any brain cache (smart checkpointing).
  • Level 1: Perform smart checkpointing if you selected Restart from Last Checkpoint at the start of an experiment. In the GUI, an experiment is by default resumed in experiment settings unless resumed experiment is set to None.
  • Level 2: Perform smart checkpointing on a new experiment if the new experiment identically matches (default).

If you chooses Level 2 (default), then Level 1 is also done when appropriate.

To make use of smart checkpointing, be sure that the new data has:

  • The same data column names as the old experiment
  • The same data types for each column as the old experiment. (This won’t match if, e.g,. a column was all int and then had one string row.)
  • The same target as the old experiment
  • The same target classes (if classification) as the old experiment
  • For time series, all choices for intervals and gaps must be the same

When the above conditions are met, then you can:

  • Start the same kind of experiment, just rerun for longer.
  • Use a smaller or larger data set (i.e. fewer or more rows).
  • Effectively do a final ensemble re-fit by varying the data rows and starting an experiment with a new accuracy, time=1, and interpretability. Check the experiment preview for what the ensemble will be.
  • Restart/Resume a cancelled, aborted, or completed experiment

To run smart checkpointing on an existing experiment, click the right side of the experiment that you want to retry, then select Restart from Last Checkpoint. The experiment settings page opens. Specify the new dataset. If desired, you can also change experiment settings, though the target column must be the same. Click Launch Experiment to resume the experiment from the last checkpoint and build a new experiment.

The smart checkpointing continues by adding a prior model as another model used during tuning. If that prior model is better (which is likely if it was run for more iterations), then that smart checkpoint model will be used during feature evolution iterations and final ensemble.

Notes:

  • Driverless AI does not guarantee exact continuation; only smart continuation from any last point.
  • The directory where the H2O.ai Brain meta model files are stored is tmp/H2O.ai_brain. In addition, the default maximum brain size is 20GB. Both the directory and the maximum size can be changed in the config.toml file.

Rerunning Experiments

To run a new experiment using an existing experiment’s settings, click the right side of the experiment that you want to use as the basis for the new experiment, then select New Model with Same Params. This opens the experiment settings page. From this page, you can rerun the experiment using the original settings, or you can specify to use new data and/or specify different experiment settings. Click Launch Experiment to create a new experiment with the same options.

Deleting Experiments

To delete an experiment, hover over the experiment that you want to delete. An “X” option displays. Click this to delete the experiment. A confirmation message will display asking you to confirm the delete. Click OK to delete the experiment or Cancel to return to the experiments page without deleting.

Delete experiment