Experiment Settings

This section includes settings that can be used to customize the experiment like total runtime, reproducibility level, pipeline building, feature brain control, adding config.toml settings and more.

Max Runtime in Minutes Before Triggering the Finish Button

Specify the maximum runtime in minutes for an experiment. This is equivalent to pushing the Finish button once half of the specified time value has elapsed. Note that the overall enforced runtime is only an approximation.

This value defaults to 1440, which is the equivalent of a 24 hour approximate overall runtime. The Finish button will be automatically selected once 12 hours have elapsed, and Driverless AI will subsequently attempt to complete the overall experiment in the remaining 12 hours. Set this value to 0 to disable this setting.

Max Runtime in Minutes Before Triggering the Abort Button

Specify the maximum runtime in minutes for an experiment before triggering the abort button. This option preserves experiment artifacts that have been generated for the summary and log zip files while continuing to generate additional artifacts. This value defaults to 10080.

Pipeline Building Recipe

Specify the Pipeline Building recipe type (overrides GUI settings). Select from the following:

  • Auto: Specifies that all models and features are automatically determined by experiment settings, config.toml settings, and the feature engineering effort. (Default)

  • Compliant: Similar to Auto except for the following:

    • Interpretability is set to 10.

    • Only uses GLM.

    • Fixed ensemble level is set to 0.

    • Feature brain level is set to 0.

    • Max feature interaction depth is set to 1.

    • Target transformers is set to ‘identity’ for regression.

    • Does not use distribution shift.

  • monotonic_gbm: Similar to Auto except for the following:

  • Kaggle: Similar to Auto except for the following:

    • Any external validation set is concatenated with the train set, with the target marked as missing.

    • The test set is concatenated with the train set, with the target marked as missing

    • Transformers that do not use the target are allowed to fit_transform across the entirety of the train, validation, and test sets.

    • Has several config.toml expert options open-up limits.

  • nlp_model: Only enable NLP models that process pure text.

  • nlp_transformer: Only enable NLP transformers that process pure text.

  • image_model: Only enable image models that process pure images (ImageAutoModel). See Automatic Image Model for more information.

    Notes:

    • This option disables the Genetic Algorithm (GA).

    • Image insights are only available when this option is selected.

  • image_transformer: Only enable image transformers that process pure images.

Kaggle Username

Optionally specify your Kaggle username to enable automatic submission and scoring of test set predictions. If this option is specified, then you must also specify a value for the Kaggle Key option. If you don’t have a Kaggle account, you can sign up at https://www.kaggle.com.

Kaggle Key

Specify your Kaggle API key to enable automatic submission and scoring of test set predictions. If this option is specified, then you must also specify a value for the Kaggle Username option. For more information on obtaining Kaggle API credentials, see https://github.com/Kaggle/kaggle-api#api-credentials.

Kaggle Submission Timeout in Seconds

Specify the Kaggle submission timeout in seconds. This value defaults to 120.

Make Python Scoring Pipeline

Specify whether to automatically build a Python Scoring Pipeline for the experiment. Select On or Auto (default) to make the Python Scoring Pipeline immediately available for download when the experiment is finished. Select Off to disable the automatic creation of the Python Scoring Pipeline.

Make MOJO Scoring Pipeline

Specify whether to automatically build a MOJO (Java) Scoring Pipeline for the experiment. Select On to make the MOJO Scoring Pipeline immediately available for download when the experiment is finished. With this option, any capabilities that prevent the creation of the pipeline are dropped. Select Off to disable the automatic creation of the MOJO Scoring Pipeline. Select Auto (default) to attempt to create the MOJO Scoring Pipeline without dropping any capabilities.

Attempt to Reduce the Size of the MOJO

Specify whether to attempt to reduce the size of the MOJO scoring pipeline when it is being built. A smaller MOJO has a smaller memory footprint during scoring. This is disabled by default.

Note: Enabling this setting can affect the overall predictive accuracy of the model because it is implemented by reducing values like interaction depth.

Measure MOJO Scoring Latency

Specify whether to measure the MOJO scoring latency at the time of MOJO creation. This is set to Auto by default. In this case, MOJO scoring latency will be measured if the pipeline.mojo file size is less than 100 MB.

Timeout in Seconds to Wait for MOJO Creation at End of Experiment

Specify the amount of time in seconds to wait for MOJO creation at the end of an experiment. If the MOJO creation process times out, a MOJO can still be made from the GUI or the R and Python clients (the timeout contraint is not applied to these). This value defaults to 1800 (30 minutes).

Number of Parallel Workers to Use During MOJO Creation

Specify the number of parallel workers to use during MOJO creation. Higher values can speed up MOJO creation but use more memory. Set this value to -1 (default) to use all physical cores.

Make Pipeline Visualization

Specify whether to create a visualization of the scoring pipeline at the end of an experiment. This is set to Auto by default. Note that the Visualize Scoring Pipeline feature is experimental and is not available for deprecated models. Visualizations are available for all newly created experiments.

Make AutoDoc

Specify whether to create the experiment AutoDoc after the experiment is finished. This is enabled by default.

Min Number of Rows Needed to Run an Experiment

Specify the minimum number of rows that a dataset must contain in order to run an experiment. This value defaults to 100.

Reproducibility Level

Specify one of the following levels of reproducibility. Note that this setting is only used when the Reproducible option is enabled in the experiment:

  • 1 = Same experiment results for same O/S, same CPU(s), and same GPU(s) (Default)

  • 2 = Same experiment results for same O/S, same CPU architecture, and same GPU architecture

  • 3 = Same experiment results for same O/S, same CPU archicture (excludes GPUs)

  • 4 = Same experiment results for same O/S (best approximation)

This value defaults to 1.

Random Seed

Specify a random seed for the experiment. When a seed is defined and the reproducible button is enabled (not by default), the algorithm will behave deterministically.

Allow Different Sets of Classes Across All Train/Validation Fold Splits

(Note: Applicable for multiclass problems only.) Specify whether to enable full cross-validation (multiple folds) during feature evolution as opposed to a single holdout split. This is enabled by default.

Max Number of Classes for Classification Problems

Specify the maximum number of classes to allow for a classification problem. A higher number of classes may make certain processes more time-consuming. Memory requirements also increase with a higher number of classes. This value defaults to 200.

Max Number of Classes to Compute ROC and Confusion Matrix for Classification Problems

Specify the maximum number of classes to use when computing the ROC and CM. When this value is exceeded, the reduction type specified by roc_reduce_type is applied. This value defaults to 200 and cannot be lower than 2.

Max Number of Classes to Show in GUI for Confusion Matrix

Specify the maximum number of classes to show in the GUI for CM, showing first max_num_classes_client_and_gui labels. This value defaults to 10, but any value beyond 6 will result in visually truncated diagnostics. Note that if this value is changed in the config.toml and the server is restarted, then this setting will only modify client-GUI launched diagnostics. To control experiment plots, this value must be changed in the expert settings panel.

ROC/CM Reduction Technique for Large Class Counts

Specify the ROC/CM reduction technique used for large class counts:

  • Rows (Default): Reduce by randomly sampling rows

  • Classes: Reduce by truncating classes to no more than the value specified by max_num_classes_compute_roc

Model/Feature Brain Level

Specify whether to use H2O.ai brain, which enables local caching and smart re-use (checkpointing) of prior experiments to generate useful features and models for new experiments. It can also be used to control checkpointing for experiments that have been paused or interrupted.

When enabled, this will use the H2O.ai brain cache if the cache file:

  • has any matching column names and types for a similar experiment type

  • has classes that match exactly

  • has class labels that match exactly

  • has basic time series choices that match

  • the interpretability of the cache is equal or lower

  • the main model (booster) is allowed by the new experiment

  • -1: Don’t use any brain cache (default)

  • 0: Don’t use any brain cache but still write to cache. Use case: Want to save the model for later use, but we want the current model to be built without any brain models.

  • 1: Smart checkpoint from the latest best individual model. Use case: Want to use the latest matching model. The match may not be precise, so use with caution.

  • 2: Smart checkpoint if the experiment matches all column names, column types, classes, class labels, and time series options identically. Use case: Driverless AI scans through the H2O.ai brain cache for the best models to restart from.

  • 3: Smart checkpoint like level #1 but for the entire population. Tune only if the brain population is of insufficient size. Note that this will re-score the entire population in a single iteration, so it appears to take longer to complete first iteration.

  • 4: Smart checkpoint like level #2 but for the entire population. Tune only if the brain population is of insufficient size. Note that this will re-score the entire population in a single iteration, so it appears to take longer to complete first iteration.

  • 5: Smart checkpoint like level #4 but will scan over the entire brain cache of populations to get the best scored individuals. Note that this can be slower due to brain cache scanning if the cache is large.

When enabled, the directory where the H2O.ai Brain meta model files are stored is H2O.ai_brain. In addition, the default maximum brain size is 20GB. Both the directory and the maximum size can be changed in the config.toml file. This value defaults to 2.

Feature Brain Save Every Which Iteration

Save feature brain iterations every iter_num % feature_brain_iterations_save_every_iteration == 0, to be able to restart/refit with which_iteration_brain >= 0. This is disabled (0) by default.

  • -1: Don’t use any brain cache.

  • 0: Don’t use any brain cache but still write to cache.

  • 1: Smart checkpoint if an old experiment_id is passed in (for example, via running “resume one like this” in the GUI).

  • 2: Smart checkpoint if the experiment matches all column names, column types, classes, class labels, and time series options identically. (default)

  • 3: Smart checkpoint like level #1 but for the entire population. Tune only if the brain population is of insufficient size.

  • 4: Smart checkpoint like level #2 but for the entire population. Tune only if the brain population is of insufficient size.

  • 5: Smart checkpoint like level #4 but will scan over the entire brain cache of populations (starting from resumed experiment if chosen) in order to get the best scored individuals.

When enabled, the directory where the H2O.ai Brain meta model files are stored is H2O.ai_brain. In addition, the default maximum brain size is 20GB. Both the directory and the maximum size can be changed in the config.toml file.

Feature Brain Restart from Which Iteration

When performing restart or re-fit of type feature_brain_level with a resumed ID, specify which iteration to start from instead of only last best. Available options include:

  • -1: Use the last best

  • 1: Run one experiment with feature_brain_iterations_save_every_iteration=1 or some other number

  • 2: Identify which iteration brain dump you wants to restart/refit from

  • 3: Restart/Refit from the original experiment, setting which_iteration_brain to that number here in expert settings.

Note: If restarting from a tuning iteration, this will pull in the entire scored tuning population and use that for feature evolution. This value defaults to -1.

Feature Brain Refit Uses Same Best Individual

Specify whether to use the same best individual when performing a refit. Disabling this setting allows the order of best individuals to be rearranged, leading to a better final result. Enabling this setting lets you view the exact same model or feature with only one new feature added. This is disabled by default.

Feature Brain Adds Features with New Columns Even During Retraining of Final Model

Specify whether to add additional features from new columns to the pipeline, even when performing a retrain of the final model. Use this option if you want to keep the same pipeline regardless of new columns from a new dataset. New data may lead to new dropped features due to shift or leak detection. Disable this to avoid adding any columns as new features so that the pipeline is perfectly preserved when changing data. This is enabled by default.

Restart-Refit Use Default Model Settings If Model Switches

When restarting or refitting, specify whether to use the model class’s default settings if the original model class is no longer available. If this is disabled, the original hyperparameters will be used instead. (Note that this may result in errors.) This is enabled by default.

Min DAI Iterations

Specify the minimum number of Driverless AI iterations for an experiment. This can be used during restarting, when you want to continue for longer despite a score not improving. This value defaults to 0.

Select Target Transformation of the Target for Regression Problems

Specify whether to automatically select target transformation for regression problems. Available options include:

  • auto

  • identity

  • identity_noclip

  • unit_box

  • log

  • square

  • sqrt

  • double_sqrt

  • inverse

  • logit

  • sigmoid

If set to auto (default), Driverless AI will automatically pick the best target transformer if the Accuracy is set to the value of the tune_target_transform_accuracy_switch configuration option (defaults to 3) or larger. Selecting identity automatically turns off any target transformations. All transformers except for identity_noclip perform clipping to constrain the predictions to the domain of the target in the training data. Use identity_noclip to disable target transformations and to allow predictions outside of the target domain observed in the training data for parametric models or custom models that support extrapolation.

Enable Genetic Algorithm for Selection and Tuning of Features and Models

Specify whether to enable genetic algorithm for selection and hyper-parameter tuning of features and models. If this is disabled, the final pipeline is trained using the default feature engineering and feature selection. This is set to Auto by default.

Tournament Model for Genetic Algorithm

Select a method to decide which models are best at each iteration. This is set to Auto by default. Choose from the following:

  • auto: Choose based on scoring metric

  • fullstack: Choose from optimal model and feature types

  • feature: Individuals with similar feature types compete

  • model: Individuals with same model type compete

  • uniform: All individuals in population compete

Number of Cross-Validation Folds for Feature Evolution

Specify the fixed number of cross-validation folds (if >= 2) for feature evolution. Note that the actual number of allowed folds can be less than the specified value, and that the number of allowed folds is determined at the time an experiment is run. This value defaults to -1 (auto).

Number of Cross-Validation Folds for Final Model

Specify the fixed number of cross-validation folds (if >= 2) for the final model. Note that the actual number of allowed folds can be less than the specified value, and that the number of allowed folds is determined at the time an experiment is run. This value defaults to -1 (auto).

Force Only First Fold for Models

Specify whether to force only the first fold for models. Select from Auto (Default), On, or Off.

Max Number of Rows Times Number of Columns for Feature Evolution Data Splits

Specify the maximum number of rows allowed for feature evolution data splits (not for the final pipeline). This value defaults to 100,000,000.

Max Number of Rows Times Number of Columns for Reducing Training Dataset

Specify the upper limit on the number of rows times the number of columns for training the final pipeline. This value defaults to 500,000,000.

Maximum Size of Validation Data Relative to Training Data

Specify the maximum size of the validation data relative to the training data. Smaller values can make the final pipeline model training process quicker. Note that final model predictions and scores will always be provided on the full dataset provided. This value defaults to 2.0.

Perform Stratified Sampling for Binary Classification If the Target Is More Imbalanced Than This

For binary classification experiments, specify a threshold ratio of minority to majority class for the target column beyond which stratified sampling is performed. If the threshold is not exceeded, random sampling is performed. This value defaults to 0.1. You can choose to always perform random sampling by setting this value to 0, or to always perform stratified sampling by setting this value to 1.

Add to config.toml via toml String

Specify any additional configuration overrides from the config.toml file that you want to include in the experiment. (Refer to the Sample config.toml File section to view options that can be overridden during an experiment.) Setting this will override all other settings. Separate multiple config overrides with \n. For example, the following enables Poisson distribution for LightGBM and disables Target Transformer Tuning. Note that in this example double quotes are escaped (\" \").

params_lightgbm=\"{'objective':'poisson'}\" \n target_transformer=identity

Or you can specify config overrides similar to the following without having to escape double quotes:

""enable_glm="off" \n enable_xgboost_gbm="off" \n enable_lightgbm="off" \n enable_tensorflow="on"""
""max_cores=10 \n data_precision="float32" \n max_rows_feature_evolution=50000000000 \n ensemble_accuracy_switch=11 \n feature_engineering_effort=1 \n target_transformer="identity" \n tournament_feature_style_accuracy_switch=5 \n params_tensorflow="{'layers': [100, 100, 100, 100, 100, 100]}"""

When running the Python client, config overrides would be set as follows:

model = h2o.start_experiment_sync(
    dataset_key=train.key,
    target_col='target',
    is_classification=True,
    accuracy=7,
    time=5,
    interpretability=1,
    config_overrides="""
                     feature_brain_level=0
                     enable_lightgbm="off"
                     enable_xgboost_gbm="off"
                     enable_ftrl="off"
                     """
)