Using AutoDoc

The following sections describe Driverless AI’s AutoDoc feature.

• Understanding AutoDoc

• Generating an AutoDoc

• Configuring AutoDoc

• Generating a Custom AutoDoc

Understanding AutoDoc

The AutoDoc feature is used to generate automated machine learning documentation for individual Driverless AI experiments. This editable document contains an overview of the experiment and includes other significant details like feature engineering and final model performance.

To download and view a sample experiment report in Word format, click here.

AutoDoc Support

AutoDoc only supports resumed experiments for certain Driverless AI versions. See the following table to check the types of resumed experiments that are supported for your version:

AutoDoc Support for Resumed Experiments Via	1.7.0 and older	1.7.1	1.9.0 and later
New experiment with same settings	yes	yes	yes
Restart from last checkpoint	no	yes	yes
Retrain final pipeline	no	no	yes

注解

• To ensure that AutoDoc pipeline visualizations are generated correctly on native installations, installing fontconfig is recommended.

• AutoDoc does not support experiments that were built off of previously aborted or failed experiments.

• Reports for unsupported resumed experiments will still build, but they will only include the following text: “AutoDoc not yet supported for resumed experiments.”

Custom AutoDocs

All Driverless AI experiments can generate either a standard or custom AutoDoc. A standard AutoDoc uses the default AutoDoc template that is included with Driverless AI, while a custom AutoDoc uses a customer-specific template that Driverless AI automatically populates.

If you are interested in creating a custom AutoDoc, contact support@h2o.ai. If you have already purchased a custom AutoDoc template and want to learn how to generate custom AutoDocs from your experiments, see Generating a Custom AutoDoc.

注解

• For a list of custom AutoDoc placeholders, see AutoDoc Custom Template Placeholders.

• Custom AutoDocs are Driverless AI version-specific.

BYOR Recipes with AutoDoc

The experiment AutoDoc supports experiments that use custom scorers, transformers, or models. Custom scorers and transformers are documented the same as Driverless AI scorers and transformers. If Driverless AI used a custom transformer, is is included in the Feature Transformations table under its display name; otherwise it is only included in the Feature Evolution section. (Note: custom-transformer descriptions are currently shown as “None” in this section.) For custom models, the standard performance metrics and plots are included; however, information that Driverless AI cannot access is not included, or is shown as “custom”, “unavailable”, or “auto.” For example, in the Model Tuning table, the booster is listed as “custom”, and in the Alternative Models section, the model package documentation is listed as “unavailable.”

Generating an AutoDoc

Three different approaches can be used to generate an AutoDoc:

• Experiment UI

• MLI UI

• Python Client

Notes:

• For more information on how to configure plots/tables and enable/disable specific sections in the AutoDoc, see Configuring AutoDoc.

• These approaches also apply to custom AutoDocs. For more information, see Generating a Custom AutoDoc.

Experiment UI

Navigate to the Experiments page and click on the completed experiment you want to generate an AutoDoc for.

If AutoDoc was not previously enabled for the experiment, click the Build AutoDoc button.

If AutoDoc was previously enabled for the experiment, click the Download AutoDoc button.

MLI UI

Navigate to the MLI page and click on the completed experiment you want to generate an AutoDoc for.

Select AutoDoc from the MLI RECIPES’s menu and optionally select explainers that can be included in the AutoDoc (the standard AutoDoc supports the k-LIME Explainer and DT Surrogate Explainer).

The Standard AutoDoc with Explainers:

Python Client

• driverlessai

AutoDoc Functions

• create_and_download_autodoc()

• make_autodoc_sync()

For local downloads:

create_and_download_autodoc(
    model_key:str,
    template_path:str='',
    config_overrides:str='',
    dest_path:str='.',
    mli_key:str='',
    individual_rows:list=[],
    external_dataset_keys:list=[])

To save an AutoDoc to the DAI experiment directory (recommended if local downloads are disabled):

make_autodoc_sync(
    model_key:str,
    template_path:str='',
    config_overrides:str='',
    mli_key:str='',
    individual_rows:list=[],
    external_dataset_keys:list=[])

• model_key: The experiment key string.

• template_path: The full path the custom AutoDoc template.

• config_overrides: The TOML string format with configurations overrides for the AutoDoc.

• dest_path: The local path where the AutoDoc should be saved.

• mli_key: The mli key string.

• individual_rows: List of row indices for rows of interest in the training dataset, for which additional information can be shown (ICE, LOCO, KLIME).

• external_dataset_keys: List of DAI dataset keys.

driverlessai

Connect to a running DAI instance:

import driverlessai
address = 'http://ip_where_driverless_is_running:12345'
username = 'username'
password = 'password'
dai = driverlessai.Client(address=address, username=username, password=username)

Generate an AutoDoc and download it to your current working directory:

report = dai._backend.create_and_download_autodoc(
    model_key=exp_key,
    dest_path:str='.',
)

Configuring AutoDoc

An AutoDoc’s plots, tables, and sections can be configured through four different workflows:

• Experiment UI Expert Settings

• MLI UI Expert Settings

• Python Client

• config.toml file

Experiment UI Expert Settings

MLI UI Expert Settings

Python Client

• driverlessai

All configuration options for the AutoDoc are listed in the config.toml file. The following are several commonly used configuration parameters:

import toml

# Set the document to limit features displayed to the top ten
config_dict={
   "autodoc_num_features": 10
}

# Partial Dependence Plots (PDP) and ICE Plots
config_dict["autodoc_pd_max_runtime"] = 60
config_dict["autodoc_num_rows"] = 4

# Prediction statistics
config_dict["autodoc_prediction_stats"] = True
config_dict["autodoc_prediction_stats_n_quantiles"] = 10

# Population Stability Index (PSI)
config_dict["autodoc_population_stability_index"] = True
config_dict["autodoc_population_stability_index_n_quantiles"] = 10

# Permutation feature importance
config_dict["autodoc_include_permutation_feature_importance"] = True
config_dict["autodoc_feature_importance_scorer"] = "GINI"
config_dict["autodoc_feature_importance_num_perm"] = 1

# Response rates (only applicable to Binary classification)
config_dict["autodoc_response_rate"] = True
config_dict["autodoc_response_rate_n_quantiles"] = 10

toml_string = toml.dumps(config_dict)
print(toml_string)

After setting these parameters, generate an AutoDoc and download it to your current working directory:

driverlessai

report = dai._backend.create_and_download_autodoc(
    model_key=exp_key,
    config_overrides=config_overrides,
    dest_path:str='.',
)

Generating a Custom AutoDoc

This section describes how to generate an AutoDoc from a custom AutoDoc template. Choose from the following options:

• Have Driverless AI use a custom AutoDoc for all experiments

• Have Driverless AI generate a custom AutoDoc for an individual experiment

注解

• For a list of custom AutoDoc placeholders, see AutoDoc Custom Template Placeholders.

• AutoDoc template directories must be mapped to the Driverless AI container or pod.

Custom AutoDoc for All Experiments

To use a custom AutoDoc template, edit the config.toml file to point to the location of your custom AutoDoc. Use the following config.toml settings:

• autodoc_template: Specify the path for the main template file.

• autodoc_additional_template_folder: If you have additional custom sub-templates, use this setting to specify the location of additional AutoDoc templates. Note that if this field is left empty, only the default sub-templates folder is used.

To generate custom AutoDocs, Driverless AI must have access to the custom template(s). To make sure that Driverless AI has access, update the path in the following example with your own path:

autodoc_template="/full/path/to/your/custom_autodoc_template.docx"

# Required if you have additional custom sub-templates.
autodoc_additional_template_folder="/path/to/additional_templates_folder"

Custom AutoDoc for Individual Experiments

You can use the Python Client to generate standard or custom AutoDocs from an experiment by setting the template_path variable to your custom AutoDoc’s path:

template_path='/full/path/to/your/custom_autodoc_template.docx'

Python Client: driverlessai

report = dai._backend.create_and_download_autodoc(
    model_key=exp_key,
    template_path=template_path,
    dest_path:str='.',
)