Enabling Notifications¶

Driverless AI can be configured to trigger a user-defined script at the beginning and end of an experiment. This functionality can be used to send notifications to services like Slack or to trigger a machine shutdown.

The config.toml file exposes the following variables:

listeners_experiment_start: Registers an absolute location of a script that gets executed at the start of an experiment.
listeners_experiment_done: Registers an absolute location of a script that gets executed when an experiment is finished.

Driverless AI accepts any executable as a script. (For example, a script can be implemented in Bash or Python.) There are only two requirements:

The specified script can be executed. (i.e., The file has executable flag.)
The script should be able to accept command line parameters.

Script Interfaces¶

When Driverless AI executes a script, it passes the following parameters as a script command line:

Application ID: A unique identifier of a running Driverless AI instance.
User ID: The identification of the user who is running the experiment.
Experiment ID: A unique identifier of the experiment.
Experiment Path: The location of the experiment results.

Examples¶

The following examples execute a given action when all launched experiments are finished. The examples vary slightly based on whether you are performing a Docker image install or a Native install. These examples use the following scripts:

The on_start Script. This script increases the counter of running experiments.

#!/usr/bin/env bash

app_id="${1}"
experiment_id="${3}"
tmp_dir="${TMPDIR:-/tmp}/${app_id}"
exp_file="${tmp_dir}/${experiment_id}"

mkdir -p "${tmp_dir}"
touch "${exp_file}"

The on_done Script. This script decreases the counter and executes a specified action when the counter reaches 0-value.

#!/usr/bin/env bash

app_id="${1}"
experiment_id="${3}"
tmp_dir="${TMPDIR:-/tmp}/${app_id}"
exp_file="${tmp_dir}/${experiment_id}"

if [ -f "${exp_file}"  ]; then
    rm -f "${exp_file}"
fi

running_experiments=$(ls -1 "${tmp_dir}" | wc -l)

if [ "${running_experiments}" -gt 0  ]; then
    echo "There is still ${running_experiments} running experiments!"
else
    echo "No experiments running!"
fi

Docker Image Users¶

Copy the config.toml file from inside the Docker image to your local filesystem. (Change nvidia-docker run to docker run for non-GPU environments.)

# In your Driverless AI folder (for exmaple, dai_1.4.0),
# make config and scripts directories
mkdir config
mkdir scripts

# Copy the config.toml file to the new config directory.
nvidia-docker run \
  --pid=host \
  --rm \
  --init \
  -u `id -u`:`id -g` \
  -v `pwd`/config:/config \
  --entrypoint bash \
  h2oai/dai-centos7-x86_64:1.4.0-9.0
  -c "cp /etc/dai/config.toml /config"

Edit the Notification scripts section in the config.toml file and save your changes. Note that in this example, the scripts are saved to a dai_1.4.0/scripts folder.

# Notification scripts
# - the variable points to a location of script which is executed at given event in experiment lifecycle
# - the script should have executable flag enabled
# - use of absolute path is suggested
# The on experiment start notification script location
listeners_experiment_start = "dai_1.4.0/scripts/on_start.sh"
# The on experiment finished notification script location
listeners_experiment_done = "dai_1.4.0/scripts/on_done.sh"

Start Driverless AI with the DRIVERLESS_AI_CONFIG_FILE environment variable. Make sure this points to the location of the edited config.toml file so that the software finds the configuration file. (Change nvidia-docker run to docker run for non-GPU environments.)

nvidia-docker run \
  --pid=host \
  --init \
  --rm \
  -u `id -u`:`id -g` \
  -e DRIVERLESS_AI_CONFIG_FILE="/config/config.toml" \
  -v `pwd`/config:/config \
  -v `pwd`/data:/data \
  -v `pwd`/log:/log \
  -v `pwd`/license:/license \
  -v `pwd`/tmp:/tmp \
  -v `pwd`/scripts:/scripts \
  h2oai/dai-centos7-x86_64:1.4.0-9.0

Native Install Users¶

Export the Driverless AI config.toml file or add it to ~/.bashrc. For example:

export DRIVERLESS_AI_CONFIG_FILE="/config/config.toml"

Edit the Notification scripts section in the config.toml file to point to the new scripts. Save your changes when you are done.

# Notification scripts
# - the variable points to a location of script which is executed at given event in experiment lifecycle
# - the script should have executable flag enabled
# - use of absolute path is suggested
# The on experiment start notification script location
listeners_experiment_start = "/opt/h2oai/dai/scripts/on_start.sh"
# The on experiment finished notification script location
listeners_experiment_done = "/opt/h2oai/dai/scripts/on_done.sh"

Start Driverless AI. Note that the command used to start Driverless AI varies depending on your install type.

# Deb or RPM with systemd (preferred for Deb and RPM):
# Start Driverless AI.
sudo systemctl start dai

# Deb or RPM without systemd:
# Start Driverless AI.
sudo -H -u dai /opt/h2oai/dai/run-dai.sh

# Tar.sh
# Start Driverless AI
./run-dai.sh