{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Credit Card Demo\n", "This notebook provides an H2OAI Client workflow, of model building and scoring, that parallels the Driverless AI workflow.\n", "\n", "**Notes**:\n", "\n", "* This is an early release of the Driverless AI Python client.\n", "* Python 3.6 is the only supported version.\n", "* You must install the `h2oai_client` wheel to your local Python. This is available from the PY_CLIENT link in the top menu of the UI.\n", "\n", "![py-client](images/py_client_link.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Workflow Steps\n", "\n", "**Build an Experiment with Python API:**\n", "\n", "1. Sign in\n", "2. Import train & test set/new data\n", "3. Specify experiment parameters\n", "4. Launch Experiment\n", "5. Examine Experiment\n", "6. Download Predictions\n", "7. Launch Experiment with Config Overrides\n", " \n", "**Build an Experiment in Web UI and Access Through Python:**\n", "\n", "1. Get pointer to experiment\n", " \n", "**Score on New Data:**\n", "\n", "1. Score on new data with H2OAI model\n", "\n", "**Run Model Interpretation**\n", "\n", "1. Run Model Interpretation on Driverless AI Experiment\n", "2. Run Model Interpretation on External Model Predictions\n", "\n", "**Build Scoring Pipelines**\n", "\n", "1. Build Python Scoring Pipeline\n", "2. Build MOJO Scoring Pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build an Experiment with Python API\n", "\n", "### 1. Sign In\n", "\n", "Import the required modules and log in.\n", "\n", "Pass in your credentials through the Client class which creates an authentication token to send to the Driverless AI Server. In plain english: to sign into the Driverless AI webpage (which then sends requests to the Driverless Server) instantiate the Client class with your Driverless AI address and login credentials." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import h2oai_client\n", "import numpy as np\n", "import pandas as pd\n", "# import h2o\n", "import requests\n", "import math\n", "from h2oai_client import Client, ModelParameters, InterpretParameters" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "address = 'http://ip_where_driverless_is_running:12345'\n", "username = 'username'\n", "password = 'password'\n", "h2oai = Client(address = address, username = username, password = password)\n", "# make sure to use the same user name and password when signing in through the GUI" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "#### Equivalent Steps in Driverless: Signing In\n", "![Equivalent Steps in Driverless: Signing In](images/sign_in_home_page_0.png)\n", "\n", "![Equivalent Steps in Driverless: Signing In](images/skip_sign_in_home_page_1.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Upload Datasets\n", "\n", "Upload training and testing datasets from the Driverless AI **/data** folder.\n", "\n", "You can provide a training, validation, and testing dataset for an experiment. The validation and testing dataset are optional. In this example, we will provide only training and testing." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "train_path = '/data/Kaggle/CreditCard/CreditCard-train.csv'\n", "test_path = '/data/Kaggle/CreditCard/CreditCard-test.csv'\n", "\n", "train = h2oai.create_dataset_sync(train_path)\n", "test = h2oai.create_dataset_sync(test_path)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Equivalent Steps in Driverless: Uploading Train & Test CSV Files\n", "![Equivalent Steps in Driverless: Upload Train & Test CSV Files](images/import_data_sets_creditcard.png)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 3. Set Experiment Parameters\n", "\n", "We will now set the parameters of our experiment. Some of the parameters include:\n", "\n", "* Target Column: The column we are trying to predict.\n", "* Dropped Columns: The columns we do not want to use as predictors such as ID columns, columns with data leakage, etc.\n", "* Weight Column: The column that indicates the per row observation weights. If `None`, each row will have an observation weight of 1.\n", "* Fold Column: The column that indicates the fold. If `None`, the folds will be determined by Driverless AI.\n", "* Time Column: The column that provides a time order, if applicable.\n", " * if `[OFF]`, time series recipe is not enabled\n", "\n", "For information on the experiment settings, refer to the [Experiment Settings](http://docs.h2o.ai/driverless-ai/latest-stable/docs/userguide/launching.html#experimentsettings).\n", "\n", "For this example, we will be predicting **`default payment next month`**. The parameters that control the experiment process are: `accuracy`, `time`, and `interpretability`. We can use the `get_experiment_preview_sync` function to get a sense of what will happen during the experiment. \n", "\n", "We will start out by seeing what the experiment will look like with `accuracy`, `time`, and `interpretability` all set to 5." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "['ACCURACY [5/10]:',\n", " '- Training data size: *23,999 rows, 25 cols*',\n", " '- Feature evolution: *XGBoost*, *time-based validation*',\n", " '- Final pipeline: *XGBoost*',\n", " '',\n", " 'TIME [5/10]:',\n", " '- Feature evolution: *8 individuals*, up to *54 iterations*',\n", " '- Early stopping: After *10* iterations of no improvement',\n", " '',\n", " 'INTERPRETABILITY [5/10]:',\n", " '- Feature pre-pruning strategy: None',\n", " '- Monotonicity constraints: disabled',\n", " '- Feature engineering search space (where applicable): [Clustering, Date, FrequencyEncoding, Identity, Interactions, Lags, Text, TruncatedSVD, WeightOfEvidence]',\n", " '',\n", " 'XGBoost models to train:',\n", " '- Model and feature tuning: *16*',\n", " '- Feature evolution: *208*',\n", " '- Final pipeline: *1*',\n", " '',\n", " 'Estimated max. total memory usage:',\n", " '- Feature engineering: *48.0MB*',\n", " '- GPU XGBoost: *64.0MB*',\n", " '',\n", " 'Estimated runtime: *8 minutes*']" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "target=\"default payment next month\"\n", "exp_preview = h2oai.get_experiment_preview_sync(dataset_key= train.key, \n", " validset_key='', \n", " classification=True, \n", " dropped_cols = [], \n", " target_col=target, \n", " time_col = '', \n", " enable_gpus = True, \n", " accuracy = 5, \n", " time = 5, \n", " interpretability = 5, \n", " config_overrides = None)\n", "exp_preview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With these settings, the Driverless AI experiment should take around 8 minutes to run and will train about 225 models: \n", "* 16 for parameter tuning\n", "* 208 for feature engineering\n", "* 1 for the final model\n", "\n", "Driverless AI can suggest the parameters based on the dataset and target column. Below we will use the `get_experiment_tuning_suggestion` to see what settings Driverless AI suggests." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# let Driverless suggest parameters for experiment\n", "params = h2oai.get_experiment_tuning_suggestion(dataset_key = train.key, \n", " target_col = target, \n", " is_classification = True, \n", " is_time_series = False, \n", " config_overrides = None)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Accuracy: 6\n", "Time: 3\n", "Interpretability: 6\n" ] } ], "source": [ "print(\"Accuracy: \" + str(params.accuracy))\n", "print(\"Time: \" + str(params.time))\n", "print(\"Interpretability: \" + str(params.interpretability))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Driverless AI has found that the best parameters are to set `accuracy = 6`, `time = 3`, `interpretability = 6`. We can see our experiment preview with the suggested settings below. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['ACCURACY [6/10]:',\n", " '- Training data size: *23,999 rows, 25 cols*',\n", " '- Feature evolution: *XGBoost*, *time-based validation*',\n", " '- Final pipeline: *XGBoost*',\n", " '',\n", " 'TIME [3/10]:',\n", " '- Feature evolution: *8 individuals*, up to *36 iterations*',\n", " '- Early stopping: After *5* iterations of no improvement',\n", " '',\n", " 'INTERPRETABILITY [6/10]:',\n", " '- Feature pre-pruning strategy: FS',\n", " '- Monotonicity constraints: disabled',\n", " '- Feature engineering search space (where applicable): [Date, FrequencyEncoding, Identity, Interactions, Lags, Text, WeightOfEvidence]',\n", " '',\n", " 'XGBoost models to train:',\n", " '- Model and feature tuning: *24*',\n", " '- Feature evolution: *128*',\n", " '- Final pipeline: *1*',\n", " '',\n", " 'Estimated max. total memory usage:',\n", " '- Feature engineering: *48.0MB*',\n", " '- GPU XGBoost: *64.0MB*',\n", " '',\n", " 'Estimated runtime: *6 minutes*']" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "exp_preview = h2oai.get_experiment_preview_sync(dataset_key= train.key, \n", " validset_key='', \n", " classification=True, \n", " dropped_cols = [], \n", " target_col=target,\n", " time_col = \"\",\n", " enable_gpus = True, \n", " accuracy = 6, \n", " time = 3, \n", " interpretability = 6,\n", " config_overrides = None)\n", "exp_preview" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Equivalent Steps in Driverless: Set the Knobs, Configuration & Launch\n", "![Equivalent Steps in Driverless: Set the Knobs](images/set_parameters_creditcard.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Launch Experiment: Feature Engineering + Final Model Training\n", "\n", "Launch the experiment using the parameters that Driverless AI suggested along with the test data, scorer, and seed." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": true }, "outputs": [], "source": [ "experiment = h2oai.start_experiment_sync(dataset_key = train.key, \n", " testset_key = test.key,\n", " target_col = target,\n", " is_classification = True,\n", " accuracy = 6,\n", " time = 3,\n", " interpretability = 6,\n", " scorer = \"AUC\",\n", " seed = 1234)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Equivalent Steps in Driverless: Launch Experiment\n", "\n", "![Equivalent Steps in Driverless: Launch Your Experiment](images/exp_running_creditcard.png)\n", "\n", "![Equivalent Steps in Driverless: Launch Your Experiment](images/experiment_list_running.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Examine Experiment\n", "\n", "View the final model score for the validation and test datasets. When feature engineering is complete, an ensemble model can be built depending on the accuracy setting. The experiment object also contains the score on the validation and test data for this ensemble model. In this case, the validation score is the score on the training cross-validation predictions." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Final Model Score on Validation Data: 0.78\n", "Final Model Score on Test Data: 0.8\n" ] } ], "source": [ "print(\"Final Model Score on Validation Data: \" + str(round(experiment.valid_score, 3)))\n", "print(\"Final Model Score on Test Data: \" + str(round(experiment.test_score, 3)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The experiment object also contains the scores calculated for each iteration on bootstrapped samples on the validation data. In the iteration graph in the UI, we can see the mean performance for the best model (yellow dot) and +/- 1 standard deviation of the best model performance (yellow bar).\n", "\n", "This information is saved in the experiment object." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGqlJREFUeJzt3X+QVeWd5/H3h75NQyO2Ig0aGgKZkFFjGTJ2iMmMmsQk\nSxxZIjvZgcRdd5KSxZX4YxMdzO5OMlU7VTFaM5upNaFIZJiq8cdmAlEmywhOZsIm7khAQxwQf7BG\npQEjCCrS0H27+7t/3NP07Ut30/L06dvYn1dVV9/zPM8553suBz6cc+45VxGBmZnZqRpT7QLMzOz0\n5iAxM7MkDhIzM0viIDEzsyQOEjMzS+IgMTOzJA4SMzNL4iAxM7MkuQaJpHmSnpW0S9LyPvpvk7Qt\n+9kuqVPSpKzv5qxth6RbyuaZI+nxbJ6tkubmuQ1mZjYw5XVnu6Qa4DngU0ALsAVYHBFP9zN+PnBr\nRHxC0kXAg8BcoB14BFgaEbskbQT+IiL+XtJVwO0R8bGBapk8eXLMnDlziLbMzGx0eOKJJw5EROPJ\nxhVyrGEusCsiXgCQ9CCwAOgzSIDFwAPZ6wuAzRHRms27CVgIfAsI4MxsXAOw92SFzJw5k61bt57i\nZpiZjU6SXhrMuDyDZBqwu2y6BfhwXwMl1QPzgGVZ03bgzySdAxwFrgK6k+AWYIOkuymdmvvo0Jdu\nZmaDNVIuts8HHouIgwARsRO4E9hI6bTWNqAzG3sDpVNg04FbgXv7WqCkJdk1lK379+/Pu34zs1Er\nzyDZA0wvm27K2vqyiJ7TWgBExL0RcUlEXA4conS9BeA6YG32+m8pnUI7QUSsjIjmiGhubDzpKT4z\nMztFeQbJFmC2pFmSxlIKi3WVgyQ1AFcAD1e0T8l+z6B0feT+rGtvNh7gE8DzuVRvZmaDkts1kojo\nkLQM2ADUAKsiYoekpVn/imzoNcDGiDhSsYg12TWSInBjRLyetV8PfFtSATgGLMlrG8zM7ORy+/jv\nSNLc3Bz+1JaZ2dsj6YmIaD7ZuJFysd3MzE5TDhIzM0uS530kZmZWNd84yfTQ8RGJmZklcZCYmVkS\nB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEj+0\nsV/fOMn0cKlcb+X0cKlcb+X0cKlcb+X0aKmhr/VWTg+XyvVWTg+XyvVWTg+XyvVWTr/z+IjEzMyS\nOEjMzCyJg8TMzJI4SMzMLImDxMzMkjhIzMwsiYPEzMySOEjMzCyJg8TMzJI4SMzMLImDxMzMkjhI\nzMwsiYPEzMyS5BokkuZJelbSLknL++i/TdK27Ge7pE5Jk7K+m7O2HZJuqZjvy5Keyfq+lec2mJnZ\nwHJ7jLykGuAe4FNAC7BF0rqIeLp7TETcBdyVjZ8P3BoRByVdBFwPzAXagUck/Tgidkn6OLAA+EBE\ntEmaktc2mJnZyeV5RDIX2BURL0REO/AgpQDoz2Lggez1BcDmiGiNiA5gE7Aw67sB+GZEtAFExKu5\nVG9mZoOSZ5BMA3aXTbdkbSeQVA/MA9ZkTduByySdk/VdBUzP+t6X9W2WtEnSh3Kp3szMBmWkfEPi\nfOCxiDgIEBE7Jd0JbASOANuAzmxsAZgEXAp8CPiBpPdERJQvUNISYAnAjBkzhmUjzMxGozyPSPbQ\ncxQB0JS19WURPae1AIiIeyPikoi4HDgEPJd1tQBro+QXQBcwuXKBEbEyIpojormxsTFxU8zMrD95\nBskWYLakWZLGUgqLdZWDJDUAVwAPV7RPyX7PoHR95P6s6yHg41nf+4CxwIGctsHMzE4it1NbEdEh\naRmwAagBVkXEDklLs/4V2dBrgI0RcaRiEWsknQMUgRsj4vWsfRWwStJ2Sp/ouq7ytJaZmQ2fXK+R\nRMR6YH1F24qK6dXA6j7mvayfZbYD1w5ZkWZmlsR3tpuZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkS\nB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJ\nmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZ\nJXGQmJlZEgeJmZklcZCYmVmSXINE0jxJz0raJWl5H/23SdqW/WyX1ClpUtZ3c9a2Q9Itfcz7FUkh\naXKe22BmZgPLLUgk1QD3AJ8BLgQWS7qwfExE3BURcyJiDnAHsCkiDkq6CLgemAt8ALha0nvLlj0d\n+DTwcl71m5nZ4OR5RDIX2BURL0REO/AgsGCA8YuBB7LXFwCbI6I1IjqATcDCsrF/AdwOxNCXbWZm\nb0eeQTIN2F023ZK1nUBSPTAPWJM1bQcuk3RO1ncVMD0buwDYExG/yqtwMzMbvEK1C8jMBx6LiIMA\nEbFT0p3ARuAIsA3ozELla5ROaw1I0hJgCcCMGTPyqtvMbNTL84hkD9lRRKYpa+vLInpOawEQEfdG\nxCURcTlwCHgO+C1gFvArSS9my3xS0rmVC4yIlRHRHBHNjY2NyRtjZmZ9y/OIZAswW9IsSgGyCPh8\n5SBJDcAVwLUV7VMi4lVJMyhdH7k0Il4HppSNeRFojogDuW2FmZkNKLcgiYgOScuADUANsCoidkha\nmvWvyIZeA2yMiCMVi1gj6RygCNyYhYiZmY0wuV4jiYj1wPqKthUV06uB1X3Me9kglj8zqUAzM0vm\nO9vNzCyJg8TMzJI4SMzMLImDxMzMkjhIzMwsiYPEzMySOEjMzCyJg8TMzJI4SMzMLImDxMzMkjhI\nzMwsiYPEzMySOEjMzCyJg8TMzJI4SMzMLImDxMzMkjhIzMwsSb9BIulfSfqDPtr/QNKn8i3LzMxO\nFwN91e6fAJ/to/2nwN8Bj+ZRkJmZpTt8+BUOHHiGtrY3qKtrYPLkp5g48eJc1jXQqa26iNhf2RgR\nB4AJuVRjZmbJDh9+ipaWf6ZYbGXs2DMpFo/S0nI3hw8/lcv6BgqSMyWdcMQiqRYYn0s1ZmaWbN++\n79HWdpiuriKSqK0dT03N2Rw4sDaX9Q0UJGuB70k6fvQh6QxgRdZnZmZVFhG0t/+GQ4d+SrF4EID2\n9hakMUg9/8QXCg20tb2cSw0DXSP5r8B/B16S9BIgYDpwL/DfcqnGzMxOKqKLtrYWWlt30tr6DMXi\nISRRKJxFbe0kJky4mGJxL7W1PSePOjreoK5uRi719BskEdEBLJf0p8B7s+ZdEXE0l0rMzKxfXV0d\ndHW1UiicSWdnK6+88lfAGMaPfw8NDb/H+PG/TaFwBgCTJ/8bWloeAqBQGEdHxzE6Ow9x3nlfyqW2\nfoNE0sKKpgDOkrQtIg7nUo2ZmR3X1dVGa+vztLY+w9Gjz1NXN51zz72WQuEMpk69lrq6JsaMqTth\nvokTL6ap6SO9PrV13nlfze1TWwOd2prfR9sk4GJJX4qIf8ylIjMz47XX1nP48BNEdFJTM4EJE97P\nhAnvP94/fvxvDTj/xInnMnHiuWUt+YQIDHxq64/6apf0buAHwIfzKsrMbDQpFg9lRx3/j6lTFyPV\nUChM4swz51Jffz51ddN7XTgfaQY6IulTRLyUfQTYzMxOUbH4Om+9tY3W1mdob38FgLFjp9LR8Sa1\ntWfT0HBplSscvLcdJJLOB9pyqGVEGc67Ql3H6VXHSKjBdZx+dXR/0qqm5kxqa8+iWNzPG29soq5u\nOpMmfZr6+vOprZ007PUOhYGetfV3ktZV/Pwc+N/Afx7MwiXNk/SspF2SlvfRf5ukbdnPdkmdkiZl\nfTdnbTsk3VI2z12SnpH0lKQfSTrr7W/2wHruCj06LHeFuo7Tp46RUIPrOJ3qaOXFF7/O7t1/ye7d\nf86+fat4660nARg//j00NX2F8877Ig0NHz1tQwQGPiK5u2I6gIOULrhfC/zzQAuWVAPcA3wKaAG2\nSFoXEU8fX2DEXcBd2fj5wK0RcVDSRcD1wFygHXhE0o8jYhelZ3zdEREdku4E7gD+eLAbPBgHDqyl\npmYctbXjKRZbieiiq6uDvXu/w7RpNzBmzATq60ufiG5tfZaurmO95h+q/gMH1hIBEZ0Ui0eA0kcA\n9+37PhMn/mXu6+/u37v3O3R1dTFmTKkOqeb4XbI1NXW5r7+7f+/e79LV1UVNDcfv1u3o4PifS97r\n764hQmWfz49e+0be6+/u795HIXrtG3v3fofp0786LPsnwL59K3vtGwAR4sCBtUycePGw7J9dXceO\n7xtjxnTS0XEs+/M5m337ViJdn/v6u5XvHxFBa+sBOjvFa689zLvetZT6+guor58NkF0HOYN3goEu\ntm/qfi3pg8Dngc8BvwbWDGLZcyndd/JCtowHgQXA0/2MXww8kL2+ANgcEa3ZvJuAhcC3ImJj2TyP\nAyc8oThVW9vLjB07DoBjxw7R2VkkYgKtrTsZO/Zcxo2bfnxHOnToH2hv7/1IsqHqb2t7mWLxCG1t\nHcf7Iib0ujs1z/V397/55mZqat6kvV0AFAp1TJhQukt2ONbf3d9dR1dX+/F/yIvF/Rw58gpjx56b\nvPzB9L/55mbq6lqB0oHwsWOH6OjYc3zfyHv93f3d++hbb+2ls7MIcHwffeONTcOyfwK89dYTQM++\nAVBTc5C2tpphWX93f/k+WijUUVs7nkKhgddff5T9+yfnvv5u5fuHJMaNa0BqpqvrCFOmfI53qoHu\nI3kfpX/cFwMHgP8FKCI+PshlTwN2l0230M8nvSTVA/OAZVnTduDPJJ0DHAWuArb2MesXs7qGVF3d\nDIrFHdTWjmfChKlA0NFxPoXCWTQ13UT5I8imTr2WiM6K7Rma/rq6GUhnUSj0fE68VMekQc0/VP3H\njr1MR8c/UCiM6x5x/C7Z4Vh/d39X19Gsjp67dceNezdnnHExTU035b7+nhp+cnx6woSpvfaNvNff\n3X/s2G6KxR3H90/geB2NjZ876fxD1d/QcAUdHY+W7RvQ0dHE2LFTh2X93f09+8Y4Sg/hKN3J3dBw\nRa99I6/1d6vcP+rqGigWC4wbN5N3soFObT0D/Ay4OjulhKRbc6pjPvBYRBwEiIid2WmrjcARYBvQ\n609T0n8BOoD7+lqgpCXAEoAZM97eYwEmT15IS8uPgJ67Qru6jjJlyo0nnMcsFBoGXFZKf3cdnZ01\nFXUsHpb1d/dPmfIFWloeobOz44S7ZIdj/d166igi1WTvRytTpvynfs8vD3V93TUUi0cpFMbR2Vns\nd9/IY/3dBtpHC4Uzc19/tylTPk9Ly9/32je6uo4wefLCYVl/d3/lPlosHj2+jw507SHv/SPvO8pH\nioE+mLwQ2Af8k6TvSbqS7qgfnD2Uns3VrSlr68siek5rARAR90bEJRFxOXAIeK67T9J/AK4GvhAR\n0dcCI2JlRDRHRHNjY+PbKLvnrtDa2vG0t79Jbe14mpryuyvUdZw+dYyEGlyH6xhpBrpG8hDwUPb0\n3wXALcAUSd8FflRxraIvW4DZkmZRCpBFlK6z9CKpAbiC0gX88vYpEfGqpBmUQu3SrH0ecDtwRfc1\nlDwM512hruP0qmMk1OA6XMdIctL7SCLiCHA/cL+ksyldcP9jSqedBpqvQ9IyYANQA6yKiB2Slmb9\nK7Kh1wAbs/WUW5NdIykCN0bE61n7/wTqgEclATweEUtPvqlmZpaHt3VDYkQcAlZmP4MZvx5YX9G2\nomJ6NbC6j3kv62eZ7+2r3czMqmPkPrzFzMxOCw4SMzNL4iAxM7MkDhIzM0viIDEzsyQOEjMzS+Ig\nMTOzJA4SMzNL4iAxM7MkDhIzM0viIDEzsyQOEjMzS+IgMTOzJA4SMzNL4iAxM7MkDhIzM0viIDEz\nsyQOEjMzS+IgMTOzJA4SMzNL4iAxM7MkDhIzM0viIDEzsyQOEjMzS+IgMTOzJA4SMzNL4iAxM7Mk\nDhIzM0viIDEzsyS5BomkeZKelbRL0vI++m+TtC372S6pU9KkrO/mrG2HpFvK5pkk6VFJz2e/z85z\nG8zMbGC5BYmkGuAe4DPAhcBiSReWj4mIuyJiTkTMAe4ANkXEQUkXAdcDc4EPAFdLem8223LgJxEx\nG/hJNm1mZlWS5xHJXGBXRLwQEe3Ag8CCAcYvBh7IXl8AbI6I1ojoADYBC7O+BcBfZ6//GvjskFdu\nZmaDlmeQTAN2l023ZG0nkFQPzAPWZE3bgcsknZP1XQVMz/qmRsS+7PUrwNShLtzMzAavUO0CMvOB\nxyLiIEBE7JR0J7AROAJsAzorZ4qIkBR9LVDSEmAJwIwZM/Kq28xs1MvziGQPPUcRAE1ZW18W0XNa\nC4CIuDciLomIy4FDwHNZ128knQeQ/X61rwVGxMqIaI6I5sbGxoTNMDOzgeQZJFuA2ZJmSRpLKSzW\nVQ6S1ABcATxc0T4l+z2D0vWR+7OudcB12evrKuczM7PhlduprYjokLQM2ADUAKsiYoekpVn/imzo\nNcDGiDhSsYg1ks4BisCNEfF61v5N4AeSvgS8BPzbvLbBzMxOLtdrJBGxHlhf0baiYno1sLqPeS/r\nZ5mvAVcOWZFmZpbEd7abmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVx\nkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCY\nmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZ\nklyDRNI8Sc9K2iVpeR/9t0nalv1sl9QpaVLWd6ukHVn7A5LGZe1zJD2ezbNV0tw8t8HMzAaWW5BI\nqgHuAT4DXAgslnRh+ZiIuCsi5kTEHOAOYFNEHJQ0DbgJaI6Ii4AaYFE227eAP83m+ZNs2szMqiTP\nI5K5wK6IeCEi2oEHgQUDjF8MPFA2XQDGSyoA9cDerD2AM7PXDWXtZmZWBYUclz0N2F023QJ8uK+B\nkuqBecAygIjYI+lu4GXgKLAxIjZmw28BNmT9Y4CP5lO+mZkNxki52D4feCwiDgJIOpvS0css4F3A\nBEnXZmNvAG6NiOnArcC9fS1Q0pLsGsrW/fv3574BZmajVZ5BsgeYXjbdlLX1ZRG9T2t9Evh1ROyP\niCKwlp4jj+uyaYC/pXQK7QQRsTIimiOiubGx8RQ3wczMTibPINkCzJY0S9JYSmGxrnKQpAbgCuDh\nsuaXgUsl1UsScCWwM+vbm40H+ATwfE71m5nZIOR2jSQiOiQtAzZQ+tTVqojYIWlp1r8iG3oNpWsg\nR8rm3Szph8CTQAfwS2Bl1n098O3sIvwxYEle22BmZieX58V2ImI9sL6ibUXF9GpgdR/zfh34eh/t\nPwcuGco6zczs1I2Ui+1mZnaayvWIxMzMquUbw7YmH5GYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZ\nEgeJmZklcZCYmVkSB4mZmSVRRFS7htw1NzfH1q1bq12GmdlpRdITEdF8snE+IjEzsyQOEjMzS+Ig\nMTOzJA4SMzNL4iAxM7MkDhIzM0viIDEzsyQOEjMzS+IgMTOzJKPiznZJ+4GXTnH2ycCBISzndOf3\no4ffi978fvT2Tng/3h0RjScbNCqCJIWkrYN5RMBo4fejh9+L3vx+9Daa3g+f2jIzsyQOEjMzS+Ig\nObmV1S5ghPH70cPvRW9+P3obNe+Hr5GYmVkSH5GYmVkSB8kAJM2T9KykXZKWV7ueapE0XdI/SXpa\n0g5JN1e7ppFAUo2kX0r6cbVrqTZJZ0n6oaRnJO2U9JFq11Qtkm7N/p5sl/SApHHVrilvDpJ+SKoB\n7gE+A1wILJZ0YXWrqpoO4CsRcSFwKXDjKH4vyt0M7Kx2ESPEt4FHIuJ84AOM0vdF0jTgJqA5Ii4C\naoBF1a0qfw6S/s0FdkXECxHRDjwILKhyTVUREfsi4sns9WFK/0hMq25V1SWpCfh94PvVrqXaJDUA\nlwP3AkREe0S8Xt2qqqoAjJdUAOqBvVWuJ3cOkv5NA3aXTbcwyv/xBJA0E/ggsLm6lVTd/wBuB7qq\nXcgIMAvYD/xVdqrv+5ImVLuoaoiIPcDdwMvAPuCNiNhY3ary5yCxQZN0BrAGuCUi3qx2PdUi6Wrg\n1Yh4otq1jBAF4HeA70bEB4EjwKi8pijpbEpnLmYB7wImSLq2ulXlz0HSvz3A9LLppqxtVJJUSylE\n7ouItdWup8p+F/jXkl6kdMrzE5L+prolVVUL0BIR3UepP6QULKPRJ4FfR8T+iCgCa4GPVrmm3DlI\n+rcFmC1plqSxlC6YratyTVUhSZTOf++MiD+vdj3VFhF3RERTRMyktF/8Y0S84//X2Z+IeAXYLem3\ns6YrgaerWFI1vQxcKqk++3tzJaPggweFahcwUkVEh6RlwAZKn7xYFRE7qlxWtfwu8O+Af5G0LWv7\nWkSsr2JNNrJ8Gbgv+0/XC8AfVbmeqoiIzZJ+CDxJ6dOOv2QU3OHuO9vNzCyJT22ZmVkSB4mZmSVx\nkJiZWRIHiZmZJXGQmJlZEgeJ2dsg6a3s90xJnx/iZX+tYvr/DuXyzfLiIDE7NTOBtxUk2UP8BtIr\nSCLiHX9HtL0zOEjMTs03gcskbcu+f6JG0l2Stkh6StJ/BJD0MUk/k7SO7G5vSQ9JeiL7zoolWds3\nKT0xdpuk+7K27qMfZcveLulfJP1h2bJ/WvY9IPdld1ObDSvf2W52apYDX42IqwGyQHgjIj4kqQ54\nTFL3U19/B7goIn6dTX8xIg5KGg9skbQmIpZLWhYRc/pY10JgDqXv+ZiczfN/sr4PAu+n9Kjyxyg9\nheDnQ7+5Zv3zEYnZ0Pg08O+zR8hsBs4BZmd9vygLEYCbJP0KeJzSg0FnM7DfAx6IiM6I+A2wCfhQ\n2bJbIqIL2EbplJvZsPIRidnQEPDliNjQq1H6GKXHqpdPfxL4SES0SvopkPJVrG1lrzvx32mrAh+R\nmJ2aw8DEsukNwA3Z4/aR9L5+vtypATiUhcj5lL66uFuxe/4KPwP+MLsO00jp2wh/MSRbYTYE/L8X\ns1PzFNCZnaJaTek7y2cCT2YXvPcDn+1jvkeApZJ2As9SOr3VbSXwlKQnI+ILZe0/Aj4C/AoI4PaI\neCULIrOq89N/zcwsiU9tmZlZEgeJmZklcZCYmVkSB4mZmSVxkJiZWRIHiZmZJXGQmJlZEgeJmZkl\n+f9WHN25IhwYFwAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import matplotlib.pyplot as plt\n", "\n", "iterations = list(map(lambda iteration: iteration.iteration, experiment.iteration_data))\n", "scores_mean = list(map(lambda iteration: iteration.score_mean, experiment.iteration_data))\n", "scores_sd = list(map(lambda iteration: iteration.score_sd, experiment.iteration_data))\n", "\n", "plt.figure()\n", "plt.errorbar(iterations, scores_mean, yerr=scores_sd, color = \"y\", \n", " ecolor='yellow', fmt = '--o', elinewidth = 4, alpha = 0.5)\n", "plt.xlabel(\"Iteration\")\n", "plt.ylabel(\"AUC\")\n", "plt.show();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can access the data used for each visualization shown in the GUI. Below we examine the gains and lifts table for the test data." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
gainsliftsquantiles
00.0410744.1074250.01
10.0805694.0284360.02
20.1208534.0284360.03
30.1508693.7717220.04
40.1832543.6650870.05
50.3412323.4123220.10
60.4541863.0279090.15
70.5379152.6895730.20
80.6548182.1827280.30
90.7440761.8601900.40
100.8143761.6287520.50
110.8720381.4533970.60
120.9233811.3191150.70
130.9589261.1986570.80
140.9865721.0961910.90
151.0000001.0000001.00
\n", "
" ], "text/plain": [ " gains lifts quantiles\n", "0 0.041074 4.107425 0.01\n", "1 0.080569 4.028436 0.02\n", "2 0.120853 4.028436 0.03\n", "3 0.150869 3.771722 0.04\n", "4 0.183254 3.665087 0.05\n", "5 0.341232 3.412322 0.10\n", "6 0.454186 3.027909 0.15\n", "7 0.537915 2.689573 0.20\n", "8 0.654818 2.182728 0.30\n", "9 0.744076 1.860190 0.40\n", "10 0.814376 1.628752 0.50\n", "11 0.872038 1.453397 0.60\n", "12 0.923381 1.319115 0.70\n", "13 0.958926 1.198657 0.80\n", "14 0.986572 1.096191 0.90\n", "15 1.000000 1.000000 1.00" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(experiment.test_gains.dump())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A list of all the attributes of the experiment object are shown below: " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['autoreport_path',\n", " 'clone',\n", " 'dataset_name',\n", " 'deprecated',\n", " 'description',\n", " 'dump',\n", " 'fitted_model_path',\n", " 'ids_col',\n", " 'iteration_data',\n", " 'key',\n", " 'labels',\n", " 'load',\n", " 'log_file_path',\n", " 'max_iterations',\n", " 'mojo_pipeline_path',\n", " 'parameters',\n", " 'patched_pred_contribs',\n", " 'pickle_path',\n", " 'score',\n", " 'scorer',\n", " 'scoring_pipeline_path',\n", " 'summary_path',\n", " 'test_act_vs_pred',\n", " 'test_gains',\n", " 'test_predictions_path',\n", " 'test_roc',\n", " 'test_score',\n", " 'test_score_sd',\n", " 'trace_events',\n", " 'train_predictions_path',\n", " 'training_duration',\n", " 'unfitted_pipeline_path',\n", " 'valid_act_vs_pred',\n", " 'valid_gains',\n", " 'valid_predictions_path',\n", " 'valid_roc',\n", " 'valid_score',\n", " 'valid_score_sd',\n", " 'warnings']" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "[attr for attr in dir(experiment) if not attr.startswith('__')]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Equivalent Steps in Driverless: View Results\n", "![Equivalent Steps in Driverless: View Results](images/experiment_complete_creditcard.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 6. Download Results\n", "\n", "Once an experiment is complete, we can see that the UI presents us options of downloading the: \n", "\n", "* predictions on the (holdout) train data\n", "* predictions on the test data\n", "* experiment summary - summary of the experiment including feature importance\n", "\n", "We will show an example of downloading the test predictions below. Note that equivalent commands can also be run for downloading the train (holdout) predictions. " ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'./test_preds.csv'" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "h2oai.download(src_path=experiment.test_predictions_path, dest_dir=\".\")" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
default payment next month.0default payment next month.1
00.3376450.662355
10.8687400.131260
20.9389290.061071
30.6278140.372186
40.8824890.117511
\n", "
" ], "text/plain": [ " default payment next month.0 default payment next month.1\n", "0 0.337645 0.662355\n", "1 0.868740 0.131260\n", "2 0.938929 0.061071\n", "3 0.627814 0.372186\n", "4 0.882489 0.117511" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test_preds = pd.read_csv(\"./test_preds.csv\")\n", "test_preds.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also download and examine the summary of the experiment and feature importance for the final model." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "# Download Summary\n", "import subprocess\n", "summary_path = h2oai.download(src_path=experiment.summary_path, dest_dir=\".\")\n", "dir_path = \"./h2oai_experiment_summary_\" + experiment.key\n", "_ = subprocess.call(['unzip', '-o', summary_path, '-d', dir_path], shell=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The table below shows the feature name, its relative importance, and a description. Some features will be engineered by Driverless AI and some can be the original feature." ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Relative ImportanceFeatureDescription
01.0000010_PAY_0PAY_0 (original)
10.8057123_NumToCatTE:PAY_0.0Out-of-fold mean of the response grouped by: [...
20.4497511_PAY_2PAY_2 (original)
30.213508_LIMIT_BALLIMIT_BAL (original)
40.2092812_PAY_3PAY_3 (original)
50.189281_BILL_AMT1BILL_AMT1 (original)
60.1724719_PAY_AMT4PAY_AMT4 (original)
70.1713316_PAY_AMT1PAY_AMT1 (original)
80.1527917_PAY_AMT2PAY_AMT2 (original)
90.150512_BILL_AMT2BILL_AMT2 (original)
\n", "
" ], "text/plain": [ " Relative Importance Feature \\\n", "0 1.00000 10_PAY_0 \n", "1 0.80571 23_NumToCatTE:PAY_0.0 \n", "2 0.44975 11_PAY_2 \n", "3 0.21350 8_LIMIT_BAL \n", "4 0.20928 12_PAY_3 \n", "5 0.18928 1_BILL_AMT1 \n", "6 0.17247 19_PAY_AMT4 \n", "7 0.17133 16_PAY_AMT1 \n", "8 0.15279 17_PAY_AMT2 \n", "9 0.15051 2_BILL_AMT2 \n", "\n", " Description \n", "0 PAY_0 (original) \n", "1 Out-of-fold mean of the response grouped by: [... \n", "2 PAY_2 (original) \n", "3 LIMIT_BAL (original) \n", "4 PAY_3 (original) \n", "5 BILL_AMT1 (original) \n", "6 PAY_AMT4 (original) \n", "7 PAY_AMT1 (original) \n", "8 PAY_AMT2 (original) \n", "9 BILL_AMT2 (original) " ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# View Features\n", "features = pd.read_table(dir_path + \"/features.txt\", sep=',', skipinitialspace=True)\n", "features.head(n = 10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7. Launch Experiment with Config Overrides\n", "\n", "We will launch an additional experiment, this time we will override the default configurations. For this experiment, we will only use the GLM algorithm and we blacklist all target encoding feature transformers." ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ "experiment_glm = h2oai.start_experiment_sync(dataset_key=train.key, \n", " testset_key=test.key,\n", " target_col=target,\n", " is_classification=True,\n", " accuracy=6,\n", " time=3,\n", " interpretability=6,\n", " scorer=\"AUC\",\n", " seed=1234,\n", " config_overrides='enable_glm=\"on\"\\n'\n", " 'enable_xgboost=\"off\"\\n'\n", " 'enable_tensorflow=\"off\"\\n'\n", " 'enable_rulefit=\"off\"\\n'\n", " 'enable_lightgbm=\"off\"\\n'\n", " 'black_list_transformers=[\\\"NumCatTETransformer\\\", \\\"CVTargetEncodeF\\\", \\\"NumToCatTETransformer\\\", \\\"ClusterTETransformer\\\"]'\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can compare the performance between the two experiments:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
experimenttest_score
0default0.799748
1glm only0.753666
\n", "
" ], "text/plain": [ " experiment test_score\n", "0 default 0.799748\n", "1 glm only 0.753666" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame({'experiment': [\"default\", \"glm only\"], \n", " 'test_score': [experiment.test_score, experiment_glm.test_score]})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build an Experiment in Web UI and Access Through Python\n", "\n", "It is also possible to use the Python API to examine an experiment that was started through the Web UI using the experiment key.\n", "\n", "![Experiments List](images/experiment_list_complete.png)\n", "\n", "### 1. Get pointer to experiment\n", "\n", "You can get a pointer to the experiment by referencing the experiment key in the Web UI." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['bofuseke', 'pacoganu', 'fokahohu', 'ticipucu']" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get list of experiments\n", "experiment_list = list(map(lambda x: x.key, h2oai.list_models(offset=0, limit=100)))\n", "experiment_list" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Get pointer to experiment\n", "experiment = h2oai.get_model_job(experiment_list[0]).entity" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Score on New Data\n", "\n", "You can use the python API to score on new data. This is equivalent to the **SCORE ON ANOTHER DATASET** button in the Web UI. The example below scores on the test data and then downloads the predictions.\n", "\n", "Pass in any dataset that has the same columns as the original training set. If you passed a test set during the H2OAI model building step, the predictions already exist. Its path can be found with `experiment.test_predictions_path`.\n", "\n", "### 1. Score Using the H2OAI Model\n", "\n", "The following shows the predicted probability of default for each record in the test. " ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
default payment next month.0default payment next month.1
00.5508740.449126
10.8342740.165726
20.8254030.174597
30.5620740.437926
40.8390180.160982
\n", "
" ], "text/plain": [ " default payment next month.0 default payment next month.1\n", "0 0.550874 0.449126\n", "1 0.834274 0.165726\n", "2 0.825403 0.174597\n", "3 0.562074 0.437926\n", "4 0.839018 0.160982" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prediction = h2oai.make_prediction_sync(experiment.key, test_path, output_margin = False, pred_contribs = False)\n", "pred_path = h2oai.download(prediction.predictions_csv_path, '.')\n", "pred_table = pd.read_csv(pred_path)\n", "pred_table.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also get the contribution each feature had to the final prediction by setting `pred_contribs = True`. This will give us an idea of how each feature effects the predictions using Shapley." ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
contrib_0_NumToCatWoE:PAY_0:PAY_2.0contrib_1_NumToCatWoE:PAY_0:PAY_6.0contrib_2_NumToCatWoE:BILL_AMT3:PAY_0:PAY_2:PAY_3.0contrib_3_NumToCatWoE:BILL_AMT3:PAY_0:PAY_2:PAY_3.0contrib_4_NumToCatWoE:PAY_0:PAY_2:PAY_3.0contrib_5_NumToCatWoE:PAY_0:PAY_2:PAY_3.0contrib_6_PAY_2contrib_7_PAY_0contrib_8_NumToCatWoE:PAY_0:PAY_2:PAY_6.0contrib_bias
00.0923780.1314960.1643080.1758020.1664870.1437050.00.0005240.120301-1.199351
1-0.029177-0.051891-0.080473-0.086486-0.065098-0.0562330.00.000000-0.047549-1.199351
2-0.029177-0.051891-0.047550-0.056703-0.065098-0.0562330.00.000000-0.047549-1.199351
30.0842170.1246200.1504460.1997870.1470460.1034720.00.0013100.138215-1.199351
4-0.029177-0.048915-0.078790-0.090655-0.084294-0.0728790.00.000000-0.047054-1.199351
\n", "
" ], "text/plain": [ " contrib_0_NumToCatWoE:PAY_0:PAY_2.0 contrib_1_NumToCatWoE:PAY_0:PAY_6.0 \\\n", "0 0.092378 0.131496 \n", "1 -0.029177 -0.051891 \n", "2 -0.029177 -0.051891 \n", "3 0.084217 0.124620 \n", "4 -0.029177 -0.048915 \n", "\n", " contrib_2_NumToCatWoE:BILL_AMT3:PAY_0:PAY_2:PAY_3.0 \\\n", "0 0.164308 \n", "1 -0.080473 \n", "2 -0.047550 \n", "3 0.150446 \n", "4 -0.078790 \n", "\n", " contrib_3_NumToCatWoE:BILL_AMT3:PAY_0:PAY_2:PAY_3.0 \\\n", "0 0.175802 \n", "1 -0.086486 \n", "2 -0.056703 \n", "3 0.199787 \n", "4 -0.090655 \n", "\n", " contrib_4_NumToCatWoE:PAY_0:PAY_2:PAY_3.0 \\\n", "0 0.166487 \n", "1 -0.065098 \n", "2 -0.065098 \n", "3 0.147046 \n", "4 -0.084294 \n", "\n", " contrib_5_NumToCatWoE:PAY_0:PAY_2:PAY_3.0 contrib_6_PAY_2 \\\n", "0 0.143705 0.0 \n", "1 -0.056233 0.0 \n", "2 -0.056233 0.0 \n", "3 0.103472 0.0 \n", "4 -0.072879 0.0 \n", "\n", " contrib_7_PAY_0 contrib_8_NumToCatWoE:PAY_0:PAY_2:PAY_6.0 contrib_bias \n", "0 0.000524 0.120301 -1.199351 \n", "1 0.000000 -0.047549 -1.199351 \n", "2 0.000000 -0.047549 -1.199351 \n", "3 0.001310 0.138215 -1.199351 \n", "4 0.000000 -0.047054 -1.199351 " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "prediction_contributions = h2oai.make_prediction_sync(experiment.key, test_path, \n", " output_margin = False, pred_contribs = True)\n", "pred_contributions_path = h2oai.download(prediction_contributions.predictions_csv_path, '.')\n", "pred_contributions_table = pd.read_csv(pred_contributions_path)\n", "pred_contributions_table.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will examine the contributions for the first customer more closely." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAIyCAYAAADLxr0mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmYXGWZ/vHvnbDKIltkCypoFBlUlrCIOqgIAjqEcUFQ\nAcERcUQERQcdR53R34iMihvC4MgALiwqDFGjbIICghAW2RQJYY0sERRkkS3374/zNnVS6a6uDp06\n1an7c1119Vmrnjr91HnOec8m20RERExqOoCIiOgPKQgREQGkIERERJGCEBERQApCREQUKQgREQGk\nIEw4kpaX9JCk9ZqOZWkmaWNJTy6B9327pHnlf/iS8X7/Z0LSiyT9pYvpDpR0bi9iit5KQRiD8iMe\nei2Q9Git/53j/FmXSvpb22dubvsx2yvb/uM4f97O5Ts9JOmvkn433t9pSZM0SdJHJN0g6WFJd0g6\nRdImTcdW82Vg//I//F0vP7isyJ8s/+MHJV0p6Q1D423/wfZqPY5pqqQTJd1TYrpB0r9JWuEZvOcK\nkixp6njGOghSEMag/IhXtr0ycDvwD7Vh31sCH/lP9c+0fdUS+Iy6ueW7rQp8HDhB0oZL+DPH07HA\n+4D3A6sDGwOzgF2aDGqIpGWB9YDrF3P+yeMQxgXlf7wacCLwA0krjcP7jpmk5wCXAga2sr0qsCuw\nDvC8JmIab5KWaTqGsUhBGEeSVpR0tKS7JN0p6b/KSmBoC3yOpH+XdL+kWyS9bTE+4+mtH0nbS7pd\nkmrj95J0WemeXLa25kr6k6TvSRp1C9CV/wMeBTatvfemkn4h6c9lD2L32rh/lPTbspV3u6RP1Mat\nVLbU75f0F0m/kbR6GfdcSbPKuD9I2rc23xEl5pPLXss1kjYbYblsCrwHeJvtX9p+3PbDtk+y/aUy\nzRqSvi9pfln+HxtadpKWkfRVSfdJmgPs2Pb+a0g6SdLdZc/j05ImlXEbS7pI0gPlvU8aJr5nA38u\nvTdKur4Mf6mkC8tyuUbSLrV5TpH0NUlnS3oYeMUw7/s+Sb8vy2eOpP2H/68uzPYC4DvAKsBGte/x\ndDOZpPdKurW899zh8lWVr0s6X9Kzy/9/Wm38VEmPjJB3HwPuBvazfXuJ61bbH7B9Y3s85f0ulfSu\nWrzDLfdflb83qtob2r1M/wFJN5f/8emS1i7Dh35TB5bxD0r6pKQXS7qsvP/3VFu5l3y/pvzfLlRt\nL7TkyGHlf/xgN/+PvmE7r8V4AbcCr28bdiRwIbAWsDZwOfCvZdzOwJPA54HlgNcDjwAbjvD+lwLv\nGmb4ClRbVFOpCvqdwKtr438MHFK6/6XEs16Z7wTgf0f4vJ2BOaV7EvC28jmblGGrAncB7wQmA1sB\n9wMvLON3AP6uzLtFGbdzGfch4IfAisAyZd6VyrjfAEcBywPTy3yvLOOOKMtox/KZR1Ft4Q4X/yHA\njaP8z04DfgCsDLwQuAV4Z23+a8uymgJcBDxZm/dnwNeBZwHrAlcB+5ZxZwCHASrf8ZUjfP7T/7ta\n/+3AR4BlgTcADw3lBHBKWR7blOW6/DDvuRuwYfns11MV8b8b4fMPBM4t3cuUz30UWL0M23joO1Pt\nYf0FeEHpXw94Sf19ynucRJVzK5RxxwP/XvvMfwF+MEI8VwMf7/D/ejqe4X4XIy339uVchu1KVXxe\nVsYfB5zdNv1QbmwOPAGcRbWnsgZwE/D2Mv22VL+FLany8gDgD8AyZfzdVL/99YAVm15XjeXVeAAT\n9cXwBWEe8Lpa/wzg96V7Z+BvQz+cMmwm8NER3v9S4OHyo/wL8OsyvH2l8kXgm6V7jfIDX6f030Jt\n5VRWHI8AGubzdgaeKp/1GFXxen9t/L7AOW3znAj8ywjxHwt8vnT/M/BLYNO2aaaVZbJibdhRwLGl\n+wjgJ7VxWwB/GeHzPssIxaKMX758v41qwz4E/Lx0/xp4d23cbrRWjs8r/4tla+P3A35Wuk8DvgGs\nO0rOtP/vdgRuq/8/qFZyh5fuU4DjxpiXPwfeN8K4A6lWdH8pfx8Gdq+NH64gzKjnbO19LiyxnkxZ\nEZZx21M2LEr/tcBuI8RzR32ZDzN+tIIw7HJvX85l2PeA/6j1rwYsoGqeGpp+y9r464EP1fqPBo4o\n3f9L2dCrjb8N2KZ03w28Yyz/t355pclonJSmh3WoEmPIbcD6tf75tv/WNr7T2ULvs71aeW03wjTf\nB95WdmffBlxk++4SzwbArLJb+xeqrdpJwJojvNctrg4qPptqC+p1tXHPA/5+6L3K+72FamsZSa+U\n9Muy6/4A8G6qPSWAb1MVhB+qakr7T1Xt4euVZfJo2zKpL7O7a92PUG3BDee+oVhGsE757reP8Fnr\nUa2g6uOGPI9qpTG/9t2/SrUXCHAo1Z7DVaUZ4V0d4qhbD7jdZS0yTEy0xbQISbuVZo37S1yvo7Xc\nh/PL8j9eEzgbeOVwE9n+M9Xe4MHA3ZJmSnphbZKXUG1E/IfterPOr4DJkl5RmvfWpdq7Gs5o/7PR\njGW5r0ftf2r7L1TNOfVlfU+t+9Fh+ody73nAJ9p+C1MYw/+tX6UgjJPyo76bhQ+GPZdqr2HIWlr4\n7InnAs/obCHbV1L9sF4PvIOqQAzFM7THslrttYLtP43ynn8DPgy8QtLOZfAdVLvY9fda2fYhZfxp\nwKnABrafTdU8pfJ+j9n+lO2Ngb+nKlx7lu8+RdKKbcukvsy6dS7wQkkvG2H83VRbhM8d4bPuoiqg\n9XFD7qBqylm99t1Xtb1F+X7zbO9PtXI7GDheUn3+kfyx7XPaY4Jqy3VYqg4G/4Bq7+g5ZUX/C8py\n78T2g1Rb+gdqhLOwbP/U9g6UwgUcUxt9NdXB+7MkbVSbx1TNSO8C9gZOsf3ECGGcC7y5Q5gPUxWX\n5WvD1ql91kjLfbhl9kdqv81yTGNVFi/X7gA+1fZbeJbt02vTTMjbSKcgjK+TgU9LWlPVGRT/Cny3\nNn5Z4N8kLSfpdVRNBj8ap8/9KFXbfD0pjwWOkLQBVGd1SPqHbt6wFIWvAJ8qg/4P2FzVefTLlu+w\nrapz10W19XSf7b9J2o5qpU/53NdL2kTVQdgHqZqjFgBzqJoUPqfq+ootqJqm6susK7avo2q/Pk3S\nq0t8K0p6p6SP2H6MqonjP1Ud5H4BVZPR0GedBhwqaV1Ja1Ed8Bx671uomiqOlLSKqtNbp0l6Vfl+\nb5e0XlkZDp3H/1QXYV8ITJJ0iKqD2jsCO5VYurEiVU7dCyyQtBvwmi7nxfY9VM1+/9Y+TtL6kt4o\n6VlUTYgPUf3P6vOfAHwO+IWk+obQScAewF6leyRHAutK+nYtRzdQdZD6xVQr8fnAO1WdIPHP1LbC\nR1ru5X/9AOVgeXEy8F5VJ0asQNUc+Qvb9T3Qbh0HfFDSdFVWLntqz1qM9+orKQjj61PADVTtj1cD\nF1Ml/ZBbqVaGd1OtvPazPXccPvf7VE0FP7P9QG34kVRbYb+Q9FeqdvItxvC+xwGbSNqxNCG8gart\n/C6qH+vnqNrVTbW1+cXyOR+j2nIdsj5wJvBX4DqqU0FPLfO9DdiEapmcSnVM5aKxfPma9wH/U+L+\nM9WBwDcCP62Nh6rp4Bdl2qHThb9BtYK+nupAd/tKeS+qduffUx3oPZVWk9ErgCskPVS+9wG2R93y\nLEX3TcBbqfbyvkx14LKrnCh7eodRHdS9D9idatmOxZeBt5QVcN1k4HCq/8t9VBsbBw0Tw3HAl6hy\nbGoZdjNwI/BX25d1iP9eqmW3LNXy+yvVgdy7gdtsPwX8E/Bp4E9Ue3BX1N6i03L/FNUptX+RtJvt\nn1Cd0DGTKnfXodqDGTPbF1Ptkfw3VSH6A9Xe+YTcK6jTws2XsaSUppdv2H7hqBNHTHCSvg/cYPtz\nTccS3cseQkSMq3Lw+U1UZ+PEBJKCEBHjRtKRVGez/Uc3zWbRX9JkFBERQPYQIiKiSEGIiAggBSEi\nIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGI\niAggBSEiIooUhIiIAFIQIiKiWKbpAMZirbXW8vOf//ymw4iImFCuuOKKP9meMtp0E6ogPP/5z2f2\n7NlNhxERMaFIuq2b6dJkFBERQApCREQUKQgREQGkIERERJGCEBERQJcFQdLOkm6UNEfS4cOM31jS\nJZIek3RYbfiLJV1dez0o6ZAy7jOS5tXG7Tp+XysiIsZq1NNOJU0GjgZ2BO4ELpc00/YNtcnuBw4G\ndq/Pa/tGYLPa+8wDzqhNcpTtLz6jbxAREeOimz2ErYE5tufafhw4BZhRn8D2vbYvB57o8D47ADfb\n7up82IiI6K1uLkxbH7ij1n8nsM1ifNaewMltwz4oaR9gNvAR239ejPcdk+cf/tMl/RGjuvWINzYd\nQkTEInpyUFnScsBuwA9qg48BNqJqUroL+NII8x4gabak2fPnz1/isUZEDKpuCsI8YINa/9QybCx2\nAa60fc/QANv32H7K9gLgW1RNU4uwfZzt6banT5ky6q04IiJiMXVTEC4HpknasGzp7wnMHOPn7EVb\nc5GkdWu9/whcN8b3jIiIcTTqMQTbT0o6CDgLmAwcb/t6SQeW8cdKWofqOMCqwIJyaukmth+UtBLV\nGUrva3vrIyVtBhi4dZjxERHRQ13d7dT2LGBW27Bja913UzUlDTfvw8Cawwzfe0yRRkTEEpUrlSMi\nAkhBiIiIIgUhIiKAFISIiChSECIiAphgz1SO8ZXbeEREXfYQIiICyB5CBJC9pQjIHkJERBQpCBER\nAaQgREREkYIQERFACkJERBQpCBERAaQgREREkYIQERFACkJERBQpCBERAaQgREREkYIQERFACkJE\nRBRdFQRJO0u6UdIcSYcPM35jSZdIekzSYW3jbpV0raSrJc2uDV9D0jmSbip/V3/mXyciIhbXqAVB\n0mTgaGAXYBNgL0mbtE12P3Aw8MUR3ua1tjezPb027HDgPNvTgPNKf0RENKSbPYStgTm259p+HDgF\nmFGfwPa9ti8HnhjDZ88ATizdJwK7j2HeiIgYZ90UhPWBO2r9d5Zh3TJwrqQrJB1QG7627btK993A\n2sPNLOkASbMlzZ4/f/4YPjYiIsaiFweVX2V7M6ompw9I+vv2CWybqnAswvZxtqfbnj5lypQlHGpE\nxODqpiDMAzao9U8tw7pie175ey9wBlUTFMA9ktYFKH/v7fY9IyJi/HVTEC4HpknaUNJywJ7AzG7e\nXNJKklYZ6gZ2Aq4ro2cC+5bufYEzxxJ4RESMr2VGm8D2k5IOAs4CJgPH275e0oFl/LGS1gFmA6sC\nCyQdQnVG0lrAGZKGPuv7tn9e3voI4DRJ7wFuA/YY368WERFjMWpBALA9C5jVNuzYWvfdVE1J7R4E\nXj7Ce94H7NB1pBERsUTlSuWIiABSECIiouiqySgiBsfzD/9p0yFw6xFvbDqEgZQ9hIiIAFIQIiKi\nSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiI\nAHK304iIEQ3anV+zhxAREUAKQkREFCkIEREBdFkQJO0s6UZJcyQdPsz4jSVdIukxSYfVhm8g6XxJ\nN0i6XtKHauM+I2mepKvLa9fx+UoREbE4Rj2oLGkycDSwI3AncLmkmbZvqE12P3AwsHvb7E8CH7F9\npaRVgCsknVOb9yjbX3zG3yIiIp6xbvYQtgbm2J5r+3HgFGBGfQLb99q+HHiibfhdtq8s3X8Ffges\nPy6RR0TEuOqmIKwP3FHrv5PFWKlLej6wOfCb2uAPSrpG0vGSVh/re0ZExPjpyUFlSSsDPwIOsf1g\nGXwMsBGwGXAX8KUR5j1A0mxJs+fPn9+LcCMiBlI3BWEesEGtf2oZ1hVJy1IVg+/ZPn1ouO17bD9l\newHwLaqmqUXYPs72dNvTp0yZ0u3HRkTEGHVTEC4HpknaUNJywJ7AzG7eXJKAbwO/s/3ltnHr1nr/\nEbiuu5AjImJJGPUsI9tPSjoIOAuYDBxv+3pJB5bxx0paB5gNrAoskHQIsAnwMmBv4FpJV5e3/ITt\nWcCRkjYDDNwKvG98v1pERIxFV/cyKivwWW3Djq11303VlNTuIkAjvOfe3YcZERFLWq5UjogIIAUh\nIiKKFISIiABSECIiokhBiIgIIAUhIiKKFISIiABSECIiokhBiIgIIAUhIiKKFISIiABSECIiokhB\niIgIIAUhIiKKFISIiABSECIiokhBiIgIIAUhIiKKFISIiABSECIiokhBiIgIoMuCIGlnSTdKmiPp\n8GHGbyzpEkmPSTqsm3klrSHpHEk3lb+rP/OvExERi2vUgiBpMnA0sAuwCbCXpE3aJrsfOBj44hjm\nPRw4z/Y04LzSHxERDelmD2FrYI7tubYfB04BZtQnsH2v7cuBJ8Yw7wzgxNJ9IrD7Yn6HiIgYB90U\nhPWBO2r9d5Zh3eg079q27yrddwNrD/cGkg6QNFvS7Pnz53f5sRERMVZ9cVDZtgGPMO4429NtT58y\nZUqPI4uIGBzdFIR5wAa1/qllWDc6zXuPpHUByt97u3zPiIhYAropCJcD0yRtKGk5YE9gZpfv32ne\nmcC+pXtf4Mzuw46IiPG2zGgT2H5S0kHAWcBk4Hjb10s6sIw/VtI6wGxgVWCBpEOATWw/ONy85a2P\nAE6T9B7gNmCP8f5yERHRvVELAoDtWcCstmHH1rrvpmoO6mreMvw+YIexBBsREUtOXxxUjoiI5qUg\nREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAAp\nCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERABdFgRJO0u6UdIcSYcPM16S\nvlbGXyNpizL8xZKurr0elHRIGfcZSfNq43Yd368WERFjscxoE0iaDBwN7AjcCVwuaabtG2qT7QJM\nK69tgGOAbWzfCGxWe595wBm1+Y6y/cXx+CIREfHMdLOHsDUwx/Zc248DpwAz2qaZAZzkyqXAapLW\nbZtmB+Bm27c946gjImLcdVMQ1gfuqPXfWYaNdZo9gZPbhn2wNDEdL2n1LmKJiIglpCcHlSUtB+wG\n/KA2+BhgI6ompbuAL40w7wGSZkuaPX/+/CUea0TEoOqmIMwDNqj1Ty3DxjLNLsCVtu8ZGmD7HttP\n2V4AfIuqaWoRto+zPd329ClTpnQRbkRELI5uCsLlwDRJG5Yt/T2BmW3TzAT2KWcbbQs8YPuu2vi9\naGsuajvG8I/AdWOOPiIixs2oZxnZflLSQcBZwGTgeNvXSzqwjD8WmAXsCswBHgH2G5pf0kpUZyi9\nr+2tj5S0GWDg1mHGR0RED41aEABsz6Ja6deHHVvrNvCBEeZ9GFhzmOF7jynSiIhYonKlckREACkI\nERFRpCBERASQghAREUUKQkREACkIERFRpCBERASQghAREUUKQkREACkIERFRpCBERASQghAREUUK\nQkREACkIERFRpCBERASQghAREUUKQkREACkIERFRpCBERASQghAREUUKQkREAF0WBEk7S7pR0hxJ\nhw8zXpK+VsZfI2mL2rhbJV0r6WpJs2vD15B0jqSbyt/Vx+crRUTE4hi1IEiaDBwN7AJsAuwlaZO2\nyXYBppXXAcAxbeNfa3sz29Nrww4HzrM9DTiv9EdEREO62UPYGphje67tx4FTgBlt08wATnLlUmA1\nSeuO8r4zgBNL94nA7mOIOyIixlk3BWF94I5a/51lWLfTGDhX0hWSDqhNs7btu0r33cDaw324pAMk\nzZY0e/78+V2EGxERi6MXB5VfZXszqmalD0j6+/YJbJuqcCzC9nG2p9uePmXKlCUcakTE4OqmIMwD\nNqj1Ty3DuprG9tDfe4EzqJqgAO4ZalYqf+8da/ARETF+uikIlwPTJG0oaTlgT2Bm2zQzgX3K2Ubb\nAg/YvkvSSpJWAZC0ErATcF1tnn1L977Amc/wu0RExDOwzGgT2H5S0kHAWcBk4Hjb10s6sIw/FpgF\n7ArMAR4B9iuzrw2cIWnos75v++dl3BHAaZLeA9wG7DFu3yoiIsZs1IIAYHsW1Uq/PuzYWreBDwwz\n31zg5SO8533ADmMJNiIilpxcqRwREUAKQkREFCkIEREBpCBERESRghAREUAKQkREFCkIEREBpCBE\nRESRghAREUAKQkREFCkIEREBpCBERESRghAREUAKQkREFCkIEREBpCBERESRghAREUAKQkREFCkI\nEREBpCBERETRVUGQtLOkGyXNkXT4MOMl6Wtl/DWStijDN5B0vqQbJF0v6UO1eT4jaZ6kq8tr1/H7\nWhERMVbLjDaBpMnA0cCOwJ3A5ZJm2r6hNtkuwLTy2gY4pvx9EviI7SslrQJcIemc2rxH2f7i+H2d\niIhYXN3sIWwNzLE91/bjwCnAjLZpZgAnuXIpsJqkdW3fZftKANt/BX4HrD+O8UdExDjppiCsD9xR\n67+TRVfqo04j6fnA5sBvaoM/WJqYjpe0+nAfLukASbMlzZ4/f34X4UZExOLoyUFlSSsDPwIOsf1g\nGXwMsBGwGXAX8KXh5rV9nO3ptqdPmTKlF+FGRAykbgrCPGCDWv/UMqyraSQtS1UMvmf79KEJbN9j\n+ynbC4BvUTVNRUREQ7opCJcD0yRtKGk5YE9gZts0M4F9ytlG2wIP2L5LkoBvA7+z/eX6DJLWrfX+\nI3DdYn+LiIh4xkY9y8j2k5IOAs4CJgPH275e0oFl/LHALGBXYA7wCLBfmf2VwN7AtZKuLsM+YXsW\ncKSkzQADtwLvG7dvFRERYzZqQQAoK/BZbcOOrXUb+MAw810EaIT33HtMkUZExBKVK5UjIgJIQYiI\niCIFISIigBSEiIgoUhAiIgJIQYiIiCIFISIigBSEiIgoUhAiIgJIQYiIiCIFISIigBSEiIgoUhAi\nIgJIQYiIiCIFISIigBSEiIgoUhAiIgJIQYiIiCIFISIigBSEiIgoUhAiIgLosiBI2lnSjZLmSDp8\nmPGS9LUy/hpJW4w2r6Q1JJ0j6abyd/Xx+UoREbE4Ri0IkiYDRwO7AJsAe0napG2yXYBp5XUAcEwX\n8x4OnGd7GnBe6Y+IiIZ0s4ewNTDH9lzbjwOnADPappkBnOTKpcBqktYdZd4ZwIml+0Rg92f4XSIi\n4hlYpotp1gfuqPXfCWzTxTTrjzLv2rbvKt13A2sP9+GSDqDa6wB4SNKNXcS8JK0F/OmZvIG+ME6R\nNC/LoiXLoiXLoqVflsXzupmom4KwxNm2JI8w7jjguB6HNCJJs21PbzqOfpBl0ZJl0ZJl0TLRlkU3\nTUbzgA1q/VPLsG6m6TTvPaVZifL33u7DjoiI8dZNQbgcmCZpQ0nLAXsCM9ummQnsU8422hZ4oDQH\ndZp3JrBv6d4XOPMZfpeIiHgGRm0ysv2kpIOAs4DJwPG2r5d0YBl/LDAL2BWYAzwC7Ndp3vLWRwCn\nSXoPcBuwx7h+syWnb5qv+kCWRUuWRUuWRcuEWhayh226j4iIAZMrlSMiAkhBiIiIIgUhIiKAFISu\nSVpV0qpNxxH9JXkRS5McVO5A0lSqs6HeADwECHgWcDbwCdu3NxheIyStSXUFOsA82/c1GU8TkheL\nSl5UJO1AdRuep5cFcKbtc5uLqnspCB1Iuhj4JnCa7SfKsGWBtwP/bHu7JuPrJUkvo1oWa9O6uHAq\n1W1HPmD7t03F1mvJi5bkRYukLwGbAt+huk0PVMtib+B62x9uKrZupSB0IOmmcjfWMY1bGkm6muoH\nfnHb8FcBR9t+eTOR9V7yoiV50SLpD7ZfNMxwAX+YCHmRYwidXV2e87ClpOeU15aSvg4MzJZPsVL7\njx7A9kXASg3E06TkRUvyouWx+rNgarYAHut1MIsjewgdSFqe6k6rM2i1Cd4J/Bg4zvbfmoqt1yR9\nk2oZnETrDrYbAPsAf7T9/qZi67XkRUvyokXSVlTPglme1rJ4LvA3qqbEy5qKrVspCNE1Sf/AwivB\neVT3pPqxk0gDK3mxsHLSQf0A+52dpu8nKQiLSdLOtn/edBzRX5IXMZHlGMLie1XTAfQLSfs3HUMf\nSV4UyYsWSX3fXAQpCIvN9iebjqGPrNB0AP0iebGQ5EXLtk0H0I00GY1C0lpUy2m+pDWotgD/YPv3\nDYcWDUpexGgkrQy8ALjF9oNNx9ON7CF0IOmfgCuA2eXZzj8D3gLMlPTuJmNrgqQtJG1eul8s6WBJ\nOzUdV68lLxaWvKhIOqFsKCBpR+B3wFeB6yS9udHgupQ9hA4kXQNsB6wI3AJMs31XuUz/PNubNRpg\nD0n6JNWZJMsAP6HaIv4V8Hqqs0mOaDC8nkpetCQvWiRda/ulpftiYG/bcyU9Bzh7IuTFqE9MG3BP\n2n4IeEjSzeWxoNi+T9KgVdI9gZdTtQvfBWxg+wFJXwAupbq3z6BIXrQkL1omSVrF9l8BU20sYPve\ncmuTvpeC0JklLVvuV7Pb0MByYdLk5sJqxOO2nwIeLivBBwBsPyLpqYZj67XkRUvyouVzwC/KFesX\nAadImgm8Fjin0ci6lILQ2VuGOmzfVhu+FvDR3ofTqCckrWj7UWDroYEDeuvn5EVL8qKwfbKkK6mu\nYn8R1fr1NcAZtn/aZGzdyjGEcSDpIttL9fnnkp4FPNp+5amkKcB6Q3e1lLTqRDmjYklLXiQvhiPp\nY7aPbDqO4eQso/Gx1N/Ey/Yjw92GwPb8tlscX9C7qPpe8qLlgt5F1ff2bDqAkaQgjI/sZrWo6QD6\nSPKiJXnR0rfLIgUhxltWgjGc5EVL3y6LFITx0bcVPxqVvIjh9G1epCB0IOnr5fLz0bx7SccygfRt\nso+X5MViGYS82KrLSU9fooE8AykInf0RuErSHp0mGoTnxkr6saTndjHpINyyIHlRJC8W8j+Sjh7t\nlFvbn+1VQGOV005HUZL9KGBlqqchLRgaZ3tmU3H1mqS9gP8A/gf4YrkYaWAlLyrJixZJk4FDqa5D\n+LTtkxuiiDoeAAAgAElEQVQOacxSELog6R3Af1GdOjf0w7ftfRoLqgGSVgE+A7wOOJGFV4Jfayis\nxiQvKsmLhUnaBLiE6uDxAqrmMtteo9HAupArlTuQtDHwTeB+YJuJ9Ci8JeRR4D7gWcAUaj/8QZK8\nWETyopC0L/BJ4NPA0UywZZGC0NmZwKG2ZzUdSNMk7UB1K9+fA1uWm7sNquRFkbxokXQh1Q3+trf9\nx6bjWRxpMuqgdo+W9uHbAnvZ/lADYTVC0q+B9w/CgdLRJC9akhctnZ6nLWlz21f1Oqaxyh5CB/Uf\nvaSXAu8A9qA6y+RHTcXVBNvbtQ+TtCKwO7Cn7Rm9j6oZyYuW5EVLezGQ9CJgL6r8eBTI8xAmMkkb\nUf1D9wIeAk4FlrX96kYDa5CkZYBdqJJ8F+DHwAlNxtRryYtFJS8qkqbSyo1JwAZUx5nmNBpYl9Jk\n1IGkBcCFwHtt/6EMm2t7o2Yj6z1Jr6NK8l2plsmpwFdsP6/RwBqQvGhJXrSUYwhTqJbBKbZ/J+kW\n2xs2HFrXcmFaZ3sA84FzJX1T0vYMwBWXIziX6h7vr7S9p+0zgEE95zx50ZK8aHmA6rGqzwZWKcMm\n1BZ3CkIHtn9o+63A31E9DvBwYO1y64LXNRtdz20NXA6cL+ln5fS6QXs6GJC8aJO8KGy/ieo4wfXA\nEZLmAKtL2qLZyLqXJqMxkrQW1Rbi221v33Q8vSZJwKupmgneDFxG9USo4xsNrGHJi+RFO0nrAm+n\nWibrTIRmtBSEDsozct8LvBC4FjhhkC/Nb1cOJO5EdTbJwFydm7zobFDzAkDSGsBzgbn1J8RJ2sj2\n3OYi604KQgeSTqZqG74Q2BmYY/vQZqNqRtkCPpzWSvALg3oRUvKiJXnRImk/4EjgVqqi8B7bP2k0\nqDFKQehA0rW2X1q6lwV+Y3vCtAeOJ0mzqH7wvwLeBKxge79mo2pG8qIledEi6XrgdbbvkfRC4Du2\nX9F0XGOR6xA6e2Kow/YTVTPpwFrf9q7w9ErgyobjaVLyoiV50fKY7XsAbM8pTYsTSgpCZy+XdH/p\nFrBK6Z8wdy8cT+WulkNrv0n1/np76QBIXtQkL542VdKXR+q3/eEGYhqTNBl1UO5vPqJBOpAo6U5a\nt/IdYlorwW4ekrJUSF60JC9aJL2n03jb3+5VLIsrBSEiIoBcmNY1Sf/XqT8GU/IihiNp/079/SoF\noXsHjdI/MCRd1ql/wCQviuTFQlYYpb8vpckoxkzSJNsLRuqPwZS8mPhSEDooZ0t8DJgK/Mz2abVx\nX7f9wcaCi8YkL2Ik5fqDGcD6ZdA8YKbtm5qLqntpMurseKq7F/4UeLekU8uFSACvbC6s3pO0vqTv\nSjpf0sfK7QmGxg3UQ2FIXjwtedEi6TDgdKrcuKa8VgR+VMb1vewhdCDpatub1fo/Dbwe2A04b5Cu\nTpV0FtVDTy4F3gNsCuxm+8+SrrK9eaMB9lDyoiV50SLpD8Cmth9vG748cJ3tac1E1r1cmNbZCvV2\nUNv/Xs67/hWt+50PiufY/kbpni3p3cCvJO3GBLvn+zhIXrQkL1oWAM8B7mwb/pwyru+lIHT2U2AH\n4JyhAba/Lelu4BsjzrV0Wl7S8rYfA7B9QlkO5wDPaja0nktetCQvWj4M/FLSDcAdZdhzgZcABzcW\n1RikyWgcSHqX7e82HceSJOmjwOW2L2gbPh34L9uvbSSwPpa8GLy8KMdQtmXhg8qX2n6yNs2q/XpL\njxSEcSDpykFqN+5E0sdsH9l0HP0gedGSvGjp57zIWUbjY6Bvd9lmz6YD6CPJi5bkRUvf5kUKwvjI\nblZL3yZ7A5IXLcmLlr7NixSE8ZFkb+nbZG9A8qIleTEBpCB0IGmDLie9dIkGMrEs9SvB5MViWerz\nYgz6dlmkIHR2gaTDurj//ft7FVBTJG3V5aSnL9FA+kPyokhetEj6saRunv+w0xIPZjGlIHS2BfA8\nqgtutms6mIb9j6SjJa3aaSLbn+1VQA1KXrQkL1q+D5wn6V86bSzYnt/DmMYkp512oWwFnQPcSuvp\nUO7XU8eWhJLghwIHAJ+2fXLDITUueZG8aFdufPgZ4HXAidSuULb9tYbC6loKwigk/T3wdeB84GgW\n/gff3FRcTZG0CXAJ1UHC+kpw0J4jnLyoSV5UyoVpHwP2BX7Iwnnxb03F1a3cuqIDSd8FXgDsa/vq\npuNpmqR9gU8Cn6ZtJThIkhcLS15UJO0AfBX4ObCl7YcaDmnMUhA6uwjY28PsRkma0s9tgeNN0oXA\nXcD2tv/YdDwNS14UyYuFfBZ4p+3fNh3I4kqT0RiU9sE3A+8AXmZ73YZD6hlJO9v++QjjNrd9Va9j\n6hfJi+TFSCStCOwO7Gl7RtPxjCYFYRTlXub/QPVj35rq9sZvAc63/VSTsTVJ0ouAvaiWy6P15wMM\nguTF8AY9L+Dp4wi7UC2DXaieF3G67TMaDawLKQgdSDoJeC3wC+AU4FzgD7Y3bDSwhkiaSvVj34vq\nlOUNgG1sz2k0sB5LXiwseVGR9DqqZbArcCFwKvAV289rNLAxyHUInW0B/Am4Crja9hMM6CX4pa34\nXGBlqnbSzYAHB+1HXyQviuTFQs4FXgS80vaeZY9gQu0tpiB0YHtTYG9gCtVToM4HVpG0VrORNeIB\nqufDPpvWU8EGciWYvFhI8qJla+By4HxJPytnX3W8mr3fpMloDCRtQ7VL+FZgru2/bziknpK0OtV3\n34vqSVBrAjvYvrLRwBqWvEhe1EkS8Gqq5fFm4DLgDNvHNxpYF1IQuiBpNdt/qfULeK3tXzQYVqMk\nrQu8nSrp15lI7aTjJXmxqOTFwsoB5p2ozjLap+l4RpOC0IGkXYETqK66fBTYw/bA3sFS0hpUW4Bz\n648AlLSR7bnNRdZbyYuFJS8qpcnwcOCFwLXAFybaxWk5htDZ56m2+KZQPfHpCw3H0xhJ+wE3At8C\nbpL0pqFxg/SjL5IXRfJiISdRHUT+FrAW1a1NJpRcqdzZU7avB7D963IB0qA6DNjU9j2SXgh8B/hJ\nwzE1JXnRkrxoWd/2rgCSZgET7hhKCkJnz5F08Ej9E+HuhePoMdv3ANieUy7MGlTJi5bkRU3ZOBh6\nAM6ken+9Oa1f5RhCB5I63sN9Ity9cLxIuhf4bm3Qu+r9tj/c86AakrxoSV60SLqT1p1eh5jWnV+7\neXhOo1IQoiuS3tNpvO1v9yqW6B/Ji6VLCkKX2m/i1emmXjE4khexNMlZRt171Sj9A0PS/p36B0zy\nokhetEi6rFN/v0pB6JLtT3bqHzArjNI/MJIXC0letGw7Sn9fSpPRKMpZAjsB65dB84Czbf+1uaii\nacmLaCdpmu2bSvcytp+sjdvK9uXNRdedFIQOJL2T6ilI51H94AGmUj1A+99sf6+p2JpQzjOfwcIr\nwZlDP4JBkbxYWPKiIulK21u0dw/X369yHUJnnwKm276/PlDSmlQPFB+YH76kw4B9gNOAa8rgqcCP\nJJ1k+4uNBdd7yYsiebEQjdA9XH9fSkHoTMATwwx/ggnyDx5HB1Bdkfp4faCk/wKuAwbth5+8qCQv\nWjxC93D9fSkFobMvAFeXy9DvKMOeC+xMdT+bQbIAeA5wZ9vw55RxgyR50ZK8aJkq6ctUGwVD3ZT+\n9UeerX/kGMIoSjPALizcPvpz239qLqreK3f4/DpwAwuvBF8CHGz7p03F1oTkRSV50bI0XKSXgjAO\nJF1ke6k//7zc231bFl4JXtp2NsWqE+GeLb2QvBjsvJC0IoDtR5uOpVtpMhofKzUdQC+UH/hFo0x2\nAdUzhyN5UXcBA5IXkjYBTgTWLf3zgP1s39BoYF3IhWnjI7tZLYN2ULWT5EXLIOXFccAnbE+1PRX4\n1zKs76UgxHjLSjCGM0h5sYrtc4Z6bJ8LTIhnZqQgjI9B2vqJ7iUvBtOtkj4uaWp5HQ7c2nRQ3UhB\n6EDS1yWt3MWk717SsUwgS/1KMHmxWJb6vKjZH9gAmAX8lOpCvQlxo78UhM7+CFwlaY9OE9n+bY/i\naYykH0vq5gEfOy3xYJqXvCiSFy2SNgWwfZ/tf7b9Mtsvt32Q7fuajq8bOe10FCXZjwJWBo6hdrGN\n7ZlNxdVrkvYC/gP4H+CLtp9qOKRGJS8qyYsWSVdT7Q1cDvwauBj4je2HGw1sDFIQuiDpHcB/UZ06\nN/TDt+19GguqAeUOn5+huonbiSy8Ehyk5wgDyYshyYuW0pS4LbBdeW0J3A5cbPvgTvP2g1yH0IGk\njYFvAvcD29huvzx/0DwK3Ac8C5jC4N2aAEheDCN5Udh+CDhX0kXANlQPTNoP2B1IQZjgzgQOtT2r\n6UCaJmkH4KvAz4EtS+IPquRFkbxoKceUtqO6AG8BMBv4DfCaibLRkCajDiStONxl55K2Bfay/aEG\nwmqEpF8D7x+EA6WjSV60JC9aJD1CdU+nbwIX2J7bcEhjlj2EDuo/ekkvBd4B7EF1lsmPmoqrCba3\nax9W7tWyO7Cn7Rm9j6oZyYuW5MVCVqXaO9gOOELSC6hu+HcJcIntXzUZXDeyh9CBpI2AvcrrIeBU\nqqaCbk6zWyqVG5ntQrUS3AX4MXC67TMaDayHkheLSl4sqtwRdy/gEGBD25MbDmlUKQgdSFoAXAi8\n1/YfyrC5tjdqNrLek/Q6quTelWqZnAp8xfbzGg2sAcmLluRFS7mp3Xa116pUp59eQnWW0aUNhteV\nFIQOJL0V2BPYGvgJVbKfYHvDRgNrQG0luK/tW8uwQV0JJi+K5EWLpGuo7vo6VAAm3DGEXKncge0f\n2n4r8HfApcDhwNrl1gWvaza6ntua6oKb8yX9TNK+QN/vAi8JyYuFJC+KcmXyPwPPbi8Gkg5qKKwx\nyR7CGElai+oA4tttb990PL0mScCrqZoJ3gxcBpxh+/hGA2tY8iJ5MUTSlba3aBt2le3Nm4qpWykI\nHUhaHngv8ELgWqpmgYG9NL9dOZC4E9XZJANzdW7yorMBzou3UzUlvgY4vzZqFWAZ269tIq6xSEHo\nQNLJVHdpvJDqAepzbB/abFTNKFvAh9NaCX5hUC9CSl60JC9aJG0IvAD4PNUyGfJX4CrbTzQS2Bik\nIHQg6VrbLy3dy1LdqGogHgPYTtIsqh/8r4A3ASvY3q/ZqJqRvGhJXixdcmFaZ09XdNtPVM2kA2t9\n27vC0yuBKxuOp0nJi5bkRRtJWwFfB14CLE+1N/mY7VUbDawLKQidvVzS/aVbwCqlX1R3tVyjudB6\nr9zVcmjtN6neb/vBxgLrveRFTfJiEd8E3gWcQnUW1ruBCXFdRpqMOpDU8fS5QTqQKOlOqht21TeH\nTWslODBX6SYvWpIXi5J0he0t25oWJ8RZRtlD6GCQftijsT216Rj6RfKiJXkxrIclLQf8VtJ/Ancx\nQa7NyIVpXZL0f536YzAlL2IY76Zatx4EPEV1BtZbmwyoW2ky6pKkqfV7mrf3DxJJl9neeqT+QZK8\naBn0vCin4K5p+8a24RsD8z0BnqucghBjJmmS7QUj9cdgGvS8kPR94Fu2z28b/hrgn2y/q5HAxiBN\nRh1Iek65P81XJa0u6ZOSrpL0fUnrNB1fL0maVuttz5stexlL05IXLcmLhbyovRgA2L4A2Kz34Yxd\nCkJnJwJzgXupLkU31X1aLgeOaTCuJpxa676sbdx/9zKQPpC8aEletKzSYdyyPYviGUhB6Gxd20fZ\n/n9UbYP/z/Ytto8CBu1Wxxqhe7j+pV3yoiV50XKzpDe0D5S0E3BLA/GMWU477aye0N9tGzdoxdQj\ndA/Xv7RLXrQkL1o+DPxE0gXAFWXYdGB7qtt69L0UhM5+Imll2w/Z/vjQQEkvBG5uMK4mTJX0ZaqV\n4VA3pX/95sJqRPKiJXlR2P59ecb2u4BNy+DfAAfVn8Pdz3KWUQeSZtg+s+k4+oGk93Qab/vbvYql\nacmLluTF8MopqFtR7SXNtv2nhkPqSgpCB8M96CIqklYEmChbPuMpeTGyQc6LIZLeAhxFdXt0UT1f\n+VDbZzQaWBfSZBRjUh4kfiKwbumfB+xn+4ZGA4tGJS8W8ilgK9v3AEhaGzgbSEGY4DaWNNztfIdu\n3DWIW4nHAZ+wfQ6ApNeXYa9qNKreSl4sKnnRMmmoGBT3MkFONkhB6OwW4G1NB9FnVhn60QPYPlfS\nl5oMqAHJi0UlL1rOlvRT4OTSvydwVoPxdC0FobPHbQ/aWSOjuVXSx4HvlP53Abc2F04jkheLSl60\nHEa1wTC0d3Qi8MPmwuleCkJnlzYdQB/aH/gsMIvqDIoLy7BBkrxY1MDnhaSzbe/k6kyd08prQslZ\nRh1IOrjTeNtf61UsTZO0qe3rmo6jHyQvWpIXLRPlITidZA+hs68AV1O1/z3B4F2KX/ddSVOp7tfz\na+BiqofLP9xsWI1IXrQkL1qeLenNI420fXovg1kc2UPoQNJ0qgNCO1I1E5xc7lw4kCStDGxLdV71\ndlR3s7wduNh2x63mpUnyYmHJi4qk+4AzGX4Dwbb7vgktBaELkgS8mmolsD3wL7Z/0mxUzZG0ArAN\n1UGz/YDlBvTZucmLmkHPi6XhgsU0GXVndeAlwIuBe4D7mw2n9yTtQbX1twXVQ9VnU92n5TWD+oQw\nkhfJi4VN+KbD7CF0IGkf4O3AqsCPgFNt39VsVM2Q9AhwA/BN4ALbcxsOqTHJi5bkRUv7AXZJq1Lb\n6Lbd9xsMKQgdSFoAXEv1MBRou52v7REPIC1tJC1DtRU41E78AuAO4BLgEtu/ajC8nkpetCQvFiXp\nfcC/A3+jlRu2vVFzUXUnBaEDSTt0Gm/7vF7F0m8krQnsBRwCbGh7csMh9UzyYmSDnBdDJN0EvGKi\n3OG0LscQOhjphy1pPaomg4H54Zebl21Xe61KdZrhMVSnGg6M5EVL8mJYNwOPNB3E4sgeQpckrQG8\nlWrrZ0Pg/2wf0mxUvSPpGuAiqqaAiwe5rbgueZG8aCdpc+B/qQ6uPzY0fCKcgps9hA4krQTsDrwD\n+Duqc4xfZHugngQFYPtlAJIOav/Rl2HfaCay3ktetCQvhvXfwC+ojjMtaDiWMckeQgeSHqY6je4z\nwC9tL5B0i+1Be5D604Y713ppuGR/LJIXi0petEzk7509hM4+TXXR0ZeBkyWdyuA9OBwASW+nWhYb\nSqpfgr8K8JdmompM8qJIXgzrZ5IOAH7Mwk1GOe10aSBpGlUb8Z7ARsC/AmcMUnuppA2pTin8PHB4\nbdRfgatsP9FIYA1KXiQvhiPplmEG57TTpZGkzahWAnsMchNBLCx5EUuDFIRxIOki2wPxqEBJWwFf\np7plw/JUl+s/ZnvVRgPrQ8mLwcyLciX7Imyf1OtYxirHEMbHSk0H0EPfpHoa1inA1sC7gec1GVAf\nS14Mpq1q3SsAOwBXAikIA2KQdrMm2b5R0jKlffhbkq4CPtl0YH0oeTGAeWH7g/V+SatRFcq+l4IQ\nY/WwpOWA30r6T+AuYOBuTxCLSF6M7GGqixb7XgrC+Jjwt70dg3cDk4CDgI8AL6S6UjcWlbwYQJJ+\nTGvvcBKwCRPk+co5qLyYJP3S9val++W2f9t0TEuSpLWANW3f2DZ8Y2C+7fuaiay/JC+eHj6weSFp\n+1rvk8BtE+XZEJOaDmACe/qc4qX9R198DVhvmOHrAF/tcSz9LHlRGdi8sP3L2uviiVIMIHsIi03S\n7QP2eMDZtqePMO4625v2OqZ+lLxYaNxA5UW5IG2kFaptv6CX8SyOHEPoQNJuI42iOp1skKzSYdyy\nPYuiDyQvFpK8aGkvjJOAPYDDgKt6H87YpSB09rYO487qWRT94WZJb7C90PeWtBMw3KX6S7PkRUvy\nohg6XiJpErA38FHgauCNtm9oMrZupcloMUmaYfvMpuPolXKQ8CfABcAVZfB0YHvgTbZ/31BofSV5\nAQxoXkhaFtgfOJTqGRFH2J7TbFRjk4KwmAatrRhA0opUV6MOtQtfD3zH9qPNRdVfkhfAgOaFpDup\nzir6CnB7+3jbpy8yU59JQVhMku6wvUHTcTShnGq4FdUBtNkT8dmxS0ryYnDzQtIJdD6ovH8Pw1ks\nKQiLaRC3BAEkvQU4CriQ6iDqdsChts9oNLA+kbxIXoxG0r62T2w6juGkIHRQ7sUy3AIS8BLbg3ZG\nCZJ+C+xk+57SvzZwtu2XNxtZ7yQvFpW86N5wT5frFznLqLOBvPR+FJOGfvTFvQzeBY7Ji0UlL7rX\nt7c0SUHowPbNTcfQh86W9FPg5NK/JwN2qmXyYlgDnxdj0LfNMmky6kDSnxm5acC21+hxSI2TJKrz\n8Ice/HIh8EMPUCIlLxaVvOiepKtsb950HMNJQehAUsfb99p+qlexNE3S2bZ3ajqOfpC8aElejJ2k\nb9g+qOk4hpM2vg5sP1V/UV2K/1Lg2YP0oy+mNB1Av0heLCR50QVJ+w1192sxgOwhdCTpTVQXmdxP\n9eSnY0v3BsBHbH+3wfB6StJcqnuyDGsiXHQzXpIXLcmL7kyU05FTEDoop9LtBTwbOAfY3PZNktYB\nzrH90kYD7CFJ9wFnMvwZEhPiopvxkrxoSV60SLpmpFHAi2wv38t4FkfOMupswdBNqSTdavsmANt3\nS3qy2dB67rZB+nGPInnRkrxoWRt4A/DntuECft37cMYuBaGzSZJWoTrW8lTpHtoSGrTjL3177nQD\nkhctyYuWnwAr2766fYSkC3ofztilyaiDcrOqBSyc9KZ1emHftwmOF0mb2r6u1r8qtQ0K2/c3ElgD\nkhctyYuxk7S67fa9iL6QghBjIul9wL8Df6N1Lr5tbzTyXLG0S150r59vXZGCEGMi6SbgFYN2J8vo\nLHnRvX6+MG3Q2jsXm6TLOvUPkJuBR5oOol8kL56WvOhe326FZw+hS5Im2V4wUv+gkLQ58L/Ab4DH\nhobbPrixoBqUvKgkL7rXz01GOcuoA0nThk4ppNqbqv/QtwQu731Ujftv4BfAtSy8PAZG8mJYA58X\nY9C3Z2ZlD6GDeiVvr+r9XOWXpH5u/+yV5MWikhcLk7QF1Y3+DFxs+8rauDX69eyrHEPoTCN0D9c/\nKH4m6QBJ60paY+jVdFA9lrxYVPKikPQp4ERgTWAt4H8lfXJofL8WA0iT0Wg8Qvdw/YNir/L347Vh\nBgbp9MLkxaKSFy3vBF5u+28Ako4ArgY+12hUXUhB6GyqpC9TbfUNdVP6128urObY3rDpGPpA8qJN\n8mIhfwRWoLomA2B5YF5z4XQvxxA6kPSeTuNtf7tXsfQLSfsMN9z2Sb2OpSnJi0UlL0DS16n2ip4L\nbEV140MDOwKX2X5zg+F1JQVhDCStCGD70aZjaUpJ+iErADsAV9oe2OcMJy+SFwCS9u003vaJvYpl\ncaUgdEHSJlQHidYtg+YB+w3d8XKQSVoNOMX2zk3H0mvJi5ENcl5MZDmG0J3jgE/YPgdA0uvLsFd1\nnGswPAwMavtx8mJkA5cXkk6zvYekaxnm5ALbL2sgrDFJQejOKkM/egDb50r6UpMBNUXSj2kl+yRg\nE+C05iJqVPKiSF4A8KHy902NRvEMpCB051ZJHwe+U/rfBdzaXDiN+mKt+0mqB6Tc2VQwDUtetAx8\nXti+S9Jk4ATbr206nsWRYwhdkLQm8FlaVx5eCHza9n2NBhaNSl7EcCSdB7zZ9gNNxzJWKQgdtD/8\nY5BJuoWRL7qy7Rf0Mp4mJS9akheLknQmsDnVaacPDw2fCDf6S5NRZ9+VNJXqZmW/Bi4GfmP74c6z\nLZWmt/VPAvYADgOu6n04jUpetCQvFnV6edVNiC3vFIQObG8maWVgW2A74GPAlpJup7phVd9X/PEy\n1AwiaRKwN/BRqsvx3zhop1kmL1qSF8NazfZX6wMkfWikiftJmoy6JGkFYBuq9uL9gOUG7Nm5ywL7\nA4cCFwFH2J7TbFTNS14kL9oNd8fbiXI32BSEDiTtQbUFuAXVPd5nUz0A5JJBO4OiPFj+SeArwO3t\n42237yIvtZIXLcmLFkl7Ae+g2ji4sDZqFWCB7R0aCWwMUhA6kPQIcAPwTeAC23MbDqkxkk6g88HD\n/XsYTqOSFy3JixZJz6O6GO/zwOG1UX8FrrH9ZCOBjUEKQgeSlqHaCtyuvF4A3AFcQrU1+KsGw+tL\n/7+9+4+9q67vOP58tWOAjpYf9QeuCWNbIiGjMBwsHSZTlnZsyrLUwax2KzFZlkyhumx/bHOa6TST\nQBxrtmWZCVYTh8DAwVgmKEocRRC7alUkStChYnWFtECBBXntj3Ou39tvb09P69f7ufdzXo/kpt9z\nTv948+V1+rmfz+ecz0fS5nlYs+VHkVwcuSHkogZpEI5A+9z5RuCtwOm2lxcuaeYMccew5OLwhpQL\nSRuA9wEvplkSXTS9pRVFC+shTxl1aBcv+5Wxzwqaxwz/keZRwzhY9TuGJRdHpfpcjLkSuNj2A6UL\nOVJpELpdR/PkxKeAvx7yWPERGEKXM7k4ckPIxcjueWwMIHsqd7K9xvYfASsX3/SS3lKorFlX/TfB\n5OKoVJ+LMfdL+qikjZI2jD6li+ojDUI/k56U6Nw1a8CGNGSSXPQ3pFysAPYD64GL289crICaSeUO\nkn4XeD3wKprhgZETgJ+Y1xUNl4qkD9meuHVizZKLQ5P0SuB84Eu2by9dTxyZzCF0uw/YA6wG/n7s\n/BMMbJ0WSbcsPgW8ut0ZC9u/Nf2qikkuWpLus31++/MfAG8GbgbeKelc239TtMAC2nWutgIXtKc+\nA2yZh5cW00OIXiTtoHkZ6wM0E4QC/oXmmzK27ypXXZQyviSDpM8Bv2n7+5JeCHzW9lllK5w+SXcA\nH+HAfTLeaHtduar6yRxCD5LOk/RZSXslPSPpWUn7Stc1Zb8EfB74C2Cv7U8DT9u+a6iNQXIBwDJJ\nJ7XvYiy3/X2AduXXmX8z98fkRbavtf1c+/kg8KLSRfWRIaN+/oGmlb+OZnz0MuC0kgVNm+3ngfdL\nuv3pdqAAAAqOSURBVKH9czfJz+BzAayk+aIgwJJObXcO+ymG9WTRuD2SNtH0oKF5aXEuNk3KkFEP\nkj5v+xWSdo26wPOyeuGPi6TXABfY/vNF50+y/XihsqYquTg0SS8AXmL74fZ4SLk4jWYOYS3N8Op2\n4HLbjxQtrIehf8Pr6ylJPwl8QdJ7gUeBQS9PYPs24LYJlz5Js87PECQXh2B7P/Dw2Kkh5eJdwOZR\nAyjpZJo9p2d+ob/MIfRzGc3v6i3AD4CfB36nZEEzbEjDBJeRXPQ1pFysGe8N2X6MZkvNmZceQgdJ\nq4BTbD/YnnoG+EtJZzAnY4IFVD8GmVwclepzMWbZ+BBZ20OYi39r00Po9nfAyyacfylwzYTzMQzJ\nRXS5GrhH0rslvZtmDuHKwjX1kknlDpLut714E/HRtS/Z/oVp1zTrhjCpmlwcuSHkYly7Iu6F7eGd\n87K/9Fx0Ywo6oePaMVOrYsZIOpdmm0DTbCq/Y+zyzG8TuASSiwmSiwVtAzAXjcC4DBl1e0jSry8+\nKWk9Bz5BMRiS3gFsA04BVgHXSnr76Ho7gVa75GKR5KIOGTLq0E4S/jvwaZqXb6B5Y/dXgdfa/mqh\n0oqR9CBwtu1n2uPjgZ22X162sulJLg6WXNQhPYQO7Y19FnAvcEb7uRc4a4g3fes7wHFjx8cC3y5U\nSxHJxUSDz0UNModwGLafBv65fdTwPJrx0RcCTxctbMokbaX5b98LfLldwMvAOprVPwcluWgkF3XJ\nkFEPkl4HvJ9mGVvR7KP7Nts3Fy1siiRt7rpue9u0apkVyUVyUZs0CD1I+gKw3vbu9vglwO22zy5b\nWZSUXERtMmTUz7LRTd/6HgObf5F0ve1LJe1iwlunttcUKKu05CK5qEoahH5ul3QbC8vZvh74eMF6\nStjS/jkXe8NOSXKRXFQlQ0Y9SBJwCc1LN9CMGd/ogf3yJC0HPjHkPYPHJReN5KIeaRA6SLrd9vrS\ndcwSSZ8ENtjeW7qWUpKLgyUXdciQUbe52PZuyp4EdrWPFz41Omn7inIlTV1ycbDkogJpELqtlLTh\nUBdt3zTNYmbETe1n3NC6mcnFwZKLCqRB6LaSZrJs0uYe5uAbYAhOtH3AEs+SthzqL1cquThYclGB\nzCF0kLTD9lC2/etl0u9kgEsbJxeLJBd1SA+h25C2/eskaSPwBuB0SbeMXToBGNpKlslFK7moSxqE\nbr83fiBpBWO/s4Et6budZhP5VTQ7Qo08AXyxSEXlJBcLkouKZMioB0l/CPwVzd65o1+Ybf9suaqi\ntOQiapMGoQdJXwPW2v7f0rWU1j5d8z7gxTRDJ6L5R3BF0cIKSC4WJBd1yJBRPw8B+0sXMSOuBC62\n/UDpQmZAcrEguahAGoR+/gzYLule4NnRyYG+dLM7N/0PJRcLkosKpEHo55+AO4FdwPOFayntfkkf\nBT7Ggf8IDvHZ++RiQXJRgTQI/Rxj+49LFzEjVtAMk4yv5TPUl7GSiwXJRQUyqdyDpPcC3wBu5cBv\nP0N6vDAWSS6iNmkQepD08ITTg3y8UNJqYCtwQXvqM8AW298qV1UZycWC5KIOaRDiiLSrWX4E+HB7\nahPwRtvrylUVpSUXdUiD0IOk35903vaHpl1LaZJ22j7ncOeGILlYkFzUIZPK/Zw39vNxwK8BO4DB\n3fjAHkmbWNg2ciOwp2A9JSUXC5KLCqSHcBQknQhcZ/ui0rVMm6TTaMaK19I8RbIduNz2I0ULmwHJ\nRXIx79JDODpPAaeXLqKQdwGbbT8OIOlk4CrgTUWrmg3JRXIx19Ig9CDpVhYWL1sGnAlcX66iotaM\nbnpoHrGUNMg175OLAyQXFUiD0M9VYz8/B3xzwI/TLZN00qJvgkPNUXKxILmoQP6H9WD7rtI1zJCr\ngXsk3dAeXwK8p2A9xSQXB0guKpBJ5Q7ti0eH+gXZ9s9Ns55ZIelM4ML28E7bXylZz7QlF5MNPRc1\nSIPQQdIpi04tAy4F/gTYYft1068qSksuolYZMupgew+ApGU02yb+KbATeE2+/QxXchG1SoPQQdIx\nNI/NvQ34L+C3bX+9bFVRWnIRtcqQUQdJ36J5euRvgf9ZfD1rvQ9TchG1SoPQQdIH6Z48zEs3A5Rc\nRK3SICwBSZttbytdR8yW5CLmTRqEJSBph+1zS9cRsyW5iHmzrHQBlVDpAmImJRcxV9IgLI10s2KS\n5CLmShqEpZFvgjFJchFzJQ3C0ri7dAExk5KLmCuZVD4MSefTPEr4uXatlouAr9r+j8KlRUGSzgB+\nGrjX9pNj5y+y/Z/lKos4emkQOkh6J/AbNG903wH8MvApYB3wcdtZzXGAJF0BvBl4ADgH2GL739pr\nebIo5lYahA6SdtHc8McC3wVW294n6Xiab4ZrihYYRbS5WGv7SUk/A9wIfNj2NZL+23Y2hom5lLWM\nuj1n+wfAfkkP2d4HYPtpSc8Xri3KWTYaJrL9DUmvAm5s9xXORHLMrUwqd/s/SS9of37F6KSklUAa\nhOHaLemc0UHbOLwWWAWcVayqiB9Rhow6SDrW9rMTzq8CTrW9qz3+4daBUT9Jq2l6j9+dcO0C23e3\nPycXMVfSICyBTCTGJMlFzJsMGS2NjBvHJMlFzJU0CEsj3ayYJLmIuZIGISIigDQISyVDAzFJchFz\nJZPKPUk6F3glzTDA3bZ3jF072fZjxYqLYpKLqEl6CD1IegewDTiF5lnzayW9fXQ9N/0wJRdRm/QQ\nepD0IHC27Wfa4+OBnbZfXrayKCm5iNqkh9DPd4Djxo6PBb5dqJaYHclFVCVrGXWQtJVmbHgv8GVJ\nd7TH64D7StYW5SQXUasMGXWQtLnruu1t06olZkdyEbVKgxAREUCGjDpJut72pe369we1nNkPYZiS\ni6hVeggdJJ1q+9F2nfuD2P7mtGuK8pKLqFUahMOQtBz4hO1Xl64lZkdyETXKY6eH0e6Y9ny7KU4E\nkFxEnTKH0M+TwK728cKnRidtX1GupJgByUVUJQ1CPze1n3EZa4vkIqqSBqGfE21fM35C0pZSxcTM\nSC6iKplD6GfSi0iXTbuImDnJRVQlPYQOkjYCbwBOl3TL2KUTgKxkOVDJRdQqDUK37cCjNEsbXz12\n/gngi0UqilmQXESV8h5CREQAmUPoRdIGSV+TtFfSPklPSNpXuq4oK7mI2qSH0IOkrwMX236gdC0x\nO5KLqE16CP3szk0fEyQXUZX0EHqQdA3wUuBjwLOj87YXv5QUA5JcRG3ylFE/K4D9wPqxc+bgt1Rj\nWJKLqEp6CBERAWQOoRdJqyXdLOl77edfJa0uXVeUlVxEbdIg9HMtcAvwsvZza3suhi25iKpkyKgH\nSTttn3O4czEsyUXUJj2EfvZI2iRpefvZBOwpXVQUl1xEVdJD6KHdO3crsJbmKZLtwOW2HylaWBSV\nXERt0iD0IGkb8Fbbj7fHJwNX2X5T2cqipOQiapMho37WjG56ANuPAb9YsJ6YDclFVCUNQj/LJJ00\nOmi/CealvkguoioJbz9XA/dIuqE9vgR4T8F6YjYkF1GVzCH0JOlM4ML28E7bXylZT8yG5CJqkgYh\nIiKAzCFEREQrDUJERABpECIiopUGISIigDQIERHR+n+o7PFO4uK6AwAAAABJRU5ErkJggg==\n", "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%matplotlib inline\n", "\n", "contrib =pred_contributions_table.iloc[0, :-1].to_frame(name = \"contribution\")\n", "contrib[\"abs_contribution\"] = contrib.contribution.abs()\n", "contrib = contrib.sort_values(by=\"abs_contribution\", ascending=False)[[\"contribution\"]]\n", "\n", "_ = contrib[:5].plot(kind='bar', \n", " title='Top Five Reason Codes for a Risky Customer\\n', \n", " legend=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The Weight of Evidence of the columns: `BILL_AMT3`, `PAY_0`, `PAY_2`, `PAY_3` had the greatest impact on their prediction. Since the contribution is positive, we know that it increases the probability that they will default." ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Run Model Interpretation\n", "\n", "Once we have completed an experiment, we can interpret our H2OAI model. Model Interpretability is used to provide model transparency and explanations. More information on Model Interpretability can be found here: http://docs.h2o.ai/driverless-ai/latest-stable/docs/userguide/interpreting.html.\n", "\n", "\n", "### 1. Run Model Interpretation on Driverless AI Experiment\n", "\n", "We can run the model interpretation in the python client as shown below. " ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": true }, "outputs": [], "source": [ "mli_experiment = h2oai.run_interpretation_sync(dai_model_key=experiment.key, \n", " dataset_key=train.key, \n", " target_col=target)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is equivalent to clicking **Interpret this Model** in the UI once the experiment has completed.\n", "\n", "![Equivalent Steps in Driverless: View Results](images/experiment_complete_creditcard.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once our interpretation is finished, we can navigate to the MLI tab in the UI to see our interpreted model. \n", "\n", "![Equivalent Steps in Driverless: MLI List](images/mli_list.png)\n", "\n", "We can also see the list of interpretations using the Python Client:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "['gonelugu']" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get list of interpretations\n", "mli_list = list(map(lambda x: x.key, h2oai.list_interpretations(offset=0, limit=100)))\n", "mli_list" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "### 2. Run Model Interpretation on External Model Predictions\n", "\n", "Model Interpretation does not need to be run on a Driverless AI experiment. We can also train an external model and run Model Interpretability on the predictions. In this next section, we will walk through the steps to interpret an external model.\n", "\n", "#### Train External Model\n", "\n", "We will begin by training a model with scikit-learn. Our end goal is to use Driverless AI to interpret the predictions made by our scikit-learn model." ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "# Dataset must be located where python client is running\n", "train_pd = pd.read_csv(train_path)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "GradientBoostingClassifier(criterion='friedman_mse', init=None,\n", " learning_rate=0.1, loss='deviance', max_depth=3,\n", " max_features=None, max_leaf_nodes=None,\n", " min_impurity_decrease=0.0, min_impurity_split=None,\n", " min_samples_leaf=1, min_samples_split=2,\n", " min_weight_fraction_leaf=0.0, n_estimators=100,\n", " presort='auto', random_state=10, subsample=1.0, verbose=0,\n", " warm_start=False)" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import GradientBoostingClassifier\n", "\n", "predictors = list(set(train_pd.columns) - set([target]))\n", "\n", "gbm_model = GradientBoostingClassifier(random_state=10)\n", "gbm_model.fit(train_pd[predictors], train_pd[target])" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[0.38111179, 0.61888821],\n", " [0.44396186, 0.55603814],\n", " [0.91738328, 0.08261672],\n", " [0.88780536, 0.11219464],\n", " [0.80028008, 0.19971992]])" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predictions = gbm_model.predict_proba(train_pd[predictors])\n", "predictions[0:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Interpret on External Predictions\n", "\n", "Now that we have the predictions from our scikit-learn GBM model, we can call Driverless AI's `h2o_ai.run_interpretation_sync` to create the interpretation screen." ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": true }, "outputs": [], "source": [ "train_gbm_path = \"./CreditCard-train-gbm_pred.csv\"\n", "predictions = pd.concat([train_pd, pd.DataFrame(predictions[:, 1], columns = [\"p1\"])], axis = 1)\n", "predictions.to_csv(path_or_buf=train_gbm_path, index = False)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": true }, "outputs": [], "source": [ "train_gbm_pred = h2oai.upload_dataset_sync(train_gbm_path)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [], "source": [ "mli_external = h2oai.run_interpretation_sync(dai_model_key = \"\", #we are interpreting an external model\n", " dataset_key=train_gbm_pred.key, \n", " target_col=target, # target column used by external model\n", " prediction_col=\"p1\", # column with external model's predictions\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also run Model Interpretability on an external model in the UI as shown below:\n", "\n", "![Equivalent Steps in Driverless: MLI External Model](images/mli_external.png)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['kilegodu', 'gonelugu']" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Get list of interpretations\n", "mli_list = list(map(lambda x: x.key, h2oai.list_interpretations(offset=0, limit=100)))\n", "mli_list" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build Scoring Pipelines\n", "\n", "In our last section, we will build the scoring pipelines from our experiment. There are two scoring pipeline options: \n", "\n", "* Python Scoring Pipeline: requires Python runtime\n", "* MOJO Scoring Pipeline: requires Java runtime\n", "\n", "Documentation on the scoring pipelines is provided here: http://docs.h2o.ai/driverless-ai/latest-stable/docs/userguide/python-mojo-pipelines.html.\n", "\n", "![Equivalent Steps in Driverless: View Results](images/experiment_complete_creditcard.png)\n", "\n", "The experiment screen shows two scoring pipeline buttons: **Download Python Scoring Pipeline** or **Build MOJO Scoring Pipeline**. Driverless AI determines if any scoring pipeline should be automatically built based on the `config.toml` file. In this example, we have run Driverless AI with the settings: \n", "\n", "```\n", "# Whether to create the Python scoring pipeline at the end of each experiment\n", "make_python_scoring_pipeline = true\n", "\n", "# Whether to create the MOJO scoring pipeline at the end of each experiment\n", "# Note: Not all transformers or main models are available for MOJO (e.g. no gblinear main model)\n", "make_mojo_scoring_pipeline = false\n", "```\n", "\n", "Therefore, only the Python Scoring Pipeline will be built by default.\n", "\n", "### 1. Build Python Scoring Pipeline\n", "\n", "The Python Scoring Pipeline has been built by default based on our `config.toml` settings. We can get the path to the Python Scoring Pipeline in our experiment object." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'h2oai_experiment_bofuseke/scoring_pipeline/scorer.zip'" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "experiment.scoring_pipeline_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also build the Python Scoring Pipeline - this is useful if the `make_python_scoring_pipeline` option was set to false." ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": true }, "outputs": [], "source": [ "python_scoring_pipeline = h2oai.build_scoring_pipeline_sync(experiment.key)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'h2oai_experiment_bofuseke/scoring_pipeline/scorer.zip'" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "python_scoring_pipeline.file_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we will download the scoring pipeline zip file." ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'./scorer.zip'" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "h2oai.download(python_scoring_pipeline.file_path, dest_dir=\".\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Build MOJO Scoring Pipeline\n", "\n", "The MOJO Scoring Pipeline has not been built by default because of our `config.toml` settings. We can build the MOJO Scoring Pipeline using the Python client. This is equivalent to selecting the **Build MOJO Scoring Pipeline** on the experiment screen." ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": true }, "outputs": [], "source": [ "mojo_scoring_pipeline = h2oai.build_mojo_pipeline_sync(experiment.key)" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'h2oai_experiment_bofuseke/mojo_pipeline/mojo.zip'" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "mojo_scoring_pipeline.file_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can download the scoring pipeline zip file." ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'./mojo.zip'" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "h2oai.download(mojo_scoring_pipeline.file_path, dest_dir=\".\")" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "Once the MOJO Scoring Pipeline is built, the **Build MOJO Scoring Pipeline** changes to **Download MOJO Scoring Pipeline**.\n", "\n", "\n", "\n", "![Equivalent Steps in Driverless: Download MOJO](images/download_mojo.png)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.13" } }, "nbformat": 4, "nbformat_minor": 2 }