Enterprise Steam Release Notes =============================== Change Log ---------- Version 1.7.1 (Jun 8, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Web Client: Added H2O logs to the Logs screen and download bundle - Web Client: Fixed Conda environment upload - Python Client: Added backward compatibility with Enterprise Steam v1.6 - Python Client: Fix for cacert bundle not being used when connecting to H2O - Driverless Kubernetes: Fixed liveness probe being too strict in some cases - Autodoc: Updated autodoc - Autodoc: Added Autodoc logs to the Logs screen and download bundle - Installation: Added support for high availability installation - Documentation: Documented high availability installation - Authentication: Login sessions now persist across Enterprise Steam restarts - Dependencies: Upgraded dependencies Version 1.7.0 (May 25, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Major: Added support for running Enterprise Steam inside Kubernetes cluster - Major: Added support for deploying and managing Driverless AI instances on Kubernetes - Major: Added support for ML-Autodoc for automated H2O model documentation - Web Client: Redesigned the Web client for managing H2O/Sparkling Water clusters and Driverless AI instances - Web Client: Added automatic H2O and Sparkling Water cluster sizing when launching clusters - Web Client: Added support for displaying and downloading YARN logs of H2O and Sparkling Water clusters - Web Client: Added support for displaying and downloading H2O driver logs - Web Client: Added support for displaying and downloading Sparkling Water spark-submit logs - Web Client: Added support for displaying lifecycle events of H2O/Sparkling Water clusters and Driverless AI instances - Web Client: Added support for launching copies of H2O and Sparkling Water clusters - Web Client: Improved validation when launching H2O/Sparkling Water clusters and Driverless AI instances - Python Client: New API client - R Client: New API client - H2O: Added configuration to add YARN job name prefix - H2O: Added support for importing engine from the server - Sparkling: Added support for importing engine from the server - Sparkling: Added configuration to add extra JARs to the classpath - Sparkling: Added configurable maximum idle and uptime for clusters - Documentation: New documentation for Python client - Documentation: New documentation for R client - Documentation: New documentation for using ML-Autodoc with Enterprise Steam - Installation: Simplified installation and upgrade - Installation: Moved initial administrator setup from CLI to the Web Client - Installation: Added separate directories for static assets and data to support high availability deployments - Dependencies: Upgraded dependencies - Dependencies: Replaced HAProxy with Traefik - Various other fixes and improvements Version 1.6.7 (Apr 28, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fixed issue with environments containing multiple Spark installations Version 1.6.6 (Apr 20, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added support for Hive in H2O and Sparkling Water - Added support for Sparkling Water engines 3.28.1.1 and higher - Added option to allow insecure XGBoost - Fixed parsing of Spark properties containing '=' - Fixed possible corruption of HAProxy configuration Version 1.6.5 (Mar 2, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fixed custom Hadoop temporary directory to be part of HADOOP_CLIENT_OPTS Version 1.6.4 (Feb 12, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fixed error in Hadoop Helper for Java 7 Hadoop distributions - Fixed error when creating Python environment with the same name as deleted environment - Improved validation and documentation for LDAPs with internal certificate authority - Improved documentation for Driverless AI config.toml override - Improved documentation for Driverless AI installation prerequisites Version 1.6.3 (Jan 31, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Supports RSparkling 3.28 - Use Hadoop Tmp Dir for Hadoop Helper commands to prevent slowdowns - Fixed an issue with old DAI servers preventing instances from loading - Fixed server panic with invalid LDAP over TLS configuration - Fixed initial LDAP config requiring server restart Version 1.6.2 (Jan 20, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Supports Sparkling Water 3.28 - Fixed LDAP over TLS - Fixed SAML token login in R client - Fixed H2O profile not saving YARN queues Version 1.6.1 (Jan 14, 2020) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fixed mutual TLS setup - Fixed Driverless AI version parser - Add datatable_strategy WRITE to default DAI launch config - Add Driverless AI instance password to the instance details - Docs: Updated copyright year to include 2020 Version 1.6.0 (Dec 18, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Complete UI overhaul of configuration page for administrators. - New administrator settings for individual products and deployment backends and other configuration. - Improved UI/UX experience. - Moved most of the settings from steam.yaml into the administrator configuration in Steam UI. - Configuration file steam.yaml is no longer required. - Enabling Hadoop backend now thoroughly validates all settings to ensure they are correct (Proxyuser, Kerberos, ...). - Added ability to export and import configuration. - Several settings no longer require Enterprise Steam restart. - Added indicator when Enterprise Steam needs to restart to apply changes. - Added option to use existing Livy server for Sparkling Water as long as it is on the same host as Enterprise Steam. - HAProxy comes bundled and does not need to be installed separately. - Steam Hadoop Helper (Steam Manager) comes bundled and does not need to be installed separately. - Additional HADOOP_CLASSPATH options can be added when launching H2O clusters. - Added ability to download complete Enterprise Steam logs from the UI. - Added SAML configuration validation. - Added cacert option when connecting to Steam or H2O clusters via Python client. - Added option to ensure internal secure connections with H2O and Sparkling Water clusters. - Added warning when uploading non-Hadoop version of H2O engine. - Improved database migration. - Improved validation of H2O Launch Parameters. - Fixed bug where Sparkling Water could not launch when users reused cluster names. - Fixed bug where Sparkling Water could not launch when the cluster name contained spaces. - Fixed UI issue when uploading Python environments with the same name. - Fixed Python environment selection when no Python environments assigned. - Fixed auto-assigment of profiles to LDAP groups Version 1.5.1 (Nov 21, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fixed a database migration bug affecting new installations. Version 1.5.0 (Oct 10, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added ability to launch Driverless AI instances. - Added ability to use Conda environments with Sparkling Water Version 1.4.11 (Oct 1, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fix token authentication in Python client when SAML is used - Fix R and Python client when internal and external ports do not match - Support recent versions of Sparkling Water Version 1.4.10 (Jul 19, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added an ability to download H2O logs from Steam UI - Added a config option for custom KRB5_CONFIG (STEAM_KRB5_CONFIG) - Added support for templating in YARN queue names - Added a check to ensure the Spark version and Sparkling Water version match - Added a check to warn admin if Steam port is occupied - Improved fetching of H2O YARN job status to now use impersonation - Fixed h2osteam Conda package - Fixed issue when Sparkling Water cluster failed to start when SPARK_YARN_MODE property was set Version 1.4.9 (May 21, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added Conda packages for Python client - Added ability to delete multiple stopped clusters at the same time - Added Python PySpark path configuration to profile - Added YARN queue configuration to user setting - Removed the need for secondary external port - Configuration fields are now protected from trailing and leading whitespace characters - Updated pysparkling dependencies for Python2 - Fixed an issue with YARN usernames being lowercased when they were not configured to do so Version 1.4.8 (Apr 15, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Add support for RSparkling through Jupyter notebook - Add option to disable Jupyterhub - Expose internal address and node ID for H2O cluster in the Steam Web UI - Add option to specify path to custom Pyspark Python for Sparkling Water - Fix pysparkling dependencies for Python3 - Updated engine upload backend Version 1.4.7 (Apr 3, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Fix Sparkling Water proxy issue with upercase usernames - Improve uploading h2o-3 engines - Set SPARK_YARN_MODE correctly based on the hadoop distribution Version 1.4.6 (Apr 1, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added ability to choose H2O-3 Leader Node when starting a cluster - Added ability to control the number of clusters a user can spin per cluster profile - Added option to select default Sparkling Water backend - Added automatic redirection back to login with an expired session cookie - Added an ability to auto-assign Steam profiles according to SAML profiles - Docs: Add "Before you begin installation" section - Docs: Documented steam.yaml configuration options - Docs: Updated documentation - Fix an issue when Steam was hitting API endpoints of dead clusters - Fix and issue when hadoop-unjar files were not deleted from temp directory - Fix issue with upercase usernames and Sparkling Water on Hadoop Version 1.4.5 (Mar 22, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added Configurable Steam Web UI timeout (STEAM_WEB_UI_TIMEOUT_MIN) Version 1.4.4 (Feb 20, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Make log file permissions configurable (STEAM_LOG_PERMISSIONS) - H2O: Communicate with cluster using leader node only - SW: Added support for Hive tables - SW: Disable Spark dynamic allocation for internal backend - SW: Bundle and distribute all pysparkling dependencies - LDAP group configuration is no longer mandatory - Bug fixes for Jupyterhub - Bug fixes for Sparkling Water params - Bug fixes for CDH5 Version 1.4.3 (Jan 29, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Added ability to provide secure HTTP headers (SERVER_STRICT_TRANSPORT, SERVER_X_XSS_PROTECTION, SERVER_CONTENT_SECURITY_POLICY) - Improved logging output, time in UTC - Accept encrypted TLS private keys - Log files permissions elevated for the group - Bug fixes for HDP Spark - Bug fixes for Sparkling Water integration - Bug fixes for Steam Python API - Bug fixes for LDAP connector - Bug fixes for Jupyterhub - Bug fixes for Steam Web UI Version 1.4.2 (Jan 1, 2019) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Add ability to upload H2O engine jar from Python/R API - Remove cluster dialog waits until cluster has been shut down - Log successful logins - Log files permissions reduced - Steam nbextension shows "please wait" when changing clusters - Document setup and troubleshooting steps for Hadoop/Kerberos configuration - Bug fixes Version 1.4.1 (Dec 3, 2018) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Enable Sparkling Water API from API - Hide disabled user with a checkbox - Display message if there are no clusters to show - Fix glibc dependency for steam binary - Fix documentation version - Add release notes to documentation Version 1.4.0 (Nov 23, 2018) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Add Sparkling Water integration - Add SAML authentication - Add More detailed cluster profiles - New cluster overview - New launch cluster page - Add option to generate Personal access tokens