Cloudera Altus Director 6.0 adds support for deploying CDH 6 clusters on public cloud infrastructure. Altus Director also features improved integration with SDX configured namespaces, allows encryption of its configuration properties, and includes a tool for copying its database. Altus Director’s interfaces have been streamlined, removing deprecated features and old API endpoints.

Cloudera Altus Director, formerly known as Cloudera Director, helps you deploy, scale, and manage Cloudera clusters in AWS, Microsoft Azure, or Google Cloud Platform. Altus Director enables and enforces the best practices of big data deployments and cloud infrastructure. Altus Director’s enterprise-grade features deliver a mechanism for establishing production-ready clusters in the cloud for big data workloads and applications in a simple, reliable, automated fashion. In this post, you will learn about new functionality and changes in release 6.0.

Cloudera Altus Director Overview

For those new to Altus Director, let’s revisit what it does.

• On-demand creation and termination of clusters: Using Altus Director, you can allocate and configure Cloudera Manager instances and highly available CDH clusters in the cloud provider of your choice. A single Altus Director instance can manage multiple cloud provider environments and the separate lifecycles of multiple Cloudera Manager installations and clusters.
• Multi-cloud support: Altus Director supports creating clusters in Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) through its cloud provider plugin architecture. A single Altus Director instance can work with multiple cloud providers at once. Because the plugin specification is open source, you can create a plugin to support other providers, either in-house or public.
• On-demand grow and shrink of clusters: One of the main benefits of running CDH clusters in the cloud is being able to provision additional instances when demand increases, and to terminate instances when demand decreases. Altus Director, in concert with Cloudera Manager, does the work required to add new instances to and remove existing ones from your CDH clusters.
• Programmatic and repeatable instantiation of clusters: Altus Director can consume cluster definitions specified in HOCON configuration files submitted through the Altus Director CLI or in JSON or HOCON input sent to the Altus Director API. A cluster definition can include custom scripts to run after instance provisioning and cluster setup, or before cluster termination, to perform tasks like installing additional packages, configuring system settings, or saving off important data.
• Long-running cluster support: Long-running clusters often require actions like upgrading CDH and Cloudera Manager, changing the topology of the cluster, and reconfiguring the cluster. Altus Director supports such modifications when using Cloudera Manager 5.11 and above.
• Usage-based billing for Cloudera services: Usage-based billing can help you optimize your expenditures for transient clusters. With a pay-as-you-go billing ID from Cloudera, you can use your Cloudera Enterprise license as usual, but you are only charged for CDH services when they are running.
• Security: Altus Director, like other Cloudera offerings, is committed to enabling secure deployments and applications. Altus Director’s own database is automatically encrypted, and Altus Director helps you configure Cloudera Manager and CDH clusters with Kerberos authentication, TLS for on-the-wire privacy, and Cloudera Navigator for auditing, data lineage, and data discovery.
• Powerful web user interface: Altus Director’s user interface provides a single dashboard to assess the health of all your clusters across all cloud providers and all Cloudera Manager deployments. It can also be used to bootstrap new clusters, grow and shrink existing clusters, and terminate clusters that are no longer needed. Exploring the web user interface is a great stepping stone to using the configuration file or API to deploy production-ready clusters.

Deploying Cloudera Enterprise 6.0

Altus Director 6.0 supports Cloudera Manager 6.0 and 5.7+ along with any CDH versions they support. Altus Director 6.0 will deploy Cloudera Manager 6.0 and CDH 6.0 by default.

Introduced in Altus Director 2.8, Altus Director’s simple cluster setup provides the simplest and most convenient method to deploy your first CDH 6 cluster. In simple cluster setup, Altus Director infers default values for much of the cluster configuration, requiring you to enter fewer pieces of information to specify your cluster to Altus Director. Once bootstrapped, the cluster can be managed by Altus Director just like any other CDH cluster. Simple setup is available through the web user interface (UI), the command-line interface (CLI), and the REST API for AWS and Microsoft Azure environments.

The UI continues to include an advanced setup option that provides more configurability than the simple setup. The CLI and REST API expose all of Altus Director’s features. These interfaces should be used to enable more powerful features that you may be looking for in a production-ready cluster, such as Sentry, Kerberos, TLS, or high availability.

Integration with Altus Configured SDX Namespaces

Altus Director’s UI shows the cluster details for each cluster that Altus Director manages. The cluster details include information about the databases used by the various services in the cluster. Altus Director 6.0 now exposes the JDBC URL for Hive and Sentry databases, allowing them to be used as configured SDX namespaces for Altus Data Engineering and Altus Data Warehouse clusters. All clusters that share the same SDX namespace, whether managed by Altus Director or Altus cloud services can share metadata, obviating the need to recreate metadata for each cluster.

Encrypting Cloudera Altus Director’s Configuration

Altus Director reads its configuration properties from the application.properties file. This file is in plain text format. Altus Director 6.0 now allows you to encrypt property values in the application.properties file using a master password. Altus Director can then decrypt these property values when started with the same master password. This feature can be used to protect the credentials for Altus Director’s database.

Altus Director distributes the encryptPropertyValues tool alongside the cloudera-director-server executable for encrypting property values. You can generate the encrypted values by using this encryptPropertyValues script along with your master password.

Encrypted values must be marked in the application.properties file so that Altus Director knows to decrypt the values. You can mark encrypted values by enclosing the value in ENC().

lp.database.password: ENC(1LC5sbvMlv0I0e88HmCmvA==)

Finally, the master password must be specified to Altus Director on startup.

More information is available in the README.md colocated with the cloudera-director-server executable.

Copying Cloudera Altus Director’s Database

Altus Director is configured to create a local H2 database file by default, but Cloudera highly recommends using an external database server, such as MySQL, for production. Altus Director 6.0 can now copy its database to another database server. This feature is intended to aid users in migrating their database from H2 to MySQL and is accessed by using the copy-database script that is distributed alongside the cloudera-director-server executable.

There are two prerequisites to using the copy-database script. First, the source database must have already been upgraded to the current version of Altus Director, that is, this script cannot be used against H2 database files from older versions of Altus Director. Second, the destination database must be completely empty. This is a safety measure to ensure that Altus Director does not overwrite any data.

Instructions to use the copy-database script are:

1. Stop the Altus Director server
2. Create a MySQL database
3. Configure Altus Director to use the new database
4. Run the copy-database script
5. Start Altus Director server

More information is available in the README.md colocated with the cloudera-director-server executable.

Changes in Cloudera Altus Director 6.0

CLI Changes

The standalone CLI commands (bootstrap, update, terminate, validate), deprecated in Altus Director 2.8, have been removed from Altus Director 6.0. The existence of parallel commands to perform the same functions in standalone mode and with the server was confusing. We have been recommending the use of the server over the standalone CLI for better persistence and cluster management capabilities. The remote server CLI commands (bootstrap-remote, terminate-remote) are still available, and have been augmented with a validate-remote command.

API Changes

Altus Director’s API is built to be backwards compatible with previous versions. A new API version is added whenever new features require changing the API, resulting in API versions v1 through v12. Altus Director 6.0 drops API versions v1-v5, but still supports v6-v12. Additionally, Altus Director’s API versioning scheme has been changed to use the release version rather than its own versioning scheme. The latest API is available at http://servername:7189/api/d6.0/.

As part of Altus Director’s internal upgrades, code generation for Java and Python API clients have been overhauled. The changes include upgrading dependencies for improved security and automatic generation of documentation for both the Java and Python clients. Automation accessing Altus Director’s API through the Java and Python clients might have to be updated to account for changes such as:

• Java client helper Strings were replaced by Enums for better type safety
• Python client method and field names were changed from camelCase to snake_case to adhere to PEP8 standards
• Python client methods will now raise cloudera.director.common.rest.APIException instead of urllib2.HTTPError

Previous Java and Python client versions that work with API versions v6-v12 are compatible with Altus Director 6.0.

Supported Versions

Altus Director 6.0 may now be installed on CentOS and RHEL 7.5 and Ubuntu 16.04 but requires Java 8. See the documentation for the full list of Requirements and Supported Versions.

Using Cloudera Altus Director

For more information on what’s new in this release, check out the Altus Director 6.0 section of the New Features and Changes in Altus Director page in the Altus Director documentation. If you’re ready to give the latest version of Altus Director a try, here are the ways you can get started.

• Download Altus Director from our website, where you can also find its user guide, to start fresh or upgrade from an existing installation. Upgrades are only allowed from Altus Director 2.8. See the upgrade documentation for detailed instructions on how to upgrade Altus Director.

David Han is a software engineer at Cloudera.