Plugin for CDH with Cloudera Manager

This specification proposes to add CDH plugin with Cloudera Distribution of Hadoop and Cloudera Manager in Sahara.

Problem description

Cloudera is open-source Apache Hadoop distribution, CDH (Cloudera Distribution Including Apache Hadoop). CDH contains the main, core elements of Hadoop that provide reliable, scalable distributed data processing of large data sets (chiefly MapReduce and HDFS), as well as other enterprise-oriented components that provide security, high availability, and integration with hardware and other software. Cloudera Manager is the industry’s first end-to-end management application for Apache Hadoop. Cloudera Manager provides many useful features for monitoring the health and performance of the components of your cluster (hosts, service daemons) as well as the performance and resource demands of the user jobs running on your cluster. [1]

Proposed change

CDH plugin implementation will support Cloudera Manager version 5 and CDH version 5.

Plugin will support key Sahara features:

  • Cinder integration

  • Cluster scaling

  • EDP

  • Cluster topology validation

  • Integration with Swift

  • Data locality

Plugin will be able to install following services:

  • Cloudera Manager

  • HDFS

  • YARN

  • Oozie

CDH plugin will support the following OS: Ubuntu 12.04 and CentOS 6.5. CDH provisioning plugin will support mirrors with CDH and CM packages.

By default CDH doesn’t support Hadoop Swift library. Integration with Swift should be added to CDH plugin. CDH maven repository contains Hadoop Swift library. [2]

CDH plugin will support the following processes:

  • MANAGER - Cloudera Manager, master process

  • NAMENODE - HDFS NameNode, master process

  • SECONDARYNAMENODE - HDFS SecondaryNameNode, master process

  • RESOURCEMANAGER - YARN ResourceManager, master process

  • JOBHISTORY - YARN JobHistoryServer, master process

  • OOZIE - Oozie server, master process

  • DATANODE - HDFS DataNode, worker process

  • NODEMANAGER - YARN NodeManager, worker process



Data model impact


REST API impact


Other end user impact


Deployer impact


Developer impact


Sahara-image-elements impact

CDH plugin must be support vanilla images and images with Cloudera packages. For building pre-installed images with Cloudera packages use specific CDH elements.

Sahara-dashboard / Horizon impact




Primary assignee:


Other contributors:


Work Items

  • Add implementation of plugin

  • Add jobs in Sahara-ci

  • Add integration tests

  • Add elements to Sahara-image-elements for building images with pre-installed Cloudera packages


Depends on OpenStack requirements, needs a cm_api python library version 6.0.2, which is not present in OS requirements. [3] Need to add cm_api to OS requirements. [4]


  • Add unit tests to Sahara to cover basic functionality of plugin

  • Add integration tests to Sahara

Documentation Impact

CDH plugin documentation should be added to the plugin section of Sahara docs.