Add support HDP 2.2 plugin¶

https://blueprints.launchpad.net/sahara/+spec/hdp-22-support

This specification proposes to add new HDP plugin based on Ambari Blueprints [1] with Ambari Management Console.

Problem description¶

Currently we support old HDP plugin which contains old HDP distribution. Also old HDP plugin looks like unsupported by HortonWorks team every year [2]. Many customers want new version of HDP. New HDP plugin will be based on Ambari Blueprints. Ambari Blueprints are a declarative definition of a cluster. With a Blueprint, you specify a Stack, the Component layout and the Configurations to materialize a Hadoop cluster instance via REST API.

Proposed change¶

New HDP plugin will support provisioning HDP stack via Ambari Blueprints.

Plugin will support key Sahara features:

Cinder integration
Swift integration
EDP
Scaling
Event logs

New HDP plugin will support the following OS: Ubuntu 12.04 and CentOS 6. Aslo new plugin will support mirrors with HDP packages.

New HDP plugin will support all services which supports Ambari. Also new plugin will support HA for NameNode and ResourceManager. Client will be installed on all nodes if selected our process. For example if selected Oozie then will be installed Oozie client on all nodes.

Plugin wil be support the following services:

Service	Process
Ambari	Ambari
Falcon	Falcon Server
Flume	Flume
HBase	HBase Master
HBase	HBase RegionServer
HDFS	NameNode
	DataNode
	SecondaryNameNode
	JournalNode
Hive	Hive Metastore
Hive	HiveServer
Kafka	Kafka Broker
Knox	Knox Gateway
Oozie	Oozie
Ranger	Ranger Admin
Ranger	Ranger Usersync
Slider	Slider
Spark	Spark History Server
Sqoop	Sqoop
Storm	DRPC Server
	Nimbus
	Storm UI Server
	Supervisor
YARN	YARN Timeline Server
	MapReduce History Server
	NodeManager
	ResourceManager
ZooKeeper	ZooKeeper

Alternatives¶

Add support of HDP 2.2 in old plugin, but it is very difficult to do without Ambari Blueprints.

Data model impact¶

None

REST API impact¶

None

Other end user impact¶

None

Deployer impact¶

None

Developer impact¶

None

Sahara-image-elements impact¶

Need to add elements for building images with pre-installed Ambari packages. For installing HDP Stack plugin should use mirror with HDP packages. Also should add elements for building local HDP mirror.