Data Processing Service (Sahara) Specifications

Implemented specs by release:

• Rocky specs
  • placeholder
  • Plugins outside Sahara core
• Queens specs
  • Add HBase on Vanilla cluster
  • There and back again, a roadmap to API v2
  • Decommission of specific instance
  • Force Delete Clusters
  • Remove Job Binary Internal
• Pike specs
  • Deprecation of CentOS 6 images
  • Node Group Template And Cluster Template Portability
  • Support for S3-compatible object stores
• Ocata specs
  • Data Source and Job Binary Pluggability
• Newton specs
  • Designate Integration
  • CLI for Plugin-Declared Image Generation
  • Improve anti-affinity behavior for cluster creation
  • Initial Kerberos Integration
  • Adding pagination and sorting ability to Sahara
  • Admin API for Managing Plugins
  • Allow creation of python topologies for Storm
  • Refactor the sahara.service.api module
  • Refactor the logic around use of floating ips in node groups and clusters
  • Run Spark jobs on vanilla Hadoop 2.x
• Mitaka specs
  • Add ability of suspending and resuming EDP jobs for sahara
  • Allow ‘is_public’ to be set on protected resources
  • add cdh 5.5 support into sahara
  • Code refactoring for CDH plugin
  • Implement Sahara cluster verification checks
  • Remove unsupported versions of MapR plugin
  • Support of distributed periodic tasks
  • Improved secret storage utilizing castellan
  • Move scenario tests to a separate repository
  • Add recurrence EDP jobs for sahara
  • Reduce Number Of Dashboard panels
  • Removal of Direct Engine
  • Remove plugin Vanilla V2.6.0
  • Modify ‘is_default’ behavior relative to ‘is_protected’ for templates
  • Add ability of scheduling EDP jobs for sahara
  • SPI Method to Validate Image
• Liberty specs
  • Adding custom scenario to scenario tests
  • Adding cluster/instance/job_execution ids to log messages
  • Allow the creation of multiple clusters simultaneously
  • Objects update support in Sahara API
  • Retry of all OpenStack clients calls
  • Use trusts for cluster creation and scaling
  • Deprecation of Direct Engine
  • Drop Hadoop v1 support in provisioning plugins
  • [EDP] Add Spark Shell Action job type
  • Allow placeholders in datasource URLs
  • [EDP] Allow editing datasource objects
  • [EDP] Allow editing job binaries
  • CDH HDFS HA Support
  • CDH YARN ResourceManager HA Support
  • Add support HDP 2.2 plugin
  • Migrate to HEAT HOT language
  • Decompose cluster template for Heat
  • Updating authentication to use keystone sessions
  • Manila as a runtime data source
  • Addition of Manila as a Binary Store
  • API to Mount and Unmount Manila Shares to Sahara Clusters
  • Provide ability to configure most important configs automatically
  • Heat WaitConditions support
  • Use Templates for Scenario Tests Configuration
  • Support of shared and protected resources
  • Running Spark Jobs on Cloudera Clusters 5.3.0
  • Storm EDP
  • Storm Scaling
  • Support NTP service for cluster instances
  • Unified Map to Define Job Interface
  • upgrade oozie Web Service API version of sahara edp oozie engine
• Kilo specs
  • Add CM API Library into Sahara
  • Add More Services into CDH plugin
  • Add check service test in integration test
  • Add timeouts for infinite polling for smth
  • Authorization Policy Support
  • CDH HBase Support
  • Better Version Management in Cloudera Plugin
  • CDH Zookeeper Support
  • Default templates for each plugin
  • Remove support of Hadoop 2.3.0 in Vanilla plugin
  • Add a common HBase lib in hdfs on cluster start
  • [EDP] Add Oozie Shell Action job type
  • JSON sample files for the EDP API
  • [EDP] Add options supporting DataSource identifiers in job_configs
  • Enable Swift resident Hive tables for EDP with the vanilla plugin
  • [EDP] Improve Java type compatibility
  • [EDP] Add a new job-types endpoint
  • Enable Spark jobs to access Swift URL
  • Storage of recently logged events for clusters
  • Exceptions improvement
  • Use first_run to One-step Start Cluster
  • Enable HDFS NameNode High Availability with HDP 2.0.6 plugin
  • Indirect access to VMs
  • Plugin for Sahara with MapR
  • Refactor MapR plugin code
  • Clean up clusters that are in non-final state for a long time
  • Support multi-worker Sahara API deployment
  • New style logging
  • HTTPS support for sahara-api
  • Scenario integration tests for Sahara
  • Creation of Security Guidelines Documentation
  • Spark Temporary Job Data Retention and Cleanup
  • Specification Repository Backlog Refactor
  • Storm Integration
  • Support Cinder API version 2
  • Support Cinder availability zones
  • Support Nova availability zones
  • Spec - Add support for filtering results
  • Spec - Add support for editing templates
  • Cinder volume instance locality functionality
• Juno specs
  • Make anti affinity working via server groups
  • Append to a remote existing file
  • Plugin for CDH with Cloudera Manager
  • Specification for integration Sahara with Ceilometer
  • Store Sahara configuration in cluster properties
  • Security groups management in Sahara
  • Move the EDP examples from the sahara-extra repo to sahara
  • [EDP] Refactor job manager to support multiple implementations
  • [EDP] Add a Spark job type (instead of overloading Java)
  • [EDP] Add an engine for a Spark standalone deployment
  • [EDP] Using trust delegation for Swift authentication
  • Improve error handling for provisioning operations
  • Move Sahara REST API samples from /etc to docs

Unimplemented backlog specs:

• Backlog
  • Support for Boot from Volume
  • Revival of “Chronic EDP” work
  • Two step scaling with Heat engine

Other components:

• Sahara client
  • SaharaClient CLI as an OpenstackClient plugin
  • CLI: delete by multiple names or ids

• Sahara Image Elements
  • Add an option to sahara-image-create to generate bare metal images

• Sahara tests
  • Add an API to sahara-scenario for integration to another frameworks
  • Feature sets for scenario tests

Indices and tables

• Search Page