Template based test cases with external configuration

Improve the system test for using external configuration

Problem description

We have a permanent growing product with a lot of features. From release to release we should cover more and more cases and current approach can not keep up with the changes.

  • Tests have a lot of copy-paste code
  • Many tests have a hardcoded cluster configuration
  • For test additional configuration with the same steps, we should write another test
  • A lot of tests have output without step separation and timings

Proposed changes

Write extension for current test framework which might works with external configuration and has template structure for the test cases.


  • Configuration for test be at the external human readable yaml file
  • Unified library for steps of test, checkers and actions
  • Inheritance, simple expanding and composition new cases with already existing
  • Get a test matrix cases and configuration

Configuration files for tests

Basic idea, why we use configuration. We can store everything what we need for test in one place.

Configuration contains a basic settings for OpenStack which provided into Fuel when test creates OS environment.

For match the test with the configuration, framework uses name of configuration file like suffix for base_groups of test (see “Group names example” below).

Configuration for each test is stored in YAML file under the key ‘template’.

Main sections:

  • name: short description of the environment configuration
  • devops-template: fuel-devops environment configuration for the test case, where are described configurations for all virtual or hardware nodes and L2 network topology.
  • cluster-template: all data that are necessary for Fuel cluster creation, including: - “name” of the OpenStack cluster in Fuel - “release” of operation system for nodes in the cluster - “nodes” roles mapping on fuel-devops nodes - “network” type selection (gre/vlan/vxlan/..) - various “settings” for the cluster, such as enabled/disabled components for the cluster, storage configuration, additional data for configuring different plugins or other components, and so on.

Basic structure for configuration file (example_config.yaml)

    name: 1 Controller, 1 Compute, 1 Cinder on Neutron/VLAN
    slaves: 3
    devops-template: *devops-config
        name: env1
        release: ubuntu
            components: *components-config
            storages: *storages-config
              # Specific for plugin-aaa configuration
              # Specific for plugin-bbb configuration
        network: *network-config
        nodes: *nodes

Components config

components-config: &components-config
    sahara: false
    murano: false
    ceilometer: false

Storages configuration

storages-config: &storages-config
    volume-lvm: true
    volume-ceph: false
    image-ceph: false
    ephemeral-ceph: false
    rados-ceph: false
    replica-ceph: 2

Network configuration

network-config: &network-config
    provider: neutron
    segment-type: vlan
    public-ip-to-all: false

Node configuration

nodes: &nodes
    - roles:
        - controller
      count: 1
      node_group: rack-01 # Assign node into devops node group
    - roles:
        - compute
      count: 1
      node_group: rack-01
    - roles:
        - cinder
      count: 1
      node_group: rack-01

Placement of template files in fuel-qa repository

|-- fuelweb_test/
`-- system_test/
    |-- helpers/
    |   |-- utils.py
    |   ..
    |   # Core functiunality of framework
    |-- core/
    |   |-- factory.py
    |   |-- decorators.py
    |   ..
    |   # Actions library for test
    |-- actions/
    |   |-- base.py
    |   |-- cluster.py
    |   ..
    |   # Test cases that contain different action lists
    |-- tests/
    |   |-- test_foo.py
    |   |-- test_bar.py
    |   ..
    |   # Environments and test cases configurations
    `-- tests_templates/
        |   # Configs for test cases
        |-- tests_configs/
        |   |-- ceph_all_ceilo_on_neutron_tun.yaml
        |   |-- ceph_all_on_neutron_vlan.yaml
        |   |-- example_test_environment.yaml
        |   ..
        |   # Additional data for including into test cases configs
        |-- cluster_configs/
        |   |-- networks/
        |   |   |-- neutron_gre.yaml
        |   |   |-- neutron_tun.yaml
        |   |   |-- neutron_vlan.yaml
        |   |   ..
        |   |
        |   |-- nodes/
        |   |   |-- 1ctrl_1comp.yaml
        |   |   |-- 1ctrl_2comp_1cndr_3ceph_1mongo.yaml
        |   |   ..
        |   |
        |   `-- settings/
        |       |-- cinder_ceilometer.yaml
        |       |-- cinder_cephImg_ceilometer.yaml
        |       ..
        |   # fuel-devops configs for including into test cases configs
        `-- devops_configs/
            |-- default.yaml

Test cases re-design

New approach for writing of test scripts.

  • coding separate steps like atomic actions
  • combine and sort steps as needed for a scenario
  • better a test report which contains each step and result for it
  • more readable test output to improve quality of investigation

Actions example:

class BaseActions(object):

  # Default value
  deploy_timeout = 1200

  def prepare_env(self):
      """Prepare VMs"""

  def bootstrap_slaves(self):
      """Bootstrap slaves and make snapshot"""

  def deploy_cluster(self):
      """Deploy environment"""

  def network_check(self):
      """Run network checker"""

  def health_check(self):
      """Run health checker"""

  def prepare_and_bootstrap():
      return [

Test example:

@testcase(groups = ['system_test', 'system_test.deploy_ostf'])
class CreateDeployOstf(BaseBase, BaseActions):
  """Case deploy Environment
      1. Deploy Environment
      2. Run network checker
      3. Run OSTF

  # To control behavior of action use a class attribute
  deploy_timeout = 1800

  actions_order = [

Group names example:

# Run all test cases for base_group 'system_test' using
# config file ceph_all_ceilo_on_neutron_tun.yaml :
./utils/jenkins/system_tests.sh  ... \

# Run test cases for base_group 'system_test.deploy_ostf' using
# config file ceph_all_on_neutron_vlan.yaml :
./utils/jenkins/system_tests.sh  ... \

# Run all test cases for base_group 'system_test' using
# all existing config files from system_test/tests_templates/tests_configs/:
./utils/jenkins/system_tests.sh  ... --group=system_test

Running new test cases

For selecting test with specific configuration please use special test group. It contains combination of base_groups from the test plus name of configuration file without extension. Test group and configuration group divided by point - BASE_GROUP(CONFIG_NAME):

  • system_test.example_config
  • system_test.deploy_ostf.example_config

Web UI




Data model



No FUEL REST API changes.



RPC Protocol


Fuel Client




Fuel Library



  • Passing configuration of openstack environment through system environment variable. If use this way, we’ll not have dynamic mapping confguraion and tests. That will make CI impact, we should add variable for configuration to each Jenkins jobs and maintain it every time when we add new configuration.

Upgrade impact


Security impact


Notifications impact


End user impact


Performance impact


Deployment impact


Developer impact


Infrastructure impact


Documentation impact

  • fuel-qa
  • fuel-devops



Primary assignee:
Dmytro Tyzhnenko
Other contributors:
Denys Dmytriiev
Mandatory design review:
Anastasiia Urlapova, Denys Dmytriiev

Work Items

  • Create configuration structure
  • Code base models for templated tests
  • Implement collector of test + configuration combination
  • Integrate with current framework
  • Update reporting tools
  • Checkers for tests and configs

Testing, QA

All existed tests and tools should work as worked before.

Acceptance criteria

Tool which can combine templated tests and external configuration files on same infrastructure as exist today.