Template based test cases with external configuration

Improve the system test for using external configuration

Problem description

We have a permanent growing product with a lot of features. From release to release we should cover more and more cases and current approach can not keep up with the changes.

• Tests have a lot of copy-paste code
• Many tests have a hardcoded cluster configuration
• For test additional configuration with the same steps, we should write another test
• A lot of tests have output without step separation and timings

Proposed changes

Write extension for current test framework which might works with external configuration and has template structure for the test cases.

Pros:

• Configuration for test be at the external human readable yaml file
• Unified library for steps of test, checkers and actions
• Inheritance, simple expanding and composition new cases with already existing
• Get a test matrix cases and configuration

Configuration files for tests

Basic idea, why we use configuration. We can store everything what we need for test in one place.

Configuration contains a basic settings for OpenStack which provided into Fuel when test creates OS environment.

For match the test with the configuration, framework uses name of configuration file like suffix for base_groups of test (see “Group names example” below).

Configuration for each test is stored in YAML file under the key ‘template’.

Main sections:

• name: short description of the environment configuration
• devops-template: fuel-devops environment configuration for the test case, where are described configurations for all virtual or hardware nodes and L2 network topology.
• cluster-template: all data that are necessary for Fuel cluster creation, including: - “name” of the OpenStack cluster in Fuel - “release” of operation system for nodes in the cluster - “nodes” roles mapping on fuel-devops nodes - “network” type selection (gre/vlan/vxlan/..) - various “settings” for the cluster, such as enabled/disabled components for the cluster, storage configuration, additional data for configuring different plugins or other components, and so on.

Basic structure for configuration file (example_config.yaml)

template:
    name: 1 Controller, 1 Compute, 1 Cinder on Neutron/VLAN
    slaves: 3
    devops-template: *devops-config
    cluster-template:
        name: env1
        release: ubuntu
        settings:
            components: *components-config
            storages: *storages-config
            plugin-aaa:
              # Specific for plugin-aaa configuration
              ...
            plugin-bbb:
              # Specific for plugin-bbb configuration
              ...
        network: *network-config
        nodes: *nodes

Components config

components-config: &components-config
    sahara: false
    murano: false
    ceilometer: false

Storages configuration

storages-config: &storages-config
    volume-lvm: true
    volume-ceph: false
    image-ceph: false
    ephemeral-ceph: false
    rados-ceph: false
    replica-ceph: 2

Network configuration

network-config: &network-config
    provider: neutron
    segment-type: vlan
    public-ip-to-all: false

Node configuration

nodes: &nodes
    - roles:
        - controller
      count: 1
      node_group: rack-01 # Assign node into devops node group
    - roles:
        - compute
      count: 1
      node_group: rack-01
    - roles:
        - cinder
      count: 1
      node_group: rack-01

Placement of template files in fuel-qa repository

.fuel-qa
|-- fuelweb_test/
..
|
`-- system_test/
    |-- helpers/
    |   |-- utils.py
    |   ..
    |
    |   # Core functiunality of framework
    |-- core/
    |   |-- factory.py
    |   |-- decorators.py
    |   ..
    |
    |   # Actions library for test
    |-- actions/
    |   |-- base.py
    |   |-- cluster.py
    |   ..
    |
    |   # Test cases that contain different action lists
    |-- tests/
    |   |-- test_foo.py
    |   |-- test_bar.py
    |   ..
    |
    |   # Environments and test cases configurations
    `-- tests_templates/
        |
        |   # Configs for test cases
        |-- tests_configs/
        |   |-- ceph_all_ceilo_on_neutron_tun.yaml
        |   |-- ceph_all_on_neutron_vlan.yaml
        |   |-- example_test_environment.yaml
        |   ..
        |
        |   # Additional data for including into test cases configs
        |-- cluster_configs/
        |   |-- networks/
        |   |   |-- neutron_gre.yaml
        |   |   |-- neutron_tun.yaml
        |   |   |-- neutron_vlan.yaml
        |   |   ..
        |   |
        |   |-- nodes/
        |   |   |-- 1ctrl_1comp.yaml
        |   |   |-- 1ctrl_2comp_1cndr_3ceph_1mongo.yaml
        |   |   ..
        |   |
        |   `-- settings/
        |       |-- cinder_ceilometer.yaml
        |       |-- cinder_cephImg_ceilometer.yaml
        |       ..
        |
        |   # fuel-devops configs for including into test cases configs
        `-- devops_configs/
            |-- default.yaml
            ..

Test cases re-design

New approach for writing of test scripts.

• coding separate steps like atomic actions
• combine and sort steps as needed for a scenario
• better a test report which contains each step and result for it
• more readable test output to improve quality of investigation

Actions example:

class BaseActions(object):

  # Default value
  deploy_timeout = 1200

  @action
  def prepare_env(self):
      """Prepare VMs"""
      pass

  @action
  def bootstrap_slaves(self):
      """Bootstrap slaves and make snapshot"""
      pass

  @action
  def deploy_cluster(self):
      """Deploy environment"""
      self.fuel_web.deploy_cluster_wait(self.cluster_id,
          timeout=self.deploy_timeout)

  @action
  def network_check(self):
      """Run network checker"""
      self.fuel_web.verify_network(self.cluster_id)

  @action
  def health_check(self):
      """Run health checker"""
      self.fuel_web.run_ostf(self.cluster_id)

  @nested_action
  def prepare_and_bootstrap():
      return [
          'prepare_env',
          'bootstrap_slaves'
      ]

Test example:

@testcase(groups = ['system_test', 'system_test.deploy_ostf'])
class CreateDeployOstf(BaseBase, BaseActions):
  """Case deploy Environment
      Scenario:
      1. Deploy Environment
      2. Run network checker
      3. Run OSTF
  """

  # To control behavior of action use a class attribute
  deploy_timeout = 1800

  actions_order = [
      'prepare_and_bootstrap',
      'deploy_cluster',
      'network_check',
      'health_check',
  ]

Group names example:

# Run all test cases for base_group 'system_test' using
# config file ceph_all_ceilo_on_neutron_tun.yaml :
./utils/jenkins/system_tests.sh  ... \
  --group=system_test(ceph_all_ceilo_on_neutron_tun)

# Run test cases for base_group 'system_test.deploy_ostf' using
# config file ceph_all_on_neutron_vlan.yaml :
./utils/jenkins/system_tests.sh  ... \
  --group=system_test.deploy_ostf(ceph_all_on_neutron_vlan)

# Run all test cases for base_group 'system_test' using
# all existing config files from system_test/tests_templates/tests_configs/:
./utils/jenkins/system_tests.sh  ... --group=system_test

Running new test cases

For selecting test with specific configuration please use special test group. It contains combination of base_groups from the test plus name of configuration file without extension. Test group and configuration group divided by point - BASE_GROUP(CONFIG_NAME):

• system_test.example_config
• system_test.deploy_ostf.example_config

Web UI

None

Nailgun

None

Data model

None

REST API

No FUEL REST API changes.

Orchestration

None

RPC Protocol

None

Fuel Client

None

Plugins

None

Fuel Library

None

Alternatives

• Passing configuration of openstack environment through system environment variable. If use this way, we’ll not have dynamic mapping confguraion and tests. That will make CI impact, we should add variable for configuration to each Jenkins jobs and maintain it every time when we add new configuration.

Upgrade impact

N/A

Security impact

N/A

Notifications impact

N/A

End user impact

N/a

Performance impact

N/A

Deployment impact

N/A

Developer impact

N/A

Infrastructure impact

N/A

Documentation impact

• fuel-qa
• fuel-devops

Implementation

Assignee(s)

Primary assignee:

Dmytro Tyzhnenko

Other contributors:

Denys Dmytriiev

Mandatory design review:

Anastasiia Urlapova, Denys Dmytriiev

Work Items

• Create configuration structure
• Code base models for templated tests
• Implement collector of test + configuration combination
• Integrate with current framework
• Update reporting tools
• Checkers for tests and configs

Dependencies

• Environment templates for devops https://blueprints.launchpad.net/fuel/+spec/template-based-virtual-devops-environments

Testing, QA

All existed tests and tools should work as worked before.

Acceptance criteria

Tool which can combine templated tests and external configuration files on same infrastructure as exist today.

References

https://blueprints.launchpad.net/fuel/+spec/template-based-testcases