Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/fuel/+spec/ha-test-improvements
Need to add new HA tests and modify the existing one
We need to clarify the list of new tests and new checks and then implement it in system tests
No alternatives
No impact
No impact
No impact
No impact
No impact
No impact
No impact
No impact
Can be implemented by fuel-qa team in parallel
1. Shut down public vip two times (link to bug https://bugs.launchpad.net/fuel/+bug/1311749)
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Find node with public vip 3. Shut down eth with public vip 4. Check vip is recovered 5. Find node on which vip is recovered 6. Shut down eth with public vip one more time 7. Check vip is recovered 8. Run OSTF 9. Do the same for management vip
2. Galera does not reassemble on galera quorum loss (link to bug https://bugs.launchpad.net/fuel/+bug/1350545)
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Shut down one controller 3. Wait for galera cluster to reassemble (HA health check has passed) 4. Kill mysqld on second controller 5. Start first controller 6. Wait for 5 minutes that galera reassembles and check it reassembles 7. Run OSTF 8. Check rabbit status with MOS script
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Corrupt root file system on primary controller 3. Run OSTF
4. Block corosync traffic (link to bug https://bugs.launchpad.net/fuel/+bug/1354520)
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Login to rabbit master node 3. Block corosync traffic by extracting interface from management bridge 4. Unblock corosync traffic back 5. Check rabbitmqctl cluster_status at rabbit master node 6. Run OSTF HA tests
Steps: 1. Deploy HA cluster with Nova-network, 1 controller and 3 mongo nodes 2. Add 2 controller nodes and re-deploy cluster 3. Run OSTF 4. Add 2 mongo nodes and re-deploy cluster 5. Run OSTF
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Lock DB access on primary controller 3. Run OSTF
Steps: 1. Deploy HA cluster with Neutron Vlan, 3 controllers, 2 compute, eth1-eth4 interfaces are bonded in active backup mode 2. Destroy primary controller 3. Check pacemaker status 4. Run OSTF 5. Check rabbit status with MOS script (retry it during 5 min till successful result)
8. HA load testing with rally (May be not a part of this blueprint)
9. Need to test HA Neutron cluster under high load and simultaneous removing of virtual router ports (related link http://lists.openstack.org/pipermail/openstack-operators/ 2014-September/005165.html)
Steps: 1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute, cinder-neutron plugin enabled 2. Run network verification 3. Run OSTF
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute with cinder roles 2. Disable one compute node with nova-manage service disable –host=<compute_node_name> –service=nova-compute 3. On controller node under test (which compute node under test is connected to via rmq port 5673) repeat spawn / destroy instance requests continuosly (sleep 60) while the test is running 4. Add iptables block rule from compute IP to controller IP:5673 (take care for conntrack as well) iptables -I INPUT 1 -s compute_IP -p tcp –dport 5673 -m state –state NEW,ESTABLISHED,RELATED -j DROP 5. Wait 3 min for compute node under test should be marked as down in the nova service-list 6. Wait for another 3 min for it to be brought up back 7. Check for the compute node under test queue - it should be zero messages in it 8. Check if the instance could be spawned at the node
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. Ssh to every compute node 3. Kill nova-compute service 4. Check that service was restarted by monit
Steps: 1. Deploy HA cluster with Nova-network, 3 controllers, 2 compute 2. SSH to controller with running heat-engine 3. Check heat-engine status 4. Block heat-engine amqp connections 5. Check if heat-engine was moved to another controller or stopped on current controller 6. If moved - ssh to node with running heat-engine 6.1 Check heat-engine is running 6.2 Check heat-engine has some amqp connections 7. If stopped - check heat-engine process is running with new pid 7.1 Unblock heat-engine amqp connections 7.2 Check amqp connection re-appears for heat-engine
Steps: 1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute 2. Check the neutron-agents list consitency (no duplicates, alive statuses, etc) 3. On host with l3 agent create one more router 4. Check there are 2 namespaces 5. Destroy controller with l3 agent 6. Check it was moved to another controller, check all routers and namespaces were moved 7. Check metadata agent was also moved, there is process in router namespace that listen to 8775 port
Steps: 1. Deploy HA cluster with Neutron GRE, 3 controllers, 2 compute 2. Destroy controller with dhcp agent 3. Check it was moved to another controller 4. Check metadata agent was also moved, there is process in router namespace that listen to 8775 port