Cross-service Tempest testing – Devstack

At the OpenStack summit in Sydney, I ran a workshop about writing a Tempest plugin for cross-service integration testing.
This is the first of a series of posts where I will cover the material from the workshop. To make this more practical for people to use, I will also include examples on how to integrate this kind of test in an OpenStack CI job. I expect to split this into four posts, roughly covering the following topics:

  • Introduction and how to set up the test environment using Devstack (this post)
  • How to create a Tempest plugin from scratch
  • How to write a Tempest scenario test for cross-service integration testing
  • Writing an OpenStack CI job in Zuul V3 native format

This series of posts assume you have a basic familiarity with OpenStack testing infrastructure and Tempest.
The infra user manual and Tempest docs can provide more background information.


As part of the “Technical Committee Vision for 2019”, the OpenStack TC introduced the idea of “constellations”, which are reference sets of OpenStack projects, defined to help users get started with OpenStack.

The idea of the workshop was to demonstrate what Tempest plugins are and how they can be used and combined together to test specific OpenStack constellation. Since a lot of (testing) code starts as copy/paste from an example, I wanted to have a good reference example for test developers to re-use.

Scenario Description

I set-up a fictitious constellation for users who want to use the compute service with DNS. It includes the standard compute set: keystone, nova, neutron, glance and cinder, plus Designate for DNS services and Heat for Orchestration.
Nova, Neutron and Designate integration is well documented, however, there are no integration tests written to validate it. Using this constellation would give me the opportunity to build the workshop demo and also write something actually useful.

Test Scenario Diagram
Test Scenario (CC-BY 2018 Andrea Frittoli)

Setting up the test environment

The first thing I needed was a cloud running all the services I wanted, so I could fire tests against it. I needed the smallest possible footprint so that before the workshop I could spawn one cloud for each workshop attendee. I decided to use devstack, which is OpenStack development and test environment. I chose OpenStack version Pike, which at the time of writing was the latest stable OpenStack release.

Configuring devstack for my purposes was easy enough, using a combination of Neutron and Designate configuration.
What I did was:

  • Disable Horizon and Swift to reduce the footprint
  • Enable heat and designate by loading the corresponding devstack plugins
  • Configure the DNS driver and DNS domain in Neutron
  • Set Designate specific settings in Neutron

Which resulted in the following changes:

The only issue I encountered was with the Designate backend. The version of PowerDNS on Ubuntu Xenial does not match the existing driver, so it cannot be used until the driver is re-written. The default has since been changed to bind9 on stable/pike, so the extra configuration change should not be needed anymore.

The full configuration file is available here.

With devstack configuration ready, to stand up a test cloud I only need to run Configuration settings are then propagated into service configuration files as well as test configuration files (where relevant). The devstack plugin interface defines a test-config interface that can be used to configure the corresponding plugin or even core Tempest settings.

Create a workshop test image

I wanted the devstack setup to be easily reproducible so I could easily rebuild the test environment at any time. I also wanted to give workshop attendees an easy way to set up the test environment and play re-run the workshop if they wanted to.
I wrote the automation in Ansible – the full code is available on GitHub.

The playbook provisions a virtual machine (VM) in an OpenStack cloud using Ansible OpenStack cloud modules. It then adds the created VM to ansible inventory dynamically:

Once the VM is provisioned and ssh-able, it runs a number of roles against it to prepare it for Devstack and to prefetch relevant repositories. The script “” will be executed at the first boot of the VM thanks to a systemd unit file:

Finally, a new snapshot is taken of the VM and the old snapshot (if any) is deleted.

Reconfiguring the test environment

I wanted people attending the workshop to be able to configure their own DNS domain for the exercise. However, running is time-consuming (20+ minutes) and I did not want to spend that time during the workshop. To solve this I asked attendees to change the configuration directly in Neutron:

In the next blog post I will explain how to create a new Tempest plugin.

Leave a Reply