Support traits in Glance

https://blueprints.launchpad.net/nova/+spec/glance-image-traits

This blueprint proposes to extend the granular scheduling of compute instances to use traits provided by glance image metadata in addition to the traits provided by the flavor [1].

Problem description

Currently, traits provided by the flavor are used by the scheduler only if:

  1. Administrator creates a flavor with the traits required.

  2. User picks the correct flavor with the required traits from the list.

Cloud administrative users might want to put some qualitative attributes on workload images directly to assert control on where these workloads will be scheduled. These attributes can then be used as hints by the scheduler so that it can pick the right set of compute hosts which can support these workloads without the need for flavors to be pre-created and user needing to pick the right flavor.

The attributes on the image can be used to specify traits required on the hosts similar to traits on the flavor. For example support for a specific CPU instruction set, NIC features or whether it’s a trusted host, etc.

Use Cases

As a cloud administrator, I want to be able to assert further control on where certain workloads can be launched and be operational.

For example, if a workload instance will be applying an encryption algorithm or will process privacy data, there might be specific capabilities/traits that the instance needs on the hosts.

The administrator would be able to add these capability requirements to the glance image as required traits. This would allow the nova scheduler to automatically select only hosts which have those traits.

For administrative users

  • Allow administrators to specify a set of traits that an image requires.

Proposed change

Currently the traits information specified in the flavor is being collected as part of the boot request. This information is provided to the Placement API which uses it to filter out resource providers which do not have those traits associated with them.

We propose to allow traits to be specified in the image metadata to be part of the boot request work flow. The boot request will combine the traits information provided by the flavor AND the image metadata into a union of the two. This union set will be passed to the Placement API to generate allocation candidates.

The glance image already allows users to specify additional properties as key:value pairs. We will be re-using the same mechanism used by nova to encode traits information in nova’s flavor metadata items.

We propose to add trait: as a prefix so we can reuse the same name space as the traits specified in the flavor extra_specs.

The trait syntax in the glance image’s additional properties looks like:

trait:HW_CPU_X86_AVX2=required
trait:CUSTOM_TRUSTED_HOST=required

For now the only valid value is required. Validation of traits specified as part of the image additional properties is out of scope for this change.

Due to the difficulty of attempting to reconcile granular request groups between an image and a flavor, only the (un-numbered) trait group is supported. The traits listed there are merged with those of the un-numbered request group from the flavor.

Based on the ironic driver traits spec implemented we need to send image traits to ironic similar to how we are sending extra_specs traits to ironic.

Dealing with rebuild

In case of rebuild with new image(host and flavor staying the same), we need to make sure the image traits(if updated) are taken into account. Ideally the scheduler would request new candidates from placement and makes sure the current host is part of that list, but this is problematic in-case the compute is close to full as the current host will be excluded. This is described in the issue rebuild should not check with placement.

To resolve the above, the conductor can do pre-flight checks on the rebuild request to make sure the image traits can still be accommodated within the current allocations for that instance.

The conductor can request current allocations for the instance using GET /allocations/{instance_uuid} and collect all the resource providers and their corresponding traits from the allocations. It can then check to see if any of the requested image traits are missing from the set of traits above. If there are any missing traits, we can fail the rebuild.

Alternatives

Continue with the current model of utilizing the traits provided as part of the flavor in the boot request.

This would mean that the cloud administrator would need to create specific flavors with traits required for the workload image and the end user needs to select the flavor with the configured traits.

One other aspect to look at would be, because the flavor describes both the quantitative and qualitative aspects of the request, the number of flavors will need to increase substantially if we are given a mix of workloads.

In a typical Openstack installation with 7 flavors(nano -> xlarge) if we need to add one trait to each of the flavors we will end up with 14 flavors. If we need to add combinations of traits along with the quantitative aspects, this number will grow pretty quickly.

Another potential alternative is to provide traits directly as part of the instance boot request. But this has the same issue where the end user could forget to select the right traits.

With Image based traits the administrator sets the traits once on the image and the approach is immune to user errors during launch.

Another alternative is to use the AggregateImagePropertiesIsolation filter to filter select hosts within specific host aggregates. Host aggregate metadata is not standardized unlike the traits and also requires host aggregates to be pre-created with duplicated standard traits which is not ideal.

Dealing with rebuild

see rebuild should not check with placement

Alternative 1

If the image’s required traits have changed from the original image, we can reject the rebuild request at the API layer with a clear error message. This is a simpler approach but comes with draw backs.

In scenarios where a user is trying to do a rebuild that should be valid, the request would get rejected because old image traits != new image traits. It seems like unnecessary user and admin pain.

Alternative 2

The scheduler can request traits of current host using the existing GET /resource_providers/{UUID}/traits API and try to match the traits returned for the current host against the traits specified in the image.

If the traits do not match, NoValidHost exception will be raised before the filters are run. If the traits match, then the request will continue to be processed as it does currently(passing through the various filters etc)

Potential issue with this is that the traits on the image maybe attached to a nested resource provider under the compute node. For example, in case the instance is running on a host which has two SRIOV nic’s. One is normal SRIOV nic, another one with some kind of offload feature.

So, the original request is:

resources=SRIOV_VF:1

The instance gets a VF from the normal SRIOV nic.

But with the new image, the new request is:

resources=SRIOV_VF:1
traits=HW_NIC_OFFLOAD_XX

To handle nested resource providers and gather their traits we might need to make multiple GET /resource_providers/{UUID}/traits for every resource provider present in the tree.

Ideally this request should fail since we can’t ensure we allocated VF from the other SRIOV PF.

This alternative can also be implemented in the ImagePropertiesFilter in case of rebuild. But this is not ideal since none of the other filters make any API calls during the filtering process.

Other alternatives

Few other alternatives have been discussed on the ML [2].

Data model impact

Update ImageMetaProps class to return traits as key:value

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Upgrade impact

None.

Implementation

Assignee(s)

Primary assignee:

Arvind Nadendla <arvind.nadendla@intel.com>

Other contributors:

Mohammed Karimullah <karimullah.mohammed@intel.com>

Work Items

  • Update ImageMetaProps class to return traits

  • Update Nova Scheduler to extract properties from ImageMeta and pass them to the Placement API

  • Update Nova Conductor to validate the image traits match the existing allocations for the instance during a rebuild

  • Need to update the ironic virt driver to push traits from images to nodes based on ironic driver traits spec

Dependencies

None.

Testing

Unit tests and functional tests for building up requests shall be added.

Documentation Impact

References

History

Revisions

Release Name

Description

Rocky

Introduced