Server move operations with ports having resource request¶
Since microversion 2.72 nova supports creating servers with neutron ports having resource request. However moving such servers is not possible due to missing resource handling implementation in nova.
The owner or the admin needs to be able to resize, migrate, live-migrate, evacuate and unshelve after shelve offload servers having ports with resource request.
Servers could be created before Stein with SRIOV ports having QoS minimum bandwidth policy rule and for them the resource allocation is not enforced in placement during scheduling. After this spec is implemented the admin will be able to migrate such servers and the migration will heal the missing port allocations in placement.
The implementation of the bandwidth resource provider spec introduced the
requested_resources field in the
RequestSpec versioned object. During
server create this field is populated based on the resource request of the
neutron ports. The nova scheduler generates the allocation candidate query
including the request groups from the
requested_resources field as well.
However this field is not persisted to the database intentionally as the port
resource request is owned and persisted by neutron. So during any operation
that creates a new allocation of an existing server the port resource
request needs to be queried from neutron and the
to be populated in the
RequestSpec before the new allocation candidate
query is sent to placement.
After the new allocation is created in placement the
allocation key in the
binding:profile of the neutron port needs to be updated to point to the
new resource provider providing the resources for the port. To figure out
which resource provider fulfills which port’s resource request in a given
allocation the already introduced mapping code can be reused.
These move operations are the following:
unshelve after the server is shelve offloaded
The generic steps are:
the server move request reaches the conductor
the conductor calls query the ports from neutron that are bound to the server
the conductor updates the
RequestSpec.requested_resourcesfield based on the resource request of the ports
the conductor requests select_destination from the scheduler
the scheduler sends the allocation_candidate query to Placement based on
RequestSpec.requested_resources, then selects and allocates a candidate
the conductor updates the port - resource provider mapping in the
RequestSpecbased on the
Selectionobject returned from the scheduler
the conductor requests the move operation from the compute based on the
the target compute updates the port binding in neutron. The
allocationkey in the
binding:profileis also updated based on the mapping in the
During resize and migrate the compute manager’s
finish_resize call does the
port binding update. But the
RequestSpec is not passed to this call. So
here the RPC API needs to be extended with a new
finish_resize is called from source host’s
which does not get the
RequestSpec either so this RPC call also needs to
be extended. Fortunately the
prep_resize call on the destination host
already gets the
RequestSpec so it can call
resize_instance with the
During confirm resize (and migrate) the source allocation is deleted as today, no extra step is needed.
During revert resize (and migrate) the allocation is deleted on the
destination host and the source host allocation moved back from the
migration_uuid as consumer to the instance_uuid. No extra step is needed here.
However the port needs to be bound again to the source host and during that
allocation key of the
binding:profile need to be reverted
too. This means that the mapping needs to be re-calculated based on the
reverted allocation and data from placement. Alternatively we could store the
old mapping in the MigrationContext or start using the multi port binding API
and keep old mapping in the old binding. These alternatives both give extra
complexity to the solution.
During resize to same host the allocations are doubled in placement today. This will be true for the port related allocation as well.
During evacuate the compute manager’s
rebuild_instance call does the port
binding update. This call has a
request_spec parameter so this
RPC API does not need to be extended.
During unshelve of a shelve offloaded instance the compute manager’s
unshelve_instance call does the port binding update and here the RPC API
needs to be extended with a new
During live-migrate nova uses the multiple bindings API of neutron to manage
the bindings on the source and the target host in parallel. The conductor
creates the new, inactive binding on the destination host in neutron and it
will add the
allocation key in the new binding according to the
RequestSpec. When the live-migrate finishes the source port binding is
deleted along with the source host allocation. If the live-migration is
rolled back the source host binding still have the proper
The multiple bindings neutron API extension cannot be turned off so if it is not present nova can fail the live-migrate operation if ports have resource request.
Currently these move operations are rejected by nova if the server has ports attached with resource request. After the above proposed change is implemented these operations will be allowed. The way we will signal that nova is capable of supporting these operations is described in the REST API impact section.
For alternatives of the REST API change see the REST API impact section.
An alternative implementation could rely on the multiple port binding api of
neutron for all the move operations. In that solution the
object would not need to be passed down to the nova-compute in each move
operations as the inactive binding can be created with the necessary
allocation information in the conductor. However this solution would still
result in nova-compute impact to active the bindings during move instead of
creating a new binding as today.
Data model impact¶
REST API impact¶
There are two different approaches to signal that nova can handle the move operations for these servers:
Introduce a new microversion. If the move operations are requested with an older microversion for these servers then the request is rejected in the same way as today. If the move operation is requested with the new (or newer) microversion the request is accepted and handled properly.
Consider the missing support for these operations as bugs. Implement the above proposed changes as bugfixes without any new microversion. After the implementation is done requesting such move operations with any microversion is accepted and handled properly.
Other end user impact¶
During move operations the conductor needs to query neutron to get the resource request of the ports that are attached to the server. Also, after the scheduling the request group - resource provider mapping needs to be recalculated and the binding:profile of the ports needs to be updated in neutron.
Other deployer impact¶
As the solution requires RPC changes the move operations can only be supported
after both the source and the destination host are upgraded. So the conductor
needs to ensure that the service version of both compute is high enough. This
can be done similarly to how conductor checks the service version during live
migration. However if the conductor is configured with
[upgrade_levels]compute=auto (e.g. rolling upgrade) then even if both the
source and the destination computes are new enough but there are older computes
in the system then the older RPC version will be used and the
will be stripped from the calls. Therefore an additional check is needed. The
nova-compute needs to check if the instance has ports that would require
mapping and if the
RequestSpec is not provided in the call then fail the
The support for move operations makes it possible to heal missing or
inconsistent port allocation as during the move the requested resources are
re-calculated and the new allocation created accordingly in placement. This
will complement the port allocation healing capabilities of the
nova-manage placement heal_allocations CLI that has multiple limitation in
In general the operators having incomplete port allocations are recommended to
try to heal that with the
heal_allocations CLI in place if possible to
minimize the number for server move operations required.
- Primary assignee:
- Other contributors:
Add the RequestSpec to the compute RPC calls in a single step.
Implement support for each move operation as a separate task.
Each move operation will have a functional test asserting that the proper allocation exists after the move, old allocations are removed, and the port binding in neutron refers to the appropriate resource provider.
The bandwidth resource provider spec describing the support for creating such servers.
The documentation of microversion 2.72 introducing the support for creating such servers.
The nova API guide for the existing feature: Using ports with resource request
The neutron admin guide for this feature Quality of Service Guaranteed Minimum Bandwidth
ML thread about the possible options for the API impact.