Placement API support for instance reservation

https://blueprints.launchpad.net/blazar/+spec/placement-api

Placement API 1 was introduced to Nova at the 14.0.0 Newton release to separate API and data model for tracking resource provider inventories and usages. It can be used for improving instance reservation.

Problem description

Current instance reservation has the following constraints:

  • A user has to create instance reservation with the anti-affinity policy. Therefore, the amount of instances in one reservation cannot be larger than the amount of hosts.

  • A user has to specify the server group when the user launches instances on reserved resources. If it is not specified, more instances than the reserved amount are possibly launched.

Use Cases

  • A user wants to reserve instance resources with arbitrary affinity policy.

  • A user wants to reserve more instances than the number of hosts.

Proposed change

Use the custom resource class to represent reservation resources and use the nested resource provider to manage capacity and usage of reservation resources. The following sections describe how Blazar interacts with Nova and Placement for supporting instance reservation.

When creating a host:

  1. Get hypervisor information and store it into the computehosts table.

  2. Create a nested resource provider as a child of the compute node resource provider by calling the Create resource provider API. The UUID of the compute node resource provider can be retrieved by calling the List resource providers API with the name option query, e.g. GET /placement/resource_proiders?name=compute-1.

    The child resource provider is referred to as ‘reservation provider’ in the following sections.

    Create reservation provider request body example:

    POST /placement/resource_providers

{
    "name": "blazar_compute-1",
    "parent_provider_uuid": "542df8ed-9be2-49b9-b4db-6d3183ff8ec8"
}

Note

“542df8ed-9be2-49b9-b4db-6d3183ff8ec8” is the UUID of the “compute-1” compute node.

  1. Add the host into the freepool.

When creating a lease:

  1. Look for available resources with arbitrary affinity policy.

  2. Update the computehost_allocations table.

  3. Create a custom resource class CUSTOM_RESERVATION_{reservation UUID} by calling the Create resource class API.

    Create resource class request body example:

    POST /placement/resource_classes

{
    "name": "CUSTOM_RESERVATION_4D17D41A_830D_47B2_91C7_4F9FC0AE611E"
}

Note

Use upper case and under score for the custom resource class name because lower case and hyphen cannot be used.

  1. Create a private flavor which has resources:CUSTOM_RESERVATION_{reservation UUID}=1 in its extra_spec.

Note

  • A host aggregate is not created for each instance reservation anymore because reserved hosts can be distinguished by the reservation provider inventory.

  • A server group is not created anymore because the proposed approach does not depend on the ServerGroup(Anti)AffinityFilter.

When starting a lease:

  1. Add the custom resource class CUSTOM_RESERVATION_{reservation UUID} into the reservation provider’s inventory by calling the Update resource provider inventories API with the total parameter which equals to the amount of instances reserved for the host.

    Update resource provider inventories request body example:

    PUT /placement/resource_providers/{reservation_provider_uuid}/inventories

{
    "inventories": {
        "CUSTOM_RESERVATION_4D17D41A_830D_47B2_91C7_4F9FC0AE611E": {
            "total": 3,
            "allocation_ratio": 1.0,
            "min_unit": 1,
            "max_unit": 1,
            "step_size": 1
        },
        "snip"
    },
    "resource_provider_generation": 5
}

Note

Existing hosts which were created before this spec is implemented do not have the reservation provider. So, check if the reservation provider exists and create it if it does not exist before this step.

  1. Add the lease owner’s project to the private flavor access rights list.

Note

The previous implementation of starting lease should be kept until the previous instance reservation is deprecated and completely removed. The previous instance reservations can be distinguished by checking the aggregate_id or server_group_id column in the instance_reservations table.

When launching instances (from user point of view):

  1. A lease owner uses the private flavor and the instance is launched on the reserved host which has the CUSTOM_RESERVATION_{reservation UUID} in it’s child resource provider inventory, i.e. reservation provider inventory.

    Consumption of CUSTOM_RESERVATION_{reservation UUID} resources in the reservation provider inventory is claimed by the Nova scheduler. It means that usage of reserved resources is automatically tracked by the Placement.

Note

It still depends on the BlazarFilter though the BlazarFilter will be ideally removed in the future. The BlazarFilter is changed to check if resources:CUSTOM_RESERVATION_* is in flavor extra specs to distinguish the request from normal, i.e. non-reserved, instance creation requests.

Traits or other features would be able to be used for solving BlazarFilter dependency. It would be addressed by another blueprint.

On the other hand, dependency on the following filters are solved. These filters are not needed any more.

  • AggregateInstanceExtraSpecsFilter

  • AggregateMultiTenancyIsolationFilter

  • ServerGroupAntiAffinityFilter

Note that above filters and existing logic in the BlazarFilter should be kept to keep backward compatibility until the previous instance reservation is deprecated and completely removed.

When terminating a lease:

  1. Delete related instances and the private flavor.

  2. Remove the CUSTOM_RESERVATION_{reservation UUID} class from the reservation provider’s inventory by calling the Delete resource provider inventory API.

  3. Delete the CUSTOM_RESERVATION_{reservation_UUID} resource class by calling the Delete resource class API.

Note

The previous implementation of terminating lease should be kept until the previous instance reservation is deprecated and completely removed. The previous instance reservations can be distinguished by checking the aggregate_id or server_group_id column in the instance_reservations table.

When deleting a host:

  1. Delete the reservation provider which is associated with the host by calling the Delete resource provider API.

  2. Remove the host from the freepool.

  3. Update the computehosts table.

Alternatives

Dummy resources approach

Update inventories of the general resources, e.g. VCPU, of compute nodes in the freepool to be zero or reserved. And add dummy resources like CUSTOM_VCPU_{reservation UUID} into the inventory. This approach complicates resource usage tracking because real usage of each general resource cannot be seen through the top level compute node inventory.

Traits approach

Use Traits to express reserved resources. The problem is that traits are just traits and they cannot be used for managing capacity and usage of reserved resources.

Data model impact

The affinity column of the instance_reservations table is changed to allow NULL. NULL means no affinity policy is applied while True means affinity is applied and False means anti-affinity is applied.

The instance_reservations table:

ALTER TABLE instance_reservations
    ALTER COLUMN affinity NULL;

After the previous instance reservation is deprecated and completely removed, drop the following columns in the instance_reservations table:

ALTER TABLE instance_reservations
    DROP COLUMN aggregate_id, server_group_id;

REST API impact

The affinity parameter of the Create lease API is changed to be an optional parameter. If the affinity parameter is not given, no affinity policy is applied.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

  • The Placement API has to be newer than or equal to Ver. 1.29.

  • To upgrade from the previous version, run the DB upgrade script and the instance_reservations table schema will be updated.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

<tetsuro>

Other contributors:

<hiro-kobayashi>

Work Items

Base:

  • Update DB schema: update the instance_reservations table.

  • Add placement library in blazar/utils/openstack module.

To support the host creation:

  • Update the create_computehost() of the host plugin to call Placement APIs and update related tables.

To support the host deletion:

  • Update the delete_computehost() of the host plugin to delete Placement related resources.

To support the lease creation:

  • Update the query_available_hosts() to return how many instance can be launched on each available host.

  • Update the pickup_hosts() to support arbitrary affinity policy.

  • Update the reserve_resource() and update_reservation() to support multiple allocations which have the same pair of reservation_id and computehost_id.

  • Update the _create_resources() of the instance plugin to create the CUSTOM_RESERVATION_{reservation UUID} class and add it into the private flavor extra specs.

To support starting the lease:

  • Update the on_start() of the instance plugin to add the CUSTOM_RESERVATION_{reservation UUID} into the reservation provider inventory. The total parameter equals to the number of entries of the computehost_allocations table which have the same reservation id and computehost id.

To support launching reserved instances:

  • Update the BlazarFilter.

To support termination of the lease:

  • Update the on_end() of the instance plugin to remove the custom resource from the reservation provider inventory and delete the class itself.

Others:

  • Update the api module and the python-blazarclient to support arbitrary affinity policies.

  • Update the blazar-dashboard to support arbitrary affinity policies.

  • Update documentation.

Dependencies

WIP: Check Placement API development status.

Testing

  • Add unit tests for new features of each method described in the work items section.

  • Add test scenarios of instance reservation with the affinity policy and no affinity policy.

Documentation Impact

  • Parameter description of the Create Lease API reference will be updated.

  • Instance reservation part of the Command-Line Interface Reference will be updated.

  • Release notes will be added.

References

1

https://docs.openstack.org/nova/latest/user/placement.html

History

Revisions

Release Name

Description

Rocky

Introduced