Count quota usage from placement¶
In Pike, we re-architected the quota system to count actual resource usage instead of using reservations and tracking quota usages in a separate database table. We’re counting resources like instances, CPU, and RAM by querying each cell database and aggregating the results per project and per user. This approach is problematic in the context of “handling of a down cell. If a cell becomes unavailable, resources in its database cannot be counted and will not be included in resource usage until the cell returns. Cells could become unavailable if an operator is performing maintenance on a cell or if a cell database is experiencing problems and we cannot connect to it.
We can make resource usage counting for quotas resilient to temporary cell outages by querying placement and the API database for resource usage instead of reading separate cell databases.
When we count quota resource usage for CPU and RAM, we do so by reading separate cell databases and aggregating the results. CPU and RAM amounts per instance are derived from the flavor and are stored in the database as columns in the instances table. So, each time we check quota usage against limits, we query a count of instance ID and a sum of CPU and RAM per cell database and aggregate them to calculate the resource usage.
This approach is sensitive to temporary cell outages which may occur during operator maintenance or if a cell database is experiencing issues and we cannot connect to it. While a cell is unavailable, we cannot count resource usage residing in that cell database and things would behave as though more quota is available than should be. That is, if someone has used all of their quota and part of it is in cell A and cell A goes offline temporarily, that person will suddenly be able to allocate more resources than their limit (assuming cell A returns, the person will have more resources allocated than their allowed quota).
We could take a different approach by querying the placement API and the API database to get resource usage counts. Since placement is managing resource allocations, it has the information we need to count resource usage for CPU and RAM quotas. By querying placement and the API database, we can avoid reading separate cell databases for resource usage.
Counting quota resource usage from placement would make quota behavior consistent in the event of temporary cell database disruptions. It would be easier for Operators to take cells offline if needed for maintenance without concern about the possibility of quota limits being exceeded during the maintenance. It could spare Operators the trouble of potentially having to fix cases where quota has been exceeded during maintenance or if a cell database connection could not be established.
We will add a new method for counting instances that queries the
instance_mappings table in the API database and make a separate limit check
for number of instances.
The new method will contain:
One query to the API database to get resource usage for instances. We can get the number of instances for a project and user if we add a new column
nova_api.instance_mappingstable. We already have a
project_idcolumn on the table. This will allow us to count instance mappings for a project and a user to represent the instance count.
We will rename the
_instances_cores_ram_count method to
_instances_cores_ram_count_legacy that counts cores and ram from the cell
databases and is only used if
[quota]count_usage_from_placement is False or
if the data migration has not yet completed.
Because there is not yet an ability to partition resource providers in
placement, in order to support deployments where multiple Nova deployments
share the same placement service, like possibly in an Edge scenario, we will
[quota]count_usage_from_placement config option which defaults to
False. If False, we use the legacy quota counting method for instances, cores,
and ram. If True, we use a quota counting method that calls placement. This is
a way to keep “legacy” quota counting available for the scenario of multiple
Nova deployments sharing one placement service. The config option will simply
control which counting method will be called by the pluggable quota system.
For example (pseudo-code):
We will add a new method for counting cores and ram from placement that is used
[quota]count_usage_from_placement is True. This method could be called
The new method will contain:
Up to two calls to placement to get resource usage for CPU and RAM. One call will count usage across a project. Then, if user-scoped quota limits are found for a resource, a second call will count usage across a project and a user. We can get CPU and RAM usage for a project and user by querying the
GET /usages?project_id=<project id> GET /usages?project_id=<project id>&user_id=<user id>
One alternative is to hold off on counting any quota usage from placement until placement has allocation partitioning support. The problem with that is in the meantime, the only solution we have for handling of down cells is to implement the policy-driven behavior where an operator has to choose between failing server create requests when a project has instances in a down cell or allowing server create requests to potentionally exceed quota limits.
Another alternative which has been discussed is, to use placement aggregates to
surround each entire Nova deployment and use that as a means to partition
placement usages. We would need to add a
aggregate= query parameter to the
/usages API in this case. This approach would also require some
work by either Nova or the operator to keep the placement aggregate
Data model impact¶
A nova_api database schema change will be required for adding the
column of type String to the
REST API impact¶
Other end user impact¶
End users will see consistent quota behavior even when cell databases are unavailable.
There will be a performance impact for checking if data needs to be migrated at the time of the quota check. The impact can be reduced by caching the results of checks that indicate data migration has been completed for a project and avoid a useless check per project in that case.
The change involves making external REST API calls to placement instead of doing a parallel scatter-gather to all cells. It might be slower to make the external REST API calls if all cells are fast responding. It might be faster to make external REST API calls if any cells are slower responding.
Other deployer impact¶
The addition of the
user_id column to the
table will require a data migration of all existing instance mappings to
user_id field. The migration routine would look for mappings
user_id is None and query cells by corresponding
the mapping. The query could filter on instance UUIDs, finding the
values to populate in the mappings. This would implement the batched
nova-manage db online_data_migrations way of doing the migration.
We will also heal/populate an instance mapping on-the-fly when it is accessed
during a server GET request. This would provide some data migration in the
situation where an upgrade has not run
nova-manage db online_data_migrations yet.
In order to handle a live in-progress upgrade, we will need to be able to fall
back on the legacy counting method for instances, cores, and ram if
nova_api.instance_mappings do not yet have
user_id populated (if the
operator has not yet run the data migration). We will need a way to detect that
the migration has not yet been run in order to fall back on the legacy counting
method. We could have a check such as
if exists(InstanceMapping.id) where
project_id=<project id> and user_id=None, then fall back on the legacy
counting method to query cell databases. We should cache the results of the
each migration completeness check per
project_id so we avoid needlessly
project_id that has already been migrated every time quota is
We will populate the
user_id field even for instance mappings that are
queued_for_delete=True because such instance mappings include instances
SOFT_DELETED and these can be restored at any time in the future.
If we do not migrate
SOFT_DELETED instances with
and they are restored in the future, their instance mappings would be
unmigrated and would prevent us being able to eventually drop the related data
The data migrations and fallback to the legacy counting method will be
temporary for Train, to be dropped in U or V with a blocker migration. That is,
you cannot pass
nova-manage api_db sync if there are any instance mappings
user_id=None to force the batched migration using
- Primary assignee:
- Other contributors:
Add a new column
Implement an online data migration to populate the
_server_group_count_members_by_userquota counting method to use only the
nova_api.instance_mappingstable instead of querying cell databases.
Add a config option
[quota]count_usage_from_placementthat defaults to False. This will be able to be deprecated when partitioning of resource providers is available in placement and other quirks around placement resource allocations in Nova are resolved in the future (example: “doubling” of allocations during resizes).
Add a new method to count instances with a count of
Add a new count method that queries the placement API for CPU and RAM usage. In the new count method, add a check for whether the online data migration has been run yet and if not, fall back on the legacy count method.
_instances_cores_ram_count_legacyand let it count only cores and ram in the legacy way, for use if
[quota]count_usage_from_placementis False or the data migration is not yet completed.
Adjust the nova-next CI job to run with
Unit tests and functional tests will be included to test the new functionality.
We will also adjust one CI job (nova-next) to run with
[quota]count_usage_from_placement=True to make sure we have integration
test coverage of that path.
The documentation of Cells v2 caveats will be updated to update the paragraph about the inability to correctly calculate quota usage when one or more cells are unreachable. We will document that beginning in Train, there are new deployment options.
This builds upon the work done in Pike to re-architect quotas to count resources.
This may also inadvertantly fix a bug we have where if the “recheck” quota check fails during the conductor check and the request is a multi-create, we will have all servers fall into ERROR state for the user to clean up. Because this change will count instance mappings for the instance count and instance mappings have almost [*] the same lifetime as build requests, we should not see the behavior of multi-create servers in ERROR state if they fail the quota “recheck” in conductor.