Provider Configuration File¶
This is a proposal to configure resource provider inventory and traits using a standardized YAML file format.
This work is derived from Jay’s Rocky provider-config-file proposal and Konstantinos’s device-placement-model spec (which is derived from Eric’s device-passthrough spec), but differs in several substantive ways.
This work is influenced by requirements to Nova to support non native compute resources that are managed by Resource Management Daemon for finer grain control. PTG discussion notes available at Resource Management Daemon_PTG Summary
We currently limit the ownership and consumption of the provider config YAML as described by the file format to Nova only.
The provider config will currently only accept placement overrides to create and manage inventories and traits for resources not natively managed by the Nova virt driver.
This is intended to define a) a file format for currently active use cases, and b) Nova’s consumption of such files. Subsequent features can define the semantics by which the framework can be used by other consumers or enhanced to satisfy particular use cases.
In order to facilitate the proper management of resource provider information in the placement API by agents within Nova (such as virt drivers and the PCI passthrough subsystem), we require a way of expressing various overrides for resource provider information. While we could continue to use many existing and new configuration options for expressing this information, having a standardized, versioned provider descriptor file format allows us to decouple the management of provider information from the configuration of the service or daemon that manages those resource providers.
Note that the file format/schema defined here is designed to accommodate the following use cases. The file format/schema currently addresses a few use cases that require changes to resource provider information as consumed by virt drivers in Nova but it should allow options for extensions to be consumed by Nova or other services as described in the problem statement in the future.
An operator would like to describe inventories for new platform features
These features could be experimental or not yet completely supported by Nova.
The expectation is that Nova can manage these inventories and help schedule
workloads requesting support for new platform features against their
capacities. For instance, to report
CUSTOM_LLC (last-level cache)
The file defined by this spec must allow its author to:
Identify a provider unambiguously.
Create and manage inventories for resource classes not natively managed by Nova virt driver (
An operator wishes to associate new custom traits with a provider.
These features could be experimental or not yet completely supported by Nova. The expectation is that Nova can manage these traits and help schedule workloads with support to new platform features against their traits.
The file defined by this spec must allow its author to:
Identify a provider unambiguously.
Specify arbitrary custom traits which are to be associated with the provider.
Provider Config File Schema¶
A versioned YAML file format with a formal schema is proposed. The scope of this spec is the schema, code to parse a file into a Python dict, code to validate the dict against the schema, and code to merge the resulting dict with the provider tree as processed by the resource tracker.
The code shall be introduced into the
openstack/nova project initially and
consumed by the resource tracker. Parts of it (such as the schema definition,
file loading, and validation) may be moved to a separate oslo-ish library in
the future if it can be standardized for consumption outside of Nova.
The following is a simplified pseudo-schema for the file format.
meta: # Version ($Major, $minor) of the schema must successfully parse documents # conforming to ($Major, *). I.e. additionalProperties must be allowed at # all levels; but code at a lower $minor will ignore fields it does not # recognize. Schema changes representing optional additions should bump # $minor. Any breaking schema change (e.g. removing fields, adding new # required fields, imposing a stricter pattern on a value, etc.) must bump # $Major. The question of whether/how old versions will be deprecated or # become unsupported is left for future consideration. schema_version: $Major.$minor providers: # List of dicts # Identify a single provider to configure. # Exactly one of uuid or name is mandatory. Specifying both is an error. # The consuming nova-compute service will error and fail to start if the # same value is used more than once across all provider configs for name # or uuid. # NOTE: Caution should be exercised when identifying ironic nodes, # especially via the `$COMPUTE_NODE` special value. If an ironic node # moves to a different compute host with a different provider config, its # attributes will change accordingly. - identification: # Name or UUID of the provider. # The uuid can be set to the specialized string `$COMPUTE_NODE` which # will cause the consuming compute service to apply the configuration # in this section to each node it manages unless that node is also # identified by name or uuid. uuid: ($uuid_pattern|"$COMPUTE_NODE") # Name of the provider. name: $string # Customize provider inventories inventories: # This section allows the admin to specify various adjectives to # create and manage providers' inventories. This list of adjectives # can be extended in the future as the schema evolves to meet new # use cases. For now, only one adjective, `additional`, is supported. additional: # The following inventories should be created on the identified # provider. Only CUSTOM_* resource classes are permitted. # Specifying inventory of a resource class natively managed by # nova-compute will cause the compute service to fail. $resource_class: # `total` is required. Other optional fields not specified # get defaults from the Placement service. total: $int reserved: $int min_unit: $int max_unit: $int step_size: $int allocation_ratio: $float # Next inventory dict, keyed by resource class... ... # Customize provider traits. traits: # This section allows the admin to specify various adjectives to # create and manage providers' traits. This list of adjectives # can be extended in the future as the schema evolves to meet new # use cases. For now, only one adjective, `additional`, is supported. additional: # The following traits are added on the identified provider. Only # CUSTOM_* traits are permitted. The consuming code is # responsible for ensuring the existence of these traits in # Placement. - $trait_pattern - ... # Next provider... - identification: ...
This section is intended to describe at a very high level how this
file format could be consumed to provide
This section is intended to describe at a very high level how this file format could be consumed to provide P-state compute trait information.
meta: schema_version: 1.0 providers: # List of dicts - identification: uuid: $COMPUTE_NODE inventories: additional: CUSTOM_LLC: # Describing LLC on this compute node # max_unit indicates maximum size of single LLC # total indicates sum of sizes of all LLC total: 22 reserved: 2 min_unit: 1 max_unit: 11 step_size: 1 allocation_ratio: 1 traits: additional: # Describing that this compute node enables support for # P-state control - CUSTOM_P_STATE_ENABLED
Provider config consumption from Nova¶
Provider config processing will be performed by the nova-compute process as described below. There are no changes to virt drivers. In particular, virt drivers have no control over the loading, parsing, validation, or integration of provider configs. Such control may be added in the future if warranted.
A new config option is introduced:
[compute] # Directory of yaml files containing resource provider configuration. # Default: /etc/nova/provider_config/ # Files in this directory will be processed in lexicographic order. provider_config_location = $directory
- Loading, Parsing, Validation
On nova-compute startup, files in
CONF.compute.provider_config_locationare loaded and parsed by standard libraries (e.g.
yaml), and schema-validated (e.g. via
jsonschema). Schema validation failure or multiple identifications of a node will cause nova-compute startup to fail. Upon successful loading and validation, the resulting data structure is stored in an instance attribute on the ResourceTracker.
- Provider Tree Merging
A generic (non-hypervisor/virt-specific) method will be written that merges the provider config data into an existing
ProviderTreedata structure. The method must detect conflicts whereby provider config data references inventory of a resource class managed by the virt driver. Conflicts should log a warning and cause the conflicting config inventory to be ignored. The exact location and signature of this method, as well as how it detects conflicts, is left to the implementation. In the event that a resource provider is identified by both explicit UUID/NAME and $COMPUTE_NODE, only the UUID/NAME record will be used.
In the ResourceTracker’s
_update_to_placementflow, the merging method is invoked after
update_provider_treeand automatic trait processing, only in the
update_provider_treeflow (not in the legacy
compute_node_to_inventory_dictflows). On startup (
startup == True), if the merge detects a conflict, the nova-compute service will fail.
Ad hoc provider configuration is being performed today through an amalgam of
oslo.config options, more of which are being proposed or considered to deal
with VGPUs, NUMA, bandwidth resources, etc. The awkwardness of expressing
hierarchical data structures has led to such travesties as
[pci]passthrough_whitelist and “dynamic config” mechanisms where config
groups and their options are created on the fly. YAML is natively suited for
this purpose as it is designed to express arbitrarily nested data structures
clearly, with minimal noisy punctuation. In addition, the schema is
Data model impact¶
REST API impact¶
Admins should ensure that provider config files have appropriate permissions and ownership. Consuming services may wish to check this and generate an error if a file is writable by anyone other than the process owner.
Other end user impact¶
Other deployer impact¶
An understanding of this file and its implications is only required when the operator desires provider customization. The deployer should be aware of the precedence of records with UUID/NAME identification over $COMPUTE_NODE.
Subsequent specs will be needed for services consuming this file format.
None. (Consumers of this file format will need to address this - e.g. decide how to deprecate existing config options which are being replaced).
- Primary assignee:
- Other contributors:
dustinc efried dakshinai
- Feature liaison:
Construct a formal schema
Implement parsing and schema validation
Implement merging of config to provider tree
Incorporate above into ResourceTracker
Compose a self-documenting sample file
Schema validation will be unit tested.
Functional and integration testing to move updates from provider config file to Placement via Nova virt driver.
The formal schema file and a self-documenting sample file for provider config file.
Admin-facing documentation on guide to update the file and how Nova processes the updates.
User-facing documentation (including release notes).