cyborg-specs

Example Spec - The title of your blueprint

Thu, 26 Feb 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand. The title and this first paragraph should be used as the subject line and body of the commit message respectively.

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

At this point, if you would like to just get feedback on if the problem and proposed change fit in Cyborg, you can stop here and post this for review to get preliminary feedback. If so please say: Posting to get preliminary feedback on the scope of this spec.

Alternatives

What other ways could we do this thing? Why aren’t we using those? This doesn’t have to be a full literature review, but it should demonstrate that thought has been put into why the proposed solution is an appropriate one.

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Note that the schema should be defined as restrictively as possible. Parameters which are required should be marked as such and only under exceptional circumstances should additional parameters which are not defined in the schema be permitted (eg additionaProperties should be False).

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss the important scenarios needed to test here, as well as specific edge cases we should be ensuring work correctly. For each scenario please specify if this requires specialized hardware, a full OpenStack environment, or can be simulated inside the Cyborg tree.

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

Which audiences are affected most by this change, and which documentation titles on docs.openstack.org should be updated because of this change? Don’t repeat details discussed above, but reference them here in the context of documentation for multiple audiences. For example, the Operations Guide targets cloud operators, and the End User Guide would need to be updated if the change offers a new feature available through the CLI or dashboard. If a config option changes or is deprecated, note here that the documentation needs to be updated to reflect this specification’s change.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2026.2	Introduced

Example Spec - The title of your blueprint

Thu, 26 Feb 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2026.2	Introduced

Example Spec - The title of your blueprint

Thu, 26 Feb 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2026.2	Introduced

Cyborg-Nova Interaction for Scheduling

Thu, 26 Feb 2026 00:00:00

https://blueprints.launchpad.net/cyborg/+spec/cyborg-nova-interaction

Cyborg provides a general management framework for accelerators, such as FPGAs, GPUs, etc. For scheduling an instance that needs accelerators, Cyborg needs to work with Nova on three levels:

Representation and Discovery: Cyborg shall represent accelerators as resources in Placement. When a device is discovered, Cyborg updates resource providers, inventories, traits, etc. in Placement.
Instance placement/scheduling: Cyborg may provide a filter and/or weigher that limit or prioritize hosts based on available accelerator resources, but it is expected that Placement itself can handle most requirements.
Attaching accelerators to instances. In the compute node, Cyborg shall define a workflow based on interacting with Nova through a new os-acc library (similar to os-vif and os-brick).

This spec addresses the first two aspects. There is another spec to address the attachment of accelerators to instances [1]. Cyborg also needs to handle some aspects for FPGAs without involving Nova, specifically FPGA programming and bitstream management. They will be covered in other specs. This spec is independent of those specs.

This spec is common to all accelerators, including GPUs, High Precision Time Synchronization (HPTS) cards, etc. Since FPGAs have more aspects to be considered than other devices, some sections may focus on FPGA-specific factors. The spec calls out the FPGA-specific aspects.

Smart NICs based on FPGAs fall into two categories: those which expose the FPGA explicitly to the host, and those that do not. Cyborg’s scope includes the former. This spec includes such devices, though the Cyborg-Neutron interaction is out of scope.

The scope of this spec is Rocky release.

Terminology

Accelerator: The unit that can be assigned to an instance for offloading specific functionality. For non-FPGA devices, it is either the device itself or a virtualized version of it (e.g. vGPUs). For FPGAs, an accelerator is either the entire device, a region within the device or a function.
Bitstream: An FPGA image, usually a binary file, possibly with vendor-specific metadata. A bitstream may implement one or more functions.
Function: A specific functionality, such as matrix multiplication or video transcoding, usually represented as a string or UUID. This term may be used with multi-function devices, including FPGAs and other fixed function hardware like Intel QuickAssist.
Region: A part of the FPGA which can be programmed without disrupting other parts of that FPGA. If an FPGA does not support Partial Reconfiguration, the entire device constitutes one region. A region may implement one or more functions.

Here is an example diagram for an FPGA with multiple regions, and multiple functions in a region:

       PCI A     PCI B
        |        |
+-------|--------|-------------------+
|       |        |                   |
|  +----|--------|---+   +--------+  |
|  | +--|--+ +---|-+ |   |        |  |
|  | | Fn A| | Fn B| |   |        |  |
|  | +-----+ +-----+ |   |        |  |
|  +-----------------+   +--------+  |
|  Region 1              Region 2    |
|                                    |
+------------------------------------+

Problem description

Cyborg’s representation and handling of accelerators needs to be consistent with Nova’s Placement API. Specifically, they must be modeled in terms of Resource Providers (RPs), Resource Classes (RCs) and Traits.

Though PCI Express is entrenched in the data center, some accelerators may be exposed to the host via some other protocol. Even with PCI, the connections between accelerator components and PCI functions may vary across devices. Accordingly, Cyborg should not represent accelerators as PCI functions.

For instances that need accelerators, we need to define a way for Cyborg to be included seamlessly in the Nova scheduling workflow.

Use Cases

We need to satisfy the following use cases for the tenant role:

Device as a Service (DaaS): The flavor asks for a device.
- FPGA variation: The flavor asks for a device to which specific bitstream(s) can be applied. There are three variations, the first two of which delegate bitstream programming to Cyborg for secure programming:
  - Request-time Programming: The flavor specifies a bitstream. (Cyborg applies the bitstream before instance bringup. This is similar to AWS flow.)
  - Run-time Programming: The instance may request one or more bitstreams dynamically. (Cyborg receives the request and does the programming.)
  - Direct Programming: The instance directly programs the FPGA region assigned to it, without delegating it to Cyborg. The security questions that this raises need to be addressed in the future. (This is listed only for completeness; this is not going to be addressed in Rocky, or even future releases till the security concerns are fully addressed.)
Accelerated Function as a Service (AFaaS): The flavor asks for a function (e.g. ipsec) attached to the instance. The operator may satisfy this use case in two ways:
- Pre-programmed: Do not allow orchestration to modify any function, for any of these reasons:
  - Only fixed function hardware is available. (E.g. ASICs.)
  - Operational simplicity.
  - Assure tenants of programming security, by doing all programming offline through some audited process.
- For FPGAs, allow orchestration to program as needed, to maximize flexibility and availability of resources.

An operator must be able to provide both Device as a Service and Accelerated Function as a Service in the same cluster, to serve all kinds of users: those who are device-agnostic, those using 3rd party bitstreams, and those using their own bitstreams (incl. developers).

The goal for Cyborg is to provide the mechanisms to enable all these use cases.

In this spec, we do not consider bitstream developer or device developer roles. Also, we assume that each accelerator device is dedicated to a compute node, rather than shared among several nodes.

Proposed change

Representation

Cyborg will represent a generic accelerator for a device type as a custom Resource Class (RC) for that type, of the form CUSTOM_ACCELERATOR_<device-type>. E.g. CUSTOM_ACCELERATOR_GPU, CUSTOM_ACCELERATOR_FPGA, etc. This helps in defining separate quotas for different device types.

Device-local memory is the memory available to the device alone, usually in the form of DDR, QDR or High Bandwidth Memory in the PCIe board along with the device. It can also be represented as an RC of the form CUSTOM_ACCELERATOR_MEMORY_<memory-type>. E.g. CUSTOM_ACCELERATOR_MEMORY_DDR. A single PCIe board may have more than one type of memory.

In addition, each device/region is represented as a Resource Provider (RP). This enables traits to be applied to it and other RPs/RCs to be contained within it. So, a device RP provides one or more instances of that device type’s RC. This depends on nested RP support in Nova [2].

For FPGAs, both the device and the regions within it will be represented as RPs. This allows the hierarchy within an FPGA to be naturally modelled as an RP hierarchy.

Using Nested RPs is the preferred way. But, until Nova supports nested RPs, Cyborg shall associate the RCs and traits (described below) with the compute node RPs. This requires that all devices on a single host must share the same traits. If nested RP support becomes usable after Rocky release, the operator needs to handle the upgrade as below:

Terminate all instances using accelerators.

Remove all Cyborg traits and inventory on all compute node RPs, perhaps by running a script.

Perform the Cyborg upgrade. Post-upgrade, the new agent/driver(s) will create RPs for the devices and publish the traits and inventory.

Cyborg will associate a Device Type trait with each device, of the form CUSTOM_<device-type>-<vendor>. E.g. CUSTOM_GPU_AMD or CUSTOM_FPGA_XILINX. This trait is intended to help match the software drivers/libraries in the instance image. This is meant to be used in a flavor when a single driver/library in the instance image can handle most or all of device types from a vendor.

For FPGAs, this trait and others will be applied to the region RPs which are children of the device RPs as well.

Cyborg will associate a Device Family trait with each device as needed, of the form CUSTOM_<device-type>_<vendor>_<family>. E.g. CUSTOM_FPGA_INTEL_ARRIA10. This is not a product name, but the name of a device family, used to match software in the instance image with the device family. This is a refinement of the Device Type Trait. It is meant to be used in a flavor when there are different drivers/libraries for different device families. Since it may be tough to forecast whether a new device family will need a new driver/library, it may make sense to associate both these traits with the same device RP.

For FPGAs, Cyborg will associate a region type trait with each region (or with the FPGA itself if there is no Partial Reconfiguration support), of the form CUSTOM_FPGA_REGION_<vendor>__<uuid>. E.g. CUSTOM_FPGA_REGION_INTEL_<uuid>. This is needed for Device as a Service with FPGAs.

For FPGAs, Cyborg may associate a function type trait with a region when the region gets programmed, of the form CUSTOM_FPGA_FUNCTION_<vendor>_<uuid>. E.g. CUSTOM_FPGA_FUNCTION_INTEL_<gzip-uuid>. This is needed for AFaaS use case. This is updated when Cyborg reprograms a region as part of AFaaS request.

For FPGAs, Cyborg should associate a CUSTOM_PROGRAMMABLE trait with every region. This is needed to lay the groundwork for multi-function accelerators in the future. Flavors should ask for this trait, except in the pre-programmed case.

For FPGAs, since they may implement a wide variety of functionality, we may also attach a Functionality Trait. E.g. CUSTOM_FPGA_COMPUTE, CUSTOM_FPGA_NETWORK, CUSTOM_FPGA_STORAGE.

The Cyborg agent needs to get enough information from the Cyborg driver to create the RPs, RCs and traits. In particular, it needs to get the device type string, region IDs and function IDs from the driver. This requires the driver/agent interface to be enhanced [3].

The modeling in Placement represents generic virtual accelerators as resource classes, and devices/regions as RPs. This is PCI-agnostic. However, many FPGA implementations use PCI Express in general, and SR-IOV in particular. In those cases, it is expected that Cyborg will pass PCI VFs to instances via PCI Passthrough, and retain the PCI PF in the host for management.

Flavors

For the sake of illustrating how the device representation in Nova can be used, and for completeness, we now show how to define flavors for various use cases. Please see [4] for more details.

A flavor that needs device access always asks for one or more instances of ‘resource:CUSTOM_ACCELERATOR_<device-type>’. In addition, it needs to specify the right traits.

Example flavor for DaaS:

resources:CUSTOM_ACCELERATOR_HPTS=1

trait:CUSTOM_HPTS_ZTE=required

NOTE: For FPGAs, the flavor should also include CUSTOM_PROGRAMMABLE trait.

Example flavor for AFaaS Pre-programed:

resources:CUSTOM_ACCELERATOR_FPGA=1

trait:CUSTOM_FPGA_INTEL_ARRIA10=required

trait:CUSTOM_FPGA_FUNCTION_INTEL_<gzip-uuid>=required

Example flavor for AFaaS Orchestration-Programmed:

resources:CUSTOM_ACCELERATOR_FPGA=1

trait:CUSTOM_FPGA_INTEL_ARRIA10=required

trait:CUSTOM_PROGRAMMABLE=required

function:CUSTOM_FPGA_FUNCTION_INTEL_<gzip-uuid>=required (Not interpreted by Nova.)

NOTE: When Nova supports preferred traits, we can use that instead of ‘function’ keyword in extra specs.

NOTE: For Cyborg to fetch the bitstream for this function, it is assumed that the operator has configured the function UUID as a property of the bitstream image in Glance.

Another example flavor for AFaaS Orchestration-Programmed which refers to a function by name instead of UUID for ease of use:

resources:CUSTOM_ACCELERATOR_FPGA=1

trait:CUSTOM_FPGA_INTEL_ARRIA10=required

trait:CUSTOM_PROGRAMMABLE=required

function_name:<string>=required (Not interpreted by Nova.)

NOTE: This assumes the operator has configured the function name as a property of the bitstream image in Glance. The FPGA hardware is not expected to expose function names, and so Cyborg will not represent function names as traits.

A flavor may ask for other RCs, such as local memory.

A flavor may ask for multiple accelerators, using the granular resource request syntax. Cyborg can tie function and bitstream fields in the extra_specs to resources/traits using an extension of the granular resource request syntax (see References) which is not interpreted by Nova.

resourcesN: CUSTOM_ACCELERATOR_FPGA=1

traitsN: CUSTOM_FPGA_INTEL_ARRIA10=required

othersN: function:CUSTOM_FPGA_FUNCTION_INTEL_<gzip-uuid>=required

Scheduling workflow

We now look at the scheduling flow when each device implements only one function. Devices with multiple functions are outside the scope for now.

A request spec with a flavor comes to Nova conductor/scheduler.

Placement API returns the list of RPs which contain the requested resources with matching traits. (With nested RP support, the returned RPs are device/region RPs. Without it, they are compute node RPs.)

FPGA-specific: For AFaaS orchestration-programmed use case, Placement will return matching devices but they may not have the requested function. So, Cyborg may provide a weigher which checks the allocation candidates to see which ones have the required function trait, and ranks them higher. This requires no change to Cyborg DB.

The request_spec goes to compute node (ignoring Cells for now).

NOTE: When one device/region implements multiple functions and orchestration-driven programming is desired, the inventory of that device needs to be adjusted. This can be addressed later and is not a priority for Rocky release. See References.

Nova compute calls os-acc/Cyborg [1].

FPGA-specific: If the request spec asks for a function X in extra specs, but X is not present in the selected region RP, Cyborg should program that region.

Cyborg should associate RPs/RCs and PFs/VFs with Deployables in its internal DB. It can use such mappings associating the requested resource (device/function) with some attach handle that can be used to attach the resource to an instance (such as a PCI function).

NOTE : This flow is PCI-agnostic: no PCI whitelists involved.

Handling Multiple Functions Per Device

Alternatives

N/A

Data model impact

Following changes are needed in Cyborg.

Do not publish PCI functions as resources in Nova. Instead, publish RC/RP info to Nova, and keep RP-PCI mapping internally.
Cyborg should associate RPs/RCs and PFs/VFs with Deployables in its internal DB.
Driver/agent interface needs to report device/region types so that RCs can be created.
Deployables table should track which RP corresponds to each Deployable.

REST API impact

None

Security impact

This change allows tenants to initiate FPGA bitstream programming. To mitigate the security impact, it is proposed that only 2 methods are offered for programming (flavor asks for a bitstream, or the running instance asks for specific bitstreams) and both are handled through Cyborg. There is no direct access from an instance to an FPGA.

Notifications impact

None

Other end user impact

None

Performance Impact

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

None

Work Items

Decide specific changes needed in Cyborg conductor, db, agent and drivers.

Dependencies

NOTE: the granular requests feature is needed to define a flavor that requests non-identical accelerators, but is not needed for Cyborg development in Rocky.

Testing

For each vendor driver supported in this release, we need to integrate the corresponding FPGA type(s) in the CI infrastructure.

Documentation Impact

None

References

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader know what happened over time.

Revisions
Release Name	Description
Rocky	Introduced

Nova - Cyborg Interaction

Thu, 26 Feb 2026 00:00:00

https://blueprints.launchpad.net/nova/+spec/nova-cyborg-interaction

This specification describes the Nova - Cyborg interaction needed to create and manage instances with accelerators, and the changes needed in Nova to accomplish that.

Problem description

Scope

Nova and Cyborg need to interact in many areas for handling instances with accelerators. While this spec covers the gamut, specific areas are covered in detail in other specs. We list all the areas below, identify which specific parts are covered by other specs, and describe what is covered in this spec.

Representation: Cyborg shall represent devices as nested resource providers under the compute node (except possibly for disaggregated servers), accelerator types as resource classes and accelerators as inventory in Placement. The properties needed for scheduling are represented as traits. This is specified by [1]. This spec does not dwell on this topic.
Discovery and Updates: Among the devices discovered in a host, Cyborg intends to claim only those that are not included under the PCI Whitelisting mechanism. Cyborg shall update Placement in a way that is compatible with the virt driver’s update of Placement. These aspects are addressed in sections Coexistence with PCI whitelists and Placement update respectively.
User requests for accelerators: Users usually request compute resources via flavors. However, since the requests for devices may be highly varied, placing them in flavors may result in flavor explosion. We avoid that by expressing device requests in a device profile [2] . The relationship between device profiles and flavors is explored in Section User requests.

When an instance creation (boot) request is made, the contents of a device profile shall be translated to request groups in the request spec; the syntax in request groups is covered in Section Updating the Request Spec.
Instance scheduling: Nova shall use the Placement data populated by Cyborg to schedule instances. This spec does not dwell on this topic.
Assignment of accelerators: We introduce the concept of Accelerator Request objects in Section Accelerator Requests. The workflow to create and use them is summarized in Section Nova changes for Assignment workflow. The same section also highlights the Nova changes needed.
Instance operations: The behavior with respect to accelerators for all standard instance operations are defined in [3]. This spec does not dwell on this topic.

Use Cases

A user requests an instance with one or more accelerators of different types assigned to it.
An operator may provide users with both Device as a Service or Accelerated Function as a Service in the same cluster (see [1]).

The following use cases are not addressed in this release but are of long term interest:

A user requests to add one or more accelerators to an existing instance.
Live migration with accelerators.

Proposed change

Coexistence with PCI whitelists

The operator tells Nova which PCI devices to claim and use by configuring the PCI Whitelists mechanism. In addition, the operator installs Cyborg drivers in compute nodes and configures/enables them. Those drivers may then discover and report some PCI devices. The operator must ensure that both configurations are compatible.

Ideally, there should be a single way for the operator to identify which PCI devices should be claimed by Nova and which by Cyborg. Until that is figured out, the operator shall use Cyborg’s configuration file to specify which Cyborg drivers are enabled. Since each driver claims specific PCI IDs, the operator can and must ensure that none of these PCI IDs are included in Nova’s PCI whitelist.

Placement update

Cyborg shall call Placement API directly to represent devices and accelerators. Some of the intended use cases for the API invocation are:

Create or delete child RPs under the compute node RP.
Create or delete custom RCs and custom traits.
Associate traits with RPs or remove such association.
Update RP inventory.

Cyborg shall not modify the RPs created by any other component, such as Nova virt drivers.

User requests

The user request for accelerators is encapsulated in a device profile [2], which is created and managed by the admin via the Cyborg API.

A device profile may be viewed as a ‘flavor for devices’. Accordingly, the instance request should include both a flavor and a device profile. However, that requires a change to the Nova API for instance creation. To mitigate the impact of such changes on users and operators, we propose to do this in phases.

In the initial phase, Nova API remains as today. The device profile is folded into the flavor as an extra spec by the operator, as below:

openstack flavor set --property 'accel:device_profile=<profile_name>' flavor

Thus the standard Nova API can be used to create an instance with only the flavor (without device profiles), like this:

openstack server create --flavor f ....  # instance creation

In the future, device profile may be used by itself to specify accelerator resources for the instance creation API.

Updating the Request Spec

When the user submits a request to create an instance, as described in Section User requests, Nova needs to call a Cyborg API, to get back the resource request groups in the device profile and merge them into the request spec. (This is along the lines of the scheme proposed for Neutron [4].)

This call, like all the others that Nova would make to Cyborg APIs, is done through a Keystone-based adapter that would locate the Cyborg service, similar to the way Nova calls Placement. A new Cyborg client module shall be added to Nova, to encapsulate such calls and to provide Cyborg-specific functionality.

VM images in Glance may be associated with image properties (other than image traits), such as bitstream/function IDs needed for that image. So, Nova should pass the VM image UUID from the request spec to Cyborg. This is TBD.

The groups in the device profile are numbered by Cyborg. The request groups that are merged into the request spec are numbered by Nova. These numberings would not be the same in general, i.e., the N-th device profile group may not correspond to the N-th request group in the request spec.

When the device profile request groups are added to other request groups in the flavor, the group_policy of the flavor shall govern the overall semantics of all request groups.

Accelerator Requests

An accelerator request (ARQ) is an object that represents the state of the request for an accelerator to be assigned to an instance. The creation and management of ARQs are handled by Cyborg, and ARQs are persisted in Cyborg database.

An ARQ, by definition, represents a request for a single accelerator. The device profile in the user request may have N request groups, each asking for M accelerators; then N * M ARQs will be created for that device profile.

When an ARQ is initially created by Cyborg, it is not yet associated with a specific host name or a device resource provider. So it is said to be in an unbound state. Subsequently, Nova calls Cyborg to bind the ARQ to a host name, a device RP UUID and an instance UUID. If the instance fails to spawn, Nova would unbind the ARQ without deleting it. On instance termination, Nova would delete the ARQs after unbinding them.

Each ARQ needs to be matched to the specific RP in the allocation candidate that Nova has chosen, before the ARQ is bound. The current Nova code maps request groups to RPs, while the Cyborg client module in Nova (cyborg-client-module) matches ARQs to request groups. The matching is done using the requester_id field in the RequestGroup object as below:

The order of request groups in a device profile is not significant, but it is preserved by Cyborg. Thus, each device profile request group has a unique index.
When the device profile request groups returned by Cyborg are added to the request spec, the requester_id field is set to ‘device_profile_<N>’ for the N-th device profile request group (starting from zero). The device profile name need not be included here because there is only one device profile per request spec.
When Cyborg creates an ARQ for a device profile, it embeds the device profile request group index in the ARQ before returning it to Nova.
The matching is done in two steps:
- Each ARQ is mapped to a specific request group in the request spec using the requester_id field.
- Each request group is mapped to a specific RP using the same logic as the Neutron bandwidth provider ([5]).

Nova changes for Assignment workflow

This section summarizes the workflow details for Phase 1. The changes needed in Nova are marked with NEW.

NEW: A Cyborg client module is added to nova (cyborg-client-module). All Cyborg API calls are routed through that.

The Nova API server receives a POST /servers API request with a flavor that includes a device profile name.
NEW: The Nova API server calls the Cyborg API GET /v2/device_profiles?name=$device_profile_name and gets back the device profile. The request groups in that device profile are added to the request spec.
The Nova scheduler invokes Placement and gets a list of allocation candidates. It selects one of those candidates and makes claim(s) in Placement. The Nova conductor then sends a RPC message build_and_run_instances to the Nova compute manager.
NEW: Nova compute manager calls the Cyborg API POST /v2/accelerator_requests with the device profile name. Cyborg creates a set of unbound ARQs for that device profile and returns them to Nova. (The call may originate from Nova conductor or the compute manager; that will be settled in code review.)
NEW: The Cyborg client in Nova matches each ARQ to the resource provider picked for that accelerator. See match-rp.
NEW: The Nova compute manager calls the Cyborg API PATCH /v2/accelerator_requests to bind the ARQ with the host name, device’s RP UUID and instance UUID. This is an asynchronous call which prepares or reconfigures the device in the background.

NEW: Cyborg, on completion of the bindings (successfully or otherwise), calls Nova’s POST /os-server-external-events API with:

{
   "events": [
      { "name": "accelerator-requests-bound",
        "tag": $device_profile_name,
        "server_uuid": $instance_uuid,
        "status": "completed" # or "failed"
      },
      ...
   ]
}

NEW: The Nova compute manager waits for the notification, subject to the timeout mentioned in Section Other deployer impact. It then calls the Cyborg REST API GET /v2/accelerator_requests?instance=<uuid>&bind_state=resolved.
NEW: The Nova virt driver uses the attach handles returned from the Cyborg call to compose PCI passthrough devices into the VM’s definition.
NEW: If there is any error after binding has been initiated, Nova must unbind the relevant ARQs by calling Cyborg API. It may then retry on another host or delete the (unbound) ARQs for the instance.

This flow is captured by the following sequence diagram, in which the Nova conductor and scheduler are together represented as the Nova controller.

Alternatives

It is possible to have an external agent create ARQs from device profiles by calling Cyborg, and then feed those pre-created ARQs to the Nova instance creation API, analogous to Neutron ports. We do not take that approach yet because it requires changes to Nova instance creation API.

It is possible to have the Nova virt driver poll for the Cyborg ARQ binding completion. That is not preferable, partly because that is not the pattern of interaction with other services like Neutron.

Data model impact

None

REST API impact

None. A new extra_spec key accel:device_profile_name is added to the flavor, but no API is modified.

Security impact

None

Notifications impact

Nova may choose to add additional notifications for Cyborg API calls.

Other end user impact

None

Performance Impact

The extra calls to Cyborg REST API may potentially impact Nova conductor/scheduler throughput. This has been mitigated by making some critical Cyborg operations as asynchronous tasks.

Other deployer impact

The deployer needs to set up the clouds.yaml file so that Nova can call the Cyborg REST API.

The deployer needs to configure a new tunable in nova-cpu.conf:

* arq_binding_timeout (integer): Time in seconds for Nova compute
  manager to wait for Cyborg to notify that ARQ binding is done.
  Timeout is fatal, i.e., VM startup is aborted with an exception.
  Default: 300.

Developer impact

The resource classes FPGA and PGPU have already been standardized. But, as new device types are proposed, they will be represented as custom RCs to begin with, but may get standardized later. Such standardization requires changes to os-resource-classes.

For end-to-end testing with tempest, Cyborg shall provide a fake driver which returns attach handles of type TEST_PCI. The Nova virt driver should ignore such attach handles, and create VMs as if such ARQs did not exist.

Upgrade impact

None

Implementation

Assignee(s)

Sundar Nadathur

Work Items

See the steps marked NEW in Nova changes for Assignment workflow section.

Dependencies

None

Testing

There need to be unit tests and functional tests for the Nova changes. Specifically, there needs to be a functional test fixture that mocks the Cyborg API calls.

There need to be tempest tests for the end-to-end flow, including failure modes. The tempest tests should be targeted at a fake driver (in addition to real hardware, if any) and tied to the Nova Zuul gate.

Documentation Impact

Device profile creation needs to be documented in Cyborg, as noted in [2].

The need for operator to fold the device profile into the flavor needs to be documented.

References

Specification for Cyborg API Version 2

History

Revisions
Release Name	Description
Queues	Introduced
Train	Re-proposed

Cyborg API Version 2

Thu, 26 Feb 2026 00:00:00

Problem description

A new device model and database schema was introduced in Stein release. That has obsoleted the accelerator objects, changed the definition of deployable objects, and introduced the concepts of devices. That has made some old APIs obsolete and raised the need for new APIs.

Secondly, the proposed scheme to integrate with Nova involves new objects like device profiles and accelerator requests (ARQs). They require new APIs as well. Of these, the device profiles API is covered in [2].

This spec will specify what v1 APIs will be dropped in v2, and what new APIs need to be introduced.

Use Cases

As a user, I want to create VMs with attached accelerators, delete them and perform other instance operations. (Live migration is excluded.)
- Cyborg should provide APIs for device profiles and ARQs, so as to support end-to-end workflows with Nova for VM creation, deletion and other operations.
As an operator, I want to query and manage the inventory of accelerators and devices.
- Cyborg should provide an API to query the list of devices based on various attributes.
As a user with appropriate authorization, or as an operator, I want to initiate programming of specific devices, either with FPGA bitstreams, or with FPGA shell logic or device firmware.

Proposed changes

Microversions

Cyborg should adopt microversions to enable incremental backwards-compatible changes to APIs. This is important because Cyborg is a rapidly evolving project which needs to support new devices and features over time. For Train release, only one microversion will be supported, i.e., 2.0.

The changes should be compatible with Microversion Specification [1]. This means that the output of the API:

GET /accelerator

must change as specified in Section Changed APIs. Also, a new API:

GET /accelerator/v2

is introduced conformant to that specification, as documented in Section New APIs.

The API clients can use the following header to indicate which microversion to use:

OpenStack-API-Version: accelerator <microversion>

If they do not, the default microversion of 2.0 will be assumed in Train.

API Changes

The following v1 URLs will not be supported in v2, because the accelerator object does not exist anymore and the deployable object is exposed as part of the device object:

/v1/accelerators
/v1/deployables

A new set of APIs based on the URL:

/v2/device_profiles

is introduced in the Specification for device profiles ([2]).

This specification introduces another set of APIs based on the URLs:

/v2/accelerator_requests
/v2/devices
/v2/deployables/{uuid}

See Section New APIs for details.

Alternatives

We could add another API to enable or disable specific devices, or specific deployables (resource providers). That is left for a future version.

Data model impact

The database changes needed for these APIs have mostly been done in Train. However, we need to add a bitstream field to deployables table in the database, which can serve as a target in the JSON patch for reprogramming.

REST API impact

Deleted APIs

Cyborg should drop the APIs for accelerators and deployables in v2. The deployables, as per the new definition, will be included in the new devices API.

Changed APIs

URL: /accelerator

METHOD: GET

ROLE: Admin or any tenant

Current JSON response:

{
   "description": "Cyborg (previously known as Nomad) is an OpenStack
      project that aims to provide a general purpose management framework for
      acceleration resources (i.e. various types of accelerators such as
      Crypto cards, GPU, FPGA, NVMe/NOF SSDs, ODP, DPDK/SPDK and so on).",
   "name": "OpenStack Cyborg API"
}

Proposed JSON response:

{
   "versions": [
       {
           "id": "v1",
           "links": [
               {
                   "href": "http://<ip>/accelerator/v1",
                   "rel": "self"
               }
           ],
           "version": "1.0",
           "status": "DEPRECATED"
       },
       {
           "id": "v2.0",
           "links": [
               {
                   "href": "http://<ip>/accelerator/v2",
                   "rel": "self"
               }
           ],
           "min_version": "2.0",
           "max_version": "2.0",
           "version": "2.0",
           "status": "CURRENT"
       },
   ]
}

New APIs

URL: /accelerator/v2

METHOD: GET

ROLE: Admin or any tenant

Proposed JSON response:

{
   "id": "v2.0",
   "links": [
       {
           "href": "http://<ip>/accelerator/v2",
           "rel": "self"
       }
   ],
   "min_version": "2.0",
   "max_version": "2.0",
   "version": "2.0",
   "status": "CURRENT"
}

Device profile APIs are covered in [2]. A device profile may request one or more accelerator resources. The request for each accelerator resource is encapsulated in an Accelerator Request (ARQ) object. The following are the REST APIs for ARQs.

URL: /accelerator/v2/accelerator_requests

METHOD: GET

ROLE: Admin or the tenant who owns the specified instance.

Query Parameters:

instance: UUID of the instance whose ARQs are requested.
bind_state: Bind state of ARQs. Only supported value is
‘resolved’, which means the ARQ is either bound successfully or failed to bind. Other states like ‘bound’ may be supported in the future.

Proposed JSON response:

{
   "arqs": [
      <arq_obj>,
      ...
   ]
}

URL: /accelerator/v2/accelerator_requests

METHOD: POST

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request:

{
  "device_profile_name": <string>
}

Action: Create ARQs for the specified device profile. If the device profile specifies N accelerator resources across all its request groups, N ARQs will get created in unbound state.

Proposed JSON response:

{
   "arqs": [
      <arq_obj>,
      ...
   ]
}

URL: /accelerator/v2/accelerator_requests

METHOD: PATCH

ROLE: Admin or the tenant who owns the specified instance.

Proposed JSON request (in RFC 6902 format):

For binding:
{
  "$arq_uuid": [
     { "path": "/hostname", "op": "add", "value": <string> },
     { "path": "/instance_uuid",  "op": "add", "value": <uuid> },
     { "path": "/device_rp_uuid", "op": "add", "value": <uuid> }
  ],
  "$arq_uuid": [...],
  ...
}

For unbinding:
{
  "$arq_uuid": [
     { "path": "/hostname", "op": "remove" },
     { "path": "/instance_uuid",  "op": "remove" },
     { "path": "/device_rp_uuid", "op": "remove" }
  ],
  "$arq_uuid": [...],
  ...
}

Action: Bind or unbind

Proposed JSON response: None

URL: /accelerator/v2/accelerator_requests

METHOD: DELETE

ROLE: Admin or the tenant who owns the specified ARQs.

Query Parameters (required):

arqs: List of one or more comma-separated ARQ UUIDs.

Proposed JSON response: None

URL: /accelerator/v2/devices

METHOD: GET

Query Parameters:

Proposed JSON response:

{
  "devices": [
     <device_obj>,
     ...
  ]
}

URL: /accelerator/v2/devices/{uuid}

METHOD: PATCH

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request (in RFC 6902 format):

{
  [
     { "path": "/firmware", "op": "add", "value": <img_uuid> },
  ]
}

Action: Update the firmware or shell image (FPGA bitstream) for the: specified device. The request allows a lst of path specifiers for future extensibility.

Proposed JSON response: None.

URL: /accelerator/v2/deployables/{uuid}

METHOD: PATCH

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request (in RFC 6902 format):

{
  [
     { "path": "/bitstream", "op": "add", "value": <img_uuid> },
  ]
}

Action: Update the FPGA bitstream for the specified deployable. The: request allows a lst of path specifiers for future extensibility.

Proposed JSON response: None.

Security impact

Notifications impact

None

Other end user impact

Performance Impact

None

Other deployer impact

Developer impact

Implementation

Assignee(s)

TBD

Work Items

Dependencies

None

Testing

Unit tests and functional tests need to be written.

Documentation Impact

Cyborg API documentation needs to be updated.

References

Nova Cyborg interaction specification

History

Revisions
Release Name	Description
Train	Introduced

Cyborg-Nova Placement Interaction

Thu, 26 Feb 2026 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-nova-placement

This specification describes the way Cyborg represents devices and accelerators in Placement.

This spec is common to all accelerators, including GPUs, ASIC-based devices, etc. Since FPGAs have more aspects to be considered than other devices, some sections highlight FPGA-specific factors.

Problem description

Cyborg should represent devices and accelerators in Placement so that Nova can schedule instances with accelerators.

Though PCI Express is entrenched in the data center, devices may be identified by some identifier other than PCI bus-device-functions and accelerators may be attached to instances by attach handles other than PCI functions. Accordingly, Cyborg should not represent PCI functions in Placement.

Terminology

Accelerator: The unit that can be assigned to an instance for offloading specific functionality. For non-FPGA devices, it is either the device itself or a virtualized version of it (e.g. vGPUs). For FPGAs, an accelerator is either the entire device, a region within the device or a function.
Bitstream: An FPGA image, usually a binary file, possibly with vendor-specific metadata. A bitstream may implement one or more functions.
Function: A specific functionality, such as matrix multiplication or video transcoding, usually represented as a string or UUID. This term may be used with multi-function devices, including FPGAs and other fixed function hardware like Intel QuickAssist.
Region: A part of the FPGA which can be programmed without disrupting other parts of that FPGA. If an FPGA does not support Partial Reconfiguration, the entire device constitutes one region. A region may implement one or more functions.

Background

A Cyborg device has one or more components named deployables, each of which contains one or more accelerators. A device has a management interface, whose address is the control path identifier: for SR-IOV devices, this is usually the PCI Physical Function (PF). Each deployable has one or more attach handles, usally one for each accelerator resource in the deployable. The attach handle represents the object by which an accelerator is associated with an instance: for SR-IOV devices, this is usually the PCI Virtual Function (VF).

A device may have components, such as flash memory or BMC, which are not of relevance to Nova or Placement. Those components may have attributes, such as flash memory capacity or BMC firmware version.

This is diagrammatically shown below:

Control Path Identifier
      +----+
      |    |
    +-+----+--------------------------------------+
    |                                             |
    |     Attach Handles       Attach Handles     |
    |      +--+    +--+         +--+    +--+      |
    |      |  |    |  |         |  |    |  |      |
    |    +-+--+----+--+-+     +-+--+----+--+-+    |
    |    |  ACC1  ACC2  |     |  ACC1  ACC2  |    |
    |    +--------------+     +--------------+    |
    |      Deployable 1         Deployable 2      |
    |                                             |
    +---------------------------------------------+

Use cases

Operators should be able to define device profiles for any or all of the tenant use cases defined below, using the set of resources, traits and properties published by Cyborg. The tenant (user) should be able to specify the accelerators needed for an instance with an operator-defined device profile.

The use cases for the tenant role are as below:

Device as a Service (DaaS): The user asks for a deployable that she will manage herself. This is meant for power users who know device-specific details. Example: A GPU user asks for a specific GPU model that can support specific driver version(s).
- FPGA variation: The user chooses the bitstream that needs to be programmed into the device (or region). To address the potential security concerns, we define three variations, the first two of which delegate bitstream programming to Cyborg for mitigating some risks:
  - Request-time Programming: The device profile specifies a bitstream. Cyborg applies the bitstream before instance bringup.
  - Run-time Programming: The instance may request one or more bitstreams dynamically. Cyborg receives the request and does the programming. (This use case may not be addressed in Train.)
Accelerated Function as a Service (AFaaS): The user asks for a function (e.g. ipsec) or an algorithm that needs to be offloaded. The accelerator containing that function should be assigned to the instance.

This does not require detailed knowledge of the devices or bitstreams, and thus enables a larger audience of users.

The operator may satisfy this use case in two ways:
- Pre-programmed: Do not allow orchestration to modify any function, for any of these reasons:
  - Only fixed function hardware is available. (E.g. ASICs.)
  - Operational simplicity.
  - Assure tenants of programming security, by doing all programming offline through some audited process.
- Orchestration-programmed: For FPGAs, allow orchestration to program as needed, to maximize flexibility and availability of resources.

An operator must be able to provide both Device as a Service and Accelerated Function as a Service in the same cluster.

Proposed change

Representation

Cyborg will represent a generic accelerator for a device type as a standard or custom Resource Class (RC) for that type. Standard RCs have been proposed for GPUs and FPGAs: PGPU and FPGA [2]. For others, Cyborg will create custom RCs of the form CUSTOM_ACCELERATOR_<device-type>. E.g. CUSTOM_ACCELERATOR_AICHIP. Using a different RC for each device type helps in defining separate quotas for different device types.
- Note that an accelerator is not an object, either in Cyborg or in Placement. It is a virtual resource, represented as inventories of RPs in Placement.
- In the future, there could be other resource classes. For example, there could be a RC for device-local memory (i.e., memory available to the device alone, usually in the form of DDR, QDR or High Bandwidth Memory in the PCIe board along with the device).
Since a device may have componnets that are not of relevance to Placement or Nova (see Section Background ), Cyborg does not represent devices directly in Placement.
Each deployable is represented as a Resource Provider (RP) in Placement. Each accelerator in that deployable is represented as an inventory of the corresponding RP.
Cyborg will associate a Device Family trait with each deployable as needed, of the form CUSTOM_<device-type>_<vendor>_<family>. E.g. CUSTOM_FPGA_INTEL_ARRIA10. This is not a product name, but the name of a device family, used to match software in the instance image with the device family.
For FPGAs, Cyborg will associate a region type trait with each region (or with the FPGA itself if there is no Partial Reconfiguration support), of the form CUSTOM_FPGA_REGION_<vendor>_<id-string>. E.g. CUSTOM_FPGA_REGION_INTEL_<id-string>. This is needed for Device as a Service with FPGAs.
For FPGAs, Cyborg may associate a function type trait with a region when the region gets programmed, of the form CUSTOM_FPGA_FUNCTION_ID_<vendor>_<id-string>. E.g. CUSTOM_FPGA_FUNCTION_ID_INTEL_<gzip-id>. This is needed for AFaaS use case.
In addition to the above, Cyborg will associate additional custom traits reported by the Cyborg driver for that deployable.
In addition to the traits, user requests for accelerators may include Cyborg-specific properties in the device profile. These are not interpreted by Nova. They are explained in the Device Profiles specification ([3]).

Usage in device profiles

This section shows how an operator can realize the ‘Use cases’_ in device profiles using the RCs and traits from previous section, along with Cyborg-specific properties.

To recap from [1], the request to create an instance with accelerators may use a flavor with an embedded device profile, or a flavor and a device profile separately. In either case, the accelerator specification is entirely in the device profile.

We now show how to define device profiles for various use cases.

Some example device profiles for DaaS for a generic device:
```
| resources:ACCELERATOR_GPU=1
| trait:CUSTOM_GPU_AMD_RADEON_R9=required
```
resources:CUSTOM_ACCELERATOR_HPTS=1

trait:CUSTOM_HPTS_ZTE=required

Example device profile for DaaS with request-time programming for FPGAs:

| resources:ACCELERATOR_FPGA=1
| trait:CUSTOM_FPGA_REGION_ID_INTEL_<id>=required
| accel:bitstream_id=3AFB

Example device profile for DaaS with run-time programming for FPGAs:

| resources:CUSTOM_ACCELERATOR_FPGA=1
| trait:CUSTOM_FPGA_REGION_INTEL_<name>=required
| accel:bitstream_at_runtime=required

Example device profile for AFaaS pre-programmed FPGAs:
```
| resources:CUSTOM_ACCELERATOR_FPGA=1
| trait:CUSTOM_FPGA_INTEL_ARRIA10=required
| trait:CUSTOM_FPGA_FUNCTION_ID_INTEL_<gzip-id-string>=required
```
Since the function trait is required, if no free accelerator with that trait is available, the request would fail. There would be no attempt to reprogram another FPGA to get that function.
Example device profile for AFaaS Orchestration-Programmed:
```
| resources:CUSTOM_ACCELERATOR_FPGA=1
| trait:CUSTOM_FPGA_INTEL_ARRIA10=required
| accel:function:CUSTOM_FPGA_FUNCTION_INTEL_<gzip-id-string>=required
```
The function is not specified as a trait and so Nova ignores it. Placement returns the list of all allocation candidates which contain the only stated trait, i.e., the list of all nodes with that FPGA device model. During binding, Cyborg will notice whether the selected device RP actually has the requested function and, if not, will initiate reprogramming.

NOTE: When Nova supports preferred traits, we can use that instead of ‘function’ keyword in extra specs.

NOTE: For Cyborg to fetch the bitstream for this function, it is assumed that the operator has configured the function ID as a property of the bitstream image in Glance.
Another example device profile for AFaaS Orchestration-Programmed which refers to a function by name instead of ID for ease of use:
```
| resources:CUSTOM_ACCELERATOR_FPGA=1
| trait:CUSTOM_FPGA_INTEL_ARRIA10=required
| accel:function_name:<string>=required
```
NOTE: This assumes the operator has configured the function name as a property of the bitstream image in Glance. The FPGA hardware is not expected to expose function names, and so Cyborg will not represent function names as traits.

Alternatives

N/A

Data model impact

Following changes are needed in Cyborg.

Do not publish PCI functions as resources in Nova. Instead, publish RC/RP info to Nova, and keep RP-PCI mapping internally.
Cyborg should associate RPs/RCs and PFs/VFs with Deployables in its internal DB (as part of the deployable attributes table).
Driver/agent interface needs to report device/region types so that RCs can be created.
Deployables table should track which RP corresponds to each Deployable.

REST API impact

No change is needed in Placement API.

Cyborg API to create and use device profiles is covered in the Device Profiles specification [1].

Cyborg API for use by Nova during the instance spawn/update workflow is covered elsewhere.

Security impact

The use cases for DaaS run-time programming and direct programming will not be taken up till the security issues are addressed.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Operators must define device profiles based on the RCs, traits and Cyborg properties.

Developer impact

When a custom trait in one release becomes a standard trait in another release, there needs to be an upgrade script to translate device profiles.

Implementation

Assignee(s)

None

Work Items

The code changes needed to realize this device representation are described in other specs. Work items are stated in those specs.

Dependencies

Enable use of Nested Resource Providers in Nova Scheduler

Testing

The code changes needed to realize this device representation are described in other specs. Testing requirements are stated in those specs.

Documentation Impact

Usage of RCs, traits and Cyborg properties must be documented for operators.

References

Enable use of Nested Resource Providers in Nova Scheduler

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader know what happened over time.

Revisions
Release Name	Description
Rocky	Introduced as cyborg-nova-sched.rst
Stein	Updated
Train	Updated

API Microversions for Cyborg

Thu, 26 Feb 2026 00:00:00

Problem description

Cyborg has deprecated v1.0 API since Train release and use v2.0 API as current API version. As Cyborg evolves and obtains new features which are also represented in API changes, we should support microversion of Cyborg which will not break current infrastructure.

Use Cases

Allows developers to modify the Cyborg API in backwards compatible way and signal to users of the API dynamically that the change is available without having to create a new API extension.

Allows developers to modify the Cyborg API in a non backwards compatible way while still supporting the old behaviour. Users of the REST API are able to decide if they want the Cyborg API to behave in the new or old manner on a per request basis. Deployers are able to make new backwards incompatible features available without removing support for prior behaviour as long as there is support to do this by developers.

Users of the REST API are able to decide, on a per request basis, which version of the API they want to use (assuming the deployer supports the version they want). This means that a deployer who does not upgrade will not break compatibility, and clients that do not upgrade will also remain compatible.

Proposed change

Cyborg API should receive the API version from client that represents the list resources and their GET/POST/PATCH attributes it can work with.

Cyborg will use a framework we will call “API Microversions” for allowing changes to the API while preserving backward compatibility. The basic idea is that a user has to explicitly ask for their request to be treated with a particular version of the API. So breaking changes can be added to the API without breaking users who don’t specifically ask for it. This is done with an HTTP header “OpenStack-API-Version: accelerator 2.0” which is a monotonically increasing semantic version number starting from 2.0.

In these cases, we need a new microversion: 1. If the parameters of an API change, we need to add a new microversion. For example, when we add a new parameter to limit the length of return value of the device list API.

2. No parameters to any API has changed, but the functionality of some API has changed. For example, we already have a device list API with a filter dictionary as the input param, it currently support list the device filtered by “attribute_A” and “attribute_B”, now we also want this API to return the device list filtered by “attribute_C”. In this case, the API input parameter is always the filter dict. But the API’s functionality changes, so we also need to add a microverison.

If a user makes a request without specifying a version, they will get the DEFAULT_API_VERSION as defined in cyborg/api/versions.py. This value is currently 2.0 and is expected to remain so for quite a long time. We will not change this default version unless the major version changed and we deprecate V2 api version. It seems that we have very little change to update this value.

Versioning of the API should be a single monotonically increasing counter. It will be of the form X.Y where it follows the following convention:

X will only be changed if a significant backwards incompatible API change is made which affects the API as whole. That is, something that is only very rarely incremented.

Y will only be changed when you make any change to the API. Note that this includes semantic changes which may not affect the input or output formats or even originate in the API code layer. We are not distinguishing between backwards compatible and backwards incompatible changes in the versioning system. However, it will be made clear in the documentation as to what is a backwards compatible change and what is a backwards incompatible one.

Version Discovery

The Version API will return the minimum and maximum microversions. These values are used by the client to discover the API’s supported microversion(s).

Requests to / will get version info for all endpoints. A version response would look as follows for GET http://host_ip/accelerator/

{
  "default_version": {
    "status": "CURRENT",
    "min_version": "2.0",
    "max_version": "2.1",
    "id": "v2.0",
    "links": [
      {
        "href": "http://host_ip/accelerator/v2",
        "rel": "self"
      }
    ]
  },
  "versions": [
    {
      "status": "CURRENT",
      "min_version": "2.0",
      "max_version": "2.1",
      "id": "v2.0",
      "links": [
        {
          "href": "http://host_ip/accelerator/v2",
          "rel": "self"
        }
      ]
    }
  ],
  "name": "OpenStack Cyborg API",
  "description": "Cyborg is the OpenStack project for lifecycle "
                 "management of hardware accelerators, such as GPUs,"
                 "FPGAs, AI chips, security accelerators, etc."
}

Alternatives

Leave it as is, when user updates Cyborg along with new OpenStack release. It may lead to potential incapability to serve request from legacy clients.

Data model impact

None

REST API impact

Every API method should be decorated by a version-checking decorator.

For more Cyborg API details, please see Cyborg API Documentation [1]

Security impact

None

Notifications impact

None

Other end user impact

SDK authors will need to start using the OpenStack-API-Version header to get access to new features. The fact that new features will only be added in new versions will encourage them to do so.

python-cyborgclient is in an identical situation and will need to be updated to support the new header in order to support new API features.

Performance Impact

None

Other deployer impact

None

Developer impact

Any future changes to Cyborg’s REST API (whether that be in the request or any response) must result in a microversion update, and guarded in the code appropriately.

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)
Other contributors:: Shogo Saito

Work Items

Push OpenStack-API-Version header to API layer.
Add decorator for microversion check.
Implement versions.py module with history of API changes.
replace api_version_request.py by versions.py and use microversion_parser library.
Change python-cyborgclient side to support v2.0 microversion.

Dependencies

None

Testing

Appropriate unit and functional tests should be added.

Documentation Impact

Need a documentation to record microversion history.
Need a documentaiton to explain what is the backwards compatible/incompatible, what is the microverison of Cyborg, how to discover the versions and how to interact with cyborgclient.

References

History

Revisions
Release Name	Description
Ussuri	Introduced

Nova - Cyborg Interaction

Thu, 26 Feb 2026 00:00:00

https://blueprints.launchpad.net/nova/+spec/nova-cyborg-interaction

This specification describes the Nova - Cyborg interaction needed to create and manage instances with accelerators, and the changes needed in Nova to accomplish that.

Problem description

Scope

Representation: Cyborg shall represent devices as nested resource providers under the compute node (except possibly for disaggregated servers), accelerator types as resource classes and accelerators as inventory in Placement. The properties needed for scheduling are represented as traits. This is specified by [1]. This spec does not dwell on this topic.
Discovery and Updates: Among the devices discovered in a host, Cyborg intends to claim only those that are not included under the PCI Whitelisting mechanism. Cyborg shall update Placement in a way that is compatible with the virt driver’s update of Placement. These aspects are addressed in sections Coexistence with PCI whitelists and Placement update respectively.
User requests for accelerators: Users usually request compute resources via flavors. However, since the requests for devices may be highly varied, placing them in flavors may result in flavor explosion. We avoid that by expressing device requests in a device profile [2] . The relationship between device profiles and flavors is explored in Section User requests.

When an instance creation (boot) request is made, the contents of a device profile shall be translated to request groups in the request spec; the syntax in request groups is covered in Section Updating the Request Spec.
Instance scheduling: Nova shall use the Placement data populated by Cyborg to schedule instances. This spec does not dwell on this topic.
Assignment of accelerators: We introduce the concept of Accelerator Request objects in Section Accelerator Requests. The workflow to create and use them is summarized in Section Nova changes for Assignment workflow. The same section also highlights the Nova changes needed.
Instance operations: The behavior with respect to accelerators for all standard instance operations are defined in [3]. This spec does not dwell on this topic.

Use Cases

A user requests an instance with one or more accelerators of different types assigned to it.
An operator may provide users with both Device as a Service or Accelerated Function as a Service in the same cluster (see [1]).

The following use cases are not addressed in this release but are of long term interest:

A user requests to add one or more accelerators to an existing instance.
Live migration with accelerators.

Proposed change

Coexistence with PCI whitelists

Placement update

Cyborg shall call Placement API directly to represent devices and accelerators. Some of the intended use cases for the API invocation are:

Create or delete child RPs under the compute node RP.
Create or delete custom RCs and custom traits.
Associate traits with RPs or remove such association.
Update RP inventory.

Cyborg shall not modify the RPs created by any other component, such as Nova virt drivers.

User requests

The user request for accelerators is encapsulated in a device profile [2], which is created and managed by the admin via the Cyborg API.

In the initial phase, Nova API remains as today. The device profile is folded into the flavor as an extra spec by the operator, as below:

openstack flavor set --property 'accel:device_profile=<profile_name>' flavor

Thus the standard Nova API can be used to create an instance with only the flavor (without device profiles), like this:

openstack server create --flavor f ....  # instance creation

In the future, device profile may be used by itself to specify accelerator resources for the instance creation API.

Updating the Request Spec

When the device profile request groups are added to other request groups in the flavor, the group_policy of the flavor shall govern the overall semantics of all request groups.

Accelerator Requests

The order of request groups in a device profile is not significant, but it is preserved by Cyborg. Thus, each device profile request group has a unique index.
When the device profile request groups returned by Cyborg are added to the request spec, the requester_id field is set to ‘device_profile_<N>’ for the N-th device profile request group (starting from zero). The device profile name need not be included here because there is only one device profile per request spec.
When Cyborg creates an ARQ for a device profile, it embeds the device profile request group index in the ARQ before returning it to Nova.
The matching is done in two steps:
- Each ARQ is mapped to a specific request group in the request spec using the requester_id field.
- Each request group is mapped to a specific RP using the same logic as the Neutron bandwidth provider ([5]).

Nova changes for Assignment workflow

This section summarizes the workflow details for Phase 1. The changes needed in Nova are marked with NEW.

NEW: A Cyborg client module is added to nova (cyborg-client-module). All Cyborg API calls are routed through that.

The Nova API server receives a POST /servers API request with a flavor that includes a device profile name.
NEW: The Nova API server calls the Cyborg API GET /v2/device_profiles?name=$device_profile_name and gets back the device profile. The request groups in that device profile are added to the request spec.
The Nova scheduler invokes Placement and gets a list of allocation candidates. It selects one of those candidates and makes claim(s) in Placement. The Nova conductor then sends a RPC message build_and_run_instances to the Nova compute manager.
NEW: Nova compute manager calls the Cyborg API POST /v2/accelerator_requests with the device profile name. Cyborg creates a set of unbound ARQs for that device profile and returns them to Nova. (The call may originate from Nova conductor or the compute manager; that will be settled in code review.)
NEW: The Cyborg client in Nova matches each ARQ to the resource provider picked for that accelerator. See match-rp.
NEW: The Nova compute manager calls the Cyborg API PATCH /v2/accelerator_requests to bind the ARQ with the host name, device’s RP UUID and instance UUID. This is an asynchronous call which prepares or reconfigures the device in the background.

NEW: Cyborg, on completion of the bindings (successfully or otherwise), calls Nova’s POST /os-server-external-events API with:

{
   "events": [
      { "name": "accelerator-requests-bound",
        "tag": $device_profile_name,
        "server_uuid": $instance_uuid,
        "status": "completed" # or "failed"
      },
      ...
   ]
}

NEW: The Nova compute manager waits for the notification, subject to the timeout mentioned in Section Other deployer impact. It then calls the Cyborg REST API GET /v2/accelerator_requests?instance=<uuid>&bind_state=resolved.
NEW: The Nova virt driver uses the attach handles returned from the Cyborg call to compose PCI passthrough devices into the VM’s definition.
NEW: If there is any error after binding has been initiated, Nova must unbind the relevant ARQs by calling Cyborg API. It may then retry on another host or delete the (unbound) ARQs for the instance.

This flow is captured by the following sequence diagram, in which the Nova conductor and scheduler are together represented as the Nova controller.

Alternatives

It is possible to have the Nova virt driver poll for the Cyborg ARQ binding completion. That is not preferable, partly because that is not the pattern of interaction with other services like Neutron.

Data model impact

None

REST API impact

None. A new extra_spec key accel:device_profile_name is added to the flavor, but no API is modified.

Security impact

None

Notifications impact

Nova may choose to add additional notifications for Cyborg API calls.

Other end user impact

None

Performance Impact

The extra calls to Cyborg REST API may potentially impact Nova conductor/scheduler throughput. This has been mitigated by making some critical Cyborg operations as asynchronous tasks.

Other deployer impact

The deployer needs to set up the clouds.yaml file so that Nova can call the Cyborg REST API.

The deployer needs to configure a new tunable in nova-cpu.conf:

* arq_binding_timeout (integer): Time in seconds for Nova compute
  manager to wait for Cyborg to notify that ARQ binding is done.
  Timeout is fatal, i.e., VM startup is aborted with an exception.
  Default: 300.

Developer impact

Upgrade impact

None

Implementation

Assignee(s)

Sundar Nadathur

Work Items

See the steps marked NEW in Nova changes for Assignment workflow section.

Dependencies

None

Testing

There need to be unit tests and functional tests for the Nova changes. Specifically, there needs to be a functional test fixture that mocks the Cyborg API calls.

Documentation Impact

Device profile creation needs to be documented in Cyborg, as noted in [2].

The need for operator to fold the device profile into the flavor needs to be documented.

References

Specification for Cyborg API Version 2

History

Revisions
Release Name	Description
Queues	Introduced
Train	Re-proposed
Ussuri	Re-proposed

Add disable/enable device API

Wed, 16 Aug 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/disable-enable-device

Nowadays, Cyborg discover the device on compute node by each driver. All devices matching the spec of driver are discovered and reported to the Placement service as an accelerator resources. This spec proposes a set of new APIs which allow admin users to disable/enable a device.

Problem description

Cyborg maintains a configuration files to configure the enabled drivers. Once the driver is enabled, the driver will discover all devices whose vendor ID, device ID match the driver’s requirement. If admin user do not want all devices to be used by virtual machine, there is no way to disable a device currently.

Use Cases

Alice is an admin user, she wants some FPGAs to be reserved for its own use and not allow them to be allocated to a VM at the time. For example, she wants to program the FPGA device and use it as the OVS agent running on the host.

Proposed change

Since the API layer is modified, a new microversion should be introduced.
It also need a new field in Device object and data model to indicate the status of a device. If one device is disabled, the status should be set to “maintaining”, and if the device is enabled, the status should be set to “enabled”. The default value should be “enabled”.
Cyborg need call Placement API to update the “reserved” field for the device.

Alternatives

None

Data model impact

A new column status should be added in Device’s data model.

REST API impact

A microversion need to be introduced since the Device API changed.

List Device API

Return a device list URL: /devices METHOD: GET Return: 200

{
    "devices":
    [
        {
            "uuid": "359c0990-0258-44fd-8b05-fc510ac3d022",
            "type": "FPGA",
            "vendor": "0xABCD",
            "model": "miss model info",
            "std_board_info": "{'device_id': '0xabcd', 'class': 'Fake class'}",
            "vendor_board_info": "fake_vendor_info",
            "hostname": "computenode",
            "device_status": "Enabled"
            "created_at": "2020-03-13T02:26:31+00:00",
            "updated_at": null,
            "links":
            [
                {
                    "href": "http://localhost/accelerator/v2/devices/359c0990-0258-44fd-8b05-fc510ac3d022",
                    "rel": "self"
                }
            ]
        }
    ]
}

Get Device API

Get a device by uuid and return the details URL: /devices/{uuid} METHOD: GET Return: 200

{
    "uuid": "29e23349-12ee-4978-963c-11484a4ae601",
    "parent_id": null,
    "root_id": null,
    "name": "computenode_FakeDevice",
    "num_accelerators": 16,
    "device_id": 1,
    "attributes_list": "[{'traits1': 'CUSTOM_FAKE_DEVICE'}, {'rc': 'FPGA'}]",
    "rp_uuid": "853f07a6-19de-3dd6-b9f6-6c782daa3f7b",
    "driver_name": "fake",
    "status": "Enabled"
    "bitstream_id": null,
    "created_at": "2020-03-13T02:27:35+00:00",
    "updated_at": "2020-03-13T02:27:36+00:00",
    "links":
    [
       {
          "href": "http://localhost/accelerator/v2/deployables/29e23349-12ee-4978-963c-11484a4ae601",
          "rel": "self"
       },
       {
          "href": "http://localhost/accelerator/deployables/29e23349-12ee-4978-963c-11484a4ae601",
          "rel": "bookmark"
       }
    ]
}

Disable Device API

Disable a device URL: /devices/{device_uuid}/disable METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Enable Device API

Enable a device URL: /devices/{device_uuid}/enable METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer need update Cyborg to the microversion which supports disable/enable API. Otherwise the disable/enable API will be rejected.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: anguoming(agm_daydayup@foxmail.com)

Work Items

Add new column status for device table.
Add disable/enable API in DeviceController.
Update the RP reserved field according to the operation. For disable oparation, the reserved field need be set by the same value as the total field, and for enable operation, the reserved field will be set to zero.
Update GET/LIST device API with status field added in returned value.
Add disable/enable operation in cyborgclient.
Add unit tests.

Dependencies

None

Testing

Need add unit test, and tempest test if needed.

Documentation Impact

Need add related docs.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced
Bobcat	Reproposed

Support attribute API

Sat, 11 Mar 2023 00:00:00

This spec adds a new group of APIs to manage the lifecycle of accelerator’s attributes.

Problem description

Attribute is designed for describing customized information of an accelerator. Now they are generated by drivers, users can not add/delete/update them, it’s not applicable to our scenarios now.

Use Cases

An admin or operator needs a group of APIs to manage his accelerator’s attributes. Here are some useful scenarios:

For a NIC accelerator, we need to add a phys_net attribute, it’s should be created by deployer or other components.
For some Function Volatile Accelerators, we can create the Function name as an attribute.
Also for some information, such as Function_UUID is machine readable.

Proposed change

None

Alternatives

None

Data model impact

Add attribute object to deployable object.

REST API impact

URL: /v2/deployable/{uuid}/attribute

METHOD: GET

List all attributes of specified deployable.

Normal response code (200) and body:

{
    "attributes":[{
        "key":"key1",
        "value":"value1",
        "uuid":"uuid1"
        }
    ]
}

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
No response body

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: GET

GET specified attribute of specified deployable.

Query Parameters: None

Normal response code (200) and body:

{
    "attribute":
    {
        "key":"key1",
        "value":"value1",
        "uuid":"uuid1",
        "created_at":"2020-05-28T03:03:20",
        "updated_at":"2020-05-28T03:03:20"
    }
}

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists
No response body

URL: /v2/deployable/{uuid}/attribute

METHOD: POST

Create one or more deployable attribute(s).

Request body:

[
  {
    "key": "key1",
    "value": "value1"
 },
  {
    "key": "key2",
    "value": "value2"
  },
 ...
]

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
409 (Conflict): Bad input or key is not unique

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: DELETE

Delete an exist deployable attribute.

Query Parameters: None

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute

METHOD: DELETE

Delete all attributes of a deployable.

Query Parameters: None

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: PUT

Update an exist deployable attribute.

Query Parameters: None

Request body (Value of deployable attribute):

{"value": "value1"}

Normal response code and body:

204 (No content)
No response body

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists

Error response body:

{"error": "error-string"}

Security impact

None

Notifications impact

None

Other end user impact

Change Cyborg Attribute table.

Performance Impact

None

Other deployer impact

None

Developer impact

If the user want to use these feature, they should upgrade their Cyborg
project to latest to support these changes.

Implementation

Assignee(s)

Primary assignee:: hejunli

Work Items

Change Cyborg REST APIs.
Change Cyborg Attribute table.
Change Cyborg deployable object.
Change cyborgclient to support Attribute management action.
Add related tests.

Dependencies

None

Testing

Appropriate unit and functional tests should be added.

Documentation Impact

Need a documentation to record microversion history.
Need a documentaiton to explain api usage.

References

None

History

Revisions
Release Name	Description
Antelope	Introduced

Add disable/enable device API

Sat, 11 Mar 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/disable-enable-device

Nowadays, Cyborg discovers the device on compute node by each driver. All devices matching the spec of driver are discovered and reported to the Placement service as an accelerator resources. This spec proposes a set of new APIs which allow admin users to disable/enable a device.

Problem description

Cyborg maintains a configuration file to configure the enabled drivers. Once the driver is enabled, the agent will discover all devices whose vendor ID, device ID match the driver’s requirement. If admin user do not want all devices to be used by virtual machine, there is no way to disable a device currently.

Use Cases

Alice is an admin user, she wants some FPGAs to be reserved for its own use and not allow them to be allocated to a VM at the time. For example, she wants to program the FPGA device and use it as the OVS agent running on the host.

Proposed change

We propose to add new API in order to enable/disable a device. If the device is disabled, Cyborg will report this device as a reserved resource to Placement, so that Nova can not schedule to this device. On the contrary, if the device is enabled, the device should become available and the ‘reserved’ field in Placement shoule be set to 0. * Since the API layer is modified, a new microversion should be introduced. * It also need a new field “is_maintaining” in Device object and data model to indicate whether the device is disbaled. If one device is disabled, the “is_maintaining” field should be set to “True”, and if the device is enabled, the field should be set to “False”. The default value should be “False”. * Cyborg need call Placement API to update the “reserved” field for the device in this API. * Add “is_maintaining” field’s value check during conductor’s periodic report.

Alternatives

None

Data model impact

A new column is_maintaining should be added in Device’s data model.

REST API impact

A microversion need to be introduced since the Device API changed.

List Device API

Return a device list URL: /devices METHOD: GET Return: 200

{
    "devices": [
        {
            "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
            "type": "FPGA",
            "vendor": "0xABCD",
            "model": "miss model info",
            "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
            "vendor_board_info": "fake_vendor_info",
            "hostname": "devstack01",
            "links": [
                {
                    "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
                    "rel": "self"
                }
            ],
            "created_at": "2021-11-03T08:48:43+00:00",
            "updated_at": null
        }
    ]
}

Get Device API

Get a device by uuid and return the details URL: /devices/{uuid} METHOD: GET Return: 200

{
    "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
    "type": "FPGA",
    "vendor": "0xABCD",
    "model": "miss model info",
    "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
    "vendor_board_info": "fake_vendor_info",
    "hostname": "devstack01",
    "links": [
        {
            "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
            "rel": "self"
        }
    ],
    "created_at": "2021-11-03T08:48:43+00:00",
    "updated_at": null
}

Disable Device API

Disable a device URL: /devices/disable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Enable Device API

Enable a device URL: /devices/enable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer need update Cyborg to the microversion which supports disable/enable API. Otherwise the disable/enable API will be rejected.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)

Work Items

Add new column is_maintaining for device table.
Add disable/enable API in DeviceController.
Update the RP reserved field according to the operation. For disable oparation, the reserved field need be set by the same value as the total field, and for enable operation, the reserved field will be set to zero.
Update GET/LIST device API with is_maintaining field added in returned value.
Add disable/enable operation in cyborgclient.
Add unit tests.

Dependencies

None

Testing

Need add unit test, and tempest test if needed.

Documentation Impact

Need add related docs.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced
Yoga	Reproposed

Cyborg Intel PMEM Driver Proposal

Sat, 11 Mar 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/add-pmem-driver

This spec proposes to provide the initial design for Cyborg’s Intel PMEM driver.

Problem description

This spec will add Intel PMEM driver for Cyborg to manage specific Intel PMEM devices.

PMEM devices can be used as a large pool of low latency high bandwidth memory where they could store data for computation. This can improve the performance of the instance.

PMEM must be partitioned into PMEM namespaces [1] for applications to use. This vPMEM feature only uses PMEM namespaces in devdax mode as QEMU vPMEM backends [2]. If you want to dive into related notions, the document NVDIMM Linux kernel document [3] is recommended.

Starting in the 20.0.0 (Train) release, the virtual persistent memory (vPMEM) feature in Nova allows a deployment using the libvirt compute driver to provide vPMEMs for instances using physical persistent memory (PMEM) that can provide virtual devices [4].

Use Cases

As an operator, I would like to use Cyborg agent managing PMEM resource and checking periodically, the Cyborg Intel PMEM driver should provide discover() function to enumerate the list of the Intel PMEM devices, and report the details of all available Intel PMEM accelerators on the host, such as PID(Product id), VID(Vendor id), Device ID.
As a user, I would like to boot up a VM with Intel PMEM Device attached in order to accelerate compute ability. Cyborg should be able to manage this kind of acceleration resources and assign it to the VM(binding).

Proposed change

1. In general, the goal is to develop a Intel PMEM Device driver that supports discover interfaces for Intel PMEM accelerator framework. The driver should include the discover() function. This function works excuting “ndctl list” command that reports devices’ raw info sample as following:

[
  {
  "vendor": "8086",
  "product": "ns200_0",
  "device": "dax0.0"
  }
]

2. Generate Cyborg specific driver objects and resource provider modeling for the PMEM device. Below is the objects to describe a PMEM devices which complies with the Cyborg database mode and Placement data model.

Hardware     Driver objects       Placement data model
   |               |                      |
1 PMEM         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
n Namespace  n attach_handle    ---> inventories(total:n)

Need add the “enable_driver=intel_pmem_driver” in the Cyborg Agent configure file.
Need add the “pmem_namespaces=$LABEL:$NSNAME|$NSNAME,$LABEL:$NSNAME|$NSNAME” in the Cyborg Agent configure file as: “pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3”
Resource class follows standard resources classes as:
“CUSTOM_PMEM_NAMESPACE_$LABEL”
Traits follows the placement custom trait format. In the Cyborg driver, it will report two traits for PMEM accelerator using the format below: trait1:”CUSTOM_PMEM_NAMESPACE_$LABEL1” trait2:”CUSTOM_PMEM_NAMESPACE_$LABEL2”
Before cyborg discover the namespaces, they should be created. How to create the namespce can reference [5] and [6].

Alternatives

None

Data model impact

Need add new type such as PMEM in devices and attach_handle tables.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

User can manage Intel PMEM Device by Cyborg Intel PMEM driver. Such as list of the Intel PMEM devices, report the details of all available Intel PMEM accelerators on the host, binding with Intel PMEM and so on.

Performance Impact

None

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: qiujunting(qiujunting@inspur.com)

Work Items

Implement Intel PMEM driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Intel PMEM driver in Cyborg project. Add test report in cyborg wiki.

References

History

Revisions
Release	Description
Yoga	Introduced

Cyborg NVIDIA GPU Driver support vGPU management

Sat, 11 Mar 2023 00:00:00

The Cyborg NVIDIA GPU Driver has implemented pGPU management in the Train release, this spec proposes the specification of supporting vGPU management in the same driver.

Problem description

GPU devices can provide supercomputing capabilities, and can replace the CPU to provide users with more efficient computing power at a lower cost. GPU cloud servers have great value in the following application scenarios, including: video encoding and decoding, scientific research and artificial intelligence (deep learning, machine learning).

In the OpenStack ecosystem, users can now use Nova to pass gpu resources to guest by two methods:

Pass the GPU hardware to the guest (PCI pass-through).
Pass the Mediated Device(vGPU) to the guest.

With the long-term goal that Cyborg will manage heterogeneous accelerators including GPUs, Cyborg needs to support GPU management and integrate with Nova to provide users with gpu resources allocation in the aforementioned methods. The existing Cyborg GPU driver, NVIDIA GPU Driver, has supported the first method (PCI pass-through), while the second method is not yet supported. Please see ref [1] for Nova-Cyborg vGPU integration spec.

Use Cases

When the user is using Cyborg to manage GPU devices, he/she wants to boot up a VM with Nvidia GPU (pGPU or vGPU) attached in order to accelerate the video coding and decoding, Cyborg should be able to manage this kind of acceleration resources and to assign it to the VM(binding).

Proposed changes

To be clear, in the following, we will describe the whole process of how does the NVIDIA GPU Driver discover, generate Cyborg specific driver objects of the vGPU devices(comply with Cyborg Database Model), and report it to cyborg-db and Placement by cyborg-conductor. Features that are aleady supported in current branch is marked as DONE, new changes are marked as NEW CHANGES.

1. Collect raw info of GPU devices from compute node by “lspci” and grep nvidia related keyword.(DONE)

2. Parsing details from each record including vendor_id, product_id and pci_address.(DONE)

3. Generate Cyborg specific driver objects and resource provider modeling for the GPU device as well as its mdiated devices. Below is the objects to describe a vGPU devices which complies with the Cyborg database mode [4] and placement data model [5].(NEW CHANGE)

Hardware     Driver objects       Placement data model
   |               |                      |
1 GPU         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
4 vGPUs     4 attach_handles    ---> inventories(total:4)

4. Supporting set the vGPU type for a specific GPU device in cyborg.conf. The implementation is similar to that in Nova [9].(NEW CHANGE)

Firstly, we propose [gpu_devices]/enabled_vgpu_types to define which vgpu type Cyborg driver can use:
```
[gpu_devices]
enabled_vgpu_types = [str_vgpu_type_1, str_vgpu_type_2, ...]
```

And also, we propose that Cyborg driver will accept configuration sections that are related to the [gpu_devices]/enabled_vgpu_types and specifies which exact pGPUs are related to the enabled vGPU types and will have a device_addresses option defined like this:

cfg.ListOpt('device_addresses',
            default=[],
            help="""
List of physical PCI addresses to associate with a specific vGPU type.

The particular physical GPU device address needs to be mapped to the vendor
vGPU type which that physical GPU is configured to accept. In order to
provide this mapping, there will be a CONF section with a name
corresponding to the following template: "vgpu_%(vgpu_type_name)s

The vGPU type to associate with the PCI devices has to be the section name
prefixed by ``vgpu_``. For example, for 'nvidia-11', you would declare
``[vgpu_nvidia-11]/device_addresses``.

Each vGPU type also has to be declared in ``[gpu_devices]/enabled_vgpu_types``.

Related options:

* ``[gpu_devices]/enabled_vgpu_types``
"""),

For example, it would be set in cyborg.conf

[gpu_devices]
enabled_vgpu_types = nvidia-223,nvidia-224
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0,0000:86:00.0
[vgpu_nvidia-224]
device_addresses = 0000:87:00.0

5. Generate resource_class and traits for device, which later will also be reported to Placement, and used by nova-scheduler to filter appropriate accelerators.(NEW CHANGE)

resource class follows standard resources classes used by OpenStack [6]. Pass-through GPU device will report ‘PGPU’ as its resource class, Virtualized GPU device will report ‘VGPU’ as its resource class.
traits follows the placement custom trait format [7]. In the Cyborg driver, it will report two traits for vGPU accelerator using the format below:

trait1: OWNER_CYBORG.

trait2: CUSTOM_<VENDOR_NAME>_<PRODUCT_ID>_<Virtual_GPU_Type>.

Meaning of each parameter is listed below.
- OWNER_CYBORG: a new namespace in os-traits to remark that a device is reported by Cyborg when the inventory is reported to placement. It is used to distinguish GPU devices reported by Nova.
- VENDOR_NAME: vendor name of the GPU device.
- PRODUCT_ID: product ID of the GPU device.
- Virtual_GPU_Type: this parameter is actually another format of the enabled_vgpu_types for a specific device set by admin in cyborg.conf. In order to generate this param, driver will first retrieve enabled_vgpu_type and then map it to Virtual_GPU_Type by the way showed below. The name is exactly the Virtual_GPU_Type that will be reported in traits. For more details about the valid Virtual GPU Types for supported GPUs, please refer to [8].
```
# find mapping relation between Virtual_GPU_Type and enabled_vgpu_type.
# The value in "name" file contains its corresponding Virtual_GPU_Type.
cat /sys/class/mdev_bus/{device_address}/mdev_supported_types/{enabled_vgpu_type}/name
```
Here is a example to show the traits of a GPU device in the real world.
- A Nvidia Tesla T4 device has been successfully installed on host, device address is 0000:af:00.0. In addition, the vendor’s vGPU driver software must be installed and configured on the host at the same time.
```
[vtu@ubuntudbs ~]# lspci -nnn -D|grep 1eb8
0000:af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
```
- Enable GPU types (Accelerator)
  
  1. Specify which specific GPU type(s) the instances would get from this specific device.
  
  Edit devices.enabled_vgpu_types and device_address in cyborg.conf:
```
[gpu]
enabled_vgpu_types=nvidia-223
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0
```
  1. Restart the cyborg-agent service.
- Finally, traits reported for this device(RP) will be:
  
  OWNER_CYBORG and CUSTOM_NVIDIA_1EB8_T4_2B

Note

For the last parameter “T4_2B” (<Virtual_GPU_type>), we can validate the mapping relation between “nvidia-223” and “T4_2B” by check from the mdev sys path:

[vtu@ubuntudbs mdev_supported_types]$ pwd
/sys/class/mdev_bus/0000:af:00.0/mdev_supported_types
[vtu@ubuntudbs mdev_supported_types]$ ls
nvidia-222  nvidia-225  nvidia-228  nvidia-231  nvidia-234  nvidia-320
nvidia-223  nvidia-226  nvidia-229  nvidia-232  nvidia-252  nvidia-321
nvidia-224  nvidia-227  nvidia-230  nvidia-233  nvidia-319
[vtu@ubuntudbs mdev_supported_types]$ cat nvidia-223/name
GRID T4-2B

6. Generate controlpath_id, deployable, attach_handle, attribute for vGPU.(NEW CHANGE)

7. Create a mdev device in the sys by echo its UUID (actually is the attach_handle UUID) to the create file when vgpu is bind to a VM.(NEW CHANGE)

create_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/create

8. Delete a mdev device from sys by echo “1” to the remove file when vgpu is unbind from a VM.(NEW CHANGE)

remove_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/UUID/remove

Alternatives

Using Nova to manage vGPU device [10].

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This feature is highly dependent on the version of libvirt and the physical devices present on the host.

For vGPU management, deployers need to make sure that the GPU device has been successfully virtualized. Otherwise, Cyborg will report it as a pGPU device.

Please see ref [2] and [3] for how to install the Virtual GPU Manager package to virtualize your GPU devices.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>

Work Items

Implement NVIDIA GPU Driver enhancement in Cyborg
Add related test cases.
Add test report to wiki and update the supported driver doc page

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Nvidia GPU driver in Cyborg project.

References

History

Revisions
Release	Description
Wallaby	Introduced

Example Spec - The title of your blueprint

Sat, 11 Mar 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.2 Bobcat	Introduced

Example Spec - The title of your blueprint

Sat, 11 Mar 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.2 Bobcat	Introduced

Example Spec - The title of your blueprint

Sat, 11 Mar 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.2 Bobcat	Introduced

Add disable/enable device API

Sat, 11 Mar 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/disable-enable-device

Problem description

Use Cases

Alice is an admin user, she wants some FPGAs to be reserved for its own use and not allow them to be allocated to a VM at the time. For example, she wants to program the FPGA device and use it as the OVS agent running on the host.

Proposed change

Alternatives

None

Data model impact

A new column is_maintaining should be added in Device’s data model.

REST API impact

A microversion need to be introduced since the Device API changed.

List Device API

Return a device list URL: /devices METHOD: GET Return: 200

{
    "devices": [
        {
            "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
            "type": "FPGA",
            "vendor": "0xABCD",
            "model": "miss model info",
            "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
            "vendor_board_info": "fake_vendor_info",
            "hostname": "devstack01",
            "links": [
                {
                    "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
                    "rel": "self"
                }
            ],
            "created_at": "2021-11-03T08:48:43+00:00",
            "updated_at": null
        }
    ]
}

Get Device API

Get a device by uuid and return the details URL: /devices/{uuid} METHOD: GET Return: 200

{
    "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
    "type": "FPGA",
    "vendor": "0xABCD",
    "model": "miss model info",
    "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
    "vendor_board_info": "fake_vendor_info",
    "hostname": "devstack01",
    "links": [
        {
            "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
            "rel": "self"
        }
    ],
    "created_at": "2021-11-03T08:48:43+00:00",
    "updated_at": null
}

Disable Device API

Disable a device URL: /devices/disable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Enable Device API

Enable a device URL: /devices/enable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer need update Cyborg to the microversion which supports disable/enable API. Otherwise the disable/enable API will be rejected.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)

Work Items

Add new column is_maintaining for device table.
Add disable/enable API in DeviceController.
Update the RP reserved field according to the operation. For disable oparation, the reserved field need be set by the same value as the total field, and for enable operation, the reserved field will be set to zero.
Update GET/LIST device API with is_maintaining field added in returned value.
Add disable/enable operation in cyborgclient.
Add unit tests.

Dependencies

None

Testing

Need add unit test, and tempest test if needed.

Documentation Impact

Need add related docs.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced
Yoga	Reproposed

Cyborg Intel PMEM Driver Proposal

Sat, 11 Mar 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/add-pmem-driver

This spec proposes to provide the initial design for Cyborg’s Intel PMEM driver.

Problem description

This spec will add Intel PMEM driver for Cyborg to manage specific Intel PMEM devices.

PMEM devices can be used as a large pool of low latency high bandwidth memory where they could store data for computation. This can improve the performance of the instance.

Use Cases

As an operator, I would like to use Cyborg agent managing PMEM resource and checking periodically, the Cyborg Intel PMEM driver should provide discover() function to enumerate the list of the Intel PMEM devices, and report the details of all available Intel PMEM accelerators on the host, such as PID(Product id), VID(Vendor id), Device ID.
As a user, I would like to boot up a VM with Intel PMEM Device attached in order to accelerate compute ability. Cyborg should be able to manage this kind of acceleration resources and assign it to the VM(binding).

Proposed change

[
  {
  "vendor": "8086",
  "product": "ns200_0",
  "device": "dax0.0"
  }
]

Hardware     Driver objects       Placement data model
   |               |                      |
1 PMEM         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
n Namespace  n attach_handle    ---> inventories(total:n)

Need add the “enable_driver=intel_pmem_driver” in the Cyborg Agent configure file.
Need add the “pmem_namespaces=$LABEL:$NSNAME|$NSNAME,$LABEL:$NSNAME|$NSNAME” in the Cyborg Agent configure file as: “pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3”
Resource class follows standard resources classes as:
“CUSTOM_PMEM_NAMESPACE_$LABEL”
Traits follows the placement custom trait format. In the Cyborg driver, it will report two traits for PMEM accelerator using the format below: trait1:”CUSTOM_PMEM_NAMESPACE_$LABEL1” trait2:”CUSTOM_PMEM_NAMESPACE_$LABEL2”
Before cyborg discover the namespaces, they should be created. How to create the namespce can reference [5] and [6].

Alternatives

None

Data model impact

Need add new type such as PMEM in devices and attach_handle tables.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

Performance Impact

None

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: qiujunting(qiujunting@inspur.com)

Work Items

Implement Intel PMEM driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Intel PMEM driver in Cyborg project. Add test report in cyborg wiki.

References

History

Revisions
Release	Description
Yoga	Introduced

Cyborg Xilinx FPGA Driver Proposal

Sat, 11 Mar 2023 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/add-xilinx-fpga-driver

This spec proposes to provide the initial design for Cyborg’s Xilinx FPGA driver.

Problem description

We want to use Cyborg manage the Xilinx FPGA devices, but there is not driver for it, and we want to add the Xilinx FPGA driver to support managing Xilinx FPGA devices by Cyborg.

Use Cases

As an operator, I would like to use Cyborg agent starts or does resource checking periodically, the Cyborg Xilinx FPGA driver should provide discover() function to enumerate the list of the Xilinx FPGA devices, and report the details of all available Xilinx FPGA accelerators on the host, such as PID(Product id), VID(Vendor id), Device.
As a user, I would like to boot up a VM with Xilinx FPGA device. Cyborg should be able to manage this kind of acceleration resources, assign it to the VM(binding) and programming it.

Proposed changes

Xilinx FPGA is PCIe based platform that is comprised of physical partitions called Shell and User. The Shell has two physical functions: privileged PF0 also called mgmt pf and non-privileged PF1 also called user pf. Shell provides basic infrastructure for the Alveo platform. User partition (otherwise known as PR-Region) contains user compiled binary.[1]_

User PF is only displayed after installing Xilinx driver xrt. And the difference between the PCI address of user PF and mgmt PF is the func field. For example, pci address of mgmt PF is ‘0000:3b:00.0’, and the address of user PF is ‘0000:3b:00.1’. The func field increase 1. So the pci address of user PF can be calculated by the address of mgmt PF. This calculation rule also applies to ‘product’.

From the above description, the Xilinx FPGA driver should implement the following methods:

discover(): This function works by executing lspci command and reports devices’ raw info sample as following:
```
[
  {
  "vendor": "10ee",
  "product": "5000",
  "device": "0000:3b:00.0"
  }
]
```
In above example, “5000” is the mgmt PF. So the driver should add three traits for one device, including “CUSTOM_FPGA_<VENDOR_ID>”, “CUSTOM_FPGA_PRODUCT_ID_<ID>”. Also the driver should create records in deployable table only for the device. For now, only when both mgmt PF and user PF are binded to one vm, and end user can program the FPGA device. So there are two attach handles when binding it to vm.
program(): This function works by executing xbmgmt program command which is included with the XRT installation package.[2]_ Also, this function should be called for pre-program before binding to vm.

Below is the objects to describe a FPGA device which complies with the Cyborg database mode and Placement data model.

Hardware     Driver objects       Placement data model
   |               |                      |
1 FPGA         1 devices                  |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
1 FPGA       2 attach_handle    ---> inventories(total:1)

Alternatives

None

Data model impact

Xilinx FPGA driver will not touch Data model. The Cyborg Agent can call Xilinx FPGA driver to update the database during the discover, bind and unbind operations.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Add new driver xilinx_fpga_driver to enabled_drivers for agent.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Eric Xie

Work Items

Implement Xilinx FPGA driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Xilinx FPGA driver in Cyborg project. Test report in Cyborg wiki

References

History

Revisions
Release	Description
Yoga	Introduced

Cyborg Intel® x710 driver proposal

Tue, 03 Jan 2023 00:00:00

This spec proposes to provide a new Cyborg driver for Intel® x710 dirver, which is mentioned in Nova’s SRIOV Nic Support spec [1]. In Cyborg side, we need to implement a driver for SRIOV Nic in order to manage the lifecycle of this device, and to allow Nova request this kind of resource from Cyborg.

Problem description

The Intel Ethernet Converaged Network Adapter x710 [2] addressed the demanding need of an agile data center by providing unmatched features for both server and virtualization, flexibility for LAN or SAN networks and proven, reliable performance.

This device also support SR-IOV virtualization technology, it can be virtualized up to 128 VFs, each VF can support a unique and separate data path for I/O related functions within the PCI Express hierarchy.

Using SR-IOV with the networking device, for example, allow the bandwidth of a single port(function) to be partitioned into smaller slices that can be allocated to specific VMs or guest, via a standard interface.

One of the key features of x710 Network Adapter is Dynamic Device Personalization(DDP) [3].

DDP allows dynamic reconfiguration of the packet processing pipeline to meet specific use case needs on demand, adding new packet processing pipeline configuration profiles to a network adapter at run time, without resetting or rebooting the server. Software applies these custom profiles in a nonpermanent, transaction-like mode, so the original network controller’s configuration is restored after network adapter reset, or by rolling back profile changes by software. The DPDK provides all APIs to handle DDP packages.

Use Cases

As an operator, I want to use Cyborg to manage lifecycle of Intel X710. Cyborg side driver should be able to manage this kind of acceleration resources including discover it, report it to Placement and program it.
As an end user, I want to boot up a VM with Intel X710 to accelerate network related workloads, Cyborg should be able to interact with Nova to assign it the VM.
As an advanced end user, I want to customize a profile to load on Intel X710 when booting up a VM with it, Cyborg driver should be able to provide such program interface.

Proposed change

In general, the goal is to develop a Cyborg Intel® x710 driver that supports discover() and program() interfaces.

Discovery

The driver should include discover functions which will be called by agent periodically, and encapsule the device information to Cyborg’s unified data model, such as Device, Deployable, AttachHandle and so on.

Besides, we should also implement a configuration file to let Cyborg driver to discover the loaded DDP profile on X710, as well as the physnet name associated with this nic and then Cyborg can report these as the attributes.

An example of the configuration file is:

[nic_devices]
enabled_nic_types = x710_static

[x710_static]
physical_device_mappings = physnet1:eth2|eth3
function_device_mappings = GTPv1:eth3|eth2

Programming

The driver should implement a program() function which allow Cyborg API to call to load a specific profile on the nic, this is a part-implementation of the third use case mentioned above.

To implement this functionality, Nova needs to parse user’s request and get the profile’s name, and Cyborg should get this profile’s info from Nova and let this driver to load profile to the selected device.

Briefly, this function need to call some utility like (DPDK API or ethtool) to program the nic. For the first phase, we will work on the pre-programmed scenario. Another spec will be introduced to show the total design of dynamic programming functionality of this driver in the future.

Alternatives

None

Data model impact

One physical X710 card has 2 PFs, and each of them can be virtualized into several VFs. The maximum number of the VFs are different according to different models.

Here, we take an example of 4 VFs in each PFs.

The data model in Cyborg should like this:

+---------------------------------------------------+
|                                                   |
|      device A                    device B         |
|   +------------------+     +------------------+   |
|   | deployable 1-4   |     | deployables 5-8  |   |
|   | +----+ +----+    |     | +----+  +----+   |   |
|   | | vf | | vf |    |     | | vf |  | vf |   |   |
|   | +----+ +----+    |     | +----+  +----+   |   |
|   | +----+ +----+    |     | +----+  +----+   |   |
|   | | vf | | vf |    |     | | vf |  | vf |   |   |
|   | +----+ +----+    |     | +----+  +----+   |   |
|   +------------------+     +------------------+   |
| X710                                              |
+---------------------------------------------------+

The resource provider tree’s structure should be reported as:

                     +--------------+
                     |              |
                     | compute node |
                     |              |
                     +-------+------+
                             |
      +----------------+-----+-----------------------+
      |                |                             |
      v                v                             v
+-----+-----+     +----+-------+                 +---+--------+
|           |     |            |                 |            |
|deployable1|     |deployable 2|   ...           |deployable 8|
+-----------+     +------------+                 +------------+

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Developer impact

Deployer need to install DPDK toolkit to configure the DDP feature.

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)

Dependencies

None

Testing

Unit tests will be added to test this driver.
Add test result in Cyborg Wiki which is required by the Cyborg community.

Documentation Impact

Need add or update related documentations.

References

History

Revisions
Release Name	Description
Wallaby	Introduced

Support attribute API

Mon, 14 Nov 2022 00:00:00

This spec adds a new group of APIs to manage the lifecycle of accelerator’s attributes.

Problem description

Attribute is designed for describing customized information of an accelerator. Now they are generated by drivers, users can not add/delete/update them, it’s not applicable to our scenarios now.

Use Cases

An admin or operator needs a group of APIs to manage his accelerator’s attributes. Here are some useful scenarios:

For a NIC accelerator, we need to add a phys_net attribute, it’s should be created by deployer or other components.
For some Function Volatile Accelerators, we can create the Function name as an attribute.
Also for some information, such as Function_UUID is machine readable.

Proposed change

None

Alternatives

None

Data model impact

Add attribute object to deployable object.

REST API impact

URL: /v2/deployable/{uuid}/attribute

METHOD: GET

List all attributes of specified deployable.

Normal response code (200) and body:

{
    "attributes":[{
        "key":"key1",
        "value":"value1",
        "uuid":"uuid1"
        }
    ]
}

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
No response body

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: GET

GET specified attribute of specified deployable.

Query Parameters: None

Normal response code (200) and body:

{
    "attribute":
    {
        "key":"key1",
        "value":"value1",
        "uuid":"uuid1",
        "created_at":"2020-05-28T03:03:20",
        "updated_at":"2020-05-28T03:03:20"
    }
}

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists
No response body

URL: /v2/deployable/{uuid}/attribute

METHOD: POST

Create one or more deployable attribute(s).

Request body:

[
  {
    "key": "key1",
    "value": "value1"
 },
  {
    "key": "key2",
    "value": "value2"
  },
 ...
]

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
409 (Conflict): Bad input or key is not unique

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: DELETE

Delete an exist deployable attribute.

Query Parameters: None

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute

METHOD: DELETE

Delete all attributes of a deployable.

Query Parameters: None

Normal response code and body:

204 (No content)
No response body

Error response code:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed

Error response body:

{"error": "error-string"}

URL: /v2/deployable/{uuid}/attribute/{uuid_or_key}

METHOD: PUT

Update an exist deployable attribute.

Query Parameters: None

Request body (Value of deployable attribute):

{"value": "value1"}

Normal response code and body:

204 (No content)
No response body

Error response code and body:

401 (Unauthorized): Unauthorized
403 (Forbidden): RBAC check failed
404 (NotFound): No deployable of that UUID or no attribute of that UUID exists

Error response body:

{"error": "error-string"}

Security impact

None

Notifications impact

None

Other end user impact

Change Cyborg Attribute table.

Performance Impact

None

Other deployer impact

None

Developer impact

If the user want to use these feature, they should upgrade their Cyborg
project to latest to support these changes.

Implementation

Assignee(s)

Primary assignee:: hejunli

Work Items

Change Cyborg REST APIs.
Change Cyborg Attribute table.
Change Cyborg deployable object.
Change cyborgclient to support Attribute management action.
Add related tests.

Dependencies

None

Testing

Appropriate unit and functional tests should be added.

Documentation Impact

Need a documentation to record microversion history.
Need a documentaiton to explain api usage.

References

None

History

Revisions
Release Name	Description
Antelope	Introduced

Example Spec - The title of your blueprint

Tue, 04 Oct 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.1 Antelope	Introduced

Example Spec - The title of your blueprint

Tue, 04 Oct 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.1 Antelope	Introduced

Example Spec - The title of your blueprint

Tue, 04 Oct 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2023.1 Antelope	Introduced

Show device_profile by name

Thu, 07 Apr 2022 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/show-device-profile-with-name

Nowadays, A device_profile can only be got by it’s UUID, but the field ‘name’ is not supported. This spec will support getting a device profile by name.

Problem description

When using openstack accelerator device profile show command to getting one device profile, the ‘<uuid>’ of the device profile is only accepted. The device profile has ‘name’ field that can display in humanable manner. The end-users prefer to use name instead of hard-to-remember UUID. By this way you can get the device profile details directly instead of searching its UUID first.

Use Cases

As an administrator or end user, i want to get the details of the device profile by its name.

Proposed change

Change the Get One Device Profile API to accept ‘name’ as parameter.
Introduce a microverison for Get One Device Profile API
Then, change the python-cyborgclient with openstack accelerator device profile show, to support getting one device profile details by name.

Alternatives

None

Data model impact

None

REST API impact

A microversion need to be introduced since the Device Profile API changed.

Get Device Profile API

GET /v2/device_profiles/{device_profile_name_or_uuid} The ‘device_profile_name_or_uuid’ should be uuid or name, the response has not changed.

{
   "device_profile":{
      "name":"fpga-dp1",
      "uuid":"5518a925-1c2c-49a2-a8bf-0927d9456f3e",
      "description": "",
      "groups":[
         {
            "trait:CUSTOM_CHENKE_TRAITS":"required",
            "resources:FPGA":"1",
            "accel:bitstream_id":"d5ca2f11-3108-4426-a11c-a959987565df"
         }
      ],
      "created_at": "2020-03-09 11:26:05+00:00",
      "updated_at": null,
      "links":[
         {
            "href":"http://192.168.32.217/accelerator/v2/device_profiles/5518a925-1c2c-49a2-a8bf-0927d9456f3e",
            "rel":"self"
         }
      ]
   }
}

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Eric Xie(eric_xiett@163.com)

Work Items

Change the device profile get api to support ‘name’
Extend ‘python-cyborgclient’ to support ‘name’
Add related unit tests

Dependencies

None

Testing

Need add unit test.

Documentation Impact

Use ‘device_profile_name_or_uuid’ instead of ‘device_profile_uuid’ and change its description in ‘api-ref/source/device_profile.inc’.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Yoga	Introduced

Example Spec - The title of your blueprint

Thu, 07 Apr 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Zed	Introduced

Example Spec - The title of your blueprint

Thu, 07 Apr 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Zed	Introduced

Example Spec - The title of your blueprint

Thu, 07 Apr 2022 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Zed	Introduced

Device Profiles

Mon, 20 Dec 2021 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/device-profiles

This spec introduces the notion of device profiles, explains why they are needed and describes how they are used.

Problem description

The request to create or update an instance can specify various resources, spanning compute, networking, storage and accelerators. While all of them can be specified using the flavor syntax including extra specs, that results in two significant problems: lack of separation of concerns and flavor explosion.

It is desirable to separate the requirements for different resources into modular pieces and then compose them into the flavor. This is done for networking with port binding profiles. It is desirable to propose a similar model for accelerators and hardware resources.

Requests for devices can vary hugely in the type, number and combination of devices requested. For example, a web server workload may require offloads for video transcoding, SSL and log file compression. Offloads may involve different types of devices, including programmable hardware (GPUs, FPGAs, etc.), fixed function hardware (QuickAssist, TPUs, etc.) and special-purpose hardware such as High precision Time Synchronization cards. If each combination of hardware resources were to be expressed as parts of flavors, the number of flavors will increase rapidly. That is a problem because operators often use flavors for usage and billing.

So, we wish to separate the specification of accelerator resources in an instance request into a new entity called a device profile.

Use Cases

The operator should be able to create, publish, update and delete device profiles.
The operator should be able to compose a device profile into a flavor, so that the end user can request an instance using the flavor alone.
The end user should be able to request an instance to be created or updated, using a flavor and a device profile separately.
The operator should be able to enable or disable specific device profiles.
On an upgrade, device profiles from a certain number of past releases should continue to be usable by operators.

The ability to update or enable/disable device profiles may be addressed in the future after operator feedback.

The operator may use device profile usage to bill end users. That is outside the scope of this spec.

Proposed change

Overview

A device profile is a named set of the user requirements for one or more accelerators. It can be viewed as a flavor for devices. Broadly it includes two things: the desired amounts of specific resource classes and the requirements that the resource provider(s) must satisfy. While the resource classes are the same as those known to Placement, some requirements would correspond to Placement traits and others to properties that Cyborg alone knows about.

The device profile format is a Python dictionary with syntax similar to granular resource requests for Nova flavors. Here is an example:

{
  'name': 'my_device_profile',
  'description': 'Image classification',
  'groups': [
    { 'resources:CUSTOM_ACCELERATOR_FPGA':'1',
      'trait:CUSTOM_FPGA_INTEL_ARRIA10':'required',
      'accel:function_id':'3AFB'
    },
    { 'resources:CUSTOM_ACCELERATOR_GPU':'2',
      'trait:CUSTOM_GPU_MODEL_NAME':'required',
      'accel:video_ram':'2GB'
    },
  ]
  'uuid': '8411555e-3cbf-47de-988c-b5e30bbfa035' # auto-generated
}

The fields and syntax are explained in the Section Device Profile structure.

The device profiles can be queried, created and deleted from the Cyborg API (see Section REST API impact) and from the command line (see Section Cyborg client command line). The ability to update them or enable/disable them may be taken up after operator feedback.

Only the admin can create, update and delete device profiles. In the future, the admin may be able to publish them in Horizon. The operator and end user both can list device profiles from the command line today and potentially view them in Horizon in the future.

Device profile structure

The salient points of the structure of the device profile are noted in this section.

Each field uses a format similar to extra specs; it specifies either a resource, a trait or a Cyborg-specific property. The properties are specified using the accel: prefix and are of the form 'accel:<key>':'<value>'. Both the key and the value are strings, which may contain alphabets (lower or upper case), digits, hyphens and underscores. No other characters are allowed. In the future, modifiers may be introduced after the accel: prefix. For example, to indicate that some resource is preferred but not required, a possible syntax could be 'accel:preferred:resource_name':'amount'. The valid keys in this release are listed in Section Valid accel keys.

The requests are grouped in the same way as granular resource syntax. The ordering among the groups is not significant. However, no group policy is allowed here because there may be one in the flavor.

Unlike the granular request syntax, the operator need not number the groups. Cyborg will do the numbering.

The order of request groups within a device profile has no significance for scheduling or any system behavior. However, it is preserved by Cyborg. So, if a client fetches the same device profile twice (with no changes to the device profile in between), the group order will remain the same. This fact is important for Nova Cyborg interaction [1].

Lower case and hyphens are allowed: The names of custom RCs and traits can only include upper case letters, digits and underscores. Cyborg will translate lower case to upper case and hyphens to underscores.

Usage

The user request for an instance spawn needs to include both a flavor and a device profile. However, this needs a change in the Nova API for instance creation to include the device profile name. To avoid the need for such a change, initially, the operator shall explicitly fold the device profile into the flavor as below:

openstack flavor set --property 'accel:device_profile=<profile_name>' flavor

Thus the end user can launch requests with only the flavor just as today.

Later the Nova API will be enhanced to take a device profile name explicitly, removing the need for operator folding and allowing end users to launch requests with separate flavor and device profile names. The device-profile-in-flavor form will continue to be supported even after this change.

In either case, only one device profile is allowed per user request.

During the instance creation flow, Nova queries Cyborg with the device profile name and gets the device profile request groups, which are then presented to Placement along with the request groups from flavor and other sources. This is the case regardless of whether the device profile was folded into the flavor or not.

Cyborg client command line

The command line syntax for device profiles is proposed in this section.

openstack accelerator device profile create <name> <json>
openstack accelerator device profile delete <name>
openstack accelerator device profile list
openstack accelerator device profile show <uuid>

Other commands can be taken up in the future after operator feedback.

Further details will be addressed in a future spec.

Versioning and upgrades

The microversion of the Cyborg APIs used to create/update device profiles also controls the device profile version. There is no separate version field in the device profile itself.

A custom trait in one release may become a standard trait in the next one. A cyborg-specific property in one release may become a custom trait or a standard trait in the next one. This would invalidate the device profiles after upgrade.

This must be handled by migrating the device profiles to the new format during the upgrade.

Valid accel keys

The Cyborg-specific properties are expressed in the device profile using the form 'accel:<key>':'<value>, as noted earlier. The valid key-value pairs in this release are as noted below:

Cyborg Properties
Key	Value Type	Semantics
`bitstream_id`	UUID	Glance UUID of the bitstream that must be programmed. The type of the bitstream, which may influence the tool used for programming, is then assumed to be a image property.
`bitstream_name`	String	Name of the bitstream in Glance, including the suffix that indicates bitstream type.
`function_id`	UUID	UUID of the needed function.
`function_name`	String	Name of the needed function.
`attach_target`	Enum of strings: ‘VM’, ‘host’ or ‘none’	Indicates whether the accelerator should be attached to a VM (default), the host (infrastructure offload use case), or not attached to anything (indirect access use case). See sections ‘Use cases’ and ‘Indirect accelerator access’ in [1].

Alternatives

Flavor explosion can be partially addressed by adding metadata to the instance image that expresses what hardware resources are needed for that image to work correctly or well. However, that is not a complete solution.

Data model impact

Device profiles need to be persisted in Cyborg database. Each device profile should have a UUID since they may be renamed.

REST API impact

URL: /v2/device_profiles
METHOD: GET
Query Parameters: name=name1,name2,...
Normal response code and body:
   200
   { 'device_profiles': [ <dev-prof>, ... ] }
Error response code and body:
   401 (Unauthorized): RBAC check failed
   422 (Unprocessable): No device profiles exist
   No response body
Note:
   List all device profiles or the set of named device profiles.

URL: /v2/device_profiles/{uuid}
METHOD: GET
Query Parameters: None
Normal response code and body:
   200
   { 'device_profile': <dev-prof> }
Error response code and body:
   401 (Unauthorized): RBAC check failed
   422 (Unprocessable): No device profile of that UUID exists
   No response body
Note:
   Get the device profile with the specified uuid.

URL: /v2/device_profiles
METHOD: POST
Request body: A device profile
   [
       { 'name': <string>,
         'description': <string> # optional
         'groups': [
              {
                  "accel:function_id": "3AFB",
                  "resources:CUSTOM_ACCELERATOR_FPGA": "1",
                  "trait:CUSTOM_FPGA_INTEL_PAC_ARRIA10": "required"
          }
          ],
       },
   ]
Normal response code and body:
   204 (No content)
   No response body
Error response code and body:
   401 (Unauthorized): RBAC check failed
   422 (Unprocessable): Bad input or name is not unique
   { 'error': <error-string> }
Note:
   Create one or more new device profiles. May implement just one in Train.

URL: /v2/device_profiles?name=string1,...,stringN
METHOD: DELETE
Query Parameters: required
Normal response code and body:
   204 (No content)
   No response body
Error response code and body:
   401 (Unauthorized): RBAC check failed
   422 (Unprocessable): Bad input
   { 'error': <error-string> }
Note:
   Delete one or more existing device profiles.

Until device profile updates are supported, there is no need for a PUT or PATCH.

Security impact

The APIs and commands proposed here accept and parse user-provided data. To mitigate the risks, the following measures shall be adopted:

Only the admin can create, update or delete device profiles. End users can only list them or perhaps see them in Horizon.
The device profile names and fields are restricted to alphabets (lower or upper case), digits, underscores, hyphens and the characters ‘:’ and ‘=’.

A device profile is visible to all tenants by default. In the future, we may provide for device profile visibility only to certain tenants.

Notifications impact

None

Other end user impact

The end user can list the device profiles using python-cyborgclient. In the future, she may be able to see them on Horizon.

Performance Impact

None

Other deployer impact

Knobs to enable or disable device profiles may be added in the future.

Developer impact

Provide upgrade scripts for device profiles if the names of resource classes or traits change, as noted in Versioning and upgrades.

Implementation

Assignee(s)

TBD

Work Items

Create a device profiles table in the Cyborg database.
Implement REST APIs in the API server and conductor, with validation and unit tests.
Implement the python-cyborgclient CLI.

Dependencies

None

Testing

Unit tests and functional tests need to be written.

Documentation Impact

Device profiles should be explained in user docs and in operator docs.

References

History

Revisions
Release Name	Description
Stein	Introduced
Train	Updated

Add disable/enable device API

Tue, 26 Oct 2021 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/disable-enable-device

Problem description

Use Cases

Alice is an admin user, she wants some FPGAs to be reserved for its own use and not allow them to be allocated to a VM at the time. For example, she wants to program the FPGA device and use it as the OVS agent running on the host.

Proposed change

Alternatives

None

Data model impact

A new column is_maintaining should be added in Device’s data model.

REST API impact

A microversion need to be introduced since the Device API changed.

List Device API

Return a device list URL: /devices METHOD: GET Return: 200

{
    "devices": [
        {
            "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
            "type": "FPGA",
            "vendor": "0xABCD",
            "model": "miss model info",
            "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
            "vendor_board_info": "fake_vendor_info",
            "hostname": "devstack01",
            "links": [
                {
                    "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
                    "rel": "self"
                }
            ],
            "created_at": "2021-11-03T08:48:43+00:00",
            "updated_at": null
        }
    ]
}

Get Device API

Get a device by uuid and return the details URL: /devices/{uuid} METHOD: GET Return: 200

{
    "uuid": "d2446439-0142-40b7-9eee-82d855f453d9",
    "type": "FPGA",
    "vendor": "0xABCD",
    "model": "miss model info",
    "std_board_info": "{"device_id": "0xabcd", "class": "Fake class"}",
    "vendor_board_info": "fake_vendor_info",
    "hostname": "devstack01",
    "links": [
        {
            "href": "http://172.23.97.140/accelerator/v2/devices/d2446439-0142-40b7-9eee-82d855f453d9",
            "rel": "self"
        }
    ],
    "created_at": "2021-11-03T08:48:43+00:00",
    "updated_at": null
}

Disable Device API

Disable a device URL: /devices/disable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Enable Device API

Enable a device URL: /devices/enable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer need update Cyborg to the microversion which supports disable/enable API. Otherwise the disable/enable API will be rejected.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)

Work Items

Add new column is_maintaining for device table.
Add disable/enable API in DeviceController.
Update the RP reserved field according to the operation. For disable oparation, the reserved field need be set by the same value as the total field, and for enable operation, the reserved field will be set to zero.
Update GET/LIST device API with is_maintaining field added in returned value.
Add disable/enable operation in cyborgclient.
Add unit tests.

Dependencies

None

Testing

Need add unit test, and tempest test if needed.

Documentation Impact

Need add related docs.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced
Yoga	Reproposed

Example Spec - The title of your blueprint

Sat, 31 Jul 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Yoga	Introduced

Cyborg Xilinx FPGA Driver Proposal

Sat, 31 Jul 2021 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/add-xilinx-fpga-driver

This spec proposes to provide the initial design for Cyborg’s Xilinx FPGA driver.

Problem description

We want to use Cyborg manage the Xilinx FPGA devices, but there is not driver for it, and we want to add the Xilinx FPGA driver to support managing Xilinx FPGA devices by Cyborg.

Use Cases

As an operator, I would like to use Cyborg agent starts or does resource checking periodically, the Cyborg Xilinx FPGA driver should provide discover() function to enumerate the list of the Xilinx FPGA devices, and report the details of all available Xilinx FPGA accelerators on the host, such as PID(Product id), VID(Vendor id), Device.
As a user, I would like to boot up a VM with Xilinx FPGA device. Cyborg should be able to manage this kind of acceleration resources, assign it to the VM(binding) and programming it.

Proposed changes

From the above description, the Xilinx FPGA driver should implement the following methods:

discover(): This function works by executing lspci command and reports devices’ raw info sample as following:
```
[
  {
  "vendor": "10ee",
  "product": "5000",
  "device": "0000:3b:00.0"
  }
]
```
In above example, “5000” is the mgmt PF. So the driver should add three traits for one device, including “CUSTOM_FPGA_<VENDOR_ID>”, “CUSTOM_FPGA_PRODUCT_ID_<ID>”. Also the driver should create records in deployable table only for the device. For now, only when both mgmt PF and user PF are binded to one vm, and end user can program the FPGA device. So there are two attach handles when binding it to vm.
program(): This function works by executing xbmgmt program command which is included with the XRT installation package.[2]_ Also, this function should be called for pre-program before binding to vm.

Below is the objects to describe a FPGA device which complies with the Cyborg database mode and Placement data model.

Hardware     Driver objects       Placement data model
   |               |                      |
1 FPGA         1 devices                  |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
1 FPGA       2 attach_handle    ---> inventories(total:1)

Alternatives

None

Data model impact

Xilinx FPGA driver will not touch Data model. The Cyborg Agent can call Xilinx FPGA driver to update the database during the discover, bind and unbind operations.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Add new driver xilinx_fpga_driver to enabled_drivers for agent.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Eric Xie

Work Items

Implement Xilinx FPGA driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Xilinx FPGA driver in Cyborg project. Test report in Cyborg wiki

References

History

Revisions
Release	Description
Yoga	Introduced

Example Spec - The title of your blueprint

Sat, 31 Jul 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Yoga	Introduced

Example Spec - The title of your blueprint

Sat, 31 Jul 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Yoga	Introduced

Cyborg Intel PMEM Driver Proposal

Fri, 02 Jul 2021 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/add-pmem-driver

This spec proposes to provide the initial design for Cyborg’s Intel PMEM driver.

Problem description

This spec will add Intel PMEM driver for Cyborg to manage specific Intel PMEM devices.

PMEM devices can be used as a large pool of low latency high bandwidth memory where they could store data for computation. This can improve the performance of the instance.

Use Cases

As an operator, I would like to use Cyborg agent managing PMEM resource and checking periodically, the Cyborg Intel PMEM driver should provide discover() function to enumerate the list of the Intel PMEM devices, and report the details of all available Intel PMEM accelerators on the host, such as PID(Product id), VID(Vendor id), Device ID.
As a user, I would like to boot up a VM with Intel PMEM Device attached in order to accelerate compute ability. Cyborg should be able to manage this kind of acceleration resources and assign it to the VM(binding).

Proposed change

[
  {
  "vendor": "8086",
  "product": "ns200_0",
  "device": "dax0.0"
  }
]

Hardware     Driver objects       Placement data model
   |               |                      |
1 PMEM         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
n Namespace  n attach_handle    ---> inventories(total:n)

Need add the “enable_driver=intel_pmem_driver” in the Cyborg Agent configure file.
Need add the “pmem_namespaces=$LABEL:$NSNAME|$NSNAME,$LABEL:$NSNAME|$NSNAME” in the Cyborg Agent configure file as: “pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3”
Resource class follows standard resources classes as:
“CUSTOM_PMEM_NAMESPACE_$LABEL”
Traits follows the placement custom trait format. In the Cyborg driver, it will report two traits for PMEM accelerator using the format below: trait1:”CUSTOM_PMEM_NAMESPACE_$LABEL1” trait2:”CUSTOM_PMEM_NAMESPACE_$LABEL2”
Before cyborg discover the namespaces, they should be created. How to create the namespce can reference [5] and [6].

Alternatives

None

Data model impact

Need add new type such as PMEM in devices and attach_handle tables.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

Performance Impact

None

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: qiujunting(qiujunting@inspur.com)

Work Items

Implement Intel PMEM driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Intel PMEM driver in Cyborg project. Add test report in cyborg wiki.

References

History

Revisions
Release	Description
Yoga	Introduced

Add disable/enable device API

Wed, 12 May 2021 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/disable-enable-device

Problem description

Use Cases

Alice is an admin user, she wants some FPGAs to be reserved for its own use and not allow them to be allocated to a VM at the time. For example, she wants to program the FPGA device and use it as the OVS agent running on the host.

Proposed change

Since the API layer is modified, a new microversion should be introduced.
It also need a new field in Device object and data model to indicate the status of a device. If one device is disabled, the status should be set to “maintaining”, and if the device is enabled, the status should be set to “enabled”. The default value should be “enabled”.
Cyborg need call Placement API to update the “reserved” field for the device.

Alternatives

None

Data model impact

A new column device_status should be added in Device’s data model.

REST API impact

A microversion need to be introduced since the Device API changed.

List Device API

Return a device list URL: /devices METHOD: GET Return: 200

{
    "devices":
    [
        {
            "uuid": "359c0990-0258-44fd-8b05-fc510ac3d022",
            "type": "FPGA",
            "vendor": "0xABCD",
            "model": "miss model info",
            "std_board_info": "{'device_id': '0xabcd', 'class': 'Fake class'}",
            "vendor_board_info": "fake_vendor_info",
            "hostname": "computenode",
            "device_status": "Enabled"
            "created_at": "2020-03-13T02:26:31+00:00",
            "updated_at": null,
            "links":
            [
                {
                    "href": "http://localhost/accelerator/v2/devices/359c0990-0258-44fd-8b05-fc510ac3d022",
                    "rel": "self"
                }
            ]
        }
    ]
}

Get Device API

Get a device by uuid and return the details URL: /devices/{uuid} METHOD: GET Return: 200

{
    "uuid": "29e23349-12ee-4978-963c-11484a4ae601",
    "parent_id": null,
    "root_id": null,
    "name": "computenode_FakeDevice",
    "num_accelerators": 16,
    "device_id": 1,
    "attributes_list": "[{'traits1': 'CUSTOM_FAKE_DEVICE'}, {'rc': 'FPGA'}]",
    "rp_uuid": "853f07a6-19de-3dd6-b9f6-6c782daa3f7b",
    "driver_name": "fake",
    "device_status": "Enabled"
    "bitstream_id": null,
    "created_at": "2020-03-13T02:27:35+00:00",
    "updated_at": "2020-03-13T02:27:36+00:00",
    "links":
    [
       {
          "href": "http://localhost/accelerator/v2/deployables/29e23349-12ee-4978-963c-11484a4ae601",
          "rel": "self"
       },
       {
          "href": "http://localhost/accelerator/deployables/29e23349-12ee-4978-963c-11484a4ae601",
          "rel": "bookmark"
       }
    ]
}

Disable Device API

Disable a device URL: /devices/disable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Enable Device API

Enable a device URL: /devices/enable/{device_uuid} METHOD: POST Return: 200 Error Code: 404(the device is not found),403(the role is not admin)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The deployer need update Cyborg to the microversion which supports disable/enable API. Otherwise the disable/enable API will be rejected.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Xinran Wang(xin-ran.wang@intel.com)

Work Items

Add new column device_status for device table.
Add disable/enable API in DeviceController.
Update the RP reserved field according to the operation. For disable oparation, the reserved field need be set by the same value as the total field, and for enable operation, the reserved field will be set to zero.
Update GET/LIST device API with device_status field added in returned value.
Add disable/enable operation in cyborgclient.
Add unit tests.

Dependencies

None

Testing

Need add unit test, and tempest test if needed.

Documentation Impact

Need add related docs.

References

None

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced

Cyborg NVIDIA GPU Driver support vGPU management

Tue, 30 Mar 2021 00:00:00

The Cyborg NVIDIA GPU Driver has implemented pGPU management in the Train release, this spec proposes the specification of supporting vGPU management in the same driver.

Problem description

In the OpenStack ecosystem, users can now use Nova to pass gpu resources to guest by two methods:

Pass the GPU hardware to the guest (PCI pass-through).
Pass the Mediated Device(vGPU) to the guest.

Use Cases

When the user is using Cyborg to manage GPU devices, he/she wants to boot up a VM with Nvidia GPU (pGPU or vGPU) attached in order to accelerate the video coding and decoding, Cyborg should be able to manage this kind of acceleration resources and to assign it to the VM(binding).

Proposed changes

1. Collect raw info of GPU devices from compute node by “lspci” and grep nvidia related keyword.(DONE)

2. Parsing details from each record including vendor_id, product_id and pci_address.(DONE)

Hardware     Driver objects       Placement data model
   |               |                      |
1 GPU         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
4 vGPUs     4 attach_handles    ---> inventories(total:4)

4. Supporting set the vGPU type for a specific GPU device in cyborg.conf. The implementation is similar to that in Nova [9].(NEW CHANGE)

Firstly, we propose [gpu]/enabled_vgpu_types to define which vgpu type Cyborg driver can use:
```
[gpu]
enabled_vgpu_types = [str_vgpu_type_1, str_vgpu_type_2, ...]
```

And also, we propose that Cyborg driver will accept configuration sections that are related to the [gpu]/enabled_vgpu_types and specifies which exact pGPUs are related to the enabled vGPU types and will have a device_addresses option defined like this:

cfg.ListOpt('device_addresses',
            default=[],
            help="""
List of physical PCI addresses to associate with a specific GPU type.

The particular physical GPU device address needs to be mapped to the vendor
vGPU type which that physical GPU is configured to accept. In order to
provide this mapping, there will be a CONF section with a name
corresponding to the following template: "vgpu_type_%(vgpu_type_name)s

The vGPU type to associate with the PCI devices has to be the section name
prefixed by ``vgpu_``. For example, for 'nvidia-11', you would declare
``[vgpu_nvidia-11]/device_addresses``.

Each vGPU type also has to be declared in ``[devices]/enabled_vgpu_types``.

Related options:

* ``[gpu]/enabled_vgpu_types``
"""),

For example, it would be set in cyborg.conf

[gpu]
enabled_vgpu_types = nvidia-223,nvidia-224
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0,0000:86:00.0
[vgpu_nvidia-224]
device_addresses = 0000:87:00.0

5. Generate resource_class and traits for device, which later will also be reported to Placement, and used by nova-scheduler to filter appropriate accelerators.(NEW CHANGE)

resource class follows standard resources classes used by OpenStack [6]. Pass-through GPU device will report ‘PGPU’ as its resource class, Virtualized GPU device will report ‘VGPU’ as its resource class.
traits follows the placement custom trait format [7]. In the Cyborg driver, it will report two traits for vGPU accelerator using the format below:

trait1: OWNER_CYBORG.

trait2: CUSTOM_<VENDOR_NAME>_<PRODUCT_ID>_<Virtual_GPU_Type>.

Meaning of each parameter is listed below.
- OWNER_CYBORG: a new namespace in os-traits to remark that a device is reported by Cyborg when the inventory is reported to placement. It is used to distinguish GPU devices reported by Nova.
- VENDOR_NAME: vendor name of the GPU device.
- PRODUCT_ID: product ID of the GPU device.
- Virtual_GPU_Type: this parameter is actually another format of the enabled_vgpu_types for a specific device set by admin in cyborg.conf. In order to generate this param, driver will first retrieve enabled_vgpu_type and then map it to Virtual_GPU_Type by the way showed below. The name is exactly the Virtual_GPU_Type that will be reported in traits. For more details about the valid Virtual GPU Types for supported GPUs, please refer to [8].
```
# find mapping relation between Virtual_GPU_Type and enabled_vgpu_type.
# The value in "name" file contains its corresponding Virtual_GPU_Type.
cat /sys/class/mdev_bus/{device_address}/mdev_supported_types/{enabled_vgpu_type}/name
```
Here is a example to show the traits of a GPU device in the real world.
- A Nvidia Tesla T4 device has been successfully installed on host, device address is 0000:af:00.0. In addition, the vendor’s vGPU driver software must be installed and configured on the host at the same time.
```
[vtu@ubuntudbs ~]# lspci -nnn -D|grep 1eb8
0000:af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
```
- Enable GPU types (Accelerator)
  
  1. Specify which specific GPU type(s) the instances would get from this specific device.
  
  Edit devices.enabled_vgpu_types and device_address in cyborg.conf:
```
[gpu]
enabled_vgpu_types=nvidia-223
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0
```
  1. Restart the cyborg-agent service.
- Finally, traits reported for this device(RP) will be:
  
  OWNER_CYBORG and CUSTOM_NVIDIA_1EB8_T4_2B

Note

For the last parameter “T4_2B” (<Virtual_GPU_type>), we can validate the mapping relation between “nvidia-223” and “T4_2B” by check from the mdev sys path:

[vtu@ubuntudbs mdev_supported_types]$ pwd
/sys/class/mdev_bus/0000:af:00.0/mdev_supported_types
[vtu@ubuntudbs mdev_supported_types]$ ls
nvidia-222  nvidia-225  nvidia-228  nvidia-231  nvidia-234  nvidia-320
nvidia-223  nvidia-226  nvidia-229  nvidia-232  nvidia-252  nvidia-321
nvidia-224  nvidia-227  nvidia-230  nvidia-233  nvidia-319
[vtu@ubuntudbs mdev_supported_types]$ cat nvidia-223/name
GRID T4-2B

6. Generate controlpath_id, deployable, attach_handle, attribute for vGPU.(NEW CHANGE)

7. Create a mdev device in the sys by echo its UUID (actually is the attach_handle UUID) to the create file when vgpu is bind to a VM.(NEW CHANGE)

create_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/create

8. Delete a mdev device from sys by echo “1” to the remove file when vgpu is unbind from a VM.(NEW CHANGE)

remove_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/UUID/remove

Alternatives

Using Nova to manage vGPU device [10].

Data model impact

One pGPU can be virtualized 4 vGPUs, 8 vGPUs, 16 vGPUs base on the pGPU type.

Here we take 2 pGPUs and each pGPU is virtualized 4 vGPUs as example.

The data model in Cyborg should like this:

+--------------------------------------------------------+
|                                                        |
|    device A                    device B                |
|                                                        |
|  +-----------------------+  +-----------------------+  |
|  |  deployable A         |  |  deployable B         |  |
|  |                       |  |                       |  |
|  | +-------------------+ |  | +-------------------+ |  |
|  | | attach handler 1-4| |  | | attach handler 5-8| |  |
|  | |                   | |  | |                   | |  |
|  | | +------+ +------+ | |  | | +------+ +------+ | |  |
|  | | | vgpu1| | vgpu2| | |  | | | vgpu5| | vgpu6| | |  |
|  | | +------+ +------+ | |  | | +------+ +------+ | |  |
|  | | +------+ +------+ | |  | | +------+ +------+ | |  |
|  | | | vgpu3| | vgpu4| | |  | | | vgpu7| | vgpu8| | |  |
|  | | +------+ +------+ | |  | | +------+ +------+ | |  |
|  | +-------------------+ |  | +-------------------+ |  |
|  |                       |  |                       |  |
|  +-----------------------+  +-----------------------+  |
|                                                        |
|  VGPU                                                  |
+--------------------------------------------------------+

The resource provider tree’s structure should be reported as

                     +--------------------+
                     |                    |
                     |  compute node      |
                     |                    |
                     +---------+----------+
                               |
                  +------------+----------------+
                  |                             |
           +------+------+               +------+------+
           |             |               |             |
           |  deployable |               | deployable  |
           +-----+-------+               +------+------+
                 |                              |
     +-----------+--------+                +----+---------------+
     |                    |                |                    |
     |                    |                |                    |
+-----+------+       +-----+-----+     +----+------+      +------+-----+
| inventory1 | ...   | inventory4|     | inventory5| ...  | inventory8 |
+------------+       +-----------+     +-----------+      +------------+

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This feature is highly dependent on the version of libvirt and the physical devices present on the host.

For vGPU management, deployers need to make sure that the GPU device has been successfully virtualized. Otherwise, Cyborg will report it as a pGPU device.

Please see ref [2] and [3] for how to install the Virtual GPU Manager package to virtualize your GPU devices.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>
Other contributors:: songwenping

Work Items

Implement NVIDIA GPU Driver enhancement in Cyborg
Add related test cases.
Add test report to wiki and update the supported driver doc page

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Nvidia GPU driver in Cyborg project.

References

History

Revisions
Release	Description
Wallaby	Introduced
Xena	Reproposed

Cyborg NVIDIA GPU Driver support vGPU management

Fri, 12 Mar 2021 00:00:00

The Cyborg NVIDIA GPU Driver has implemented pGPU management in the Train release, this spec proposes the specification of supporting vGPU management in the same driver.

Problem description

In the OpenStack ecosystem, users can now use Nova to pass gpu resources to guest by two methods:

Pass the GPU hardware to the guest (PCI pass-through).
Pass the Mediated Device(vGPU) to the guest.

Use Cases

When the user is using Cyborg to manage GPU devices, he/she wants to boot up a VM with Nvidia GPU (pGPU or vGPU) attached in order to accelerate the video coding and decoding, Cyborg should be able to manage this kind of acceleration resources and to assign it to the VM(binding).

Proposed changes

1. Collect raw info of GPU devices from compute node by “lspci” and grep nvidia related keyword.(DONE)

2. Parsing details from each record including vendor_id, product_id and pci_address.(DONE)

Hardware     Driver objects       Placement data model
   |               |                      |
1 GPU         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
4 vGPUs     4 attach_handles    ---> inventories(total:4)

4. Supporting set the vGPU type for a specific GPU device in cyborg.conf. The implementation is similar to that in Nova [9].(NEW CHANGE)

Firstly, we propose [gpu_devices]/enabled_vgpu_types to define which vgpu type Cyborg driver can use:
```
[gpu_devices]
enabled_vgpu_types = [str_vgpu_type_1, str_vgpu_type_2, ...]
```

cfg.ListOpt('device_addresses',
            default=[],
            help="""
List of physical PCI addresses to associate with a specific vGPU type.

The particular physical GPU device address needs to be mapped to the vendor
vGPU type which that physical GPU is configured to accept. In order to
provide this mapping, there will be a CONF section with a name
corresponding to the following template: "vgpu_%(vgpu_type_name)s

The vGPU type to associate with the PCI devices has to be the section name
prefixed by ``vgpu_``. For example, for 'nvidia-11', you would declare
``[vgpu_nvidia-11]/device_addresses``.

Each vGPU type also has to be declared in ``[gpu_devices]/enabled_vgpu_types``.

Related options:

* ``[gpu_devices]/enabled_vgpu_types``
"""),

For example, it would be set in cyborg.conf

[gpu_devices]
enabled_vgpu_types = nvidia-223,nvidia-224
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0,0000:86:00.0
[vgpu_nvidia-224]
device_addresses = 0000:87:00.0

5. Generate resource_class and traits for device, which later will also be reported to Placement, and used by nova-scheduler to filter appropriate accelerators.(NEW CHANGE)

resource class follows standard resources classes used by OpenStack [6]. Pass-through GPU device will report ‘PGPU’ as its resource class, Virtualized GPU device will report ‘VGPU’ as its resource class.
traits follows the placement custom trait format [7]. In the Cyborg driver, it will report two traits for vGPU accelerator using the format below:

trait1: OWNER_CYBORG.

trait2: CUSTOM_<VENDOR_NAME>_<PRODUCT_ID>_<Virtual_GPU_Type>.

Meaning of each parameter is listed below.
- OWNER_CYBORG: a new namespace in os-traits to remark that a device is reported by Cyborg when the inventory is reported to placement. It is used to distinguish GPU devices reported by Nova.
- VENDOR_NAME: vendor name of the GPU device.
- PRODUCT_ID: product ID of the GPU device.
- Virtual_GPU_Type: this parameter is actually another format of the enabled_vgpu_types for a specific device set by admin in cyborg.conf. In order to generate this param, driver will first retrieve enabled_vgpu_type and then map it to Virtual_GPU_Type by the way showed below. The name is exactly the Virtual_GPU_Type that will be reported in traits. For more details about the valid Virtual GPU Types for supported GPUs, please refer to [8].
```
# find mapping relation between Virtual_GPU_Type and enabled_vgpu_type.
# The value in "name" file contains its corresponding Virtual_GPU_Type.
cat /sys/class/mdev_bus/{device_address}/mdev_supported_types/{enabled_vgpu_type}/name
```
Here is a example to show the traits of a GPU device in the real world.
- A Nvidia Tesla T4 device has been successfully installed on host, device address is 0000:af:00.0. In addition, the vendor’s vGPU driver software must be installed and configured on the host at the same time.
```
[vtu@ubuntudbs ~]# lspci -nnn -D|grep 1eb8
0000:af:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
```
- Enable GPU types (Accelerator)
  
  1. Specify which specific GPU type(s) the instances would get from this specific device.
  
  Edit devices.enabled_vgpu_types and device_address in cyborg.conf:
```
[gpu]
enabled_vgpu_types=nvidia-223
[vgpu_nvidia-223]
device_addresses = 0000:af:00.0
```
  1. Restart the cyborg-agent service.
- Finally, traits reported for this device(RP) will be:
  
  OWNER_CYBORG and CUSTOM_NVIDIA_1EB8_T4_2B

Note

For the last parameter “T4_2B” (<Virtual_GPU_type>), we can validate the mapping relation between “nvidia-223” and “T4_2B” by check from the mdev sys path:

[vtu@ubuntudbs mdev_supported_types]$ pwd
/sys/class/mdev_bus/0000:af:00.0/mdev_supported_types
[vtu@ubuntudbs mdev_supported_types]$ ls
nvidia-222  nvidia-225  nvidia-228  nvidia-231  nvidia-234  nvidia-320
nvidia-223  nvidia-226  nvidia-229  nvidia-232  nvidia-252  nvidia-321
nvidia-224  nvidia-227  nvidia-230  nvidia-233  nvidia-319
[vtu@ubuntudbs mdev_supported_types]$ cat nvidia-223/name
GRID T4-2B

6. Generate controlpath_id, deployable, attach_handle, attribute for vGPU.(NEW CHANGE)

7. Create a mdev device in the sys by echo its UUID (actually is the attach_handle UUID) to the create file when vgpu is bind to a VM.(NEW CHANGE)

create_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/create

8. Delete a mdev device from sys by echo “1” to the remove file when vgpu is unbind from a VM.(NEW CHANGE)

remove_file_path= /sys/class/mdev_bus/{pci_address}/mdev_supported_types/{type-id}/UUID/remove

Alternatives

Using Nova to manage vGPU device [10].

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

This feature is highly dependent on the version of libvirt and the physical devices present on the host.

For vGPU management, deployers need to make sure that the GPU device has been successfully virtualized. Otherwise, Cyborg will report it as a pGPU device.

Please see ref [2] and [3] for how to install the Virtual GPU Manager package to virtualize your GPU devices.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>

Work Items

Implement NVIDIA GPU Driver enhancement in Cyborg
Add related test cases.
Add test report to wiki and update the supported driver doc page

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Nvidia GPU driver in Cyborg project.

References

History

Revisions
Release	Description
Wallaby	Introduced

Cyborg Inspur NVMe SSD Driver Proposal

Fri, 12 Mar 2021 00:00:00

This spec proposes to provide the initial design for Cyborg’s Inspur NVMe SSD driver.

Problem description

Inspur NVMe SSD devices provide the ability to accelerate VM IO rate. In the OpenStack ecosystem, we don’t have any tool to manage this kind of accelerators. This spec will add an Inspur NVMe SSD driver in Cyborg to automatically manage Inspur NVMe SSD devices.

The management in the driver scope includes:

Cyborg Inspur NVMe SSD driver can automatically discover the NVMe SSD and report to database by calling cyborg-conductor. Cyborg Inspur NVMe SSD driver needs to do the device binding/unbinding to VM.

Use Cases

As an operator, I would like to use Cyborg to manage Inspur NVMe SSD devices.
As a developer, I would like to use Cyborg agent to start or to do resource checking periodically, the Cyborg Inspur NVMe SSD driver should provide discover() function to enumerate the list of the Inspur NVMe SSD devices, and report the details of all available NVMe SSD accelerators on the host, such as PID(Product ID), VID(Vendor ID), Device ID.
As a user, I would like to boot up a VM with Inspur NVMe SSD card attached in order to accelerate IO rate. Cyborg should be able to manage this kind of acceleration resources and assign it to the VM(binding).

Proposed changes

1. Collect raw info of Inspur NVMe SSD devices from compute node by “lspci” command and grep Non-Volatile memory controller related keyword and then grep Inspur vendor ID.

2. Parsing details from each record including vendor_id, product_id and pci_address.

3. Generate Cyborg specific driver objects and resource provider modeling for the Inspur NVMe SSD devices. Below is the objects to describe a Inspur NVMe SSD device which complies with the Cyborg database mode and Placement data model.

Hardware     Driver objects       Placement data model
   |               |                      |
1 SSD         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
1 SSD        1 attach_handle    ---> inventories(total:1)

4. We report CUSTOM_SSD as resource_class, CUSTOM_SSD_INSPUR and CUSTOM_SSD_PRODUCT_ID_1003 as traits to Placement.

Alternatives

None

Data model impact

Add SSD type to the column device_type in table devices.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

User can manage Inspur NVMe SSD cards by Cyborg Inspur NVMe SSD driver. Such as list the Inspur NVMe SSD devices, report the details of all available Inspur NVMe SSD accelerators on the host, binding an Inspur NVMe SSD to VM.

Performance Impact

None

Other deployer impact

None.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Wenping Song

Work Items

Implement Inspur NVMe SSD driver in Cyborg
Add related test cases.
Add test report to wiki page.
Update doc page and release note.

Dependencies

None

Testing

Unit tests will be added to test this driver.
Add test result in Cyborg Wiki which is required by the Cyborg community.

Documentation Impact

Document Inspur NVMe SSD driver in Cyborg project.

References

None

History

Revisions
Release	Description
Wallaby	Introduced

Example Spec - The title of your blueprint

Fri, 05 Mar 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced

Example Spec - The title of your blueprint

Fri, 05 Mar 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced

Example Spec - The title of your blueprint

Fri, 05 Mar 2021 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Xena	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Pike	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Pike	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Pike	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Queens	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Stein	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Stein	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Stein	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Train	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Train	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Train	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Ussuri	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Ussuri	Introduced

Cyborg API Version 2

Fri, 25 Dec 2020 00:00:00

Problem description

This spec will specify what v1 APIs will be dropped in v2, and what new APIs need to be introduced.

Use Cases

As a user, I want to create VMs with attached accelerators, delete them and perform other instance operations. (Live migration is excluded.)
- Cyborg should provide APIs for device profiles and ARQs, so as to support end-to-end workflows with Nova for VM creation, deletion and other operations.
As an operator, I want to query and manage the inventory of accelerators and devices.
- Cyborg should provide an API to query the list of devices based on various attributes.
As a user with appropriate authorization, or as an operator, I want to initiate programming of specific devices, either with FPGA bitstreams, or with FPGA shell logic or device firmware.

Proposed changes

Microversions

The changes should be compatible with Microversion Specification [1]. This means that the output of the API:

GET /accelerator

must change as specified in Section Changed APIs. Also, a new API:

GET /accelerator/v2

is introduced conformant to that specification, as documented in Section New APIs.

The API clients can use the following header to indicate which microversion to use:

OpenStack-API-Version: accelerator <microversion>

If they do not, the default microversion of 2.0 will be assumed in Train.

API Changes

The following v1 URLs will not be supported in v2, because the accelerator object does not exist anymore and the deployable object is exposed as part of the device object:

/v1/accelerators
/v1/deployables

A new set of APIs based on the URL:

/v2/device_profiles

is introduced in the Specification for device profiles ([2]).

This specification introduces another set of APIs based on the URLs:

/v2/accelerator_requests
/v2/devices
/v2/deployables/{uuid}

See Section New APIs for details.

Alternatives

We could add another API to enable or disable specific devices, or specific deployables (resource providers). That is left for a future version.

Data model impact

Besides, we need to add a status field to device table in the database, which can indicate whether the device is enabled or disabled.

REST API impact

Deleted APIs

Cyborg should drop the APIs for accelerators and deployables in v2. The deployables, as per the new definition, will be included in the new devices API.

Changed APIs

URL: /accelerator

METHOD: GET

ROLE: Admin or any tenant

Current JSON response:

{
   "description": "Cyborg (previously known as Nomad) is an OpenStack
      project that aims to provide a general purpose management framework for
      acceleration resources (i.e. various types of accelerators such as
      Crypto cards, GPU, FPGA, NVMe/NOF SSDs, ODP, DPDK/SPDK and so on).",
   "name": "OpenStack Cyborg API"
}

Proposed JSON response:

{
   "versions": [
       {
           "id": "v1",
           "links": [
               {
                   "href": "http://<ip>/accelerator/v1",
                   "rel": "self"
               }
           ],
           "version": "1.0",
           "status": "DEPRECATED"
       },
       {
           "id": "v2.0",
           "links": [
               {
                   "href": "http://<ip>/accelerator/v2",
                   "rel": "self"
               }
           ],
           "min_version": "2.0",
           "max_version": "2.0",
           "version": "2.0",
           "status": "CURRENT"
       },
   ]
}

New APIs

URL: /accelerator/v2

METHOD: GET

ROLE: Admin or any tenant

Proposed JSON response:

{
   "id": "v2.0",
   "links": [
       {
           "href": "http://<ip>/accelerator/v2",
           "rel": "self"
       }
   ],
   "min_version": "2.0",
   "max_version": "2.0",
   "version": "2.0",
   "status": "CURRENT"
}

URL: /accelerator/v2/accelerator_requests

METHOD: GET

ROLE: Admin or the tenant who owns the specified instance.

Query Parameters:

instance: UUID of the instance whose ARQs are requested.
bind_state: Bind state of ARQs. Only supported value is
‘resolved’, which means the ARQ is either bound successfully or failed to bind. Other states like ‘bound’ may be supported in the future.

Proposed JSON response:

{
   "arqs": [
      <arq_obj>,
      ...
   ]
}

URL: /accelerator/v2/accelerator_requests

METHOD: POST

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request:

{
  "device_profile_name": <string>
}

Action: Create ARQs for the specified device profile. If the device profile specifies N accelerator resources across all its request groups, N ARQs will get created in unbound state.

Proposed JSON response:

{
   "arqs": [
      <arq_obj>,
      ...
   ]
}

URL: /accelerator/v2/accelerator_requests

METHOD: PATCH

ROLE: Admin or the tenant who owns the specified instance.

Proposed JSON request (in RFC 6902 format):

For binding:
{
  "$arq_uuid": [
     { "path": "/hostname", "op": "add", "value": <string> },
     { "path": "/instance_uuid",  "op": "add", "value": <uuid> },
     { "path": "/device_rp_uuid", "op": "add", "value": <uuid> }
  ],
  "$arq_uuid": [...],
  ...
}

For unbinding:
{
  "$arq_uuid": [
     { "path": "/hostname", "op": "remove" },
     { "path": "/instance_uuid",  "op": "remove" },
     { "path": "/device_rp_uuid", "op": "remove" }
  ],
  "$arq_uuid": [...],
  ...
}

Action: Bind or unbind

Proposed JSON response: None

URL: /accelerator/v2/accelerator_requests

METHOD: DELETE

ROLE: Admin or the tenant who owns the specified ARQs.

Query Parameters (required):

arqs: List of one or more comma-separated ARQ UUIDs.

Proposed JSON response: None

URL: /accelerator/v2/devices

METHOD: GET

Query Parameters: * hostname: hostname of the compute node where devices are located.

type: type of devices. For example, we call get all FPGA devices by add “FPGA” as the query parameter.
vendor: vendor ID of devices. For example, we call get all Intel devices by add “0x8086” as the query parameter.

Proposed JSON response:

{
  "devices": [
     <device_obj>,
     ...
  ]
}

URL: /accelerator/v2/devices/{uuid}

METHOD: GET

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON response:

{
    "vendor": "0x8086",
    "uuid": "1c6c9033-560d-4a7a-bb8e-94455d1e7825",
    "links":
    [
        {"href": "http://10.238.145.73/accelerator/v2/devices/1c6c9033-560d-4a7a-bb8e-94455d1e7825",
         "rel": "self"
         }
    ],
    "created_at": "2019-11-12T07:38:55+00:00",
    "hostname": "host1",
    "updated_at": null,
    "vendor_board_info": "fake_vendor_info",
    "model": "fake_model_info",
    "type": "FPGA",
    "id": 2,
    "std_board_info": "{"class": "Fake class", "device_id": "0x09c4"}
}

URL: /accelerator/v2/devices/{uuid}

METHOD: PATCH

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request (in RFC 6902 format):

{
  [
     { "path": "/status", "op": "replace", "value": "enabled", "reason": "reason(optional)" },
  ]
}

Action: Update the device status as enabled. This request will invoke placement API to set all resource provider under this device to available. Optionally,operators can append a reason of the operation.

Proposed JSON response: None.

URL: /accelerator/v2/devices/{uuid}

METHOD: PATCH

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request (in RFC 6902 format):

{
  [
     { "path": "/status", "op": "replace", "value": "disabled", "reason": "reason(optional)" },
  ]
}

Action: Update the device status as disabled. This request will invoke placement API to update the reserved field of inventories of all resource providers under this device to disable this devices from the point of view of end-users. Optionally, operators can append a reason of the operation.

Proposed JSON response: None.

URL: /accelerator/v2/deployables/{uuid}

METHOD: PATCH

ROLE: Admin or any tenant with RBAC authorization.

Proposed JSON request (in RFC 6902 format):

{
  [
     { "path": "/bitstream", "op": "add", "value": <img_uuid> },
  ]
}

Action: Update the FPGA bitstream for the specified deployable. The: request allows a lst of path specifiers for future extensibility.

Proposed JSON response: None.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

TBD

Work Items

Dependencies

None

Testing

Unit tests and functional tests need to be written.

Documentation Impact

Cyborg API documentation needs to be updated.

References

History

Revisions
Release Name	Description
Train	Introduced
Ussuri	Re-proposed

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Wallaby	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Wallaby	Introduced

Example Spec - The title of your blueprint

Fri, 25 Dec 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/openstack-cyborg/+spec/example

Some notes about the cyborg-spec and blueprint process:

Not all blueprints need a spec. For more information see https://docs.openstack.org/developer/cyborg/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes, nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/openstack-cyborg/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Cyborg REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:merged+project:openstack/cyborg+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
  - URL should not include underscores, and use hyphens instead.
- Parameters which can be passed via the url
- JSON schema definition for the request body data if allowed
  - Field names should use snake_case style, not CamelCase or MixedCase style.
- JSON schema definition for the response body data if any
  - Field names should use snake_case style, not CamelCase or MixedCase style.
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-cyborgclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Should they be more generic than proposed (for example a flag that other hypervisor drivers might want to implement as well)? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack, such as:

If the blueprint proposes a change to the driver API, discussion of how other hypervisors would implement the feature is required.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in cyborg, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Cyborg, document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Wallaby	Introduced

Cyborg Inspur FPGA Driver Proposal

Tue, 10 Nov 2020 00:00:00

This spec proposes to provide the initial design for Cyborg’s Inspur FPGA driver.

Problem description

This spec will add an Inspur FPGA driver for Cyborg to manage specific Inspur FPGA devices.

Use Cases

When Cyborg agent starts or does resource checking periodically, the Cyborg Inspur FPGA driver should provider discover() function to enumerate the list of the Inspur FPGA devices, and report the details of all available Inspur FPGA accelerators on the host, such as PID(Product id), VID(Vendor id), Device and BSP(Inspur FPGA program environment).
When user uses empty Inspur FPGA as their accelerators, Cyborg agent will call driver’s program() interface by provide BSP to the driver.

Proposed changes

In general, the goal is to develop a Cyborg Inspur FPGA driver that supports discover/program interfaces for Inspur FPGA accelerator framework.

The driver should include the follow functions: 1. discover() driver reports devices’ raw info sample as following:

[{
  "vendor": "0x1db4",
  "product": "a115",
  "device": "0000:af:00:0",
  "bsp": {"env": "/var/lib/inspur-fpga",
          "cmd": "aocl"}
}]

program(bsp) program by bsp. bsp: the environment of accelerator device.

Image Format

Alternatives

None

Data model impact

Inspur FPGA driver will not touch Data model. The Cyborg Agent can call Inspur FPGA driver to update the database during the discover/program operations.

REST API impact

The related Inspur FPGA accelerator APIs is out of scope for this spec.

Security impact

None

Notifications impact

None

Other end user impact

User can manage Inspur FPGA card by Cyborg Inspur FPGA driver. Such as list of the Inspur FPGA devices, report the details of all available Inspur FPGA accelerators on the host, program with Inspur FPGA and so on.

Performance Impact

None

Other deployer impact

Deployers should install the specific Inspur FPGA management stack that the driver depends on.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Wenping Song

Work Items

Implement Inspur FPGA driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Inspur FPGA driver in Cyborg project.

References

None

History

Revisions
Release	Description
Victoria	Introduced

Cyborg Intel® QAT Driver Proposal

Tue, 10 Nov 2020 00:00:00

This spec proposes to provide the initial design for Cyborg’s Intel® QAT driver.

Problem description

Intel® QuickAssist Technology(Intel® QAT) provides security(encryption) HW acceleration and compression HW acceleration. It improves performance across applications and platforms. That includes symmetric encryption and authentication, asymmetric encryption, digital signatures, RSA, DH, and ECC, and lossless data compression. Hence, using Intel® QAT for application acceleration in cloud has been becoming desirable.

Intel® QAT card also support SRIOV [1], which means admin can virtualize it into serveral VFs and assign VFs to the VM. One Intel® QAT Card usually has 3 or 6 PFs which correspond to “Device” notion in Cyborg, and each of them can be virtualize into 8 or 16 VFs. It depends on diffirent devices.

This spec will add a Intel® QAT driver for Cyborg to manage specific Intel® QAT devices.

Use Cases

When user want to boot up a VM with Intel® QAT card (PF or VF) attached in order to accelerate TLS workload. Cyborg should be able to manage this kind of acceleration resources and to assign it to the VM(binding).

Proposed changes

In general, the goal is to develop a Cyborg Intel® QAT driver that supports discover() interfaces, this driver is reponsible to encapsule the device information to Cyborg’s unified data model, such as Device, Deployale, AttachHandle and so on.

The driver should include discover functions which will be called by agent periodically.

Image Format

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Deployers should install the specific Intel® QAT software package that the driver depends on.

Please see ref [2] for details.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Xinran Wang

Work Items

Implement Intel® QAT driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document Intel® QAT driver in Cyborg project.

References

History

Revisions
Release	Description
Victoria	Introduced

Cyborg FPGA Driver Proposal

Thu, 05 Nov 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-fpga-driver

This spec proposes to provide the initial design for Cyborg’s FPGA driver.

Problem description

A Field Programmable Gate Array(FPGA) is an integrated circuit designed to be configured by a customer or a designer after manufacturing. The advantage lies in that they are sometimes significantly faster for some applications because of their parallel nature and optimality in terms of the number of gates used for a certain process. Hence, using FPGA for application acceleration in cloud has been becoming desirable.

There is a management framwork in Cyborg [1] for heterogeneous accelerators, tracking and deploying FPGAs. This spec will add a FPGA driver for Cyborg to manage specific FPGA devices.

Use Cases

When Cyborg agent starts or does resource checking periodically, the Cyborg FPGA driver should enumerate the list of the FPGA devices, and report the details of all available FPGA accelerators on the host, such as BDF(Bus, Device, Function), PID(Product id) VID(Vendor id), IMAGE_ID and PF(Physical Function)/VF(Virtual Function) type.
When user uses empty FPGA regions as their accelerators, Cyborg agent will call driver’s program() interface. Cyborg agent should provide BDF of PF/VF, and local image path to the driver. More details can be found in ref [2].
When there maybe more thant one vendor fpga card on a host, or on different hosts in the cluster, Cyborg agent can discover the wendors easiy and intelligently by Cyborg FPGA driver, and call the correct driver to execute it’s operations, such as discover() and program().

Proposed changes

In general, the goal is to develop a Cyborg FPGA driver that supports discover/program interfaces for FPGA accelerator framework.

The driver should include the follow functions: 1. discover() driver reports devices as following:

[{
  "vendor": "0x8086",
  "product": "bcc0",
  "pr_num": 1,
  "devices": "0000:be:00:0",
  "path": "/sys/class/fpga/intel-fpga-dev.0",
  "regions": [
    {"vendor": "0x8086",
      "product": "bcc1",
      "regions": 1,
      "devices": "0000:be:00:1",
      "path": "/sys/class/fpga/intel-fpga-dev.1"
    }]
}]

pr_num: partial reconfiguration region numbers.

program(device_path, image) program the image to a PR region specified by device_path. device_path: the sys path of accelerator device. image: The local path of programming image.

Image Format

Alternatives

None

Data model impact

FPGA driver will not touch Data model. The Cyborg Agent can call FPGA driver to update the database during the discover/program operations.

REST API impact

The related FPGA accelerator APIs is out of scope for this spec. The FPGA management framework for Cyborg [1] will alter the proposal.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Deployers should install the specific FPGA management stack that the driver depends on.

Please see ref [2] for details.

Developer impact

There will be some developer impact vis-à-vis new functionality that will be available to devs.

Implementation

Assignee(s)

Primary assignee:: Shaohe Feng <shaohe.feng@intel.com> Dolpher Du <dolpher.du@intel.com>

Work Items

Implement the cyborg-fpga-driver in this spec.

Dependencies

Cyborg API Spec
Cyborg Agent Spec
Cyborg Driver Spec
Cyborg Conductor Spec

Testing

Unit tests will be added to test Cyborg FPGA driver.
Functional tests will be added to test Cyborg FPGA driver.

Documentation Impact

Document FPGA driver in the Cyborg project

References

Cyborg API Spec
Cyborg Agent Spec
Cyborg Driver Spec
Cyborg Conductor Spec

History

Revisions
Release	Description
Queens	Introduced

Cyborg NVIDIA GPU Driver Proposal

Sat, 24 Oct 2020 00:00:00

This spec proposes to provide the initial design for Cyborg’s NVIDIA physical GPU management driver. Please note that the virtualized GPU is out of scope. we only support passthrough one pGPU card to one VM directly.

Problem description

This spec will add a NVIDIA GPU driver for Cyborg to manage specific NVIDIA physical GPU devices.

Use Cases

As an operator, I would like to use Cyborg agent starts or does resource checking periodically, the Cyborg NVIDIA GPU driver should provider discover() function to enumerate the list of the NVIDIA GPU devices, and report the details of all available NVIDIA GPU accelerators on the host, such as PID(Product id), VID(Vendor id), Device.
As a user, I would like to boot up a VM with NVIDIA GPU card attached in order to accelerate compute ability. Cyborg should be able to manage this kind of acceleration resources and assign it to the VM(binding).

Proposed changes

In general, the goal is to develop a Cyborg NVIDIA GPU driver that supports discover interfaces for NVIDIA GPU accelerator framework. The driver should include the discover() function. For physical GPU, this function works by executing lspci command and reports devices’ raw info sample as following:

[
  {
  "vendor": "10de",
  "product": "1db6",
  "device": "0000:af:00:0"
  }
]

Generate Cyborg specific driver objects and resource provider modeling for the GPU device. Below is the objects to describe a pGPU devices which complies with the Cyborg database mode and Placement data model.

Hardware     Driver objects       Placement data model
   |               |                      |
1 GPU         1 device                    |
   |               |                      |
   |         1 deployable       ---> resource_provider
   |               |            ---> parent resource_provider: compute node
   |               |                      |
1 pGPU       1 attach_handle    ---> inventories(total:1)

Alternatives

None

Data model impact

NVIDIA GPU driver will not touch Data model. The Cyborg Agent can call NVIDIA GPU driver to update the database during the discover operations.

REST API impact

None.

Security impact

None

Notifications impact

None

Other end user impact

User can manage NVIDIA GPU cards by Cyborg NVIDIA GPU driver. Such as list of the NVIDIA GPU devices, report the details of all available NVIDIA GPU accelerators on the host, binding with NVIDIA GPU and so on.

Performance Impact

None

Other deployer impact

Deployers need to make sure the GPU device hasn’t been virtualized. Otherwise, we can’t use it as a pGPU.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Wenping Song

Work Items

Implement NVIDIA GPU driver in Cyborg
Add related test cases.

Dependencies

None

Testing

Unit tests will be added to test this driver.

Documentation Impact

Document NVIDIA pGPU driver in Cyborg project. Test report in Cyborg wiki

References

None

History

Revisions
Release	Description
Train	Introduced

Policy Default Refresh

Sat, 29 Aug 2020 00:00:00

Role Based Access Control (RBAC) policies in OpenStack has long been the forefront of operator concerns and pain. The implementation is complicated to understand, inconsistent across projects, and lacks secure defaults. To improve the existing default policy of OpenStack, the Consistent_and_Secure_Default_Policies_Popup_Team [1] was built up to track policy refresh for all projects, where Keystone was the lead. Keystone defined ongoing policy goals and roadmaps, default roles, as well as a reclarification of system-scoped and project-scoped RBAC. As a member of this popup_team, Cyborg is also going to follow up policy default refresh.

Problem description

The current default policy in Cyborg is incomplete and not good enough. Since Cyborg V2 API is newly implemented in Train, RBAC check for V2 API still remains incomplete. And now Cyborg mainly has three policy rules:

allow
admin_only
admin_or_owner

Firstly “allow” means any access will be passed. Now “allow” rule is used by cyborg:arq:create, which is too slack. We’ve reached an agreement that this should not be open for all users [2]. As for the new rule, please see that in the cyborg policy table introduced in Use Cases part.

Secondly “admin_only” is used for the global admin that is able to make almost any change to cyborg, and see all details of the cyborg system. The rule actually passes for any user with an admin role, it doesn’t matter which project is used, any user with the admin role gets this global access.

Thirdly “admin_or_owner” sounds like it checks if the user is a member of a project. However, for most APIs we use the default target which means this rule will pass for any authenticated user.

With all the above policy rules, there still some cases which are not well covered. For example, it is impossible to allow a user to retrieve/update devices which are shared by multiple projects from a system level without being given the global admin role. In addition, cyborg now doesn’t have a “reader” role.

Keystone comes with member, admin and reader roles by default. We can use these default roles: https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/define-default-roles.html

In addition, we can use the new “system scope” concept to define which users are system administrators: https://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/system-scope.html

Use Cases

The following user roles should be supported by cyborg default configuration:

Add System Scoped Admin: to operate system-level resources
Add System Scoped Reader: read-only role for system-level resources
Add Project Scoped Reader: read-only role for project-level resources
Refresh existed admin to Project Scoped Admin by default
Add Project Scoped Member: role for project-level non-admin APIs

Cyborg V2 APIs need RBAC check. Objects of V2 APIs are listed in the following:

device_profile [3], as the “flavor” in accelerator request, is generally supposed to be a system-level resource especially for public cloud, where billing is based on the device_profile. So we reached an agreement [4] that sys_admin is required for device_profile:create and delete. As for other specific cloud, such as private cloud providers, they can change the policy by themselves if they want other users to create device_profiles.
devices and deployables [5] objects are used to describe hardware accelerators, where a device refers to the hardware and deployables are derived from a device. device:update API is introduced for sys_admin to enable or disable a specific device, while deployable:update API offers project_admin a way to update the shell image/FPGA bitstream to custom user logic for the specified deployable. NOTE that sys_admin is required to do the pre_configuration for devices.
accelerator_requests are sent by nova to bind/unbind accelerators to VMs during instance boot. Any project user who can create VM should be able to create an arq, so “member” role is required for arq:create. And for the arq:patch and arq:delete admin_or_owner should make sense.

To be clear, the roles needed for all Cyborg V2 APIs’ operations are defined in the table cyborg-policy-table.

In introducing the above new default permissions, we must ensure:

Operators using default policy are given at least one cycle to add additional roles to users (likely via implied roles)
Operators with over-ridden policy are given at least one cycle to understand how the new defaults may or may not help them

Proposed change

According to the discussions above, we will try to make the changes as less as possible to meet the requirements. For the current stage, there should be at least the following changes. Each policy rules will be covered with appropriate oslo.policy’s “scope_types”, ‘system’ and ‘project’ in cyborg case. And we will use the DocumentedRuleDefault to update policy and follow the oslo.policy deprecation workflow, where both old and new policy check strings are active during the deprecation period.

Add system scoped admin policy

This policy will be useful for situations where devices are shared by multiple projects, and we want a system-level admin to operator the devices like programming or firmware upgrade. In addition, a system admin is required to do the service disable/enable things.
Add system scoped reader policy

This policy will be useful for situations where a read-only role is required for a more secure access to devices that are shared by multiple projects in a system.
Add project scoped reader policy

Similarly, this policy will be useful for situations where a read-only role is required for a more secure access at a project level. For example, some users should only have the permission to check and use the resources but shouldn’t update the resources.
Add project scoped member policy

This policy will be used to check if a user is eligible to post a non-admin API such as arq creation.

POC: https://review.opendev.org/#/c/699102/, https://review.opendev.org/#/c/700765/

Alternatives

Keep the old policy mechanism, but on one hand, as more apis features added, the policy check will become more complicated, on the other hand, there are some important situations which are not covered or well-covered by old policy mechanism. This refresh will make cyborg policy basically comprehensive and clean.

Data model impact

None

REST API impact

Operations for each API should be reassessed and associated which scope, or scopes are appropriate.

The following new roles will be added to replace legacy policies, more details can be found in the aforementioned cyborg-policy-table:

Project Reader check
- GET /v2/device_profiles
- GET /v2/device_profiles/{device_profiles_uuid}
- GET /v2/accelerator_requests
- GET /v2/accelerator_requests/{accelerator_request_uuid}
System Reader check
- GET /v2/devices
- GET /v2/devices/{device_uuid}
System Admin check
- PATCH /v2/devices
- POST /v2/device_profil
- DELETE /v2/device_profiles/{device_profiles_uuid}
- DELETE /v2/device_profiles?value={dev_profile_name1},{dev_profile_name2}
Project Admin check
- PATCH /v2/deployables/{uuid}
Project Member check
- POST /v2/accelerator_requests
- PATCH /v2/accelerator_requests/{arq_uuid}
- DELETE /v2/accelerator_requests?arqs={arq_uuid}
- DELETE /v2/accelerator_requests?instance={instance_uuid}

Note

The default roles discussed will be created by Keystone, during the bootstrap process, using implied roles. As indicated in the above list, having admin role implies a user also has the same rights as the member role. Therefore this user will also has the same rights as the reader role as member implies reader.

This keeps policy files clean. For example, the following are equivalent as a result of implied roles:

“cyborg:device:get_all”: “role:reader OR role:member OR role:admin” “cyborg:device:get_all”: “role:reader”

The chain of implied roles will be documented alongside of the policy-in-code defaults in addition to general Keystone documentation updates noting as much.

Security impact

Policy defaults refresh will help keep the system secure.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Deployers will need to look through the new policies (communicated via release notes) to make sure they can adopt them.

Developer impact

New APIs must add policies that follow the new pattern.

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>

Work Items

In order to make sure existed policies run normally when every changes happen, we will follow the [6] and propose changes in the following order:

Add new roles to cyborg policy including Project-Reader, Project-Member, System-Reader, System-Admin.
Update APIs and unit tests that are using the above new roles.
Update APIs and unit tests that are using other roles such as Project-Admin, Admin_or_user etc.
Refactor cyborg policy file.

Dependencies

None

Testing

Tests for policy rules and APIs should be added. Reference RBAC test in keystone-tempest-plugin: https://review.opendev.org/#/c/686305/

Documentation Impact

API Reference should be kept consistent with any policy changes, in particular around the default reader role.

References

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Ussuri	Introduced
Victoria	Re-proposed

Cyborg Agent Proposal

Tue, 28 Apr 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-agent

This spec proposes the responsibilities and initial design of the Cyborg Agent.

Problem description

Cyborg requires an agent on the compute hosts to manage the several responsibilities, including locating accelerators, monitoring their status, and orchestrating driver operations.

Use Cases

Use of accelerators attached to virtual machine instances in OpenStack

Proposed change

Cyborg Agent resides on various compute hosts and monitors them for accelerators. On it’s first run Cyborg Agent will run the detect accelerator functions of all it’s installed drivers. The resulting list of accelerators available on the host will be reported to the conductor where it will be stored into the database and listed during API requests. By default accelerators will be inserted into the database in a inactive state. It will be up to the operators to manually set an accelerator to ‘ready’ at which point cyborg agent will be responsible for calling the drivers install function and ensuring that the accelerator is ready for use.

In order to mirror the current Nova model of using the placement API each Agent will send updates on it’s resources directly to the placement API endpoint as well as to the conductor for usage aggregation. This should keep placement API up to date on accelerators and their usage.

Alternatives

There are lots of alternate ways to lay out the communication between the Agent and the API endpoint or the driver. Almost all of them involving exactly where we draw the line between the driver, Conductor , and Agent. I’ve written my proposal with the goal of having the Agent act mostly as a monitoring tool, reporting to the cloud operator or other Cyborg components to take action. A more active role for Cyborg Agent is possible but either requires significant synchronization with the Conductor or potentially steps on the toes of operators.

Data model impact

Cyborg Agent will create new entries in the database for accelerators it detects it will also update those entries with the current status of the accelerator at a high level. More temporary data like the current usage of a given accelerator will be broadcast via a message passing system and won’t be stored.

Cyborg Agent will retain a local cache of this data with the goal of not losing accelerator state on system interruption or loss of connection.

REST API impact

TODO once we firm up who’s responsible for what.

Security impact

Monitoring capability might be useful to an attacker, but without root this is a fairly minor concern.

Notifications impact

Notifying users that their accelerators are ready?

Other end user impact

Interaction details around adding/removing/setting up accelerators details TBD.

Performance Impact

Agent heartbeat for updated accelerator performance stats might make scaling to many accelerator hosts a challenge for the Cyborg endpoint and database. Perhaps we should consider doing an active ‘load census’ before scheduling instances? But that just moves the problem from constant load to issues with a bootstorm.

Other deployer impact

By not placing the drivers with the Agent we keep the deployment footprint pretty small. We do add development complexity and security concerns sending them over the wire though.

Developer impact

TBD

Implementation

Assignee(s)

Primary assignee:: <jkilpatr>
Other contributors:: <launchpad-id or None>

Work Items

Agent implementation

Dependencies

Cyborg Driver Spec
Cyborg API Spec
Cyborg Conductor Spec

Testing

CI infrastructure with a set of accelerators, drivers, and hardware will be required for testing the Agent installation and operation regularly.

Documentation Impact

Little to none. Perhaps on an on compute config file that may need to be documented. But I think it’s best to avoid local configuration where possible.

References

Other Cyborg Specs

History

Revisions
Release	Description
Pike	Introduced

Cyborg API proposal

Tue, 28 Apr 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-api

This spec proposes to provide the initial API design for Cyborg.

Problem description

Cyborg as a common management framework for dedicated devices (hardware/ software accelerators, high-speed storage, etc) needs RESTful API to expose the basic functionalities.

Use Cases

As a user I want to be able to spawn VM with dedicated hardware, so that I can utilize provided hardware.
As a compute service I need to know how requested resource should be attached to the VM.
As a scheduler service I’d like to know on which resource provider requested resource can be found.

Proposed change

In general we want to develop the APIs that support basic life cycle management for Cyborg.

Life Cycle Management Phases

For cyborg, LCM phases include typical create, retrieve, update, delete operations. One thing should be noted that deprovisioning mainly refers to detach(delete) operation which deactivate an acceleration capability but preserve the resource itself for future usage. For Cyborg, from functional point of view, the LCM includes provision, attach,update,list, and detach. There is no notion of deprovisioning for Cyborg API in a sense that we decomission or disconnect an entire accelerator device from the bus.

Difference between Provision and Attach/Detach

Noted that while the APIs support provisioning via CRUD operations, attach/detach are considered different:

Provision operations (create) will involve api-> conductor->agent->driver workflow, where as attach/detach (update/delete) could be taken care of at the driver layer without the involvement of the pre-mentioned workflow. This is similar to the difference between create a volume and attach/detach a volume in Cinder.
The attach/detach in Cyborg API will mainly involved in DB status modification.

Difference between Attach/Detach To VM and Host

Moreover there are also differences when we attach an accelerator to a VM or a host, similar to Cinder.

When the attachment happens to a VM, we are expecting that Nova could call the virt driver to perform the action for the instance. In this case Nova needs to support the acc-attach and acc-detach action.
When the attachment happens to a host, we are expecting that Cyborg could take care of the action itself via Cyborg driver. Althrough currently there is the generic driver to accomplish the job, we should consider a os-brick like standalone lib for accelerator attach/detach operations.

Alternatives

For attaching an accelerator to a VM, we could let Cyborg perform the action itself, however it runs into the risk of tight-coupling with Nova of which Cyborg needs to get instance related information.
For attaching an accelerator to a host, we could consider to use Ironic drivers however it might not bode well with the standalone accelerator rack scenarios where accelerators are not attached to server at all.

Data model impact

A new table in the API database will be created:

CREATE TABLE accelerators (
    accelerator_id INT NOT NULL,
    device_type STRING NOT NULL,
    acc_type STRING NOT NULL,
    acc_capability STRING NOT NULL,
    vendor_id STRING,
    product_id STRING,
    remotable INT,
);

Note that there is an ongoing discussion on nested resource provider new data structures that will impact Cyborg DB imp- lementation. For code implementation it should be aligned with resource provider db requirement as much as possible.

REST API impact

The API changes add resource endpoints to:

GET a list of all the accelerators
GET a single accelerator for a given id
POST create a new accelerator resource
PUT an update to an existing accelerator spec
PUT attach an accelerator to a VM or a host
DELETE detach an existing accelerator for a given id

The following new REST API call will be created:

‘GET /accelerators’

Return a list of accelerators managed by Cyborg

Example message body of the response to the GET operation:

200 OK
Content-Type: application/json

{
   "accelerator":[
    {
      "uuid":"8e45a2ea-5364-4b0d-a252-bf8becaa606e",
      "acc_specs":
      {
         "remote":0,
         "num":1,
         "device_type":"CRYPTO"
         "acc_capability":
         {
            "num":2
            "ipsec":
            {
               "aes":
               {
                  "3des":50,
                  "num":1,
               }
            }
         }
       }
     },
     {
       "uuid":"eaaf1c04-ced2-40e4-89a2-87edded06d64",
       "acc_specs":
       {
          "remote":0,
          "num":1,
          "device_type":"CRYPTO"
          "acc_capability":
          {
             "num":2
             "ipsec":
             {
                "aes":
                {
                   "3des":40,
                   "num":1,
                }
             }
          }
        }
      }
   ]
}

‘GET /accelerators/{uuid}’

Retrieve a certain accelerator info indetified by ‘{uuid}’

Example GET Request:

GET /accelerators/8e45a2ea-5364-4b0d-a252-bf8becaa606e

200 OK
Content-Type: application/json

{
   "uuid":"8e45a2ea-5364-4b0d-a252-bf8becaa606e",
   "acc_specs":{
      "remote":0,
      "num":1,
      "device_type":"CRYPTO"
      "acc_capability":{
         "num":2
         "ipsec":{
             "aes":{
               "3des":50,
               "num":1,
             }
         }
      }
    }
}

If the accelerator does not exist a 404 Not Found must be returned.

‘POST /accelerators/{uuid}’

Create a new accelerator

Example POST Request:

Content-type: application/json

{
    "name": "IPSec Card",
    "uuid": "8e45a2ea-5364-4b0d-a252-bf8becaa606e"
}

The body of the request must match the following JSONSchema document:

{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "uuid": {
            "type": "string",
            "format": "uuid"
        }
    },
    "required": [
        "name"
    ]
    "additionalProperties": False
}

The response body is empty. The headers include a location header pointing to the created accelerator resource:

201 Created
Location: /accelerators/8e45a2ea-5364-4b0d-a252-bf8becaa606e

A 409 Conflict response code will be returned if another accelerator exists with the provided name.

‘PUT /accelerators/{uuid}/{acc_spec}’

Update the spec for the accelerator identified by {uuid}.

Example:

PUT /accelerator/8e45a2ea-5364-4b0d-a252-bf8becaa606e

Content-type: application/json

{
    "acc_specs":{
       "remote":0,
       "num":1,
       "device_type":"CRYPTO"
       "acc_capability":{
          "num":2
          "ipsec":{
             "aes":{
               "3des":50,
               "num":1,
             }
          }
       }
     }
}

The returned HTTP response code will be one of the following:

200 OK if the spec is successfully updated
404 Not Found if the accelerator identified by {uuid} was not found
400 Bad Request for bad or invalid syntax
409 Conflict if another process updated the same spec.

‘PUT /accelerators/{uuid}’

Attach the accelerator identified by {uuid}.

Example:

PUT /accelerator/8e45a2ea-5364-4b0d-a252-bf8becaa606e

Content-type: application/json

{
    "name": "IPSec Card",
    "uuid": "8e45a2ea-5364-4b0d-a252-bf8becaa606e"
}

The returned HTTP response code will be one of the following:

200 OK if the accelerator is successfully attached
404 Not Found if the accelerator identified by {uuid} was not found
400 Bad Request for bad or invalid syntax
409 Conflict if another process attach the same accelerator.

‘DELETE /accelerator/{uuid}’

Detach the accelerator identified by {uuid}.

The body of the request and the response is empty.

The returned HTTP response code will be one of the following:

204 No Content if the request was successful and the accelerator was detached.
404 Not Found if the accelerator identified by {uuid} was not found.
409 Conflict if there exist allocations records for any of the accelerator resource that would be detached as a result of detaching the accelerator.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developers can use this REST API after it has been implemented.

Implementation

Assignee(s)

Primary assignee:: zhipengh <huangzhipeng@huawei.com>

Work Items

Implement the APIs specified in this spec
Proposal to Nova about the new accelerator attach/detach api
Implement the DB specified in this spec

Dependencies

None.

Testing

Unit tests will be added to Cyborg API.

Documentation Impact

None

References

None

History

Revisions
Release	Description
Pike	Introduced

Cyborg Conductor Proposal

Tue, 28 Apr 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-agent

This spec proposes the responsibilities and initial design of the Cyborg Conductor.

Problem description

Cyborg requires a conductor on the controller hosts to manage the cyborg system state and coalesce database operations.

Use Cases

Use of accelerators attached to virtual machine instances in OpenStack

Proposed change

Cyborg Conductor will reside on the control node and will be responsible for stateful actions taken by Cyborg. Acting as both a cache to the database and as a method of combining reads and writes to the database. All other Cyborg components will go through the conductor for database operations.

Alternatives

Having each Cyborg Agent instance hit the database on it’s own is a possible alternative, and it may even be feasible if the accelerator load monitoring rate is very low and the vast majority of operations are reads. But since we intend to store metadata about accelerator usage updated regularly this model probably will not scale well.

Data model impact

Using the conductor ‘properly’ will result in little or no per instance state and stateful operations moving through the conductor with the exception of some local caching where it can be garunteed to work well.

REST API impact

N/A

Security impact

Negligible

Notifications impact

N/A

Other end user impact

Faster Cybrog operation and less database load.

Performance Impact

Generally positive so long as we don’t overload the messaging bus trying to pass things to the Conductor to write out.

Other deployer impact

Conductor must be installed and configured on the controllers.

Developer impact

None for API users, internally heavy use of message passing will be required if we want to keep all system state in the controllers.

Implementation

Assignee(s)

Primary assignee:: jkilpatr
Other contributors:: None

Work Items

Implementation
Integration with API and Agent

Dependencies

Cyborg API spec
Cyborg Agent spec

Testing

This component should be possible to fully test using unit tests and functional CI using the dummy driver.

Documentation Impact

Some configuration values tuning save out rate and other parameters on the controller will need to be documented for end users

References

Cyborg API Spec Cyborg Agent Spec

History

Revisions
Release	Description
Pike	Introduced

Cyborg Generic Driver Proposal

Tue, 28 Apr 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/generic-driver-cyborg

This spec proposes to provide the initial design for Cyborg’s generic driver.

Problem description

This blueprint proposes to add a generic driver for openstack-cyborg. The goal is to provide users & operators with a reliable generic implementation that is hardware agnostic and provides basic accelerator functionality.

Use Cases

As an admin user and a non-admin user with elevated privileges, I should be able to identify and discover attached accelerator backends.
As an admin user and a non-admin user with elevated privileges, I should be able to view services on each attached backend after the agent has discovered services on each backend.
As an admin user and a non-admin user, I should be able to list and update attached accelerators by driver by querying nova with the Cyborg-API.
As an admin user and a non-admin user with elevated privileges, I should be able to install accelerator generic driver.
As an admin user and a non-admin user with elevated privileges, I should be able to uninstall accelerator generic driver.
As an admin user and a non-admin user with elevated privileges, I should be able to issue attach command to the instance via the driver which gets routed to Nova via the Cyborg API.
As an admin user and a non-admin user with elevated privileges, I should be able to issue detach command to the instance via the driver which gets routed to Nova via the Cyborg API.

Proposed change

Cyborg needs a reference implementation that can be used as a model for future driver implementations and that will be referred to as the generic driver implementation
Develop the generic driver implementation that supports CRUD operations for accelerators for single backend and multi backend setup scenarios.

Alternatives

None

Data model impact

The generic driver will update the central database when any CRUD or attach/detach operations take place

REST API impact

This blueprint proposes to add the following APIs:

cyborg install-driver <driver_id>
cyborg uninstall-driver <driver_id>
cyborg attach-instance <instance_id>
cyborg detach-instance <instance_id>
cyborg service-list
cyborg driver-list
cyborg update-driver <driver_id>
cyborg discover-services

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Developers will have access to a reference generic implementation which can be used to build vendor-specific drivers.

Implementation

Assignee(s)

Primary assignee:: Rushil Chugh <rushil.chugh@gmail.com>

Work Items

This change would entail the following:

Add a feature to identify and discover attached accelerator backends.
Add a feature to list services running on the backend
Add a feature to attach accelerators to the generic backend.
Add a feature to detach accelerators from the generic backend.
Add a feature to list accelerators attached to the generic backend.
Add a feature to modify accelerators attached to the generic backend.
Defining a reference implementation detailing the flow of requests between the cyborg-api, cyborg-conductor and nova-compute services.

Dependencies

Dependent on Cyborg API and Agent implementations.

Testing

Unit tests will be added test Cyborg generic driver.

Documentation Impact

None

References

None

History

Revisions
Release	Description
Pike	Introduced

Cyborg FPGA Model Proposal

Tue, 28 Apr 2020 00:00:00

Blueprint url is not available yet https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-fpga-modelling

This spec proposes the DB modelling schema for tracking reprogrammable resources

Problem description

Use Cases

When user requests FPGA resources, scheduler will use placement agent [1] to select appropriate hosts that have the requested FPGA resources.

When a FPGA type resource is allocated to a VM, Cyborg needs to track down which exact device has been assigned in the database. On the other hand, when the resource is released, Cyborg will need to be detached and free the exact resource.

When a new device is plugged in to the system(host), Cyborg needs to discover it and store it into the database

Proposed change

We need to add 2 more tables to Cyborg database, one for tracking all the deployables and one for arbitrary key-value pairs of deplyable associated attirbutes. These tables are named as Deployables and Attributes.

Deployables table consists of all the common attributes columns as well as a parent_id and a root_id. The parent_id will point to the associated parent deployable and the root_id will point to the associated root deployable. By doing this, we can form a nested tree structure to represent different hierarchies. In addition, there will a foreign key named accelerator_id reference to the accelerators table. For the case where FPGA has not been loaded any bitstreams on it, they will still be tracked as a Deployable but no other Deployables referencing to it. For instance, a network of FPGA hierarchies can be formed using deployables in following scheme:

                        -------------------
    ------------------->|Deployable - FPGA|<--------------------
    |                   -------------------                    |
    |                           /\                             |
    | root_id                  /  \  parent_id/root_id         |
    |                         /    \                           |
    |         -----------------    -----------------           |
    |         |Deployable - PF|    |Deployable - PF|           |
    |         -----------------    -----------------           |
    |               /\                                         |
    |              /  \  parent_id                     root_id |
    |             /    \                                       |
-----------------      -----------------                       |
|Deployable - VF|      |Deployable - VF| -----------------------
-----------------      -----------------

Attributes table consists of a key and a value columns to represent arbitrary k-v pairs.

For instance, bitstream_id and function kpi can be tracked in this table. In addition, a foreign key deployable_id refers to the Deployables table and a parent_attribute_id to form nested structured attribute relationships.

Cyborg needs to have object classes to represent different types of deployables(e.g. FPGA, Physical Functions, Virtual Functions etc).

Cyborg Agent needs to add feature to discover the FPGA resources from FPGA driver and report them to the Cyborg DB through the conductor.

Conductor needs to add couple of sets of APIs for different types of deployable resources.

Alternatives

Alternativly, instead of having a flat table to track arbitrary hierarchies, we can use two different tables in Cyborg database, one for physical functions and one for virtual functions. physical_functions should have a foreign key constraint to reference the id in Accelerators table. In addition, virtual_functions should have a foreign key constraint to reference the id in physical_functions.

The problems with this design are as follows. First, it can only track up to 3 hierarchies of resources. In case we need to add another layer, a lot of migaration work will be required. Second, even if we only need to add some new attribute to the existing resource type, we need to create new migration scripts for them. Overall the maintenance work is tedious.

Data model impact

As discussed in previous sections, two tables will be added: Deployables and Attributes:

CREATE TABLE Deployables
  (
    id           INTEGER NOT NULL ,     /*Primary Key*/
    parent_id    INTEGER ,              /*Pointer to the parent deployable's primary key*/
    root_id      INTEGER ,              /*Pointer to the root deployable's primary key*/
    name         VARCHAR2 (32 BYTE) ,   /*Name of the deployable*/
    pcie_address VARCHAR2 (32 BYTE) ,   /*pcie address which can be used for passthrough*/
    uuid         VARCHAR2 (32 BYTE) ,   /*uuid v4 format for the deployable itself*/
    node_id      VARCHAR2 (32 BYTE) ,   /*uuid v4 format to identify which host this deployable is located*/
    board        VARCHAR2 (16 BYTE) ,   /*Identify the model of the deployable(e.g. KU115)*/
    vendor       VARCHAR2 (16 BYTE) ,   /*Identify the vendor of the deployable(e.g. Xilinx)*/
    version      VARCHAR2 (32 BYTE) ,   /*Identify the version of the deployable(e.g. 1.2a)*/
    type         VARCHAR2 (32) ,        /*Identify the type of the deployable(e.g. FPGA/PF/VF)*/
    assignable   CHAR (1) ,             /*Represent if the deployable can be assigned to users*/
    instance_id  VARCHAR2 (32 BYTE) ,   /*Represent which instance this deployable has been assigned to*/
    availability INTEGER NOT NULL,      /*enum type to represent the status of the deployable(e.g. acclocated/claimed)*/
    accelerator_id INTEGER NOT NULL     /*foreign key references to the accelerator table*/
  ) ;
ALTER TABLE Deployables ADD CONSTRAINT Deployables_PK PRIMARY KEY ( id ) ;
ALTER TABLE Deployables ADD CONSTRAINT Deployables_accelerators_FK FOREIGN KEY ( accelerator_id ) REFERENCES accelerators ( id ) ;


CREATE TABLE Attributes
  (
    id            INTEGER NOT NULL ,    /*Primary Key*/
    deployable_id INTEGER NOT NULL ,    /*foreign key references to the Deployables table*/
    KEY CLOB ,                          /*Attribute Key*/
    value CLOB ,                        /*Attribute Value*/
    parent_attribute_id INTEGER         /*Pointer to the parent attribute's primary key*/
  ) ;
ALTER TABLE Attributes ADD CONSTRAINT Attributes_PK PRIMARY KEY ( id ) ;
ALTER TABLE Attributes ADD CONSTRAINT Attributes_Deployables_FK FOREIGN KEY ( deployable_id ) REFERENCES Deployables ( id ) ON
DELETE CASCADE ;

RPC API impact

Two sets of conductor APIs need to be added. 1 set for physical functions, 1 set for virtual functions

Physical function apis:

def physical_function_create(context, values)
def physical_function_get_all_by_filters(context, filters, sort_key='created_at', sort_dir='desc', limit=None, marker=None, columns_to_join=None)
def physical_function_update(context, uuid, values, expected=None)
def physical_function_destroy(context, uuid)

Virtual function apis:

def virtual_function_create(context, values)
def virtual_function_get_all_by_filters(context, filters, sort_key='created_at', sort_dir='desc', limit=None, marker=None, columns_to_join=None)
def virtual_function_update(context, uuid, values, expected=None)
def virtual_function_destroy(context, uuid)

REST API impact

Since these tables are not exposed to users for modifying/adding/deleting, Cyborg will only add two extra REST APIs to allow user query information related to deployables and their attributes.

API for retrieving Deployable’s information:

Url: {base_url}/accelerators/deployable/{uuid}
Method: GET
URL Params:
    GET: uuid --> get deplyable by uuid

Data Params:
    None

Success Response:
    GET:
        Code: 200
        Content: { deployable: {id : 12, parent_id: 11, root_id: 10, ....}}

Error Response
    Code: 401 UNAUTHORIZED
    Content: { error : "Log in" }
    OR
    Code: 422 Unprocessable Entry
    Content: { error : "deployable uuid invalid" }

Sample Call:
    To get the deployable with uuid=2864a139-c2cd-4f9f-abf3-44eb3f09b83c
    $.ajax({
      url: "/accelerators/deployable/2864a139-c2cd-4f9f-abf3-44eb3f09b83c",
      dataType: "json",
      type : "get",
      success : function(r) {
        console.log(r);
      }
    });

API for retrieving list of Deployables with filters/attirbutes:

Url: {base_url}/accelerators/deployable
Method: GET
URL Params:
    None

Data Params:
    k-v pairs for filtering

Success Response:
    GET:
        Code: 200
        Content: { deployables: [{id : 12, parent_id: 11, root_id: 10, ....}]}

Error Response
    Code: 401 UNAUTHORIZED
    Content: { error : "Log in" }
    OR
    Code: 422 Unprocessable Entry
    Content: { error : "deployable uuid invalid" }

Sample Call:
    To get a list of FPGAs with no bitstream loaded.
    $.ajax({
      url: "/accelerators/deployable",
      data: {
        "bitstream_id": None,
        "type": "FPGA"
      },
      dataType: "json",
      type : "get",
      success : function(r) {
        console.log(r);
      }
    });

API for retrieving Deployable attributes’ information:

Url: {base_url}/accelerators/deployable/{uuid}/attribute/{key}
Method: GET
URL Params:
    GET: uuid --> uuid for the associated deployable
         key  --> key for the associated deployable

Data Params:
    None

Success Response:
    GET:
        Code: 200
        Content: { attribute: {key : value}}

Error Response
    Code: 401 UNAUTHORIZED
    Content: { error : "Log in" }
    OR
    Code: 422 Unprocessable Entry
    Content: { error : "attirbute key invalid" }

Sample Call:
    To get the value of key=kpi for deployable with id=2864a139-c2cd-4f9f-abf3-44eb3f09b83c
    $.ajax({
      url: "/accelerators/deployable/2864a139-c2cd-4f9f-abf3-44eb3f09b83c/attribute/kpi",
      dataType: "json",
      type : "get",
      success : function(r) {
        console.log(r);
      }
    });

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

There will be new functionalities available to the dev because of this work.

Implementation

Assignee(s)

Primary assignee:: Li Liu <liliu1@huawei.com>

Work Items

Create migration scripts to add two more tables to the database
Create models in sqlalchemy as well as related conductor APIs
Create corespoinding objects
Create Conductor APIs to allow resourece reporting

Dependencies

Testing

Unit tests will be added test Cyborg generic driver.

Documentation Impact

Document FPGA Modelling in the Cyborg project

References

History

Revisions
Release	Description
Queens	Introduced

Cyborg Internal API spec

Tue, 28 Apr 2020 00:00:00

This document loosely specifies the API calls between the components of Cyborg. Driver, Agent, Conductor, and API endpoint.

These API’s are internal and therefore may change from version to version without warning or backwards compatibility. This document is kept as a developer reference to be edited before any internally braking changes are made.

Problem description

Developers writing one component of Cyborg need to know how to talk to another component of Cyborg, hopefully without having to go spelunking in the code of that component.

Use Cases

Happier Cyborg developers

Proposed change

Versioning internal API’s

Alternatives

A mess

Data model impact

A fixed internal API should help keep data models consistent.

REST API impact

The API changes add resource endpoints to:

Driver:

POST start accelerator discovery FROM: Agent
GET get a list of discovered accelerators and their properties FROM: Agent

Agent:

POST register driver FROM: Driver
POST start accelerator discovery across all drivers FROM: Conductor
GET get a list of all accelerators across all drivers FROM: Conductor

Conductor: * POST register agent FROM: Agent

The following new REST API call will be created:

Driver ‘POST /discovery’

Trigger the discovery and setup process for a specific driver

Content-Type: application/json

{
   "status":"IN-PROGRESS"
}

Driver ‘GET /hardware’

Gets a list of hardware, not accelerators, accelerators are ready to use entires available by the public API. Hardware are physical devices on nodes that may or may not be ready to use or even fully supported.

200 OK
Content-Type: application/json

{
   "hardware":[
    {
      "uuid":"8e45a2ea-5364-4b0d-a252-bf8becaa606e",
      "acc_specs":
      {
         "remote":0,
         "num":1,
         "device_type":"CRYPTO"
         "acc_capability":
         {
            "num":2
            "ipsec":
            {
               "aes":
               {
                  "3des":50,
                  "num":1,
               }
            }
         }
       }
       "acc_status":
       {
         "setup_required":true,
         "reboot_equired":false
       }
     }]
}

Driver ‘POST /hello’

Registers that a driver has been installed on the machine and is ready to use. As well as it’s endpoint and hardware support.

Content-Type: application/json

{
   "status":"READY",
   "endpoint":"localhost:1337",
   "type":"CRYPTO"
}

Agent ‘POST /discovery’

Trigger the discovery and setup process for all registered drivers

See driver example

Agent ‘GET /hardware’

Get list of hardware across all drivers on the node

see driver example

Conductor ‘POST /hello’

Registers that an Agent has been installed on the machine and is ready to use.

Content-Type: application/json

{
   "status":"READY",
   "endpoint":"compute-whatever:1337",
}

Security impact

Care must be taken to secure the internal endpoints from malicious calls

Notifications impact

N/A

Other end user impact

This change might have an impact on python-cyborgclient

Performance Impact

In this model the Agent takes care of wrangling however many drivers are on a compute and the Conductor takes care of wrangling all the agents to present a coherent answer to the API quickly and easily. I don’t include API <-> Conductor calls yet because I assume the API will be for the most part working from the database while the Conductor tries to keep that database up to date and takes the occasional setup call.

Other deployer impact

In this model we won’t really know when we’re missing an agent. If one has reported in previously and then goes away we can have an alarm for that. But if an agent never reports in we just have to assume no instance exists by that name. This means making sure the Cyborg Drivers/Agent’s are installed and running is the responsibility of the deployment tool.

Developer impact

More internal communication in Cyborg

Implementation

Assignee(s)

Primary assignee:: jkilpatr
Other contributors:: zhuli

Work Items

N/A

Dependencies

N/A

Testing

N/A

Documentation Impact

N/A

References

N/A

History

Revisions
Release Name	Description
Queens	Introduced

Cyborg SPDK Driver Proposal

Tue, 28 Apr 2020 00:00:00

https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-spdk-driver

This spec proposes to provide the initial design for Cyborg’s SPDK driver.

Problem description

SPDK is a high performance kit and provides a user space, polled-mode, asynchronous, lockless NVMe driver for storage acceleration on the backend. Our goal is to add a SPDK driver for Cyborg to manage SPDK, and further improve storage performance.

Use Cases

When Cinder uses Ceph as its backend, the user should be able to use the Cyborg SPDK driver to discover the SPDK accelerator backend, enumerate the list of the Ceph nodes that have installed the SPDK.
When Cinder directly uses SPDK’s BlobStore as its backend, the user should be able to accomplish the same life cycle management operations for SPDK as mentioned above. After enumerating the SPDK, the user can attach (install) SPDK on that node. When the task completes, the user can also detach the SPDK from the node. Last but not least the user should be able to update the latest and available SPDK.

Proposed change

In general, the goal is to develop the Cyborg SPDK driver that supports discover/list/update/attach/detach operations for SPDK framework.

SPDK framework

The SPDK framework comprises of the following components:

      +-----------userspace--------+  +--------------+
      | +------+ +------+ +------+ | | +-----------+ |
+---+ | |DPDK  | |NVMe  | |NVMe  | | | |   Ceph    | |
| N +-+-+NIC   | |Target| |Driver+-+-+ |NVMe Device| |
| I | | |Driver| |      | |      | | | +-----------+ |
| C | | +------+ +------+ +------+ | | +-----------+ |
+---+ | +------------------------+ | | | Blobstore | |
      | |     DPDK Libraries     | | | |NVMe Device| |
      | +------------------------+ | | +-----------+ |
      +----------------------------+ +---------------+

BlobStore NVMe Device Format

BlobStore owns the entire NVMe device including metadata management and data management, which defines three basic units of disk space (like logical block, page, cluster). The NVMe device is divided into clusters starting from the first logical block.

Cluster0 has special format which consists of pages. Page0 is the first page of Cluster0. Super Block contains the basic information of BlobStore.

Page 0	Page 1 … Page N
Super Block	Metadata Region

Each blob is allocated a non-contiguous set of pages. These pages form a linked list. In general, the BlobStore adopts direct operation of bare metal device and avoids the filesystem, which improves efficiency.

Life Cycle Management Phases

We should be able to add a judgement whether the backend node has SPDK kit in generic driver module. If true, initialize the DPDK environment (such as hugepage).
Import the generic driver module, and then we should be able to discover (probe) the system for SPDK.
Determined by the backend storage scenario, enumerate (list) the optimal SPDK node, returning a boolean value to judge whether the SPDK should be attached.
After the node where SPDK will be running is attached, we can now send a request about the information of namespaces, and then create an I/O queue pair to submit read/write requests to a namespace.
When Ceph is used as the backend, as the latest Ceph (such as Luminous) uses the BlueStore to be the storage engine, BlueStore and BlobStore are very similar things. We will not be able to use BlobStore to accelerate Ceph, but we can use Ioat and poller to boost speed for storage.
When SPDK is used as the backend, we should be able to use BlobStore to improve performance.
Whenever user requests, we should be able to detach the SPDK device.
Whenever user requests, we should be able to update SPDK to the latest and stable release.

Alternatives

None

Data model impact

The Cyborg SPDK driver will notify Cyborg Agent to update the database when discover/list/update/attach/detach operations take place.

REST API impact

This blueprint proposes to add the following APIs:

cyborg discover-driver（driver_type）
cyborg driver-list(driver_type)
cyborg install-driver(driver_id, driver_type)
cyborg attach-instance <instance_id>
cyborg detach-instance <instance_id>
cyborg uninstall-driver(driver_id, driver_type)
cyborg update-driver <driver_id, driver_type>

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The SPDK can provide a user space, polled-mode, asynchronous, lockless NVMe driver for storage acceleration on the backend.

Other deployer impact

Deployers can call SPDK from the nodes which have installed SPDK after the driver has been implemented.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: luwei he <heluwei@huawei.com>

Work Items

Implement the cyborg-spdk-driver in this spec.
Propose SPDK to py-spdk. The py-spdk is designed as a SPDK client which provides the python binding.

Dependencies

Cyborg API Spec
Cyborg Agent Spec
Cyborg Driver Spec
Cyborg Conductor Spec

Testing

Unit tests will be added to test Cyborg SPDK driver.
Functional tests will be added to test Cyborg SPDK driver. For example: discover–>list–>attach，whether the workflow can be passed successfully.

Documentation Impact

Document SPDK driver in the Cyborg project

References

Cyborg API Spec
Cyborg Agent Spec
Cyborg Driver Spec
Cyborg Conductor Spec

History

Revisions
Release	Description
Queens	Introduced

Cyborg-Nova-Glance Interaction in Compute Node

Tue, 28 Apr 2020 00:00:00

Cyborg is a service for managing accelerators, such as FPGAs, GPUs, etc. For scheduling an instance that needs accelerators, Cyborg needs to work with Nova at three levels:

Representation and Discovery: Cyborg shall represent accelerators as resources in Placement. When a device is discovered, Cyborg updates resource inventories in Placement.
Instance placement/scheduling: Cyborg may provide a weigher that prioritizes hosts based on available accelerator resources.
Attaching accelerators to instances. In the compute node, Cyborg shall define a workflow based on interacting with Nova through a new os-acc library (like os-vif and os-brick).

The first two aspects are addressed in [1]. This spec addresses the attachment of accelerators to instances, via os-acc. For FPGAs, Cyborg also needs to interact with Glance for fetching bitstreams. Some aspects of that are covered in [2]. This spec will address the interaction of Cyborg and Glance in the compute node.

Smart NICs based on FPGAs fall into two categories: those which expose the FPGA explicitly to the host, and those that do not. Cyborg’s current scope includes the former. This spec includes such devices, though the Cyborg-Neutron interaction is out of scope.

The scope of this spec is Rocky release.

Terminology

Accelerator: The unit that can be assigned to an instance for offloading specific functionality. For non-FPGA devices, it is either the device itself or a virtualized version of it (e.g. vGPUs). For FPGAs, an accelerator is either the entire device, a region within the device or a function.
Bitstream: An FPGA image, usually a binary file, possibly with vendor-specific metadata. A bitstream may implement one or more functions.
Function: A specific functionality, such as matrix multiplication or video transcoding, usually represented as a string or UUID. This term may be used with multi-function devices, including FPGAs and other fixed function hardware like Intel QuickAssist.
Region: A part of the FPGA which can be programmed without disrupting other parts of that FPGA. If an FPGA does not support Partial Reconfiguration, the entire device constitutes one region. A region may implement one or more functions.

Here is an example diagram for an FPGA with multiple regions, and multiple functions in a region:

       PCI A     PCI B
        |        |
+-------|--------|-------------------+
|       |        |                   |
|  +----|--------|---+   +--------+  |
|  | +--|--+ +---|-+ |   |        |  |
|  | | Fn A| | Fn B| |   |        |  |
|  | +-----+ +-----+ |   |        |  |
|  +-----------------+   +--------+  |
|  Region 1              Region 2    |
|                                    |
+------------------------------------+

Problem description

Once Nova has picked a compute node for placement of an instance that needs accelerators, the following steps needs to happen:

Nova compute on that node has to invoke Cyborg Agent for handling the needed accelerators. This needs to happen through a library, named os-acc, patterned after os-vif (Neutron) and os-brick (Cinder).
Cyborg Agent may call Glance to fetch a bitstream, either by id or based on tags.
Cyborg Agent may need to call into a Cyborg driver to program said bitstream.
Cyborg Agent needs to call into a Cyborg driver to prepare a device and/or obtain an attach handle (e.g. PCI BDF) that can be attached to the instance.
Cyborg Agent returns enough information to Nova compute via os-acc for the instance to be launched.

The behavior of each of these steps needs to be specified.

In addition, the OpenStack Compute API [3] specifies the operations that can be done on an instance. The behavior with respect to accelerators must be defined for each of these operations. That in turn is related to when Nova compute calls os-acc.

Use Cases

Please see [1]. We intend to support FPGAaaS with request time programming, and AFaaS (both pre-programmed and orchestrator-programmed scenarios).

Cyborg will discover accelerator resources whenever the Cyborg agent starts up. PCI hot plug can be supported past Rocky release.

Cyborg must support all instance operations mentioned in OpenStack Compute API [3] in Rocky, except booting off a snapshot and live migration.

Proposed change

OpenStack Server API Behavior

The OpenStack Compute API [3] mentions the list of operations that can be performed on an instance. Of these, some will not be supported by Cyborg in Rocky. The list of supported operations (with the intended behaviors) are as follows:

When an instance is started, the accelerators requested by that instance’s flavor must be attached to the instance. On termination, those resources are released.
When an instance is paused, suspended or locked, the accelerator resources are left intact, and not detached from the instance. So, when the instance is unpaused, resumed or unlocked, there is nothing to do.
When an instance is shelved, the accelerator resources are detached. On an unshelve, it is expected that the build operation will go through the scheduler again, so it is equivalent to an instance start.
When an instance is deleted, the accelerator resources are detached. On a restore, it is expected that the build operation will go through the scheduler again, so it is equivalent to an instance start.
Reboot: The accelerator resources are left intact. It is up the instance software to rediscover attached resources.
Rebuild: Prior to the instance image replacement, all device access must be quiesced, i.e., accesses to devices from that instance must be completed and further accesses must be prohibited. The mechanics of such quiescing are outside the scope of this document. With that precondition, accelerator resources are left attached to the instance during the rebuild.
Resize (with change of flavor): It is equivalent to a termination followed by re-scheduling and restart. The accelerator resources are detached on termination, and re-attached on when the instance is scheduled again.
Cold migration: It is equivalent to a termination followed by re-scheduling and restart. The accelerator resources are detached on termination, and re-attached on when the instance is scheduled again.
Evacuate: This is a forcible rebuild by the administrator. As the semantics of evacuation are left open even without accelerators, Cyborg’s behavior is also left undefined.
Set administrator password, trigger crash dump: These are supported and not no-ops for accelerators.

The following instance operations are not supported in this release:

Booting off a snapshot: The snapshot may have been taken when the attached accelerators were in a particular state. When booting off a previous snapshot, the current configuration and state of accelerators may not match the snapshot. So, this is unsupported.
Live migration: Until a mechanism is defined to migrate accelerator state along with the instance, this is unsupported.

os_acc Structure

Cyborg will develop a new library named os-acc. That library will offer the APIs listed later in this section. Nova Compute calls these APIs if it sees that the requested flavor refers to CUSTOM_ACCELERATOR resource class, except for the initialize() call, which is called unconditionally. Nova Compute calls these APIs asynchronously, as suggested below:

with ThreadPoolExecutor(max_workers=1) as executor:
   future = executor.submit(os_acc.<api>, *args)
   # do other stuff
   try:
      data = future.result()
   except:
      # handle exceptions

The APIs of os-acc are as below:

initialize()
- Called once at start of day. Waits for Cyborg Agent to be ready to accept requests, i.e., all devices enumerated and traits published.
- Returns None on success.
- Throws CyborgAgentUnavailable exception if Cyborg Agent cannot be contacted.
plug(instance_info, selected_rp, flavor_extra_specs)
- Parameters are all read-only. Here are their descriptions:
  - instance_info: dictionary containing instance UUID, instance name, project/tenant ID and VM image UUID. The instance name is needed for better logging, the project/tenant ID may be passed to some accelerator policy engine in the future and the VM image UUID may be used to query Glance for metadata about accelerator requirements that may be stored with the VM image.
  - selected_rp: Information about the selected resource provider is passed as a dictionary.
  - flavor_extra_specs: the extra_specs field in the flavor, including resource classes, traits and other fields interpreted by Cyborg.
- Called by Nova compute when an instance is started, unshelved, or restored and after a resize or cold migration.
- Called before an instance is built, i.e., before the specification of the instance is created. For libvirt-based hypervisors, this means the call happens before the instance’s domain XML is created.
- As part of this call, Cyborg Agent may fetch bitstreams from Glance and initiate programming. It may fetch the bitstream specified in the request’s flavor extra specs, if any. If the request refers to a function ID/name, Cyborg Agent would query Glance to find bitstreams that provide the flavor and match the chosen device, and would then fetch the needed bitstream.
- As part of this call, Cyborg Agent will locate the Deployable corresponding to the chosen RP, locate the attach handles (e.g. PCI BDF) needed, update its internal data structures in a persistent way, and return the needed information back to Nova.
- Returns an array, with one entry per requested accelerator, each entry being a dictionary. The dictionary is structured as below for Rocky:
{ “pci_id”: <pci bdf> }
unplug(instance_info)
- Parameters are all read-only. Here are their descriptions:
  - instance_info: dictionary containing instance UUID and instance name. The instance name is needed for better logging.
- Called when an instance is stopped, shelved, or deleted and before a resize or cold migration.
- As part of this call, Cyborg Agent will clean up internal resources, call the appropriate Cyborg driver to clean up the device resources and update its data structures persistently.
- Returns the number of accelerators that were released. Errors may cause exceptions to be thrown.

Workflows

The pseudocode for each os-acc API can be expressed as below:

def initialize():
  # checks that all devices are discovered and their traits published
  # waits if any discovery operation is ongoing
  return None

def plug(instance_info, rp, extra_specs):
  validate_params(....)
  glance = glanceclient.Client(...)
  driver = # select Cyborg driver for chosen rp
  rp_deployable = # get deployable for RP
  if extra_specs refers to ``CUSTOM_FPGA_<vendor>_REGION_<uuid>`` and
     extra_specs refers to ``bitstream:<uuid>``:
     bitstream = glance.images.data(image_uuid)
     driver.program(bitstream, rp_deployable,  …)
  if extra_specs refers to ``CUSTOM_FPGA_<vendor>_FUNCTION_<uuid>`` and
     extra_specs refers to function UUID/name:
     region_type_uuid = # fetch from selected RP
     bitstreams = glance.images.list(...)
     # queries Glance by function UUID/name property and region type
     # UUID to get matching bitstreams
     if len(bitstreams) > 1:
       error(...) # bitstream choice policy is outside Cyborg
     driver.program(bitstream, rp_deployable, …)
  pci_bdf = driver.allocate_handle(...)
  # update Cyborg DB with instance_info and BDF usage
  return { “pci_id”: pci bdf }

def unplug(instance_info):
  bdf_list = # fetch BDF usage from Cyborg DB for instance
  # update Cyborg DB to mark those BDFs as free
  return len(bdf_list)

Alternatives

N/A

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

None

Work Items

Decide how to associate multiple functions/bitstreams in extra specs with multiple devices in the flavor.
Decide specific changes needed in Cyborg conductor, db, agent and drivers.
Others: TBD

Dependencies

Nested Resource Provider support in Nova
Nova Granular Requests

Testing

For each vendor driver supported in this release, we need to integrate the corresponding FPGA type(s) in the CI infrastructure.

Documentation Impact

The behavior with respect to accelerators during various instance operations (reboot, pause, etc.) must be documented. The procedure to upload a bitstream, including applying Glance properties, must also be documented.

References

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Cyborg Agent-Driver API

Tue, 28 Apr 2020 00:00:00

Cyborg agent interacts with each Cyborg driver in the compute node to discover available devices. This spec defines how the agent-driver API is structured.

No change is proposed to the way the agent discovers the drivers on start or restart.

This spec is common to all accelerators, including GPUs, High Precision Time Synhronization (HPTS) cards, etc. Since FPGAs have more aspects to be considered than other devices, some sections may focus on FPGA-specific factors. The spec calls out the FPGA-specific aspects.

The scope of this spec is Rocky release, but the API has been designed to be extensible for future releases. Accordingly, the spec calls out the Rocky-specific aspects.

Problem description

The [1] specifies that devices are represented using Resource Providers (RPs), Resource Classes (RCs) and traits. The information needed to create them has to come from the Cyborg driver to the Cyborg agent, which in turn needs to push it to the Cyborg Conductor.

The main challenge is discovering the device topology for FPGAs. An FPGA may have one or more Partial Reconfiguration regions, and those regions may have one or more accelerators nested inside them. Further, it may have local memory that is either partitioned or shared among the regions.

Use Cases

Devices of different types (GPUs, FPGAs, HPTS cards, Quick Assist) are present in the same host.
FPGAs of different types, possibly from different vendors, are present in the same host.
An FPGA may have one or more regions. Each region may have one or more accelerators.
- In Rocky, we may support only one region per FPGA, and only one accelerator per region.
For Rocky, it is proposed that local memory need not be exposed as a resource to orchestration. That is because, since there is only one region per FPGA, an instance attached to that region will be able to access all the memory, no matter how much there is. For non-FPGA devices like GPUs, there does not seem to be a requirement to expose video RAM.

Cyborg will assume and handle the following component relationships:

One product (e.g. Intel PAC Arria 10) may correspond to multiple PCI vendor/device IDs.
One PCI vendor/device ID may correspond to different region type IDs. This could be either because there are multiple regions in the same device or because there are different versions/revisions of the same device.
But the same region type ID will never show up in products with different PCI IDs.

Proposed change

Today, the Cyborg agent invokes the discover() API for each driver that it finds. The discover() API returns a dictionary indexed by the PCI BDF of a device. The value element in the key-value pair of the dictionary contains the components and characteristics of the device with that BDF.

We propose to retain the same model, but enhance the dictionary to include enough information to create the resource providers and traits needed to populate Placement. Here are the additional proposed keys in the device dictionary for each PF:

"type": <enum-string> # One of GPU, FPGA, etc.
"vendor": <string>
"product": <string>

Also, in the regions entry for each PF, it is proposed to add the following keys:

"region-type-uuid": <uuid>  # Optional, default: NULL
"bitstream-id": <uuid> # Glance/other UUID, optional, default: NULL
"function-uuid": <uuid> # Optional, default: NULL

When the agent receives this dictionary for a device, it will do the following:

If there is nested RP support, create an RP for the device and each region within.
Create a device type trait: CUSTOM_<type>_<vendor>_<product>. Apply it to the device RP (if nRP support exists) or the compute node RP.
- E.g. CUSTOM_FPGA_INTEL_PAC_ARRIA10.
- NOTE: The agent will convert all characters to upper case, replace spaces with underscores, and check for conformance to custom trait syntax (see [2])
Create region type traits for each region, of the form: CUSTOM_<type>_<vendor>_REGION_<type-uuid>. Apply them to the corresponding region RP (if nRP support exists) or the compute node RP.
- E.g. CUSTOM_FPGA_INTEL_REGION_<type-uuid>
- NOTE: For UUIDs, the agent will convert all hexadecimal digits to upper case, replace hyphens with underscores and validate all characters.
Create function type traits for each function in each region, of the form: CUSTOM_<type>_<vendor>_FUNCTION_<function-uuid>. Apply them to the corresponding region RP (if nRP support exists) or the compute node RP.
- E.g. CUSTOM_FPGA_INTEL_FUNCTION_<function-uuid>

Alternatives

N/A

Data model impact

Add the new fields to the database under Deployables and Attributes.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

None

Work Items

Dependencies

None

Testing

Need to update unit tests to check for the newly added fields.

Documentation Impact

None

References

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Rocky	Introduced

Cyborg FPGA Bitstream metadata spec

Tue, 28 Apr 2020 00:00:00

Blueprint url: https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-fpga-bitstream-metadata-spec

This spec proposes the FPGA Bitstream metadata specifications for bitstream management

Problem description

A field-programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or a designer after manufacturing. Their advantage lies in that they are sometimes significantly faster for some applications because of their parallel nature and optimality in terms of the number of gates used for a certain process. Hence, using FPGA for application acceleration in cloud has become desirable. One of the encountered problems is when it comes to bitstream management, it is difficult to map bitstreams to their appropriate FPGA boards or reconfigurable regions. The aim of this proposal is to provide a standardized set of metadata which should be encapsulated together with bitstream storage.

Use Cases

When user requests to reprogram a FPGA board with certain functionality in the cloud environment, he or she will need to retrieve a suitable bitstream from the storage. In order to find the suitable one, bitstreams need to be categorized based on some properties defined in metadata.

Proposed change

For each metadata, it will be stored as a row in this Glance’s image_properties in key-value pair format: column [name] holds the key whereas column [value] holds the value. Note: no batabase schema change is required. This is a standardization document to guide how to use existing Glance table for FPGA bitstreams.

Given this, Cyborg will standardize the key convention as follows:

name	value	nullable	description
bs-name	aes-128	False	name of the bitstream(not unique)
bs-uuid	{uuid}	False	The uuid generated during synthesis
vendor	Xilinx	False	Vendor of the card
board	KU115	False	Board type for this bitstream to load
shell_id	{uuid}	True	Required shell bs-uuid for the bs
version	1.0	False	Device version number
driver	SDX	True	Type of driver for this bitstream
driver_ver	1.0	False	Driver version
driver_path	/path/	False	Where to retrieve the driver binary
topology	{CLOB}	False	Function Topology
description	desc	True	Description
region_uuid	{uuid}	True	The uuid for target region type
function_uuid	{uuid}	False	The uuid for bs function type
function_name	nic-40	True	The function name for this bitstream

Here are the details regarding some definded keys.

[shell_id] This field is optional. If a loading this PR bitstream requires a shell image, this field specifies the shell bitstream’s uuid. If it field is null, it means this bitstream is a shell bitstream.

[driver] This specifies the path to a package of scripts/binaries to be installed in order to use the loaded bitstream(e.g. insmod some kernel driver/git clone some remote source code, etc)

[region_uuid] This value specifies the type of region that is required to load this bitstream. This type is a uuid generated during the shell bitstream synthesis.

[function_uuid] This value specifies the type of function for this bitstream. It helps the upsteam scheduler to match traits with appropriate bitstream.

[topology] This field describes the topology of function structures after the bitstream is loaded on the FPGA. In particular, it uses JSON format to visualize how physical functions, virtual functions are co-related to each other. It is vendor driver’s responsibility to interpret this and prepare the porper report for Cyborg Agent. For instance:

{
  "pf_num": 2,
  "vf_num": 2,
  "pf": [
    {
      "name": "pf_1",
      "capability": "",
      "kpi": "",
      "pci_offset": "0",
      "vf": [
        {
          "name": "vf_1",
          "pci_offset": "1"
        }
      ]
    },
    {
      "name": "pf_2",
      "capability": "",
      "kpi": "",
      "pci_offset": "2",
      "vf": [
        {
          "name": "vf_2",
          "pci_offset": "3"
        }
      ]
    }
  ]
}

This JSON template guides Cyborg Agent to populate vf/pf/deployable list in Cyborg.

Given the above JSON topology, Cyborg Driver should be able to interpret the accelerator structure as follows:

              =============
              =Accelerator=
              =============
                    |
              ============
              =Deployable=
              ============
                   /\
                  /  \
===================  ===================
= Deployable pf_1 =  = Deployable pf_2 =
===================  ===================
               |        |
               |        |
===================  ===================
= Deployable vf_1 =  = Deployable vf_2 =
===================  ===================

Noted: 1. Topology is not mandatory to fill in, as long as vendor driver can figure out what resources to report after the bitstream is loaded. 2. The JSON provided here is only a reference template. It does not have to be PCI-centric etc. and up to vendors how to define it for their products. 3. A root deployable shouldbe created in the graph. In addition, the pfs and vfs here are all instances of deployable. Please refer to the DB objects specs regarding physical_function and virtual_function.

Finnally, all of the FPGA bitstreams should be TAGGED as “FPGA” in Glance. This helps distinguishing between normal VM images and bitstream images during filtering.

Alternatives

Data model impact

RPC API impact

REST API impact

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Accelerator vendors should implement the logic in program() api to populate the loaded topology

Implementation

Assignee(s)

Primary assignee:: Li Liu <liliu1@huawei.com> Shaohe Feng <shaohe.feng@intel.com>

Work Items

Provide example JSON format for bitstream
Provide example implementation of vendor driver

Dependencies

Testing

Documentation Impact

None

References

None

History

Revisions
Release Name	Description
Rocky	Introduced

Cyborg FPGA Programming Service Proposal

Tue, 28 Apr 2020 00:00:00

Blueprint url is not available yet https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-fpga-programming-ability

This spec proposes a Programming Service to be added to Cyborg to allow user dynamically change the functions loaded on FPGA in cloud environment

Problem description

A field-programmable gate array (FPGA) is an integrated circuit designed to be configured by a customer or a designer after manufacturing. Their advantage lies in that they are sometimes significantly faster for some applications because of their parallel nature and optimality in terms of the number of gates used for a certain process. In addition, FPGA can be reprogrammed based on different applications Hence, using FPGA for application acceleration in cloud has been becoming desirable. Cyborg as a management framwork for heterogeneous accelerators, tracking, deploying and reprogramming FPGAs are much needed features. Since the FPGA modelling has already been proposed in another document, this spec will be focused on proposing Reporgramming Service for FPGAs in Cyborg

Use Cases

In the scenario of OpenCL, user loads the accelerators on FPGA for their application. When different applications are executing on OpenCL environment, the accelerators will be changed from time to time. It will not be feasible to login to each host and change the FPGA configuration manually by lab admin. Instead, through the reprogramming service, users can manage the functions of FPGA using a set of REST APIs.

Similarly, during the maintenance of FPGA, admin needs to update/migrate shells and bitstreams on FPGAs within data center. Cyborg Reprogramming Service will allow them to use the APIs from a centralized console.

Since this is a pure proposal for programming APIs, it would not focus on what the upstream use case/runtime is. Those details will be in separate specs when needed.

Proposed change

First of all, Cyborg needs to add extra REST APIs to allow others to invoke the programming service. The REST api should have following format:

Url: {base_url}/fpga/{deployable_uuid}
Method: POST
URL Params:
    None

Data Params:
    glance_bitstream_uuid

Success Response:
    POST:
        Code: 200
        Body: { "msg" : "bitstream has been loaded successfully"}

Error Response
    Code: 401 UNAUTHORIZED
    Body: { error : "Log in" }
    OR
    Code: 422 Unprocessable Entry
    Body: { error : "User is not authorized to use the resource" }

Sample Call:
    To program fpga resource with deployable_uuid=2864a139-c2cd-4f9f-abf3-44eb3f09b83c
    with bitstream with uuid=0b955a5b-f5dd-49d0-8c4f-28729427d303
    $.ajax({
      url: "/fpga/2864a139-c2cd-4f9f-abf3-44eb3f09b83c",
      data: {
        "glance_bitstream_uuid": "0b955a5b-f5dd-49d0-8c4f-28729427d303"
      },
      dataType: "json",
      type : "post",
      success : function(r) {
        console.log(r);
      }
    });

Second, implement the service in Cyborg which does three tasks: 1. identify the host location of the requested FPGA/Partial Reconfiguraion(PR) Region(e.g. on which host is the board located). 2. Check if the user(API caller, OpenStack Login User, etc) has the privilige to use the given bitstream, FPGA, or host. 3. If the previous checks pass, Cyborg will send the program notification to the target host with requested FPGA.

Third, implement notification callee in Cyborg Agent. This should be a rpc call with following signature:

int program_fpga_with_bitstream(deployable_uuid, bitstream_uuid)

The function takes both deployable_uuid and bitstream_uuid as input. It uses deployable_uuid to identify which specific FPGA/PR region is going to be programmed and uses bitstream_uuid to retrieve bitstream from the bitstream storage service (Glance in the context of OpenStack). In addition, this is a synchronous meaning it will wait for the programming task to be completed and then return a status code as integer. The return code should have following interpretation:

code	meaning
0	program successfully
1	failed with unkown errors
2	invalid deployable_uuid(target fpga not found)
3	invalid bitstream_uuid(bitstream can not be downloaded)

Alternatives

Data model impact

REST API impact

A rest api will be added to the Cyborg service as we discussed previously. It should not impact any of the existing rest apis

Security impact

The access to FPGA/PR region and bitstreams should be carefully checked.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

On the Cyborg Agent side, it relies on program() api implemented by vendor.

Implementation

Assignee(s)

Primary assignee:: Li Liu <liliu1@huawei.com>

Work Items

Implement the cyborg program service rest api
Implement the cyborg program service
Implement the notification call in Cyborg Agent, which invokes vendor driver

Dependencies

Testing

Documentation Impact

The Cyborg-Nova interaction related specs need to be aware the change of the accelerators when FPGAs are being reprogrammed.

References

None

History

Revisions
Release Name	Description
Rocky	Introduced

Quota Usage for Cyborg Resources

Tue, 28 Apr 2020 00:00:00

Launchpad blueprint: https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-resource-quota

There are multiple ways to slice an OpenStack cloud. Imposing quota on these various slices puts a limitation on the amount of resources that can be consumed which helps to guarantee “fairness” or fair distribution of resources at the creation time. If a particular project needs more resources, the concept of quota gives the ability to increase the resource count on-demand, given that the system constraints are not exceeded.

Problem description

At present in Cyborg we don’t have the concept of Quota on acceleration resources, so users can consume as many resources as they want. Quotas are tied closely to physical resources and billable entities, hence from Cyborg’s perspective, it helps to limit the allocation and consumption of a particular kind of resources at a certain value.

In place of implementing quota like other services, we want to enable the unified limit which is provided by Keystone to manage our quota limit[1]. With unified limits, all limits will be set in Keystone and enforced by oslo.limit. So we decided to implement quota usage part first. Once the oslo.limit is ready for other services, Cyborg will invoke oslo.limit to get the limit information and do limit check etc.

This specs aims at the implementation of quota usage in Cyborg. As the oslo.limit is not finished yet, we can directly set the value of limit manually, and reserved the function calling oslo.limit with a “pass” inside.

Use cases

Alice is an admin. She would like to have a feature which will give her details of Cyborg acceleration resource consumptions so that she can manage her resources appropriately.

She might run into following scenarios:

Ability to know current resource consumption.
Ability to prohibit overuse by a project.
Prevent situation where users in a project get starved because users in other project consume all the resource. “Quota Management” would help to gurantee “fairness”.
Prevent DOS kind of attacks, abuse or error by users, which leads to an excessive amount of resources allocation.

Proposed change

Proposed changes are introducing a Quota_Usage Table which primarily stores the quota usage assigned for each resource in a project, and a Reservation Table to store every modification of resource usage.

When a new resource allocation request comes, the ‘reserved’ field in the Quota usages table will be updated. This acceleration resource is being used to set up VM. For example, the fpga quota hardlimit is 5 and 3 fgpas have already been used, then two new fpga requests come in. Since we have 3 fpgas already used, the ‘used’ field will be set to 3. Now the ‘reserved’ field will be set to 2 untill the fpga attachment is successful. Once the attachment is done this field will be reset to 0, and the ‘used’ count will be updated from 3 to 5. So at this moment, hardlimit is 5, used is 5 and in-progress is 0. So there is one more request comes in, this request will be rejected since there is not enough quota available.

In general,

Resource quota available = Resource hard_limit - [ (Resource reserved + Resources already allocated for project)]

In this specs, we just focus on the update of quota usage and we will not check if one user has already exceed his quota limit. The limit management will be set in Keystone in the future and we just need to invoke the oslo.limit.

Alternatives

At present there is no quota infrastructure in Cyborg.

Adding Quota Management layer at the Orchestration layer could be an alternative.However, our approach will give a finer view of resource consumptions at the IaaS layer which can be used while provisioning Cyborg resources.

Data model impact

New Quota usages and reservation table will be introduced to Cyborg database to store quota consumption for each resource in a project.

Quota usages table:

Field	Type	Null	Key	Default	Extra
created_at updated_at id project_id resource reserved used	datetime datetime int(11) varchar(255) varchar(255) int(11) int(11)	YES YES NO YES NO NO NO	PRI MUL	NULL NULL NULL NULL NULL NULL NULL	auto_increment

Quota reservation table:

Field

Type

Null

Key

Default

Extra

created_at updated_at deleted_at deleted id uuid usage_id project_id resource delta expire

datetime datetime datetime tinyint(1) int(11) varchar(36) int(11) varchar(255) varchar(255) int(11) datetime

YES YES YES YES NO NO NO YES YES NO YES

PRI

MUL MUL

NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL NULL

auto_increment

We will also introduce QuotaEngine class which represents the set of recognized quotas and DbQuotaDriver class which performs check to enforcement of quotas and also allows to obtain quota information.

REST API impact

Not sure if we need to expose GET quota usage before oslo.limit settle down.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Xinran WANG

Other contributors: None

Work Items

Introduce Quota usages and Reservation table in Cyborg databases.
Update these two tables during allocation and deallocation of resources.
Reserve the place of function which will invoke oslo.limit with a “pass” inside.
Add rollback mechanism when allocation fails.

Dependencies

None

Testing

Each commit will be accompanied with unit tests.
Gate functional tests will also be covered.

Documentation Impact

None

References

[1] https://review.opendev.org/#/c/540803

Cyborg Database Model Proposal

Tue, 28 Apr 2020 00:00:00

Blueprint: https://blueprints.launchpad.net/openstack-cyborg/+spec/cyborg-database-modelling

This spec proposes a new DB modeling schema for tracking cyborg resources

Problem description

Heterogeneous acceleration resources have become essential in the cloud, edge, and high-performance computing scenarios. These devices achieve higher efficiency by tailoring the architecture to characteristics of the domain. They provide effective parallelism, effective use of memory bandwidth, etc. Hence tracking and deploying these accelerators are much-needed features.

Use Cases

For instance, when the user requests FPGA resources, the scheduler will use placement agent to select appropriate hosts that have the requested FPGA resources.

For instance, when Nova picks a device (GPU/FPGA/etc.) resource provider to allocate to a VM, Cyborg needs to track down which exact device has been assigned in the database. On the other hand, when the resource is released, Cyborg will need to be detached and free the exact resource.

When a new device is plugged into the system(host), Cyborg needs to discover it and store all its related information into the database

In addition, when a device is removed from the system, cyborg also needs to update the database accordingly.

Proposed change

We need to add 5 more table to Cyborg database. First one is Devices. The purpose of it is to track the physical existence of heterogeneous devices. Second one is AttachHandles, which tracks the attachment information needed to attach an accelerator to a VM. Thrid one is ControlPathID table, which essentially is containing device specific information on where the accelerator is located and ready to be attached. The fourth one is ExtARQs (accelerator requests) table. It is a syncing point with Nova for accelerator requests. At last, DeviceProfiles table is also added. It tracks the set of requirements for accelerators.

In addition, we need to repurpose the existing table: Deployables. And remove the existing Accelerators table.

Devices table consists of all the fields needed to describe the physical existence of a hardware device in the data center system. For instance, type, std_board_info, vendor_board_info, etc. This table should be populated by the discovery API.

AttachHandles table tracks the attachment information needed to attach an accelerator to a VM. Its records spawned when Nova requests to attach accelerators.

ControlPathID table tracks identifiers for a control path interface to devices. E.g. PCI PF. Aka Device ID. A device may have more than one of these, in which case the Cyborg driver needs to know how to handle these.

ExtARQs table tracks the accelerator requests which are sent by Nova. It contains fields both Nova acknowledgeable as well as only Cyborg specific. For the fields that are Nova acknowledgeable, they will be used to form ARQ objects. On the other hand, for cyborg specifc fields, they will be used to form ExtARQ objects.

DeviceProfiles [1] table tracks the set of requirements for accelerators. A device profile is a named set of the user requirements for one or more accelerators. It can be viewed as a flavor for devices. Broadly it includes two things: the desired amounts of specific resource classes and the requirements that the resource provider(s) must satisfy. While the resource classes are the same as those known to Placement, some requirements would correspond to Placement traits and others to properties that Cyborg alone knows about.

Deployables table now serves the purpose of describing all the derived resources from a given Device(referred in device_uuid as the foreign key). Similarly, it consists of all the common attributes columns as well as a parent_id and a root_id. The parent_id will point to the associated parent deployable and the root_id will point to the associated root deployable. By doing this, we can form a nested tree structure to represent different hierarchies. For the case where FPGA has not been loaded any bitstreams on it, they will still be tracked as a Deployable but no other Deployables referencing to it. Once a bitstream is loaded, the structure and properties of deployables can be changed. In the context of Placement in Nova, deployable’s counterpart is Resource Provider

Accelerators table now should be removed. However, the concept of accelerator has changed and deployable table tracks a num_accelerators field. It represents the number of accelerator current deployable can spawn. Additionally, once an accelerator is assigned, Cyborg creates an attach handle pointing to the corresponding deployable

For example, a network of device hierarchies can be formed using devices, deployables, accelerators, and attributes in the following scheme:

                            -------------------
                            |Device   -   FPGA|
                            -------------------
                                    /\
                     device_uuid   /  \  device_uuid
                                  /    \
                  -----------------    -----------------
        |-------->|   Deployable  |    |   Deployable  |<----------|
        |         -----------------    -----------------           |
 root_id|               /                  \                       |
        |    parent_id /          parent_id \              root_id |
        |             /                      \                     |
        |            /                        \                    |
  -----------------                      -----------------         |
  |   Deployable  |num_accelerators=2    |   Deployable  |---------|
  -----------------                      -----------------
       /           \                      ^ ^  -------------
      /             \       deployable_id | |--|Attribute A|
     /deployable_id  \                    |    -------------
-----------------     -----------------   |     -------------
|Attach Handle A|     |Attach Handle B|   |---- |Attribute B|
-----------------     -----------------         -------------

Attributes table should stay the same as before, which consists of a key and a value columns to represent arbitrary k-v pairs.

Alternatives

Alternatively, instead of having a flat table to track arbitrary hierarchies, we can use two different tables in the Cyborg database, one for physical functions and one for virtual functions. physical_functions should have a foreign key constraint to reference the id in Accelerators table. In addition, virtual_functions should have a foreign key constraint to reference the id in physical_functions.

The problems with this design are as follows. First, it can only track up to 3 hierarchies of resources. In case we need to add another layer, a lot of migration work will be required. Second, even if we only need to add some new attribute to the existing resource type, we need to create new migration scripts for them. Overall the maintenance work is tedious.

Data model impact

As discussed in previous sections, 5 table will be added: Devices:

CREATE TABLE Devices
  (
    id                INTEGER NOT NULL ,     /*Primary Key*/
    uuid              VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the device itself*/
    std_board_info    TEXT ,   /*A dictionary with standard fields*/
    vendor_board_info TEXT ,   /*A dictionary with driver-specific keys*/
    type              VARCHAR2 (30 BYTE)     /*Device Type*/
    vendor            VARCHAR2 (255 BYTE)    /*Device vendor*/
    model             VARCHAR2 (255 BYTE)    /*Device model*/
    hostname          VARCHAR2 (255 BYTE)     /*host name to identify which host this device is located*/
  ) ;
ALTER TABLE Devices ADD CONSTRAINT Devices_PK PRIMARY KEY ( id ) ;

CREATE TABLE AttachHandles
  (
    id               INTEGER NOT NULL ,     /*Primary Key*/
    attach_info      TEXT ,                 /*information needed to attach the accelerator to VMs*/
    device_id        INTEGER NOT NULL       /*foreign key references to the devices table*/
    handle_type      INTEGER NOT NULL ,     /*An enum to indicate the handle type, such as PCI, mdev, etc*/
  ) ;
PRIMARY KEY (id),
FOREIGN KEY (device_id) REFERENCES devices(id) ON
DELETE RESTRICT ;

CREATE TABLE DeviceProfiles
  (
    id               INTEGER NOT NULL ,     /*Primary Key*/
    uuid             VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the DeviceProfile itself*/
    name             VARCHAR2 (32 BYTE) ,   /*Name of the DeviceProfile*/
    json             TEXT ,                 /*JSON blob with all the deivce/vendor specifc information*/
  ) ;

CREATE TABLE ExtARQs
  (
    id               INTEGER NOT NULL ,     /*Primary Key*/
    uuid             VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the ARQ itself*/
    state            VARCHAR2 (32 BYTE) ,   /*represents current state of the request*/
    device_profile_id    INTEGER NOT NULL     /*foreign key references to the device profile table*/
    hostname          VARCHAR2 (255 BYTE)     /*host name to identify which host this request is targeting*/
    device_rp_uuid   VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the resource provider which this ARQ is pointing to*/
    instance_uuid    VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the instance which this ARQ is pointing to*/
    attach_handle_id INTEGER NOT NULL       /*foreign key references to the attach handle table*/
  ) ;
PRIMARY KEY (id),
FOREIGN KEY (device_profile_id) REFERENCES DeviceProfiles(id),
FOREIGN KEY (attach_handle_id) REFERENCES AttachHandles(id) ON
DELETE RESTRICT ;

CREATE TABLE ControlPathID
  (
    id               INTEGER NOT NULL ,     /*Primary Key*/
    type_name        VARCHAR2 (255 BYTE) ,  /*Name of the ControlPathID*/
    device_id        INTEGER NOT NULL ,     /*Foreign Key to point to the device*/
    json             TEXT ,                 /*JSON blob for type specific information*/
  ) ;

In addition, the Deployables and Accelerators will be changed to the following scheme:

CREATE TABLE Deployables
  (
    id           INTEGER NOT NULL ,     /*Primary Key*/
    parent_id    INTEGER ,              /*Pointer to the parent deployable's primary key*/
    root_id      INTEGER ,              /*Pointer to the root deployable's primary key*/
    num_accelerators   INTEGER ,        /*Number of accelerators contained in this deployable*/
    name         VARCHAR2 (32 BYTE) ,   /*Name of the deployable*/
    uuid         VARCHAR2 (36 BYTE) ,   /*uuid v4 format for the deployable itself*/
    device_id    INTEGER NOT NULL       /*foreign key references to the device table*/
  ) ;
PRIMARY KEY (id),
FOREIGN KEY (device_id) REFERENCES Devices(id) ON
DELETE RESTRICT ;

Disclaimer: more fields may be added to specific tables and the schema may evolve a little as the implementation progresses.

RPC API impact

Out of Scope for this spec

REST API impact

Out of Scope for this spec

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployment impacts

None

Developer impact

There will be new functionalities available to the dev because of this work.

Implementation

Assignee(s)

Primary assignee:: Zhenghao Wang <wangzh21@lenovo.com> Coco Gao <gaojh4@lenovo.com>

Work Items

Create migration scripts to add two more tables to the database
Create models in sqlalchemy as well as related conductor APIs
Create corresponding objects
Create Conductor APIs to allow resource reporting

Dependencies

Testing

Unit tests will be added test Cyborg generic driver.

Documentation Impact

Document FPGA Modelling in the Cyborg project

References

History

Revisions
Release	Description
Stein	Introduced

New Cyborg Generic Driver Proposal

Tue, 28 Apr 2020 00:00:00

This spec proposes to provide the initial design for Cyborg generic device driver.

Problem description

Currently, the FPGA and GPU have been supported in Cyborg, but the capability for the generic accelerator is not supported yet.

In general, these generic devices are the specific accelerators in some specific scenarios. For example:

The AI chips. which can be used for AI training and inference.
The security accelerator, which can be used for encryption and decryption.

In order to add the support for these specific accelerators, we propose to improve the generic driver to manage these devices. We propose to improve the existing GenericDriver.

Use Cases

As an AI chip vendor, I want to add the driver in Cyborg, but the existing driver type doesn’t meet our requirement. We hope the driver can provide the firmware upgrade, device configure, device stats query.
As a security accelerator vendor, I want add the driver in Cyborg with security accelerator configure and device stats query

Proposed change

1. The Generic driver change As the initial version, the new GenericDriver should include the following attributes:

VENDOR: the vendor name of the driver.
TYPE: the type of the driver, such as, “FPGA”, “GPU”

and the following interfaces also should be included:

discover() Discover specific accelerator
update(control_path, image_path): Upgrade the device firmware with a specific image. control_path: the image update control path of device. image_path: The image path of the firmware binary, the image would be downloaded by Cyborg agent.

get_stats(): Collects device stats. The get_stats method is used to collect information from the device about the device capabilities. Such as performance info like temprature, power, volt, packet_count info. The response of get_stats should follow the current Cyborg device-deploy-accelerator model.

A real stats look like:

{
  "device": {
    "device_name": "XXX",                 # Standard properties
    "device_number": "RFD1644N48373",     # Standard properties
    "properties": {                       # Vendor/Custom properties
      "id": "1",
      "temperature": "26",
      "volt": "",
      "packet_count": "",
      "memeory": {
        "model": "DDR4",
        "description": ""
      },
      "board": {

      },
      "flash": {

      }
    }
    "deploy": [
        {
          "accelerator": {
             "acc_name": "",               # Standard properties
             "properties": {               # Vendor/Custom properties
             }
          }
        },
        {
          "accelerator": {
             "acc_name": "",               # Standard properties
             "properties": {               # Vendor/Custom properties
             }
          }
        }
    ]
  }
}

This GenericDriver should not be used directly, after adding this GenericDriver, for every new device, we need to introduce a new driver that inherits from this GenericDriver.

2. Existing driver change We should also improve FPGA and GPU driver to inherit from this driver and implements the base driver interface.

The generic FPGA driver interface: The discover, update, get_accelerator_stats should be implemented in the FPGA driver. And the below interface is the FPGA specifc interface:

program(controlpath, image_path) Program the FPGA with the provided bitstream image.

The generic GPU driver interface: The discover, update, get_accelerator_stats should be implemented in the GPU driver.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Yikun Jiang <yikunkero@gmail.com> Sundar Nadathur <sundar.nadathur@intel.com> wangzhh <wangzh21@lenovo.com>

Work Items

Improve the generic driver for generic device
Improve the existing FPGA driver
Improve the existing GPU driver

Dependencies

Testing

Documentation Impact

None

References

None

History

Revisions
Release Name	Description
Train	Introduced

Cyborg-Nova interaction

Sun, 26 Apr 2020 00:00:00

https://blueprints.launchpad.net/cyborg/+spec/cyborg-nova-interaction

Cyborg, as a service for managing accelerators of any kind needs to cooperate with Nova on two planes: Cyborg should be able to inform Nova about the resources through placement API[1], so that scheduler can leverage user requests for particular functionality into assignment of specific resource using resource provider which possess an accelerator, and second, Cyborg should be able to provide information on how Nova compute can attach particular resource to VM.

In a nutshell, this blueprint will define how information between Nova and Cyborg will be exchanged.

Problem description

Currently in OpenStack the use of non-standard accelerator hardware is supported in that features exist across many of the core servers that allow these resources to be allocated, passed through, and eventually used.

What remains a challenge though is the lack of an integrated workflow; there is no way to configure many of the accelerator features without significant by hand effort and service disruptions that go against the goals of having a easy, stable, and flexible cloud.

Cyborg exists to bring these disjoint efforts together into a more standard workflow. While many components of this workflow already exist, some don’t and will need to be written expressly for this goal.

Use Cases

All possible use cases were briefly described in backlog Nova spec [2]. It might be distinguished two main use case groups for which accelerators might be used:

Accelerator might be attached to the VM, where workload demands acceleration. That can be achieved by passing whole PCI device, certain host device from /dev/ filesystem, passing Virtual Function, etc.
Accelerator might be utilized by infrastructure, like accelerating virtual switches (i.e. Open vSwitch), and than utilized via appropriate service (like Neutron for example).

Proposed Workflow

Using a method not relevant to this proposal Cyborg Agent inspects hardware and finds accelerators that it is interested in setting up for use.

These accelerators are registered into the Cyborg Database and the Cyborg Conductor is now responsible for using the Nova placement API to create corresponding traits and resources.

One of the primary responsibilities of the Cyborg conductor is to keep the placement API in sync with reality. For example if here is a device with a virtual function or a FPGA with a given program Cyborg may be tasked with changing the virtual function on the NIC or the program on the FPGA. At which point the previously specified traits and resources need to be updated. Likewise Cyborg will be watching monitoring Nova’s instances to ensure that doing this doesn’t pull resources out from under an allocated instance.

At a high level what we need to be able to do is the following

Add a PCI device to Nova’s whitelist live (config only / needs implementation)
Add information about this device to the placement API (existing / being worked)
Hotplug and unplug PCI devices from instances (existing / not sure how well maintained)

Alternatives

Don’t use Cyborg, struggle with bouncing services and grub config changes yourself.

Data model impact

N/A

REST API impact

N/A

Security impact

N/A

Notifications impact

N/A

Other end user impact

N/A

Performance Impact

N/A

Other deployer impact

N/A

Developer impact

N/A

Implementation

Assignee(s)

Primary assignee:: None

Work Items

Implementation of Cyborg service
Implementation of Cyborg agent
Blueprint for changes in Nova
Implementation of the POC which exposes functionality and interoperability between Cyborg and Nova

Dependencies

This design depends on the changes which may or may not be accepted in Nova project. Other than that is ongoing work on Nested resource providers: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html Which would be an essential feature in Placement API, which will be leveraged by Cyborg.

Testing

There would be a need to provide another gate, which would provide an accelerator for tests.

Documentation Impact

Document new nova api for whitelisting
Document developer and user interaction with the workflow
Document placement api standard identifiers

References

History

Revisions
Release Name	Description
Queens	Introduced

Policy Default Refresh

Tue, 18 Feb 2020 00:00:00

Problem description

allow
admin_only
admin_or_owner

Thirdly “admin_or_owner” sounds like it checks if the user is a member of a project. However, for most APIs we use the default target which means this rule will pass for any authenticated user.

Keystone comes with member, admin and reader roles by default. We can use these default roles: https://specs.openstack.org/openstack/keystone-specs/specs/keystone/rocky/define-default-roles.html

Use Cases

The following user roles should be supported by cyborg default configuration:

Add System Scoped Admin: to operate system-level resources
Add System Scoped Reader: read-only role for system-level resources
Add Project Scoped Reader: read-only role for project-level resources
Refresh existed admin to Project Scoped Admin by default
Add Project Scoped Member: role for project-level non-admin APIs

Cyborg V2 APIs need RBAC check. Objects of V2 APIs are listed in the following:

device_profile [3], as the “flavor” in accelerator request, is generally supposed to be a system-level resource especially for public cloud, where billing is based on the device_profile. So we reached an agreement [4] that sys_admin is required for device_profile:create and delete. As for other specific cloud, such as private cloud providers, they can change the policy by themselves if they want other users to create device_profiles.
devices and deployables [5] objects are used to describe hardware accelerators, where a device refers to the hardware and deployables are derived from a device. device:update API is introduced for sys_admin to enable or disable a specific device, while deployable:update API offers project_admin a way to update the shell image/FPGA bitstream to custom user logic for the specified deployable. NOTE that sys_admin is required to do the pre_configuration for devices.
accelerator_requests are sent by nova to bind/unbind accelerators to VMs during instance boot. Any project user who can create VM should be able to create an arq, so “member” role is required for arq:create. And for the arq:patch and arq:delete admin_or_owner should make sense.

To be clear, the roles needed for all Cyborg V2 APIs’ operations are defined in the table cyborg-policy-table.

In introducing the above new default permissions, we must ensure:

Operators using default policy are given at least one cycle to add additional roles to users (likely via implied roles)
Operators with over-ridden policy are given at least one cycle to understand how the new defaults may or may not help them

Proposed change

Add system scoped admin policy

This policy will be useful for situations where devices are shared by multiple projects, and we want a system-level admin to operator the devices like programming or firmware upgrade. In addition, a system admin is required to do the service disable/enable things.
Add system scoped reader policy

This policy will be useful for situations where a read-only role is required for a more secure access to devices that are shared by multiple projects in a system.
Add project scoped reader policy

Similarly, this policy will be useful for situations where a read-only role is required for a more secure access at a project level. For example, some users should only have the permission to check and use the resources but shouldn’t update the resources.
Add project scoped member policy

This policy will be used to check if a user is eligible to post a non-admin API such as arq creation.

POC: https://review.opendev.org/#/c/699102/, https://review.opendev.org/#/c/700765/

Alternatives

Data model impact

None

REST API impact

Operations for each API should be reassessed and associated which scope, or scopes are appropriate.

The following new roles will be added to replace legacy policies, more details can be found in the aforementioned cyborg-policy-table:

Project Reader check
- GET /v2/device_profiles
- GET /v2/device_profiles/{device_profiles_uuid}
- GET /v2/accelerator_requests
- GET /v2/accelerator_requests/{accelerator_request_uuid}
System Reader check
- GET /v2/devices
- GET /v2/devices/{device_uuid}
System Admin check
- PATCH /v2/devices
- POST /v2/device_profil
- DELETE /v2/device_profiles/{device_profiles_uuid}
- DELETE /v2/device_profiles?value={dev_profile_name1},{dev_profile_name2}
Project Admin check
- PATCH /v2/deployables/{uuid}
Project Member check
- POST /v2/accelerator_requests
- PATCH /v2/accelerator_requests/{arq_uuid}
- DELETE /v2/accelerator_requests?arqs={arq_uuid}
- DELETE /v2/accelerator_requests?instance={instance_uuid}

Note

This keeps policy files clean. For example, the following are equivalent as a result of implied roles:

“cyborg:device:get_all”: “role:reader OR role:member OR role:admin” “cyborg:device:get_all”: “role:reader”

The chain of implied roles will be documented alongside of the policy-in-code defaults in addition to general Keystone documentation updates noting as much.

Security impact

Policy defaults refresh will help keep the system secure.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Deployers will need to look through the new policies (communicated via release notes) to make sure they can adopt them.

Developer impact

New APIs must add policies that follow the new pattern.

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>

Work Items

In order to make sure existed policies run normally when every changes happen, we will follow the [6] and propose changes in the following order:

Add new roles to cyborg policy including Project-Reader, Project-Member, System-Reader, System-Admin.
Update APIs and unit tests that are using the above new roles.
Update APIs and unit tests that are using other roles such as Project-Admin, Admin_or_user etc.
Refactor cyborg policy file.

Dependencies

None

Testing

Tests for policy rules and APIs should be added. Reference RBAC test in keystone-tempest-plugin: https://review.opendev.org/#/c/686305/

Documentation Impact

API Reference should be kept consistent with any policy changes, in particular around the default reader role.

References

History

Optional section intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
Ussuri	Introduced