Watcher Specs

Migrate Watcher to Use OpenStackSDK

Wed, 04 Mar 2026 00:00:00

https://blueprints.launchpad.net/watcher/+spec/use-openstacksdk

Watcher currently uses individual Python client libraries (python-novaclient, python-cinderclient, etc.) to interact with other OpenStack services. These client libraries have been frozen or deprecated and are scheduled for removal in future OpenStack releases. This specification proposes migrating Watcher to use OpenStackSDK, the unified and officially maintained SDK for OpenStack service interactions, ensuring long-term maintainability. Additionally, this specification also proposes adding support for Watcher in the OpenStackSDK, aligning it with most OpenStack projects.

Problem description

Watcher relies on individual Python client libraries to communicate with various OpenStack services including Cinder, Keystone, Placement, and others. These libraries face several critical issues:

Deprecation and Removal: Individual client libraries are no longer actively maintained and are scheduled for removal from OpenStack. This poses a significant risk to Watcher’s continued operation.
Missing features: The individual client libraries do not support all existing functionality. Features introduced in later microversions not supported in the client library are not accessible to Watcher.
OpenStack Community Direction: The OpenStack community has standardized on OpenStackSDK as the unified interface for inter-service communication.

Use Cases

As a Watcher Developer, I want a unified, consistent API for interacting with OpenStack services, accessing their complete functionality, reducing code complexity and improving maintainability.

Proposed change

This specification proposes a phased migration from individual Python client libraries to OpenStackSDK. The migration will be executed across multiple release cycles to minimize risk and ensure thorough testing.

Phase 1: Remaining Client Migrations (2026.2)

Following the patterns established in 2026.1, migrate the remaining clients following the same procedure as done with Nova in this order:

Keystone: Migrate to connection.identity
Cinder: Migrate to connection.block_storage
Placement: Migrate to connection.placement
Ironic: Migrate to connection.baremetal

Each migration follows the same pattern:

Add Watcher wrapper objects to the calls to the clients
Add a configuration group for the service
Update helper module to use SDK proxy
Maintain existing interface for backward compatibility
Update unit tests while keeping the existing asserts to ensure no regressions

Phase 2: Watcher addition to OpenStackSDK (2026.2 or future release)

Add Watcher service support to OpenStackSDK by contributing an infra_optim service proxy. This is a longer-term goal that requires:

Submit specification to OpenStackSDK project
Implement Watcher service proxy in OpenStackSDK
Add resource models (Audit, ActionPlan, Strategy, etc.)
Implement CRUD operations

Alternatives

One possibility would be to continue using individual client libraries. This is not viable as the libraries have either stopped development or are deprecated and will be removed. This would eventually break Watcher entirely.

Another option would be to replace the calls to the client libraries with direct REST API calls to services. This would imply duplicating functionality already present in the OpenStackSDK, leading to increased maintenance burden and effort duplication.

Data model impact

None. This change does not modify Watcher’s data model or database schema.

REST API impact

None. This is an internal refactoring that does not change Watcher’s REST API.

Security impact

None. There might be a small gain by reducing the number of dependencies.

Notifications impact

None. This change does not affect Watcher’s notification system.

Other end user impact

None.

Performance Impact

None. Once all services are migrated, there might be a minimal performance gain since the number of connections will be reduced, but this will be negligible.

Other deployer impact

None. OpenStackSDK is already a dependency since the 2026.1 release.

Developer impact

Developers will need to learn OpenStackSDK patterns instead of individual client library APIs. This is a positive change as SDK provides consistent API patterns across all services and full functionality for the services integrated.

Future service integrations will use OpenStackSDK exclusively, providing a clearer pattern than mixing SDK and client libraries.

Upgrade impact

None. OpenStackSDK was added as an explicit dependency in 2026.1.

Implementation

Assignee(s)

Primary assignee:: jgilaber

Work Items

Keystone Helper Migration
- Create wrapper objects for Keystone objects like services, roles or projects
- Add a [keystone] group in the configuration that can be used to connect to the identity service
- Create keystone_helper.py using SDK identity proxy
- Update all Keystone-related methods
- Ensure backward compatibility with existing interface
- Update existing Keystone helper unit tests
Cinder Helper Migration
- Create wrapper objects for Cinder objects like volumes, volume types or volume snapshots
- Add a [cinder] group in the configuration that can be used to connect to the block storage service
- Rewrite cinder_helper.py using SDK block_storage proxy
- Update all Cinder-related methods
- Ensure backward compatibility with existing interface
- Update existing Cinder helper unit tests
Placement Helper Migration
- Create wrapper objects for Placement objects like resource providers or inventories
- Add a [placement] group in the configuration that can be used to connect to the placement service
- Rewrite placement_helper.py using SDK placement proxy
- Update all Placement-related methods
- Ensure backward compatibility with existing interface
- Update existing Placement helper unit tests
Ironic Helper Migration
- Create wrapper objects for Ironic objects like nodes
- Add a [ironic] group in the configuration that can be used to connect to the baremetal service
- Rewrite ironic_helper.py using SDK baremetal proxy
- Update all Ironic-related methods
- Ensure backward compatibility with existing interface
- Update existing Ironic helper unit tests

Dependencies

This change relies on the OpenStackSDK infrastructure established during the 2026.1 release.

Testing

Unit Testing

All existing unit tests for each helper will be updated following the same pattern established with the Nova helper in 2026.1. Tests will be updated to use SDK mocks, and their asserts will be maintained to ensure the change does not introduce any regression.

Integration Testing

The existing tempest tests should pass without modification.

Documentation Impact

None.

References

OpenStack Documentation

History

Revisions
Release Name	Description
2026.1	Introduced. Nova migration and infrastructure implemented.
2026.2	Re-proposed for Keystone, Cinder, Placement, and Ironic migration.

Migrate Watcher to Use OpenStackSDK

Fri, 27 Feb 2026 00:00:00

https://blueprints.launchpad.net/watcher/+spec/use-openstacksdk

Problem description

Watcher relies on individual Python client libraries to communicate with various OpenStack services including Nova, Cinder, Keystone, Placement, and others. These libraries face several critical issues:

Deprecation and Removal: Individual client libraries are no longer actively maintained and are scheduled for removal from OpenStack. This poses a significant risk to Watcher’s continued operation.
Missing features: The individual client libraries do not support all existing functionality. Features introduced in later microversions not supported in the client library are not accessible to Watcher.
OpenStack Community Direction: The OpenStack community has standardized on OpenStackSDK as the unified interface for inter-service communication.

Use Cases

As a Watcher Developer, I want a unified, consistent API for interacting with OpenStack services, accessing their complete functionality, reducing code complexity and improving maintainability.

Proposed change

Phase 1: Nova Client Migration + Infrastructure (2026.1)

The initial phase establishes the OpenStackSDK infrastructure and migrates the most used service in Watcher: Nova.

Infrastructure Setup

Add OpenStackSDK as a dependency in requirements.txt
Add a helper function in watcher/common/clients.py to create and manage an OpenStackSDK Connection object. This connection will be established either using a keystone session like it’s currently done with the individual clients, or using a RequestContext, authenticating via the auth_token it contains.

Nova Helper Migration

Modify the existing Nova client implementation in watcher/common/nova_helper.py so that they return Watcher objects representing compute concepts like server, hypervisor or flavor, instead of the python-novaclient objects.
Modify the nova_helper to optionally accept a Connection object, if that is not passed, it will use the helper function introduced in the previous section to create a connection.
Replace the calls to python-novaclient with calls to the Compute SDK. The methods will maintain the functionality and the interface introduced in the previous step.

Phase 2: Other Client Migrations (2026.1 or Future)

Following the patterns established in Phase 1, migrate the remaining clients following the same procedure as done with Nova in this order:

Keystone: Migrate to connection.identity
Cinder: Migrate to connection.block_storage
Placement: Migrate to connection.placement
Ironic: Migrate to connection.baremetal

Each migration follows the same pattern:

Update helper module to use SDK proxy
Maintain existing interface for backward compatibility
Update unit tests while keeping the existing asserts to ensure no regressions

Phase 3: Watcher addition to OpenStackSDK (Future)

Add Watcher service support to OpenStackSDK by contributing an infra_optim service proxy. This is a longer-term goal that requires:

Submit specification to OpenStackSDK project
Implement Watcher service proxy in OpenStackSDK
Add resource models (Audit, ActionPlan, Strategy, etc.)
Implement CRUD operations

Phase 4: python-watcherclient Integration (Future)

Update python-watcherclient to reflect watcher support in the sdk:

Deprecate python-watcherclient library in favor of the SDK
Deprecate watcher cli in favor of the OpenStackClient plugin, the plugin will stay in the python-watcherclient repository

Phase 5: Watcherclient consumers migration (Future)

Once the Watcher service is integrated in the OpenStackSDK, all consumers of the python-watcherclient should be migrated to use the new OpenStackSDK Watcher Proxy. This includes the watcher-dashboard horizon plugin and the rally-openstack project.

Phase 6: Cleanup and Deprecation (Future)

After all previous phases are completed, and once all consumers of python-watcherclient have removed its usage, the deprecated code from the python-watcherclient can be removed.

Scope of 2026.1 Implementation

This specification focuses on Phase 1 only for the 2026.1 release:

OpenStackSDK infrastructure setup
Nova client migration
Testing framework establishment

Subsequent phases will be addressed in future specifications once Phase 1 patterns are validated.

Alternatives

Data model impact

None. This change does not modify Watcher’s data model or database schema.

REST API impact

None. This is an internal refactoring that does not change Watcher’s REST API.

Security impact

None. There might be a small gain by reducing the number of dependencies.

Notifications impact

None. This change does not affect Watcher’s notification system.

Other end user impact

None.

Performance Impact

None. Once all services are migrated, there might be a minimal performance gain since the number of connections will be reduced, but this will be negligible.

Other deployer impact

None. OpenStackSDK will be added as a dependency. This is included in standard OpenStack distributions and should not require special packaging effort.

Developer impact

Future service integrations will use OpenStackSDK exclusively, providing a clearer pattern than mixing SDK and client libraries.

Upgrade impact

None. OpenStackSDK will be added as an explicit dependency. However, it should already be present in existing developments since its a dependency of python-openstackclient.

Implementation

Assignee(s)

Primary assignee:: jgilaber

Work Items

Infrastructure Setup
- Add openstacksdk to requirements.txt
- Extend OpenStackClients class with Connection property
Nova Helper Migration
- Rewrite nova_helper.py using SDK compute proxy
- Update all Nova-related methods
- Ensure backward compatibility with existing interface
- Update existing Nova helper unit tests

Dependencies

This change introduces the openstacksdk library as a dependency for watcher. It will also remove python-openstackclient as a dependency since it’s not currently used by Watcher.

Testing

Unit Testing

All existing unit tests for Nova helper will be updated to use new objects representing compute objects. After that change, when the Nova helper is moved to use the SDK, the tests will also be updated to use SDK mocks, and their asserts will be maintained to ensure the change does not introduce any regression.

Integration Testing

The existing tempest tests should pass without modification.

Documentation Impact

None.

References

OpenStack Documentation

History

Revisions
Release Name	Description
2026.1	Introduced

Example Spec - The title of your blueprint

Tue, 13 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Introduction paragraph – why are we doing anything? A single paragraph of prose that operators can understand. The title and this first paragraph should be used as the subject line and body of the commit message respectively.

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

At this point, if you would like to just get feedback on if the problem and proposed change fit in Watcher, you can stop here and post this for review to get preliminary feedback. If so please say: Posting to get preliminary feedback on the scope of this spec.

Alternatives

What other ways could we do this thing? Why aren’t we using those? This doesn’t have to be a full literature review, but it should demonstrate that thought has been put into why the proposed solution is an appropriate one.

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Note that the schema should be defined as restrictively as possible. Parameters which are required should be marked as such and only under exceptional circumstances should additional parameters which are not defined in the schema be permitted (e.g., additionalProperties should be False).

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

For more detailed guidance, please see the OpenStack Security Guidelines as a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These guidelines are a work in progress and are designed to help you identify security best practices. For further information, feel free to reach out to the OpenStack Security Group at openstack-security@lists.openstack.org.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Please discuss the important scenarios needed to test here, as well as specific edge cases we should be ensuring work correctly. For each scenario please specify if this requires specialized hardware, a full openstack environment, or can be simulated inside the Watcher tree.

Please discuss how the change will be tested. We especially want to know what tempest tests will be added. It is assumed that unit test coverage will be added so that doesn’t need to be mentioned explicitly, but discussion of why you think unit tests are sufficient and we don’t need to add more tempest tests would need to be included.

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

What is the impact on the docs team of this change? Some changes might require donating resources to the docs team to have the documentation updated. Don’t repeat details discussed above, but please reference them here.

References

Please add any useful references here. You are not required to have any reference. Moreover, this specification should still make sense when your references are unavailable. Examples of what you could include are:

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Optional section for liberty intended to be used each time the spec is updated to describe new design, API or any database schema updated. Useful to let reader understand what’s happened along the time.

Revisions
Release Name	Description
2026.2	Introduced

Example Spec - The title of your blueprint

Tue, 13 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
2026.2	Introduced

Example Spec - The title of your blueprint

Tue, 13 Jan 2026 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
2026.2	Introduced

Improving Unit and Integration Testing for Watcher Dashboard

Tue, 09 Dec 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/watcher-dashboard-unit-integration-testing

Watcher Dashboard is the Horizon dashboard plugin for Watcher. It currently has API unit tests and broken integration tests with no CI coverage. The watcher-dashboard developers have to manually pull and test every change in a browser before approving it which is time consuming and error prone. This specification proposes to improve the testing by adding unit testing for API and dashboard, and end to end integration testing for Watcher Dashboard workflows.

Problem description

Watcher Dashboard currently has API unit tests. Selenium based end to end integration tests are broken due to changes in Horizon. There are no unit tests for dashboard views and forms. There is no CI coverage for integration tests to run these tests.

Whenever a developer makes a change to the dashboard, the reviewer manually pulls the changes locally and tests it in a browser. This is time consuming and error prone. Many times issues are introduced in the code which got discovered by users in production.

Below are the current test coverage status:

API Unit Tests: Only working test type, located in test/api_tests/
Dashboard Unit Tests: No unit tests for dashboard views and forms.
Integration Tests: Broken tests in test/integration_tests/ - cannot verify UI navigation works correctly.

Use Cases

As a Watcher Dashboard Developer, I want to write unit and end to end integration tests alongside my UI features so that I can verify my changes work correctly and get automated feedback from CI.

As a Code Reviewer, I want to rely on automated test results instead of manually testing every change in a browser so that I can validate code functionality efficiently and catch regressions early.

Proposed change

Watcher Dashboard will reorganize testing so CI and reviewers get reliable automation instead of manual browser checks for every change.

The unit tests and functional tests will use mock and fixtures from Horizon repo(which uses Django’s test framework under the hood) and test data from test/test_data/.

Integration tests will test user UI interaction workflow against a live openstack deployment deployed via DevStack in OpenDev CI. The test will run in headless mode doing real api to validate the full UI workflow.

This specification adopts Playwright for integration tests because it provides built-in browser management, automatic waiting, code generation tools (playwright codegen), semantic locators, and strong debugging support, which yields simpler setup and more maintainable tests than Selenium for this plugin.

Following Manila-UI and Horizon layouts, suites will move from test/api_tests/ and test/integration_tests/ into test/unit/api/, test/unit/dashboards/, and test/integration/, reusing test/helpers.py and test/test_data/. Suites will run with stestr per the OpenStack Python PTI.

Before watcher-dashboard can depend on Playwright as a global dependency, contributors must add playwright to global-requirements.txt and upper-constraints in the requirements repository.

What remains is to complete those moves, extend test/helpers.py where Horizon patterns require it, add dashboard unit tests using fixtures such as self.audit_templates, self.goals, and self.strategies from test/test_data/watcher_data.py, maintain Playwright integration tests behind tox, and add Zuul jobs (for example a DevStack-based watcher-dashboard-integration-playwright job plus unit jobs) that publish HTML reports.

The tree and tox entrypoints below summarize layout and how suites are invoked after the work merges.

watcher-dashboard/
└── test/
    ├── helpers.py
    ├── __init__.py
    ├── unit/
    │   ├── api/                          # API unit tests
    │   │   ├── __init__.py
    │   │   └── test_watcher.py
    │   └── dashboards/                   # Dashboard unit tests
    │       ├── __init__.py
    │       ├── test_audit_templates.py
    │       └── test_strategies.py
    │
    ├── integration/                    # Playwright integration tests
    │   ├── __init__.py
    │   ├── base.py                   # Base test class with browser setup
    │   ├── test_audit_workflow.py
    │   └── test_action_plan_workflow.py
    └── test_data/
        ├── __init__.py
        ├── utils.py
        ├── exceptions.py
        └── watcher_data.py

tox -e unit-api           # API unit tests
tox -e unit-dashboard     # Dashboard unit tests
tox -e integration        # Playwright integration tests

Alternatives

Using Selenium for integration testing. The legacy integration tests used Selenium, but were broken and difficult to maintain. Alternative frameworks were evaluated through POC implementations.

Selenium: Requires separate WebDriver installation and version management, needs explicit waits leading to flaky tests, lacks built-in code generation, relies on fragile XPath/CSS selectors, and requires additional CI dependencies.

Playwright: Automatically manages browser binaries, includes built-in auto-wait reducing flakiness, provides playwright codegen for test generation, uses maintainable semantic locators (role, label, text), has simpler CI integration, and offers trace viewer for debugging.

Given watcher-dashboard’s need for reliable and maintainable tests, Playwright is the better choice. Rejected in favor of Playwright.
Pytest-based suites matching Horizon or manila-ui. Those projects use Pytest with Selenium for parts of their UI testing; porting watcher-dashboard the same way would mirror them but conflicts with the OpenStack Python PTI preference for stestr and standard unittest patterns. Rejected.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None for production.

Other deployer impact

None.

Developer impact

Positive impacts: Automated UI testing, less manual browser testing, clearer test layout, stronger regression coverage for UI changes.
Learning curve: Contributors learn Playwright for integration tests and Horizon/Django unit patterns; docs supply guidelines and examples.
Code review expectations: New UI features ship with integration tests; dashboard views and forms gain unit tests; reviewers skip full manual browser runs on every change.
Testing requirements: Going forward, new features added to watcher-dashboard should include integration tests to validate UI workflows. This ensures consistent test coverage and prevents regressions.

Upgrade impact

None.

Implementation

Assignee(s)

Primary assignee:

chandankumar

Other contributors:

Work Items

Restructure test directory layout following the proposed structure
Move existing API tests from test/api_tests/ to test/unit/api/
Move integration tests from test/integration_tests/ to test/integration/
Update test/helpers.py to extend Horizon’s test infrastructure
Add unit tests for dashboard views and forms
Implement and maintain Playwright-based integration tests and tox targets
Add playwright to global requirements and upper-constraints where needed if accepted otherwise keep playwright to test-requirements.txt till other horizon dashboard plugins start adopting it.
Add Zuul jobs for new test targets with HTML reporting
Create contributor testing documentation

Dependencies

Existing (from Horizon):

django.test - Django’s test framework
unittest.mock - Python standard library

New:

Playwright: playwright (must land in global requirements before hard dependency in watcher-dashboard)
stestr: stestr
ddt: ddt

Testing

All test types will be validated by successful CI runs.

Documentation Impact

doc/source/contributor/testing.rst - Testing guide
Updates to HACKING.rst with test conventions

References

History

Revisions
Release Name	Description
2026.2	Introduced - Improve testing for Watcher Dashboard

Code Quality Improvements and Test Infrastructure for Dashboard

Wed, 08 Oct 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/dashboard-code-quality-testing

Improve code quality and test infrastructure for Watcher Dashboard by eliminating patterns that hinder static analysis, adopting centralized configuration management, and restructuring tests to support reusable fixtures following the pattern established in the main Watcher project.

This work addresses technical debt in the Watcher Dashboard codebase that impedes maintainability and developer productivity. The changes will improve IDE support, enable better static analysis, and create a more maintainable codebase for future development.

Problem description

Watcher Dashboard currently uses code patterns that create robustness issues and reduce code quality:

Direct symbol imports (from X import Y) create fragile dependencies that can break during refactoring and make it difficult to understand module relationships
Reflection helpers (getattr/hasattr/setattr) used for static attributes hide potential AttributeError bugs and make code behavior unpredictable
Django settings accessed directly via getattr() throughout codebase with scattered defaults and no validation, creating silent failures and configuration drift
Test infrastructure lacks reusable fixture pattern, leading to inconsistent test setup and potential test pollution
No test structure to accommodate future functional testing with playwright
Inconsistent code patterns make the codebase harder to review and maintain

These patterns reduce code robustness by hiding errors, creating fragile dependencies, and making the codebase harder to understand and maintain.

Use Cases

As a Watcher Dashboard Developer, I would like to refactor code with confidence that errors will be caught early rather than hidden by reflection helpers, so that I can make changes safely without introducing subtle bugs.
As a Watcher Core Reviewer, I would like to review code changes efficiently without getting distracted by inconsistent patterns or debugging test issues, so that I can focus on the actual functionality being implemented.
As a New Contributor, I would like to understand the codebase quickly and write code that follows established patterns, so that I can contribute effectively without extensive mentoring on style and structure.

Proposed change

Eliminate code patterns that hinder static analysis and modernize the test infrastructure to align with established OpenStack practices. The changes will improve IDE support, enable better static analysis, and create a more maintainable codebase.

The primary focus is converting direct symbol imports to module-level imports throughout the codebase, replacing static reflection helpers with direct attribute access, and centralizing Django settings access through a typed configuration module. Test infrastructure will be restructured to support reusable fixtures following the pattern used in the main Watcher project, with tests organized in watcher_dashboard/test/unit/ and new local_fixtures/ and playwright/ directories for future testing needs.

A centralized configuration module (watcher_dashboard/config.py) will provide typed, validated access to Django settings, eliminating scattered getattr(settings, ...) usage throughout the codebase. This module will be inspired by manila-ui’s features.py pattern but enhanced with type hints and comprehensive validation. CI enforcement will be added to prevent regression of these patterns, and comprehensive contributor documentation will be created to guide future development practices.

Alternatives

Status Quo: Continue with current patterns. Rejected - technical debt accumulates as new code follow existing anti-patterns.

Skip Test Restructure: Keep flat test structure. Rejected - misses opportunity to align with main Watcher project patterns and prepare for future functional testing with playwright.

Data model impact

None.

REST API impact

None.

Security impact

Positive impact: centralized configuration with validation reduces risk of misconfiguration. Explicit imports improve code audibility. No new security concerns introduced.

Notifications impact

None.

Other end user impact

None. These are internal improvements transparent to dashboard users.

Performance Impact

Negligible positive impact: @memoized.memoized decorator on config functions provides efficient caching of settings lookups. Module-level imports resolve once rather than on each use.

Other deployer impact

None.

Developer impact

Significant positive impacts:

Improved IDE support: Module-level imports enable accurate autocomplete, go-to-definition, and refactoring across all modern Python IDEs
Static analysis: Type-hinted config module and reduced reflection enable mypy checking and catch bugs before runtime
Easier testing: Fixture pattern reduces boilerplate; config module functions are easier to mock than settings access
Better documentation: Centralized config module serves as living documentation of available settings with docstrings

Implementation

Assignee(s)

Primary assignee:

sean-k-mooney

Other contributors:

None

Work Items

Refactor import patterns to use module-level imports throughout the codebase
Replace static reflection helpers with direct attribute access
Create centralized configuration module for Django settings access
Restructure test directory to support reusable fixtures and future testing
Add CI enforcement for import and reflection policies
Create comprehensive contributor documentation for code quality standards
Implement optional type hints with mypy configuration

See detailed implementation plan: https://gist.github.com/SeanMooney/f56c7fd6f55ac48958a5c549e1701b6c

Dependencies

No new runtime dependencies.

Test dependencies extended using existing libraries proven in OpenStack:

fixtures>=3.0.0 - Reusable test fixture pattern (already in use in Nova, Placement, and main Watcher project)

All other dependencies already in use:

horizon.utils.memoized - Configuration caching (in use)
django.conf.settings - Django settings (in use)
unittest.mock - Test mocking (in use)

Testing

The centralized configuration module will require comprehensive unit test coverage to ensure proper validation and error handling. Documentation will be validated through Sphinx builds and RST syntax checking.

The refactoring of the test structure will enable functional testing with playwright to be added which will eventually be tested with a new zuul job. that is out of scope for this spec.

Documentation Impact

Comprehensive contributor documentation will be created to establish and maintain the new code quality standards. Two new documentation files will be added: doc/source/contributor/code_quality.rst containing policies and standards for import patterns, configuration access, reflection usage, and testing practices, and doc/source/contributor/code_patterns.rst providing concrete examples and common anti-patterns to avoid.

References

Detailed implementation plan: https://gist.github.com/SeanMooney/f56c7fd6f55ac48958a5c549e1701b6c
Functional Test Infrastructure blueprint: https://blueprints.launchpad.net/watcher/+spec/functional-test-infrastructure
Manila-UI features.py - Config module pattern inspiration
Python fixtures library - Test fixture pattern used across OpenStack
OpenStack Testing Standards - Import ordering and style
PEP 484 - Type Hints
PEP 585 - Type Hinting Generics (3.10+ syntax)

History

Revisions
Release Name	Description
2026.1	Introduced

Functional Test Infrastructure for Watcher

Tue, 07 Oct 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/functional-test-infrastructure

Introduce functional testing infrastructure to Watcher, enabling testing of multi-component workflows with minimal mocking. This improves test coverage for complex service interactions (API, decision engine, applier) and provides regression protection for bugs requiring integration testing to reproduce.

Problem description

Watcher currently has excellent unit test coverage but lacks functional tests that verify multiple components working together. This gap makes it difficult to:

Catch integration bugs that unit tests miss due to extensive mocking
Test real workflows (audit → strategy → action plan → applier)
Validate RPC interactions between services with real message passing
Reproduce complex bugs that require service interaction
Test API behavior with real WSGI application
Verify notification emission in realistic scenarios

Unit tests with heavy mocking can give false confidence - tests pass but the integrated system fails in production. Functional tests fill this gap by testing real code paths with minimal mocking.

Use Cases

As a Watcher Developer, I want to write functional tests so that I can verify my changes work correctly when multiple components interact, without needing a full OpenStack deployment.

As a Watcher Core Reviewer, I want functional tests for complex bug fixes so that I can be confident the bug is truly fixed and won’t regress.

As a Watcher Operator, I benefit from functional tests catching integration bugs before release, resulting in more stable deployments.

As a Watcher Contributor, I want clear examples and documentation for functional testing so I can add tests for my features without deep knowledge of the test infrastructure.

Proposed change

Implement functional testing infrastructure adapted from OpenStack patterns (Nova, Placement) for Watcher’s architecture (Pecan API, decision engine, applier services).

The infrastructure provides:

Reorganized test structure: watcher/tests/unit/ and watcher/tests/functional/
Reusable fixtures for database, RPC, external services, API server
WatcherFunctionalTestCase base class with automatic fixture setup
Gabbi (YAML) tests for declarative API contract testing
Regression test framework with documentation
Contributor guide for functional testing

The implementation follows “minimize mocks, maximize reality” - functional tests use real Watcher code (API, database, RPC) and only mock external services (Nova, Gnocchi).

Key Design:

SQLite in-memory with schema caching for speed
oslo.messaging fake driver for deterministic RPC
wsgi-intercept for in-process HTTP (no network I/O)
Python and Gabbi (YAML) test formats
Reuse existing helpers (watcher/tests/db/utils.py)

Directory Structure:

watcher/tests/
├── unit/                      # All existing tests (moved)
│   ├── api/
│   ├── decision_engine/
│   ├── applier/
│   └── ...
├── functional/                # New functional tests
│   ├── base.py               # WatcherFunctionalTestCase
│   ├── test_api_audits.py    # Python functional tests
│   ├── test_workflows.py     # End-to-end workflow tests
│   ├── test_api_gabbi.py     # Gabbi test loader
│   ├── fixtures/             # Gabbi-specific fixtures
│   │   ├── gabbi.py
│   │   └── capture.py
│   ├── gabbits/              # Gabbi YAML tests
│   │   ├── audit-lifecycle.yaml
│   │   ├── microversions.yaml
│   │   └── ...
│   └── regressions/          # Bug-specific tests
│       └── test_bug_*.py
├── local_fixtures/            # Shared fixtures
│   ├── conf.py               # Configuration fixture
│   ├── database.py           # Database fixture
│   ├── rpc.py                # RPC fixtures
│   ├── notifications.py      # Notification capture
│   ├── api.py                # API server fixture
│   ├── service.py            # Service fixture
│   ├── nova.py               # Nova mock
│   └── gnocchi.py            # Gnocchi mock
├── fixtures/                  # Existing test fixtures
│   └── watcher.py
└── helpers.py                 # Test helper functions

Test Execution:

tox -e py3                    # Unit tests
tox -e functional             # All functional tests
tox -e functional-regression  # Regression tests only

Alternatives

Full Integration Tests Only: Deploy real external services via DevStack. Rejected - too slow (30+ min), complex setup, network flakiness.

Unit Tests Only: Rely on existing unit tests and Tempest. Rejected - misses integration bugs, complex scenarios hard to reproduce.

Docker Containers: Run services in containers. Rejected - adds complexity, slower, doesn’t align with OpenStack patterns.

Tempest Only: Use only Tempest for integration. Rejected - too slow for regular development, doesn’t replace functional tests.

Data model impact

None. Functional tests use the existing data model. The database fixture applies existing migrations to create the schema.

REST API impact

None. Functional tests validate existing API behavior. The Gabbi test infrastructure will improve API testing coverage and make microversion testing easier, but does not change the API itself.

Security impact

None.

Notifications impact

None. Functional tests validate existing notification behavior. The notification fixture captures emitted notifications for verification but does not change notification emission logic.

Other end user impact

None.

Performance Impact

None

Other deployer impact

None.

Developer impact

Positive impacts: easier end to end tests, better regression protection, clearer test organization, lower barrier for API testing.

Learning curve: developers learn when to use functional vs unit tests, documentation provides guidelines and examples.

Code review expectations: functional tests for complex bug fixes, multi- component workflows, API changes (Gabbi tests), and RPC features.

Implementation

Assignee(s)

Primary assignee:

sean-k-mooney

Other contributors:

None

Work Items

Extract existing fixtures and create test helpers
Reorganize tests: move to unit/, create functional/
Core fixtures: database, RPC, notifications, config
Service fixtures: Nova, Gnocchi mocks
API fixture: Pecan with wsgi-intercept
Service fixture: decision engine, applier
Base test class: WatcherFunctionalTestCase
Gabbi infrastructure: loader, fixtures, YAML examples
Example tests: API tests, workflow tests
Regression framework: directory structure, guidelines, README
Documentation: contributor guide for functional/Gabbi tests
CI integration: tox.ini, .zuul.yaml, stestr config

See detailed implementation plan: https://gist.github.com/SeanMooney/43afa55282d2286a312eae7f3c7709e2

Dependencies

No new runtime dependencies.

test dependencies will be extended using existing libraries: used in nova and placement.

gabbi - YAML HTTP testing (in requirements)
wsgi-intercept - In-process HTTP (in requirements)
oslo.messaging - RPC with fake driver (in use)
oslo.db - Database fixtures (in use)
oslo.config - Configuration management (in use)
oslo.log - Logging (in use)
oslo.policy - Policy enforcement (in use)

Testing

Validation approach: each phase validated before proceeding. Unit test suite validates reorganization, fixture tests validate cleanup/behavior a zuul job will be added to run the functional tests once a minium test case is written and will self validate the functional tests as they are expanded.

Documentation Impact

New documentation:

Contributor guide for functional testing (doc/source/contributor/functional-testing.rst): covers functional vs unit tests, Python and Gabbi test writing, fixture usage, running/debugging, best practices
Regression testing guide (watcher/tests/functional/regressions/README.rst): when to write regression tests, naming conventions, structure requirements
API documentation: Gabbi tests serve as executable examples

Updated documentation:

Testing guidelines: add functional vs unit test section
README.rst: update with functional test commands

References

Implementation Details:

Detailed implementation plan: https://gist.github.com/SeanMooney/43afa55282d2286a312eae7f3c7709e2
How Nova functional tests work: https://gist.github.com/SeanMooney/0bc41721481dd6e5918a4504c956f882
How Gabbi tests work in Placement: https://gist.github.com/SeanMooney/36f2ef20bd5cb4c853af23c581b917fc

OpenStack References:

History

Revisions
Release Name	Description
2026.1	Introduced

Add a mechanism to skip Watcher Actions when running an action plan

Mon, 08 Sep 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/add-skip-actions

The Watcher service should allow to exclude actions from an Action Plan execution in two situations:

A cloud admin explicitly wants to exclude an action or set of actions from execution before starting an action plan.
Watcher applier detects the object where the action is applied to be in a state where the action should not be applied. For example, in a migrate action, the instance does not longer exist.

Problem description

Currently Watcher does not provide a way to exclude actions execution when running an action plan:

Administrators lack a standardized mechanism to preemptively skip actions.
Pre-condition failures are not clearly distinguished from execution failures.

Note that Actions already include the CANCELLED state in its state machine however, it is only used for two specific cases:

When an Action Plan is cancelled, all the actions included are moved to the CANCELLED state.
When the watcher-applier process is started, all the actions assigned to it in ONGOING state are moved to CANCELLED state (see this link ).

The purpose of this spec is to document the required changes in Watcher to implement the Actions execution exclusion in the two mentioned use cases.

Use Cases

Admin Override: Administrators can use a patch API call to move an action state from PENDING to SKIPPED, so the action will not be executed when the action plan is executed.
Pre-condition detection: When the pre_condition method of an action returns a specific type of Exception, the action will be moved to SKIPPED and execution and post_conditions will not be executed.

In both cases, a status_message may be provided and stored in the database. Similarly a status_message field will be added to the ActionPlan object that will be used to provide information about skipped actions. For consistency the status_message will be also added to the Audit object which will be used in future to store information related to status changes.

Proposed change

Currently, the watcher state machine for Actions from PENDING state provides following transitions:

PENDING -> CANCELLED (action plan is cancelled before action is started)
PENDING -> ONGOING (start action execution)
ONGOING -> CANCELLED (action plan is cancelled while action is running)
ONGOING -> SUCCEEDED (action finishes successfully)
ONGOING -> FAILED (action execution failed)

The proposed change in this spec is to add a new state SKIPPED which would imply that an action is being excluded from an Action Plan execution for any of the mentioned use cases.

After this change, additionally, the state machine for Actions will allow following state transition:

PENDING -> SKIPPED (action is excluded from execution before started)
PENDING -> FAILED (action failed in pre-condition with unexpected error)

Note that the transition from PENDING to SKIPPED can be triggered by the cloud administrator using an API call or automatically when watcher detects certain predefined conditions in pre_condition execution. Transitioning an action from PENDING to ONGOING should imply that pre_condition execution finishes successfully. An action which fails in pre_condition execution with any error different that the predefined conditions will change the state to FAILED before moving it to the ONGOING state.

Actions in SKIPPED state will not affect the final state of the action plan. An action plan will only be considered to be FAILED when one or more actions are in FAILED state when execution finishes. However, in order to provide the user with details about the actual result of the execution, a new field will be added to the action plan status_message which will be updated when one or more actions are skipped with relevant information about the skipped actions.

For the pre-condition detection use case, a new class of Exception ActionSkipException will be created. When pre_condition raises it, the ongoing action will be switched to SKIPPED state, and status_message will be added based on the exception message.

For the admin override use case, a new api call Patch method will be added to the Actions which will allow to change the state from PENDING to SKIPPED state only when the parent action plan is also in the PENDING state. In that case, a comment skipped by user will be included in the status_message. Optionally, the user will be able to add further details to the message.

Additionally, support for the changes in the API should be included in the python-watcherclient and watcher-dashboard so that the cloud admins can skip an action from PENDING state and see the status_message value for SKIPPED actions.

Microversion of the Watcher API will be increased for this change as follows:

The new action state SKIPPED will not be handled as microversioned. This means that the value of the action state will be identical, independently of the Watcher API microversion used.
The new field status_message will be included in the new microversion both in the Action, ActionPlan and Audit objects, so it will only be included in the corresponding API calls when using the new microversion or higher. When using previous ones, the new field will not be included in API responses.
New patch API calls will only be available when using the new microversion or newer.

Alternatives

During the review of this spec, multiple aspects were discussed and decisions made. The most important aspects were:

SKIPPED vs reuse CANCELLED

Instead of creating a new SKIPPED state, we may reuse the existing CANCELLED state to cover the two new use cases. After discussing this option on irc <https://meetings.opendev.org/irclogs/%23openstack-watcher/%23openstack-watcher.2025-05-13.log.html#openstack-watcher.2025-05-13.log.html#t2025-05-13T12:53:54>_. we have considered that it would lead to ambiguity in the actual situation for CANCELLED actions, as multiple and pretty different situations may lead to it.

With this proposal, the SKIPPED state is restricted to be used for actions which were never actually executed in executed action plans.

Using PATCH vs PUT or POST API calls

An alternative approach to the proposed changes in the API in this spec was to use POST calls on /v1/action_plans/{plan_id}/actions/{action id or index}. While this would follow the best practices in terms of REST API implementation and would be more correct to show actions as internal to ActionPlans, the approved implementation is more consistent with the existing API methods in Watcher and the current representation of the Actions in the API as top level elements.

ActionPlan state when Actions are skipped

This spec proposal does not modify the state of a finished ActionPlan when any of the actions included on it is skipped by user or pre_condition failure. Additionally, the status_message field will be used to make users aware of actions were skipped if so, even when the ActionPlan is in SUCCEEDED state as it has been considered that it is a relevant information for the users.

Alternatives to this approach would be to create a specific field skipped_actions in the ActionPlans to provide specifically that information or to create a new state SUCCEDED_WITH_SKIPPED which would be use for ActionPlans with automatically skipped actions.

However, it has been considered that using the new generic field status_message is more appropiate and reusable to provide further details in other situations in future (i.e. for FAILED actions) and the SUCCEEDED state to represent correctly the real state when an action is not executed as specifically decided by the user or automatically by watcher detecting a condition which has been predefined to lead the action to SKIPPED instead of FAILED state.

Adding status_message to Audits

Although the audits state is not directly affected by the proposed use cases, it has been determined that the status_message field can be useful for the Audits too. It has been decided to include this field in the same API microversion for consistency among the API objects and clients.

Data model impact

Add a new column status_message of type String(255) in Actions table.
Add a new column status_message of type String(255) in ActionPlans table.
Add a new column status_message of type String(255) in Audits table.

REST API impact

In a new microversion the following API responses are extended:

New field status_message will be included in actions:
- GET /v1/actions/detail
- GET /v1/actions/``{action_id}
No changes in Return code(s).

Example of json addition in GET /v1/actions/``{action_id} response:
```
{
    "state": "SKIPPED",
    "description": "Logging a NOP message",
    "status_message": "Skipped by user",
    ....
}
```
New field status_message will be included in actionplans:
- GET /v1/actionplans/detail
- GET /v1/actionplans/``{action_id}
No changes in Return code(s).

Example of json addition in GET /v1/actionplans/``{actionplan_id} response:
```
{
    "state": "SUCCEEDED",
    "status_message": "Action XXX was skipped by user",
    ....
}
```
New field status_message will be included in audits:
- GET /v1/audits/detail
- GET /v1/audits/``{audit_id}
No changes in Return code(s).

Example of json addition in GET /v1/audits/``{audit_id} response:
```
{
    "state": "CANCELED",
    "status_message": "Audit was canceled by user",
    ....
}
```
A new Patch method on /v1/actions/``{action_id} will be added to skip an Action in PENDING state.

Normal response codes: 200

Error codes: 400,409,404

Example Action PENDING skipping request:
```
[
    {
        "op": "replace",
        "value": "SKIPPED",
        "path": "/state"
    },
    {
        "op": "add",
        "value": "Exclude migration of intance foo",
        "path": "/status_message"
    },
]
```
Trying to patch an unexisting Action will return a 404 error.

Request to skip an action using Patch API method in any state different that PENDING will return a 409 error.

The API Patch call will allow to modify the status_message field for an action which is in SKIPPED state.

Security impact

No security impact.

Notifications impact

The action, actionplan and audit notifications will be extended to contain the newly added field.

Other end user impact

Following changes will be implemented in the watcherclient:

New field status_message will be included in command:
```
openstack optimize action show <action id>
```

New option skip will be added to the optimize action command:

openstack optimize action update --state skipped --message <message> <action id>

Similar functionalities will be implemented in the watcher-dashboard package to perform the same actions and get similar information from the horizon dashboard.

Performance Impact

No performance impact expected.

Other deployer impact

No new configuration parameter or any other impact on deployer

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: amoralej

Work Items

Add a new field to the Actions table.
Add a new field to the Actions GET REST API
Add a new exception type and move ongoing actions to SKIPPED state if pre_condition raises it.
Add a new patch method to the actions api to skip pending actions.
Add support in watcherclient.
Add support in watcher-dashboard. * Include the new field status_message in the Actions and ActionPlans views. * Add a button to skip actions from the Related Actions in the ActionPlans detailed view. * Additionally, a skip button may be also added to the action detailed view.

Dependencies

None

Testing

Existing unit and API tests will be extended to validate that the new microversion contains status_message.

Unit tests will validate that an action raising the new exception in the pre_condition, moves the action to SKIPPED state.

Unit tests to test new patch methods on actions.

New tempest test to test the action skipping feature, including the optional field status_message.

Documentation Impact

API Reference
REST API Version History
watcher client

References

History

Revisions
Flamingo	Description
2025.2	Introduced

Extend Compute Model Attributes

Mon, 08 Sep 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/extend-compute-model-attributes

Problem description

When creating a solution to achieve a goal, Watcher commonly builds an action plan composed of multiple actions, which are mostly based on Server Migrations. These computed migrations are not always valid from Nova’s point of view since they can violate some placement constraints defined for a server, which Watcher is not aware of. Watcher could be improved to include additional constraints and other compute resources in its models, to assist strategies on building action plans. This blueprint proposes extending Watcher’s compute model to include additional compute attributes.

Use Cases

As an OpenStack Admin I would like to optimize my deployment, balancing the workload across compute nodes, without breaking any placement constraints associated with its instances like: flavor extra specs, affinity and anti-affinity policies, pinned availability zones, etc.

Proposed change

This blueprint proposes to extend the current compute model to include additional instance’s attributes, available in the new API versions of GET /servers/detail API:

Pinned Availability Zone: defines that an instance should be pinned to an availability zone and this parameter should also be considered when migrating it.
Flavor Extra Specs: specs that can include hardware requirements, resource allocation policies, and custom placement rules that influence how servers are scheduled across the infrastructure.

Note

If the configured compute API microversion is lower than the required ones, these new attributes will be left as empty, and strategies will not take the advantage of this proposed extension.

Note

The amount of new attributes is limited by Wacther’s maximum supported compute API microversion. Since Watcher’s Nova Collector implementation is based on the usage of the python-novaclient library, the maximum supported compute API microversion is limited to the same as the one supported by the library (which is the 2.96 for now). This limitation can be overcome by changing the implementation to use the openstacksdk library instead, but this is not in the scope of this specification.

A new configuration option will be added to the collector, to allow Admin to enable or disable the collection of these additional attributes. It will be set to disable by default, but can be changed in the next cycles once Strategies start to consume its content.

The API microversion will increased in favor of the new attributes to be added in the response body of GET /v1/data_model call.

Alternatives

In order to improve the host selection, Watcher could rely on Nova to provide the list of valid destination hosts for a server migration. However, this API does not exist in Nova and implementing it can also be a challenge, since Nova’s Scheduler is not ready to answer to these requests, without reserving the resources associated with them. This alternative would also increase the amount of API requests to Nova, since some strategies calculates multiple solutions before selecting the best one.

Data model impact

New fields will be added to the Instance Element class, which is an element of the Nova Cluster Data Model. The new fields are the following:

"server_pinned_az": wfields.StringField(),
"flavor_extra_specs": wfields.JsonField(),

REST API impact

A new API microversion will introduce new attributes to the response body of the GET /v1/data_model method. Example of the new attributes in the response json:

{
    "server_pinned_az": "us-west",
    "server_flavor_extra_specs": {
      "hw:watchdog_action": "reset",
    },
    ....

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

A minimal impact is expected since the in-memory model will be extended and have more fields. A configuration option will allow users to disable extended Instance fields, reducing the impact to almost zero.

Other deployer impact

None.

Developer impact

When developing a new Strategy or updating an existing one, developers can consider these new constraints when selecting a destination host for a server migration.

Implementation

Assignee(s)

Primary assignee:: dviroel

Work Items

Extend Nova Cluster Data Model to include new Instance attributes.
Update the Nova Collector to parse additional Instance attributes.
Extend Nova Notifications processing to update new attributes.

Dependencies

Some attributes are only available in newer versions of Nova’s API:

Flavor Extra Specs: microversion 2.47 (already supported)

Pinned Availability Zone: microversion 2.96

To achieve better results, it is expected that deployed Nova supports most of the above microversions.

Testing

Unit tests will cover different scenarios when collecting data from Nova service. A new tempest test will create instances with additional attributes and validate that this info is available in the new model. A devstack job will be modified to enable the additional attributes collection and run new tempest tests.

Documentation Impact

Update documentation that mention model collectors, to include information about additional attributes in newer microversions.

References

None.

History

Revisions
Release Name	Description
2025.2	Introduced

Host maintenance strategy disable migration

Mon, 08 Sep 2025 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/host-maintenance-strategy-disable-migration

Problem description

Host maintenance is a migration strategy designed for maintaining a compute node. It triggers either live or cold migration for all instances on the node, assuming that both migration methods are available. However, this may not apply to deployments where live or cold migration is not supported.

Use Cases

When using host maintenance strategy:

As a Cloud Administrator, if live migration is supported in my OpenStack deployment. I want to apply live migration to the instance.
As a Cloud Administrator, if live migration is not supported in my OpenStack deployment, but cold migration is supported. I want to apply cold migration to the instance.
As a Cloud Administrator, if both live migration cold migration are not supported in my OpenStack deployment. I want to stop the instance.

Proposed change

Estimated changes are going to be in the following places:

Host maintenance strategy
- Input parameters disable_cold_migration and disable_live_migration to disable the migration.
- If disable_live_migration is given:
  - This tell the strategy the live migration shouldn’t be considered during planning.
  - The instances in active status will be stopped.
  - If disable_cold_migration is not given, migrate all the SHUTOFF instances, including the instances stopped by previous operation.
- If disable_cold_migration is given, this tell the strategy the cold migration shouldn’t be considered during planning.
New stop actions in applier
- Action to stop the instance

Alternatives

None

Data model impact

None

REST API impact

Two new input parameters for host maintenance strategy.

Security impact

None

Notifications impact

None

Other end user impact

Two new input parameters for host maintenance strategy, the behavior is expected as the same if no parameters are provided, so no breaking change.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <jneo8>

Work Items

New applier action to stop the instance.
Modify the function which creates migration action for instance.

Dependencies

https://specs.openstack.org/openstack/watcher-specs/specs/queens/approved/cluster-maintenance-strategy.html

Testing

Unit tests on the Watcher Decision Engine and Applier.
Integration tests
- Launch an audit with the disable_live_migration input parameter enabled.
- Launch an audit with the disable_cold_migration input parameter enabled.
- Launch an audit with both disable_live_migration, disable_cold_migration input parameters enabled.

Documentation Impact

Need to update Host Maintenance Strategy documentation.

References

None

History

None

Support multitenancy with the Prometheus datasource through Aetos

Mon, 08 Sep 2025 00:00:00

launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/prometheus-multitenancy-support

The telemetry team is working on adding multi-tenancy capabilities and Keystone based authentication for metrics stored in Prometheus by developing the Aetos reverse-proxy. Support for Aetos needs to be added into watcher.

Problem description

Currently Watcher supports Prometheus as a metric datasource, with optional basic auth or mtls. But Prometheus doesn’t support multi-tenancy or rolebased access control so common througout the openstack services. To address this, the telemetry team is working on a reverse-proxy called Aetos to allow using keystone authentication and roles for multi tenancy similarly to how it’s currently being done with Gnocchi. Watcher currently doesn’t support communicating with Aetos and so Watcher needs to be enhanced to support Aetos as a new datasource

Use Cases

As a deployer, I would like to provide more security for stored metrics by disabling outside access to Prometheus and implementing RBAC via the Aetos Reverse proxy server.

Proposed change

Description of Aetos from Watcher’s point of view:

Aetos is a reverse-proxy, which is deployed in front of Prometheus. Aetos exposes a subset of Prometheus’s http API and requiring and validating keystone token with each request. The token is inspected for the roles of the user sending the request. It’s expected from Watcher to attach a token with either a admin or service role. If this happens correctly, then each request is forwarded unchanged to Prometheus and Prometheus’s response is forwarded without changes back to Watcher. The complete PromQL language is supported and all APIs required by Watcher are supported as well. It’s expected for Aetos to always have an endpoint registered in keystone with service type ‘metric-storage’.

The needed changes to add the support are these:

A new Aetos datasource will be added. The current PrometheusHelper class will be turned into a baseclass or a mixin. This will be shared between the PrometheusDirect and Aetos datasources. The __init__ and _setup_prometheus_client methods will be implemented in the Aetos datasource class and will use keystone endpoints to get Aetos’s endpoints instead of using the host and port values from the configuration. The new data souce will preserve the 1:1 mapping between datasources and python classes while ensuing the query logic is implemented in the base class.

A new configuration section for Aetos, will be added which would have options to confiugre keystone endpoint discoveray and the lables to used for host and instance metrics.

For the PrometheusDirect datasource, we will use the current implementation, which uses the host and port values from the configuration.

When inialising the PrometheusAPIClient the Aetos datasource, will constuct a keystone session and provide it to the PrometheusAPIClient object.

If the aetos service is not registered in Keystone, the Aetos datasource will exit with an error, which prevents the decision engine from starting. Configuring the Prometheus and Aetos datasources at the same time will be considered a configuration error.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

We expect an increase in security on the metric storage side, which is the whole purpose of the change. Authentication will be required for metric access

Notifications impact

None

Other end user impact

None

Performance Impact

There is a slight performance impact. Since Aetos acts as a reverse proxy in front of Prometheus, there is an additional step for each request in comparison with sending requests directly to Prometheus. So a slightly longer delay for each request is expected.

Other deployer impact

The configuration option ‘watcher_datasources.datasources’ will have a new valid value: ‘aetos’

There will be a new configuration section called ‘Aetos’ with the following options:

interface

Type:
string

Default:
public

Valid Values:
internal, public, admin

Type of endpoint to use in keystoneclient.

region_name

Type:
string

Default:
<None>

Region in Identity service catalog to use for communication with the OpenStack service.

fqdn_label

Type:
string

Default:
fqdn

The label that Prometheus uses to store the fqdn of exporters. Defaults to ‘fqdn’.

instance_uuid_label

Type:
string

Default:
resource

The label that Prometheus uses to store the uuid of OpenStack instances. Defaults to ‘resource’.

Developer impact

None

Implementation

Assignee(s)

jwysogla

Work Items

Use newer python-observabilityclient
Convert the current PrometheusHelper class into a base class or a mixin. The __init__ and _setup_prometheus_client methods are to be implemented in classes that inherit from it.
Create a class for implementing the Prometheus datasource, which inherits from the base class and uses the current implementation of the __init__ and _setup_prometheus_client methods.
Add a configuration section for Aetos datasource, with the following options “interface”, “region_name”, “fqdn_label”, “instance_uuid_label”. Descriptions and values should be the same as in existing clients config sections.
Add Aetos as a possible datasource
Create a class for implementing the Aetos datasource, which inherics from the Prometheus base class/mixin, but implements different __init and _setup_prometheus_client methods.
The Aetos datasource class uses keystone endpoints to get Aetos’s endpoints.
The Aetos datasource class creates a PrometheusAPIClient similarly to how it’s already being created for prometheus, but also specifies a keystone session and a root_path, which is extracted from Aetos’s endpoint.
Add a tempest job similar to watcher-prometheus-integration, which will be configured to use the new Aetos datasource instead of Prometheus directly

Dependencies

python-observabilityclient 1.0.0 or newer

Testing

The current watcher-prometheus-integration job can be duplicated and modified slightly to be deployed with Aetos, using Aetos’s devstack plugin.
Unit tests would be added as well

Documentation Impact

Add a new Aetos datasource section in the documentation. The contents should be fairly similar to the current Prometheus documentation. The differences would be mostly in the configuration options used. In comparison to Prometheus, there is no need for “host”, “port” or any authentication or TLS options. But instead we will need to know the “interface” and “region_name” as with most of other clients.

A mention will be added, that having a Prometheus and Aetos datasources at the same time isn’t supported and ends with a configuration error.

References

A bug with a related discussion: https://bugs.launchpad.net/watcher/+bug/2108855
The whole effort for adding tenancy and authentication support for Prometheus was discussed during the telemetry PTG https://etherpad.opendev.org/p/apr2025-ptg-telemetry

History

Revisions
Release Name	Description
2025.2	Introduced

Example Spec - The title of your blueprint

Mon, 08 Sep 2025 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
2026.1	Introduced

Add Show Datamodel API

Wed, 16 Apr 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/show-datamodel-api

Problem description

The datamodel is very important for Watcher to generate resource optimization solutions. Currently, it can only be found by looking at the log file, which is very inconvenient. Therefore, it is necessary to add an api to facilitate the user to quickly view the datamodel.

Use Cases

As a Watcher user, I want to use Show Datamodel API to see the information of the instance in the specified scope.

Proposed change

We can refer to the “strategy list” command, add the command line interface in the python-watcherclient. After receiving the request, the watcher-api calls watcher-decision-engine to retrieve the information about the datamodel, then parses and returns.

The command line interface used in watcherclient could be like this:

watcher datamodel list [–audit <audit_id>]

In watcherclient, we can add “datamodel.py,datamodel_shell.py” to send datamodel list request and receive the result.

In watcher-api, we can add “datamodel.py” to recieve the python-watcherclient’s request and call watcher-decision-engine module.

In watcher-decision-engine, we can get datamodel data according to the specified scope, then parses the datamodel and return to watcher-api.

Alternatives

None

Data model impact

None

REST API impact

Add following datamodel REST:

GET /v1/datamodels
- Normal http response code(200)
- Expected error http response code(400,401)
Request
- audit_uuid (Optional): UUID of an audit
Response
- instance_uuid: UUID of an instance
- node_uuid: UUID of an compute node
- instance_state: state of instance
- node_state: state of compute node
GET /v1/datamodels/detail
- Normal http response code(200)
- Expected error http response code(400,401)
Request
- audit_uuid (Optional): UUID of an audit
Response
- instance_uuid: UUID of an instance
- instance_state: state of instance
- instance_name: name of instance
- instance_vcpus: number of instance vcpus
- instance_memory: memory of instance
- instance_disk: disk of instance
- instance_disk_capacity: disk capacity of instance
- node_uuid: UUID of an compute node
- node_state: state of compute node
- node_name: name of node
- node_vcpus: number of compute node vcpus
- node_memory: memory of node
- node_disk: disk of node
- node_disk_capacity: disk capacity of node
Example JSON representation of Datamodel

{
  "compute": [
    {
      "node_uuid": "90d7da5c-d432-4eba-89b4-743c9f1e6cfa",
      "node_name": "node_1",
      "node_vcpus": 48,
      "node_memory": "4096",
      "node_disk": "40",
      "node_disk_capacity": "60"
      "servers": [
        {
          "instance_uuid": "9e7cbe91-b391-4394-a42c-68996a4fd555",
          "instance_state": "active",
          "instance_name": "vm_4",
          "instance_vcpus": 16,
          "instance_memory": "2048",
          "instance_disk": "10",
          "instance_disk_capacity": "35",
        },
        {
          "instance_uuid": "8e7cbe91-b391-4394-a42c-68996a4fd555",
          "instance_state": "active",
          "instance_name": "vm_5",
          "instance_vcpus": 16,
          "instance_memory": "2048",
          "instance_disk": "10",
          "instance_disk_capacity": "35",
        }
      ]
    },
    {
      "node_uuid": "78d7da5c-d432-4eba-89b4-743c9f1e6cfa",
      "node_name": "node_2",
      "node_vcpus": 96,
      "node_memory": "4096",
      "node_disk": "60",
      "node_disk_capacity": "60"
      "servers": [
        {
          "instance_uuid": "6b7cbe91-b391-4394-a42c-68996a4fd55b",
          "instance_state": "active",
          "instance_name": "vm_1",
          "instance_vcpus": 32,
          "instance_memory": "2048",
          "instance_disk": "10",
          "instance_disk_capacity": "35",
        },
        {
          "instance_uuid": "527cbe91-b391-4394-a42c-68996a4fd5e7",
          "instance_state": "active",
          "instance_name": "vm_2",
          "instance_vcpus": 16,
          "instance_memory": "2048",
          "instance_disk": "10",
          "instance_disk_capacity": "35",
        }
      ]
    }
  ]
}

Security impact

None

Notifications impact

None

Other end user impact

The user can view the datamodel through the command below in python-watcherclient:

watcher datamodel list

and add the audit parameter to filter the datamodel in the specified scope:

watcher datamodel list [–audit <audit_id>]

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <chenker>
Other contributors:: <li-canwei2> , <yumeng-bao>

Work Items

Add watcher datamodel list command line interface in watcherclient.
Add verification and processing of request from watcherclient in watcher-api.
Add parsing, encapsulation, and return of datamodel in watcher-decision-engine.

Dependencies

None

Testing

Unit test on the watcher-decision-engine, python-watcherclient, watcher-api.

Documentation Impact

A documentation explaining how to use watcher datamodel list [–audit <audit_id>]
Update API Reference
Update REST API Version History

References

None

History

None

Add Show Datamodel API

Wed, 16 Apr 2025 00:00:00

https://blueprints.launchpad.net/watcher/+spec/show-datamodel-api

Problem description

The data model is very important for Watcher to generate resource optimization solutions. Currently, it can only be found by looking at the log file, which is very inconvenient. Therefore, it is necessary to add an api to facilitate the user to quickly view the current datamodel in memory.

Use Cases

As a Watcher user, I want to use Show Datamodel API to see the information of the instance in the specified scope.

Proposed change

The command line interface used in watcherclient could be like this:

openstack optimize datamodel list [–audit <audit_uuid>]
[–type <type>] [–detail]

In watcherclient, we can add “data_model.py,data_model_shell.py” to send datamodel list request and receive the result.

In watcher-api, we can add “data_model.py” to recieve the python-watcherclient’s request and call watcher-decision-engine module.

In watcher-decision-engine, we can get datamodel data according to the specified scope and the type, then parses the datamodel and return to watcher-api.

Alternatives

None

Data model impact

None

REST API impact

Add following data model REST:

GET /v1/data_model
- Normal http response code(200)
- Expected error http response code(400,401)
Request
- audit_uuid (Optional): UUID of an audit
- type (Optional): Type of data model user want to list
Response
- server_uuid: UUID of server
- server_name: name of server
- server_vcpus: number of server vcpus
- server_memory: memory of server
- server_disk: disk of server
- server_state: state of server
- node_uuid: UUID of node
- node_hostname: name of node
- node_vcpus: number of node vcpus
- node_vcpu_ratio: vcpu ratio of node
- node_memory: memory of node
- node_memory_ratio: memory ratio of node
- node_disk: disk of node
- node_disk_ratio: disk ratio of node
- node_state: state of node
Example JSON representation of compute data model

{
    "context": [
        {
            "server_uuid": "1bf91464-9b41-428d-a11e-af691e5563bb",
            "server_name": "chenke-test1",
            "server_vcpus": "1",
            "server_memory": "512",
            "server_disk": "1",
            "server_state": "active",
            "node_uuid": "253e5dd0-9384-41ab-af13-4f2c2ce26112",
            "node_hostname": "localhost.localdomain",
            "node_vcpus": "4",
            "node_vcpu_ratio": "16.0",
            "node_memory": "16383",
            "node_memory_ratio": "1.5",
            "node_disk": "37"
            "node_disk_ratio": "1.0",
            "node_state": "up",
        },
        {
            "server_uuid": "e2cb5f6f-fa1d-4ba2-be1e-0bf02fa86ba4",
            "server_name": "chenke-test2",
            "server_vcpus": "1",
            "server_memory": "512",
            "server_disk": "1",
            "server_state": "active",
            "node_uuid": "253e5dd0-9384-41ab-af13-4f2c2ce26112",
            "node_hostname": "localhost.localdomain",
            "node_vcpus": "4",
            "node_vcpu_ratio": "16.0",
            "node_memory": "16383",
            "node_memory_ratio": "1.5",
            "node_disk": "37"
            "node_disk_ratio": "1.0",
            "node_state": "up",
        }
    ]
}

Security impact

None

Notifications impact

None

Other end user impact

The user can view the datamodel through the command below in python-watcherclient:

watcher datamodel list

and add the audit parameter to filter the datamodel in the specified scope:

watcher datamodel list [–audit <audit_uuid>]

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <chenker>
Other contributors:: <li-canwei2> , <yumeng-bao>

Work Items

Add watcher datamodel list command line interface in watcherclient.
Add verification and processing of request from watcherclient in watcher-api.
Add parsing, encapsulation, and return of datamodel in watcher-decision-engine.

Dependencies

None

Testing

Unit test on the watcher-decision-engine, python-watcherclient, watcher-api.

Documentation Impact

A documentation explaining how to use watcher datamodel list [–audit <audit_uuid>] [–type <type>] [–detail]
Update API Reference
Update REST API Version History

References

None

History

Revisions
Release Name	Description
Stein	Introduced
Train	Updated

Add Prometheus as a Watcher Data Source

Tue, 15 Apr 2025 00:00:00

launchpad blueprint: https://blueprints.launchpad.net/watcher/+spec/example

Watcher currently supports a small number of data sources for collection of metrics: Ceilometer, Gnocchi and Grafana. Prometheus is a widely adopted time/series based metric collection system that allows for collection of any type of custom metric an operator may be interested in for their cloud VMs or containers.

Besides its usage in OpenStack deployment, Prometheus is considered Kubernetes ‘native’ as both are CNCF projects and Prometheus is included as part of Kubernetes distributions.

Adding the ability for Watcher to interact with a Prometheus data source will increase the potential user base for Watcher and especially to those operators that are familiar with or already using Prometheus.

Problem description

Watcher currently supports a small number of data sources for collection of metrics: Ceilometer, Gnocchi and Grafana. Some of these are no longer actively developed and integrated with OpenStack distributions, limiting the ability to deploy watcher at all.

As Prometheus becomes the de facto standard metrics store in the Kubernetes ecosystem and OpenStack is increasingly deployed on Kubernetes, Watchers’ inability to consume metrics from Prometheus limits the project’s reach.

Use Cases

By providing the ability to couple the efficient and highly customizable Prometheus collector with the Watcher project operators can achieve a powerful optimization solution for their OpenStack deployments. There is currently no way to use Prometheus as a data source for Watcher.

As an operator with existing knowledge of Prometheus, I would like to leverage the power of Watcher as an optimization engine, by using it as a data source.

As an operator with existing Kubernetes infrastructure, I would like to reuse the same metrics storage solution across my OpenStack and Kubernetes deployments.

As a developer of Watcher, I want to allow it to be deployed in more OpenStack clouds, leveraging popular open-source tools to increase the project’s reach and adoption.

Proposed change

A new Prometheus module will be added to watcher.decision_engine.datasources which will leverage the https://opendev.org/openstack/python-observabilityclient already used by AODH to retrieve metrics from Prometheus. https://github.com/openstack/aodh/commit/f932265290a4e923eac6111eb28578489c7dce33

As a first implementation, we are not expecting to extend the DataSource METRIC_MAP beyond the existing set (host/instance cpu/ram etc). That could be considered future work depending on the success of this proposal. The new Prometheus client will provide a default set of mappings to enable a subset of strategies and goals to function by normalising the Prometheus metric names and units to align with the existing values supported by other data sources.

This initial work will not utilise Prometheus alert to enable triggering audits and instead will build on AODH’s existing integration to fulfil that use case.

Alternatives

It is not possible to use Prometheus as a metrics collector currently. The alternative is to use one of the currently supported data sources which restricts the potential user base for Watcher.

Data model impact

There are no expected changes to the data model as part of this proposal. Given the extensibility of Prometheus as a collector, it is feasible that future work could propose extension of the Watcher metrics beyond the current set (host/instance cpu or ram usage, temperatore etc). However that is not in the scope of this current proposal.

REST API impact

This proposal is not expected to impact the REST API.

Security impact

None Expected

Notifications impact

None expected.

Other end user impact

None expected.

Performance Impact

There is no expected impact to using a Prometheus data source compared to any of the currently supported sources.

Other deployer impact

No anticipated impact besides the ability to integrate with a new data source. Deployers will have to provide the required configuration values such as (Prometheus) authentication credentials required for the integration.

A new optional dependency on python-observabilityclient will be introduced which may require changes to packaging and installers.

Developer impact

The watcher devstack plugin will be extended to allow developers to use Prometheus instead of the default Gnocchi/Ceilometer collectors.

Implementation

Assignee(s)

Sean Mooney, Marios Andreou,

Reviewers

Dan Smith

Work Items

We will need:

New prometheus.py subclass of base.DataSourceBase in the [datasources](https://github.com/openstack/watcher/tree/master/watcher/decision_engine/datasources),

A prometheus_client.py to handle authentication and transport of metrics from the Prometheus instance under [conf](https://github.com/openstack/watcher/tree/master/watcher/conf),

Extend the Zuul CI testing for the Prometheus integration, that is, add a new devstack job similar to the existing [watcher-tempest-strategies](https://zuul.opendev.org/t/openstack/builds?job_name=watcher-tempest-strategies&project=openstack/watcher) to enable Watcher with a Prometheus collector.

Extend the Watcher devstack plugin to support deployment with Prometheus instead of the default Gnocchi/Ceilometer.

Dependencies

The proposal requires that the OpenStack deployment monitored by the Prometheus instance used as a data source, has deployed the appropriate exporters, the actual collection functions and API endpoints, such that they can be mapped to the expected Watcher metrics (host_cpu_usage, host_ram_usage, instance_cpu_usage etc).

Testing

As mentioned under work items this work will also include addition of a new CI job against the Watcher code repo. Beyond ensuring the integration point (e.g. communication with Prometheus is OK, metrics are received and processed correctly etc) ideally this should include functional testing similar to the existing watcher-tempest-strategies job that has execution of strategies.

Documentation Impact

We will need to extend documentation including considerations around setup, for example, setting up the appropriate exporters on the Prometheus side, best practices around authentication/certs etc.

References

This proposal was first mentioned by S Mooney during the [October 2024 Watcher PTG session](https://etherpad.opendev.org/p/oct2024-ptg-watcher) session

Example Spec - The title of your blueprint

Tue, 15 Apr 2025 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
2025.2	Introduced

Example Spec - The title of your blueprint

Thu, 24 Oct 2024 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Epoxy	Introduced

Provide a more dynamic Actions management solution

Mon, 23 Oct 2023 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/watcher-add-actions-via-conf

Watcher aims at providing an open solution for auditing any pool of resources and optimizing it through recommended Actions.

This blueprint aims at providing a generic and dynamic system for adding new Optimization actions that can be used in one or several Strategies to reach the Goal_(s) of an Audit.

Problem description

Today, the set of possible Actions is fixed for a given version of Watcher and enables optimization algorithms to include actions such as instance migration, changing hypervisor state, changing power state (ACPI level, …).

It should be possible to deploy and configure (in watcher.conf) new potential Actions and associate each Action to a Python class, without the need to redeploy a new version of Watcher.

Note that even if so far Watcher Decision Engine and Watcher Applier are packaged in the same Python package, those services may be deployed on separate machines. Beyond that, there will probably be a blueprint aiming at having the Watcher Planner as a separate component (today it is included in the Watcher Decision Engine).

This is the reason why, there should be a strong separation of concern regarding:

Action types definition (input parameters, …)
Actions instantiation of a given Action type
Actions scheduling (through Action types scheduling policies, …)
Actions simulation and execution (through some Action handler class)

Each of these concerns are related to a specific Bounded Context and the objective of this blueprint is to better address each Bounded Context independently so that we can update the model and source code of a given context without necessarily impacting the other contexts (Micro-Services architecture).

Below you will find a diagram, showing the functional need regarding Actions management in Watcher:

You can see that there is a need in Watcher for three main phases:

The first phase named “Define and instantiate Actions for Optimization” takes place in the Strategies of the Watcher Decision Engine: each Strategy needs to be able to use and instantiate a set of pre-defined optimization Action types in order to achieve the goal of the Audit. Those Actions are not scheduled within a timeframe at that point.

During this phase, developers need to be able to do four things in their Strategy:
- describe and register new types of Actions, with a dedicated Domain Specific Language (DSL). Each type of Action should have its own input parameters, name, description, version and target resource type;
- create one or several instances of these Action types (i.e. Watcher needs some kind of factory for Actions);
- add these Actions to the recommended Solution as a simple unordered list of Actions;
- persist this list of recommended Actions in the Watcher Database.
The second phase named “Schedule Actions in Time” takes place in the Watcher Planner: the Actions instantiated in the Strategy need to be scheduled so that they do not disturb the SLA and also because there is some logical order when executing the Actions. For example, you will disable an hypervisor before you start evacuating all instances from it (to make sure that the hypervisor does not receive new initial placement request from Nova). Therefore, the Watcher Planner must be able to take into account some scheduling policies/rules such as:
- do not migrate too many virtual machines at the same time in order to avoid network congestion;
- make sure there’s no more virtual machines on a compute node before powering it off (for energy saving);
- live migration is preferred to inactive migration;
- …
As a result, during this phase of the Watcher processing, there should be an efficient and extensible way to define those scheduling policies/rules and there should be some implementation which can take them into account to produce the design of a flow of Actions describing the dependency chain between each Action (after Action A trigger Action B, …).

Ideally, the developer should be able to integrate a new “rules solver” which could be dynamically loaded in the Watcher Planner according to what is set in Watcher configuration files.

Note that this scheduled flow of recommended Actions is what is named an Action Plan in the Watcher vocabulary.

It should also be possible to define error recovery rules which define what must be done whenever an error occurs in the worklow (how many attempts must be done, how often, …). It should be possible to define error recovery policies for a specific Action or for a wider scope (the whole flow or embedded flow).

Both scheduling policies/rules and the design of the flow of Actions could rely on dedicated Domain Specific Languages (DSL) and on a specific “solver” which would be able to take into account those rules.

Again, the developers need an easy way to persist scheduling policies/rules and the design of the flow of Actions in the Watcher Database.
The third phase named “Simulate and Execute Actions” takes place in the Watcher Applier: it consists in executing the flow of Actions built in phase two. This means that there should be some workflow engine system which is able to read the flow description (described with a DSL and stored in the Watcher Database), simulate the execution of this flow and if the simulation is successful, really execute all the Actions it contains in respect with the schedule (i.e. the dependency chain between Actions).

This workflow engine should be able to load the implementation (named “ActionHandler”) of each Action type and do the following:
- first launch some simulation method which makes sure all pre-conditions are met before executing the real command and avoid useless rollbacks on the real OpenStack services. The simulation phase could also make sure that all ActionHandler implementations are present and that all target resource exist.
- if pre-conditions are met, trigger some concrete command on technical components (most of the time, it will be OpenStack components such as Nova, Neutron,…).
- handle error recovery rules
In order to do the mapping between Action types used in phase 1 and 2 and ActionHandler classes, it must be possible for developer to implement a mapping system which would be loaded dynamically and configurable.

The workflow engine should be able to create a current context of execution of the workflow which enables to take some output results from a previously executed Action and inject it as input parameters of the next upcoming Action. This context must be persisted in some database and the workflow engine should be able to resume an interrupted workflow (in the case the Watcher Applier service was stopped for example) from where it was stoppped and not restart it from the zero.

Ideally, it should be possible for an operator with an admin role to browse the history of activity of the workflow engine (especially events and alarms regarding executed Actions).

The workflow engine should be able to send some notifications (on the Watcher AMQP message bus) whenever the current state of an Action changes.

Use Cases

As a developer. I want to be able to create and use new optimization Actions in the optimization Strategies (loaded in the Watcher Decision Engine). So that I can easily develop new Strategies (for a given optimization Goal) which rely on these actions to change the state of a managed resource and without needing an upgrade of Watcher.

As a developer. I want to be able to create workflows in the Watcher Planner containing atomic actions and embedded workflows (made of several actions). Workflows may schedule those actions in sequence or in parallel. So that I can schedule optimization actions according to some scheduling policies/rules.

As a developer. I want to be able to develop scheduling policies/rules and dynamically add those rules without needing an upgrade of Watcher. So that I can control how actions are scheduled in a given timeframe.

As a developer. I want to be able to provide new actions handlers as Python class which can be loaded in the Watcher Applier. So that the Watcher Applier can launch concrete commands to the service responsible for managing a resource (most of the time, it will be OpenStack services such as Nova, Cinder, Neutron, … but it can also be any resource management service).

As an operator with an admin role. I want to be able to easily install and configure new optimization actions without needing to deploy a new version of the Watcher software. So that they can be used for a new optimization Strategy that I need to install

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Estimated changes are going to be in the following places:

in the Watcher Decision Engine:
- in Strategies
- in the Watcher Planner
in the Watcher Applier

As described in the Problem description, Watcher should integrate a new Task/Flow management system which provides a clear separation between the three main phases:

Define new optimization Action types and instantiate them in Strategies
Schedule Actions in time
Simulate and execute Actions

Alternatives

We could deploy a new version of Watcher each time we need to add new Action types but this would impact the availability of Watcher and would lead to a much less evolutive and system.

Data model impact

The following data object will probably be impacted:

Action:
- there may be some changes in the way we store input parameters of an Action (probably as an array of key-value pairs) and the unique id of the target resource.
- We may also need to store in the database the list of output values returned after the Action was executed by the Watcher Applier.

It may also be necessary to add some new data objects such as:

ActionType which would contain all the information related to a certain type of Action:
- action_type: the unique id of the Action type.
- action_type_version: the version of the Action type. It would be provided with SemVer format.
- parameters: an array of input parameters provided as tuples with the following fields: (parameter_name, parameter_type). The parameter_type can be any simple type such as string, integer, boolean, float, …
- target_resource_type: the unique resource type that this Action type can change. It can be any HEAT resource type.
- display_name: a short human readable name for the Action type.
- display_description: a long human readable description for the Action type.

REST API impact

There will be an impact on every REST resource URLs that starts with /v1/actions/ and that uses the type Action:

GET /v1/actions/(action_uuid)
POST /v1/actions
PATCH /v1/actions
GET /v1/actions/detail

The type Action will have new attributes mentionned in the previous paragraph.

Please look at the Actions handling in Watcher API.

If Watcher stores new data objects such as ActionType, it would be necessary to provide new REST resource URLs for managing those objects with CRUD operations.

Note that for creating a new ActionType with the API, the user may provide the description of this ActionType in a file compliant with the DSL that will be used for it (i.e. probably a JSON or YAML file).

Security impact

In the case a new Action needs to access an OpenStack service (for example, Neutron), the watcher user under which Watcher Applier is running will need to be declared as having a role with enough rights on this service to trigger the concrete Action.

Notifications impact

None

Other end user impact

Aside from the API, here are there other ways a user will interact with this feature:

impact on python-watcherclient:
- will need to find a new way to display the list of Actions and the detailed information about an Action.
- will need to be able to handle new data objects such as ActionType as well

Performance Impact

None

Other deployer impact

When delivering a new Strategy, the operator will deploy the following softwares:

the main Python class implementing the Strategy;
the files containing the description of the needed Actions types (written in whatever DSL is appropriate for this);
the files containing the description of the scheduling rules (written in whatever DSL is appropriate for this);
the Python Planner class (Rules solver) that will be able to read the scheduling rules and generate a schedule of the Actions. Note that this is optional to deliver a new class here because the new Strategy may rely on a previously deployed Rules solver;
the Python classes of each Actions handler needed in the Actions flow to simulate/execute in the Watcher Applier.

The operator will also need to change the Watcher configuration in order to indicate:

What Watcher Planner implementation will be used for scheduling Actions
What Watcher Actions Mapper implementation will be used to do the mapping between Action types and Action handlers (i.e. Python class loaded by the Watcher Applier).
What implementation of the Workflow engine must be used to simulate and execute the Actions flow.
What storage backend will be used by the Workflow engine used in the Watcher Applier to persist the current state of the Actions flow.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: jed56

Work Items

The Watcher team should first study whether solutions like TaskFlow or Mistral would fit the need. It would certainly avoid long rewriting of source code and would even help us anticipate the future needs regarding the management/scheduling of Actions in Watcher.

Here is the list of foreseen work items:

Find an appropriate Domain Specific Language (DSL) for describing a new Action type in Watcher.
Implement the factory which is able to instantiate Actions in the Strategy according to the Action type description.
Put the Watcher Planner classes in a dedicated Python package (not in the Watcher Decision Engine)
Add a dynamic loading of the Watcher Planner implementation (via stevedore)
Find a generic format to persist the description of the flow of Actions that will be generated by the Watcher Planner and loaded by the Watcher Applier
Provide a default implementation of the Watcher Planner. This default implementation should be very simple and based on priority associated to each Action. Later, more complex implementations can be provided which would be able to read scheduling rules DSL. We may have to benchmark several existing Rules solver implementations.
Add a dynamic loading of Actions handlers in the Watcher Applier (via stevedore).
Integrate an existing Workflow engine which would be able to load and execute Actions handlers according to the design of the Actions flow produced by the Watcher Planner (i.e. what is called the Action Plan)
Add a simulation phase in the Watcher Applier in order to verify that the Actions flow can be executed without errors.
Make sure errors are handled correctly whenever an Action fails during the simulation phase or during the real execution of the Actions flow in the Watcher Applier.

Dependencies

There are some dependencies with the following blueprints:

In blueprint named “Code refactoring using terms defined in our glossary”, some classes related to this specification (meta-action, primitive, …) will be renamed or moved to another package.
We have to consider the existence of existing workflow management frameworks such as TaskFlow: see blueprint https://blueprints.launchpad.net/watcher/+spec/use-taskflow

We should also have a look to other existing workflow management frameworks:

Mistral: this OpenStack project proposes a Workflow as a service system

PyUtilib: it’s a self contained workflow engine, intended to be embedded and developed to automate the processing of scientific workflows.

Spiff Workflow: a workflow engine implemented in pure Python

hurry.workflow: a simple workflow system. It can be used to implement stateful multi-version workflows for Zope Toolkit applications.

Testing

Unit tests will be needed for:

the loading of the files containing the description of the needed Actions types (written in whatever DSL is appropriate for this);
the loading of files containing the description of the scheduling rules (written in whatever DSL is appropriate for this);
the Python Planner class (Rules solver) that will be able to read the scheduling rules and generate a schedule of the Actions;
the dynamic loading of an Action handler class inside the Watcher Applier;
the Python classes of each Actions handler needed in the Actions flow to simulate/execute in the Watcher Applier;
the dynamic loading of a Watcher Planner implementation;
the dynamic loading of a Watcher Actions Mapper implementation;
the execution of the default provided Watcher Planner implementation
the execution of the default provided Watcher Actions Mapper implementation;
the configuration of different storage backends for the Workflow engine used in the Watcher Applier to persist the current state of the Actions flow.
the execution of a simulation phase and the error management during this simulation phase

It will also be necessary to validate the whole Action management system in the existing integration tests.

Documentation Impact

The documentation will have to be updated, especially the glossary, in order to explain the new concepts regarding Actions definition, scheduling and execution.

The architecture description will also need to be updated because:

the Watcher Planner will become independent from the Watcher Decision Engine
a new component will be introduced: the Watcher Actions Mapper
many component implementations will be provided as plugins (Action types, Watcher Planner, Watcher Actions Mapper, ActionHandler)

The documentation regarding Watcher installation and configuration will also need to be updated in order to explain:

howto describe new Action types with the proposed DSL;
howto deploy new Action types into Watcher;
howto use new Action types in your optimization strategy (i.e. howto instantiate an Action from a given Action type);
howto add new scheduling policies/rules in the Watcher Planner;
howto build flows of Actions using the proposed DSL;
howto add a new Watcher Planner implementation;
howto add a new Watcher Actions Mapper implementation;
howto configure the Watcher Applier: engine implementation, persistence backend, …

References

IRC discussions:

Action point related to TaskFlow: http://eavesdrop.openstack.org/meetings/watcher/2015/watcher.2015-12-09-13.59.html
A lot of exchanges regarding Actions and Workflow management in the Git reviews related to the Watcher glossary: https://review.opendev.org/#/c/246370/

History

None

Cluster Model Objects Wrapper

Mon, 23 Oct 2023 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cluster-model-objects-wrapper

Currently a model of the cluster is constructed every time a strategy is executed, which will not scale in production or in larger environments. If Watcher intends to be used by larger environments it needs a more robust way to construct and maintain a model of the cluster. An in-memory cache of this model can be built up and kept fresh via notifications from services of interest in addition to periodic syncing logic.

Problem description

As discussed above, Watcher currently constructs a model of the cluster every time a strategy is executed via the get_latest_cluster_data_model() method. As it stands today Watcher only has one model collector defined (NovaClusterModelCollector). This method then fetches all hypervisors from Nova using the nova API and then for each hypervisor fetches all servers on that hypervisor. For each of these hypervisors and servers, a Watcher model object is created to represent the entity from the JSON response of the API. These objects are placed into a collection (the cluster data model) which is then passed to the strategy’s execute() method. The strategy then uses this model to make certain decisions.

Unfortunately, in a production environment of a decent size, constructing the cluster model like this will not scale. Consider an environment with hundreds of compute nodes and tens of thousands of servers. If Watcher needs to construct the representation of this every time a user wants to run an audit, running an audit will potentially be very slow. Also consider a busy environment where a lot of audits are being requested in short succession - Watcher will need to construct this most recent cluster model via API requests to each service of interest, which is a lot of strain on the overall system for almost no gain, assuming the environment has hardly changed between each audit request.

It would be ideal that a strategy can use an in-memory cache of the cluster model in order to make informed decisions immediately without the need for Watcher to query every service for its most recent representation. This will be especially important when continuous audits are implemented, which will require decisions will be made periodically and often.

Use Cases

The primary use case of this is addressing problems of scale. Strategies can be executed much quicker due to minimal delay in constructing the cluster model, which will result in faster audits delivered to the end users of Watcher. It will also reduce load on the overall OpenStack deployment due to the elimination of redundant API requests.

For the developers of Watcher, this will create an easy way to fetch and consume the current cluster model.

Project Priority

High

Proposed change

Watcher should provide a number of cluster model collectors (reusing the BaseClusterModelCollector class) that are responsible for maintaining an in-memory cache of their associated cluster. For example, the NovaClusterModelCollector class would maintain a cache of all hypervisors and servers on each hypervisor. In the future this may expand to say a CinderClusterModelCollector class which would maintain a cache of all storage providers and block devices (e.g., volumes).

These cluster model collectors would be managed by the existing CollectorManager class today, which will need to be updated to maintain references to instantiations of each collector class instead of instantiating a new one on every call to get_cluster_model_collector(). Methods should also be added to allow fetching the list of available collectors and fetching specific collectors.

Each implementation of the BaseClusterModelCollector would continue to provide a method analogous to the current get_latest_cluster_data_model, but instead of fetching the model from the service itself it would fetch it from its internal cache.

Here below is a sequence diagram depicting the workflow to be used in order to retrieve all the cluster data model:

Each implementation of the BaseClusterModelCollector should begin populating its in-memory cache on instantiation, preferably without blocking other code execution for quick service stand-up. The implementations should also define periodic tasks that are responsible for (preferably asynchronously through the use of threads) syncing the cache with the service. For example, for NovaClusterModelCollector, this periodic task would be responsible for making an API request to Nova to fetch all hypervisors and servers. From the response of that API request, the cache would be updated as appropriate. The rate at which these sync up tasks are ran should be configurable, but a sensible default is likely in the every 60 minute range.

Here below is a sequence diagram depicting the workflow to periodically synchronize all the cluster data models:

If the periodic sync up tasks are the only method of updating the cache, clearly the cache would quickly become stale. In order to combat this, a notification handler will need to be put in place that asynchronously handles notifications from different services that come in over the AMQP message bus. The notification handler should be able to configure what notifications it is interested in so that it can ignore any other notifications on the bus. The notification handler would determine what type of notification it is handling, then based on that it will spawn a thread that calls a method within specific model collectors that are configured to be interested in notifications of said type. The notification (including the payload) would be passed to the method, which would be responsible for updating its collector’s cache appropriately. It is important the notification handler can deal with notifications asynchronously via threads so that it does not get bogged down when the rate of notifications is high. For example, in the case of Nova, the notification handler would be able to receive notifications such as:

‘compute.instance.create.end’ for instances being created

‘compute.instance.delete.end’ for instances being deleted

‘compute.instance.live_migration._post.end’ for instances being migrated

… and dozens more

Here below is a sequence diagram depicting the workflow to update cluster data models after receiving a notification:

Note that a single notification will not prompt the entire cluster model to be refreshed - only the relevant items in the cache will be refreshed.

The notification handler should preferably have a way for exploiters of Watcher to somehow be able to handle other notifications via some sort of plugin mechanism.

The idea is that the notification handler will allow the collectors to keep their cache predominantly up to date. If the notification handler fails to receive any notifications sent by the services over the AMQP message bus for whatever reason, then the periodic sync up task will serve to correct any staleness of the cache. This boils down to the following idea: Watcher can live with eventual consistency.

Alternatives

No caching at all could be done in the collectors, as is today. As discussed at length above, this would only be acceptable for the smallest of cloud environments.

Instead of an in-memory cache, the cached data could be stored in Watcher database tables. This would ultimately mean duplication of potentially a lot of data which is a very big negative.

Data model impact

This should not affect the database as the data is being kept within in-memory caches.

REST API impact

None, unless we intend on surfacing Watcher’s current representation of the cluster, but that is likely outside the scope of this.

Security impact

None

Notifications impact

Watcher will not be generating any new notifications, but it will be consuming many more.

Other end user impact

None besides better performance and the understanding of what eventual consistency means.

Performance Impact

This is described in the “Proposed change” section, but as an overview:

Improved performance for environments of scale since the cluster model does not need to be reconstructed as part of every audit request.

Periodic tasks to sync up cluster model data can potentially be very slow. Therefore they should be done asynchronously preferably.

Notification handler will need to handle a significant number of notifications coming from the AMQP message bus. Spawning threads to the cluster model collectors to do the actual cache updates should allow control to quickly return to the handler to handle the next notification.

Other deployer impact

Several config options for the rate at which the periodic sync up tasks will need to be added. The intention is that the default values should work well in real deployments.

This change will take immediate effect after it is merged - it will be part of Watcher’s core architecture.

Developer impact

Strategies may need to have some refactoring done to handle the new cluster data models.

Implementation

Assignee(s)

Primary assignee:: Vincent Françoise <Vincent.FRANCOISE@b-com.com>
Other contributors:: Taylor Peoples <tpeoples@us.ibm.com>

Work Items

Part 1

Enhance the BaseClusterModelCollector to allow the creation of plugins:
- Make BaseClusterModelCollector inherit from the Loadable abstract class.
Implement a ClusterModelCollectorLoader which extends the DefaultLoader class so we can dynamically load user-defined cluster data model collectors.
Make CollectorManager capable of loading entry points/plugins that will be the various cluster model collectors (i.e. NovaClusterModelCollector only for now but also CinderClusterModelCollector later on).
- Add a loader attribute that will be a ClusterModelCollectorLoader instance.
- Adapt all existing strategies to now explicit the fact that we use the Nova cluster model collector.
- Add a get_collectors() method that returns a mapping of all the entry point names with their associated BaseClusterModelCollector instances .

Part 2

Enhance the BaseClusterModelCollector to allow an in-memory model synchronization:
- Make it inherit from oslo_service.service.Singleton so we only maintain a single model per type.
- Add a cluster_data_model abstract property which shall have to return a ModelRoot instance which will have to be thread-safe.
- Modify the get_latest_cluster_data_model() abstract method to now be a plain method that will have to return a deep copy of its in-memory cluster_data_model.
- Add a synchronize() abstract method that will be responsible for fetching the full representation of the given cluster data model. This new cluster data model should be a drop-in replacement. Note: this synchronize() method should be executed asynchronously.
- Implement the cluster_data_model property for NovaClusterModelCollector.
- Implement the synchronize() method for NovaClusterModelCollector.
Implement a BackgroundTaskScheduler background scheduling service using apscheduler that will be responsible for periodically triggering a job per cluster data model in order to synchronize them. This scheduling service should be launched as part of the Watcher Decision Engine daemon.
- It should inherit from both oslo_service.service.ServiceBase and apscheduler.schedulers.background.BackgroundScheduler.
- Make use of oslo_service.service.Services in order to run both BackgroundTaskScheduler and the main decision engine service within the same process.
- The period will have to be configurable via the configuration file.
- A set of basic configuration options about this scheduling service should also be exposed.
Update the Watcher Decision Engine command to now launch the

Part 3

Create a NotificationHandlerManager class that will be responsible for dispatching any incoming notification to update the model using the observer pattern.
- Define a register() method that will be used to register all the notification handlers.
- Implement a dispatch() method that will be responsible to call the right notification handler using an internal registry based on their associated publisher ID. This method should execute notification handlers asynchronously.
Create a NotificationHandler abstract class that will be responsible for processing any given notification to update the model.
- Implement a handle() method that will be responsible to call the right registered handler method based on the content of the notification
- Define a get_publisher_id() class method that will be used to associate the NotificationHandler to a given publisher ( e.g. ‘^compute.*’).
- Implement an NotificationParser abstract class that will be responsible for parsing incoming raw notifications.
  - Create a parse() abstract method that will be responsible for converting the incoming raw notification into some Watcher notification objects which shall be different for all event type.
- Using the visitor pattern, explore the in-memory model and apply the associated change wherever needed:
- Implement a NovaNotificationHandler class extending the NotificationHandler base class:
  - Define handler methods for all the notifications defined by Nova (see this list of notifications).
  - Use the event_type attribute of the Nova notifications as the main dispatching criterion.
  - Register it against the NotificationHandlerManager.
  - Add a model_collector property that will be return the right BaseClusterModelCollector singleton.
Enhance the BaseClusterModelCollector to allow the collection and processing of notifications in order to maintain the consistency of the in-memory model over time:
- Add a notification_handler abstract property to BaseClusterModelCollector which shall have to be overridden to return a NotificationHandler instance.
- Make the notification_handler property of NovaClusterModelCollector return a NovaNotificationHandler instance.
Make CollectorManager able to find all the notification handlers:
- Add a get_notification_handlers() class method to CollectorManager so that it returns a list of all the NotificationHandler instances via
Implement an EventsNotificationEndpoint class that will be responsible for subscribing to a given notification topic in order to collect and format them:
- Make CollectorManager able to find all the notification handlers via get_collectors() and their associated notification_handler.

Dependencies

None

Testing

Existing tempest tests should provide basic coverage. The bulk of the changes will affect larger environments. If those cannot be obtained for testing, some sort of simulation and analysis of the performance needs to be done.

Documentation Impact

Documentation for the new configuration options will be needed. The notion of all of this data being cached by Watcher in memory will also need to be documented.

References

None

History

None

Efficacy Indicator

Mon, 23 Oct 2023 00:00:00

https://blueprints.launchpad.net/watcher/+spec/efficacy-indicator

Provide efficacy indicators associated with an Audit to give tangible indicators of the possible improvement of the proposed optimization.

When the Administrator trigger an audit using Watcher, we provide a list of Actions that should be run to attain a specific goal (defined in the audit template). Today, there is no incentive to give a predicted result of the optimization after running the action plan. This feature will give the admin an estimated gain of running Watcher on its infrastructure.

Problem description

Today, Watcher does not provide efficacy indicators to give more accuracy to an action plan. An administrator would require to have an idea of the level of optimization Watcher can provide. The indicators must be related to the initial goal set in the audit template, for example if the goal is “thermal optimization”, we should provide an estimated average inlet temperature of the cluster (or an estimated decrease in % of the average inlet temperature). These indicators must be computed by the chosen strategy.

In later versions of Watcher, there should also be a way to compare the efficacy of different Strategies for a given optimization goal. Therefore, all the strategies associated to a given goal should provide the same efficacy indicators so that the Administrator can decide which strategy is the best.

In an even longer run, it will enable Watcher to automatically decide:

which strategy to use (via some StrategySelector component in the Watcher Decision Engine). See Strategy Selector blueprint.
or which planner implementation is the most efficient. See Planner Selector blueprint.

Use Cases

As an administrator, I would like to have efficacy indicators to be able to judge if the proposed optimization fit my objectives before running the recommended action plan.

As an administrator, I would like to be able to compare in a fair way the efficacy of several strategies for the same optimization goal

As a strategy developer, I will provide a list of indicators that can be evaluated during the execution of the audit and displayed to the administrator with the corresponding action plan.

As a developer, I would like to make sure that all strategies associated to a given goal provide the same list of efficacy indicators.

As an administrator, I would like to be able to update a given goal and its associated optimization strategies. In that case, all the previously launched audits and action plans which are still in recommended or pending state should be cancelled (in order to make it clear that the estimated efficacy indicators are not relevant anymore and that a new audit should be launched with the new deployed version).

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

A dictionary should be added to the action plan providing a list of efficacy indicators with a name and a value (the name is used as the key for each indicator and should be unique).

The same dictionary should also be added to the solution generated by each Strategy (which contains a non scheduled list of actions).

We propose to use the DDD Specification Pattern to express the constraints regarding efficacy indicators for a given goal.

For each XYZ goal, we should create a XYZEfficacySpecification class which contains the list of expected efficacy indicators. This class would contain the following properties for each expected efficacy indicator:

the name of the indicator, which must be translated via i18n oslo lib
a description, which must be translated via i18n oslo lib
the unit of the indicator (percent, kWh, second, …), which must be translated via i18n oslo lib
the type of the value (integer, float, enum, string, …) and range of allowed values.
a flag indicating whether it is mandatory or optional

Each strategy associated to a given XYZ goal should have access to a singleton of the XYZEfficacySpecification class in order to make sure the generated solution contains the expected efficacy indicators. In other words, in each strategy, it should be possible to call the check_efficacy_indicators(Solution) method of the XYZEfficacySpecification which returns a boolean saying whether all mandatory efficacy indicators are provided in a given solution and whether the indicator values are compliant with the expected types and ranges (we may rely on Voluptuous Python data validation library to manage the schema definition and validation of indicators).

It should be possible to call a get_global_efficacy(Solution) method from the XYZEfficacySpecification class (or directly from the XYZGoal class) which returns a global score of efficacy for a given solution. This global score would be computed using a weighted sum of the different efficacy indicators.

For example, for the BASIC_CONSOLIDATION goal, the following efficacy indicators would be returned:

nb_migrations: the number of virtual machine migrations
nb_empty_nodes: the number of empty nodes (i.e. where all the virtual machines have been migrated to some other machines).

And the global score of efficacy would be computed with the following formula:

global_score = (nb_empty_nodes/nb_migrations*100)

Of course, in this case, we should handle the case when nb_migrations is 0.

Note that for simpler implementation, it is also possible to put directly these methods in the XYZGoal class (which would then act as the specification class for the XYZ goal efficacy).

Note also that whenever a XYZGoal class is updated, Watcher should automatically change to a CANCELLED state the previously generated audit and action plans . This would make sure that we can handle the consistency between the indicators described in the goal and those returned in the action plan. See State machine diagrams for more details on the different possible states of the audit or an action plan objects.

All those efficacy indicators are set in the _post_execute() method of the strategy, i.e. just after a solution has been found in the _do_execute() method.

Note that the _pre_execute() method of each strategy will be the place where all necessary data is fetched and prepared in order to compute a solution (i.e. build a model of the data-center, get some previously stored metrics, …).

Below you will find a class diagram showing the hierarchy of Strategies for several goals and how they are related to efficacy specification classes:

In the future, the DDD Specification Pattern will enable Watcher to compose several efficacy indicators whenever an Audit is launched with multiple goals.

Alternatives

The administrator will let Watcher do the optimization without having a forecast of the potential gain on the infrastructure.

Data model impact

The following data object will be impacted:

ActionPlan:
- We may need to store in the database a list of efficacy indicators associated with the action plan

A new Goal object should be stored in the Watcher database with the following information:

the unique name of the goal

the translated name of the display name of the goal

the list of efficacy indicator descriptions. Note that the regarding the name, unit and description of each indicator, the database stores the i18n unique name of the string, not the translated version.

A new table should also be created in order to store the specification of each efficacy indicator and reference the goal that uses it.

REST API impact

There will be an impact on every REST resource URLs that starts with /v1/action_plans/ and that uses the type ActionPlan:

GET /v1/action_plans
GET /v1/action_plans/(action_plans_uuid)
POST /v1/action_plans
DELETE /v1/action_plans
PATCH /v1/action_plans
GET /v1/action_plans/detail

The type ActionPlan will contain a new efficacy object with a dictionary of indicators.

Here is a sample of the new JSON payload for an action plan which includes one global efficacy indicator and three detailed indicators:

{
    "action_plans": [
        {
            "audit_uuid": "abcee106-14d3-4515-b744-5a26885cf6f6",
            "first_action_uuid": "57eaf9ab-5aaa-4f7e-bdf7-9a140ac7a720",
            "links": [
                {
                    "href": "http://localhost:9322/v1/action_plans/9ef4d84c-41e8-4418-9220-ce55be0436af",
                    "rel": "self"
                },
                {
                    "href": "http://localhost:9322/action_plans/9ef4d84c-41e8-4418-9220-ce55be0436af",
                    "rel": "bookmark"
                }
            ],
            "state": "ONGOING",
            "updated_at": "2016-02-08T12:59:33.445869",
            "uuid": "9ef4d84c-41e8-4418-9220-ce55be0436af",
            "global_efficacy": 89,
            "efficacy_indicators" : [
                {
                    "name" : "Average CPU load",
                    "value" : "22"
                },
                {
                    "name" : "Average inlet temperature",
                    "value" : "34"
                },
                {
                    "name" : "Number of sleeping hosts",
                    "value" : "2"
                }
            ]
        }
    ]
}

There will also be an impact on every REST resource URLs that starts with /v1/goals/ and that uses the type Goal:

GET /v1/goals
GET /v1/goals/(goal_uuid)
GET /v1/goals/detail

Each Goal object will be a dictionary with a goal_name and an efficacy_specification which contains an array of indicator descriptions.

Here is a sample of the new JSON payload for a goal which includes the efficacy_specification object composed of four indicators:

{
    "goals": [
        {
            "goal_name": "REDUCE_ENERGY",
            "goal_display_name": "Reduce Energy Consumption",
            "efficacy_specification" : [
                {
                    "name" : "Relative energy gain",
                    "description" : "The amount of gained energy in %",
                    "unit" : "%",
                    "type" : "integer",
                    "range" : "[0..100]",
                    "mandatory" : "true"
                },
                {
                    "name" : "Absolute energy gain",
                    "description" : "The amount of gained energy in kWh",
                    "unit" : "kWh",
                    "type" : "long",
                    "mandatory" : "true"
                },
                {
                    "name" : "Number of VM migrations",
                    "description" : "The total number of VM to migrate",
                    "unit" : "counter",
                    "type" : "integer",
                    "mandatory" : "true"
                },
                {
                    "name" : "Estimated action plan duration",
                    "description" : "The estimated time needed to execute the
                    action plan, in seconds",
                    "unit" : "second",
                    "type" : "long",
                    "mandatory" : "true"
                }
            ]
        }
    ]
}

Note that, from the user point of view, the link between indicators returned in the Goal object and in the ActionPlan object can be done through the name of the indicator (which is unique for a given goal).

Security impact

None

Notifications impact

None

Other end user impact

Efficacy indicators needs to be added to the python-watcherclient to provide extra information when requesting action plans or goals.

The indicators must also be added to watcher-dashboard to allows the cloud administrator to take appropriate decision in Horizon.

Performance Impact

The calculation of efficacy indicators will be done by the targeted strategy, we should keep in mind that this calculation must not add delay to the building of the solution.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <launchpad-id or None>
Other contributors:: v-mahe, jed56, acabot

Work Items

Here is the list of foreseen work items:

add a new Goal object in /watcher/db/sqlalchemy/models.py
add a new EfficacyIndicatorSpecification object in /watcher/db/sqlalchemy/models.py*
add a new goal.py class in the /watcher/objects/ package
add a dictionary to BaseSolution and ActionPlan classes to store the list of efficacy indicator values.
the ActionPlan class should be able to read/write the efficacy indicator values from/to the Watcher database (see /db/api.py and /db/sqlalchemy/api.py classes).
the BaseSolution class should have a new method named set_efficacy_values() which enables a Strategy to set the list of efficacy indicator values in the solution that it builds.
implement an example of efficacy specification class which contains a description of the expected efficacy indicators for a given goal and the needed methods:
- check_efficacy_indicators(Solution) which returns a boolean. The developer may use the Voluptuous Python data validation library to define the constraints on the efficacy indicators in this method.
- get_global_efficacy(Solution) which returns the computed global efficacy value.
implement a base class dedicated to a goal that will hold a list of possible Strategies and an efficacy specification.
when the Watcher Decision Engine service is started, Watcher should browse the list of available goal unique names, get for each goal the list of efficacy indicator descriptions and store them in the Watcher database.
make sure that when a new version of a goal is deployed, when the Watcher Decision Engine service is restarted, all recommended and pending action_plan and audit are changed to a CANCELLED state.
update all existing strategies so that they define three private methods which are automatically called from the public execute() method (i.e. use a Template method design pattern):
- _pre_execute(): which prepares the data needed by the optimization algorithm.
- _do_execute(): which computes the solution. This is where the main optimization algorithm is executed and where the efficacy indicators are computed.
- _post_execute(): which can do some cleaning task but, more importantly, this is where efficacy indicators are set.
update the Watcher devstack plugin setup to adapt it (remove the auto watcher_goals config setup).
update python-watcherclient to provide extra information regarding efficacy indicators when requesting action plans or goals.

Dependencies

There are some dependencies with the following blueprints:

https://blueprints.launchpad.net/watcher/+spec/get-goal-from-strategy : there should be a common base class XYZBaseStrategy for all Strategies associated to the same XYZGoal.
https://blueprints.launchpad.net/watcher/+spec/optimization-threshold : this blueprint deals with input parameters provided to Strategies and there may be some common impacts with this blueprint which deals with output parameters generated by Strategies.
https://blueprints.launchpad.net/watcher/+spec/persistent-audit-parameters : this blueprint must be implemented so that it is possible to get from the database the information regarding the goal and strategy to which the indicators of the action plan were related (hence the specification of these indicators) when the audit was launched.
Strategy Selector blueprint : this component will probably use the efficacy indicators to automatically select the best strategy.
Planner Selector blueprint : this component will probably use the efficacy indicators to automatically select the best scheduler.

There is also a dependency with the following bug:

https://bugs.launchpad.net/watcher/+bug/1546630 : the API documentation should explain how to request the list of available goals and for each goal it should be possible to see the list of efficacy indicators.

Testing

Unit tests on the Watcher Decision Engine
Tempest test to run a strategy to get efficacy indicators
Tempest test to get the list of available goals and for each goal the list of efficacy indicator specification.
An admin should be able to launch an Audit with Watcher with a BASIC_CONSOLIDATION goal on an OpenStack cluster and get efficacy indicators associated with the generated action plan.

Documentation Impact

The developer guide should explain how to specify efficacy indicators for a given goal.

References

History

None

The achieved goal should be returned by each strategy

Mon, 23 Oct 2023 00:00:00

https://blueprints.launchpad.net/watcher/+spec/get-goal-from-strategy

Problem description

Today, there is no way to know what Goal is achieved by a given Strategy. The mapping between goal and strategies is done in the Watcher configuration.

Beyond that, there is no way to know that several strategies achieve the same optimization goal.

There should be a way to return the list of available goals depending on which strategies have been deployed on the node where the Watcher Decision Engine is running.

Use Cases

As an Administrator I want to be able to get the list of available optimization goals So that I can select one Goal and use it in my Audit Template

As an Administrator I want to be able to install new strategies and get an updated list of available goals So that I am sure that the returned list of goals is aligned with the deployed strategies

As an Administrator I want to be notified when the Goal of an Audit Template is not any more available So that I know that I have to update or delete this Audit Template

As an Administrator I want to be able to get the translated name of each Goal So that I can understand what kind of optimization(s) it provides

As an Administrator I want to be able to get the list of available strategies for a given Goal So that I can select a specific strategy for my Audit Template

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Each Strategy should be able to return the Goal it achieves.

All strategies related to the same Goal should derive from the same parent class and return the same Goal properties.

Below you will find a class diagram showing a hierarchy of Strategies for several goals:

In the future, it will also enable Watcher strategies to provide other common attributes and methods for a given goal (input parameters, efficacy indicators, …).

The synchronization of goals consists in establishing data consistency between goals and strategies automatically found by the DefaultLoader class via the function list_available to then applying the changes in the Watcher database. Harmonization of the goals over time should be automatically performed by the decision_engine during start. We should create a Synchronizer class which compares records in Watcher database with those found by the DefaultLoader. Then, altered goals and strategies will be replaced in order to establish identity between the entry points and the Watcher database. As a result of the synchronization, all the available goals and strategies are updated. The synchronization is required for two main reasons. The first reason is that we don’t want a strong coupling between the Watcher API component and the Watcher Decision Engine component. In fact, we want the watcher-api only interact with Watcher database and the message broker. The synchronization avoids the watcher cli having to call the watcher-api which would then have to call the watcher-decision-api, which is in charge of listing the resources. The second one is performance and HA.

Below the strategy class and sequence diagram for syncing the goals.

Alternatives

Keep the current system which reads the mapping between goals and strategies from the Watcher configuration file.

Data model impact

The list of available goals should be stored in the Watcher database. For each goal, the list of available strategies should also be stored in the Watcher database. A new table should be created in the database for this. Therefore a new table should be created in the database for this. The proposed modification in the Watcher database. is illustrated on the diagram below:

In the audit_template object, the ‘strategy’ attribute is optional. If the admin wants to force the trigger of a Strategy it could specify the Strategy uuid in the Audit Template. There may be several strategies applying for a given optimization Goal. If the admin didn’t specify a Strategy, the watcher strategy selector is responsible for selecting an appropriate one given the provided Goal. This blueprint should reuse the DefaultStrategySelector class that is currently responsible for selecting the strategy. This selection of the strategy is complex but likely outside the scope of this blueprint (watcher-strategy-selector). However, we need to provide a basic/minimum strategy selector. So, the strategy selector will select the first available strategy for a given goal. In the future, we plan have a more complex strategy selector that would be able to select the strategy depending on many parameters such as the usage of infrastructure, workloads, etc.

The soft delete is a commonly-used pattern and cascading must be implemented in order to maintain integrity. If a strategy is deleted, we need to update every audit template linked to this strategy and setting the strategy id to None. So the audit template can still achieve the same goal but with a different strategy. However, if we delete a goals we didn’t plan to maintain the integrity as it is outside the scope of this blueprint (soft-delete-goals)

Note: The id attribute is used by oslo.db to handle the soft_delete feature.

REST API impact

There will be an impact on every REST resource URLs that starts with /v1/goals/ and that uses the type Goal:

GET /v1/goals
GET /v1/goals/(goal_uuid)
GET /v1/goals/detail

The type Goal will contain two attributes:

The unique goal id.
The display name of the goal: i.e. the translated name of the Goal in the language of the Administrator.

Here is a sample of the new JSON payload for a list of 2 goals, the first goal having only one strategy and the second goal having 2 available strategies:

{
    "goals": [
        {
            "goal_id": "REDUCE_ENERGY",
            "goal_display_name": "Reduce Energy Consumption",
            "strategies": [
                {
                  "strategy_id": "POWERING_DOWN",
                  "strategy_display_name": "Powering down unused compute
                  nodes",
                }
            ]
        },
        {
            "goal_id": "SERVERS_CONSOLIDATION",
            "goal_display_name": "Reduce the number of compute nodes needed
            to support current projects workloads"
            "strategies": [
                {
                  "strategy_id": "FFD",
                  "strategy_display_name": "First-Fit Decreasing",
                },
                {
                  "strategy_id": "BFD",
                  "strategy_display_name": "Best-Fit Decreasing",
                }
            ]
        }
    ]
}

Security impact

None

Notifications impact

None

Other end user impact

python-watcherclient should be able to return all goal attributes when requesting the list of goals and to return the list of available strategies when requesting the detail of a given goal.

It should also be possible to browse the list of available goals in Horizon when the admin is creating a new Audit Template. When the admin has selected a given goal, he/she should be able to see the list of available strategies for this goal and select the preferred Strategy.

Performance Impact

None

Other deployer impact

Now the association between a Goal and a Strategy should no more be configured in the main Watcher configuration file.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: vincent-francoise

Other contributors: jed56

Work Items

Here is the list of foreseen work items:

Remove the previous Goal listing and mapping mechanism which was read from the Watcher configuration file. The following Python files should be impacted:
- /devstack/local.conf.controller
- /watcher/decision_engine/strategy/selection/default.py
- /watcher/tests/decision_engine/strategy/selector/
  
  test_strategy_selector.py
- /watcher/tests/api/v1/test_goals.py
- /watcher/tests/api/v1/test_audit_templates.py
- /watcher/api/controllers/v1/goal.py
- /watcher/api/controllers/v1/audit_template.py
- /watcher/opts.py
- /etc/watcher/watcher.conf.sample
In the BaseStrategy class, the following attributes should be added:
- The unique goal id.
- The display name containing the name of the goal translated in the language of the Administrator.
- The unique strategy id.
- The display name containing the name of the Strategy translated in the language of the Administrator.
For each XYZ Goal, there should be a base Strategy class, named XYZBaseStrategy, containing the goal attribute values indicating what goal id and display name all child strategies achieve.
Update all existing strategies with this new way of handling goals. Create a base strategy class for each common optimization Goal.
add a new Goal object in /watcher/db/sqlalchemy/models.py for the storage of goals in the Watcher database.
add a new goal.py class in the /watcher/objects/ package to handle CRUD operations on goal objects in the database.
when the Watcher Decision Engine service is started, Watcher should browse the list of available strategies, get their Goal attributes and create a new entry in the Watcher database for each new goal id. After that, the Watcher Decision Engine should be able to check whether all audit templates in the Watcher database contain an existing goal. If not, an error should be traced in the logs. During this phase, the Watcher Decision Engine should also create a record in the database for each new Strategy id and associate this record to the Goal object it achieves. Note that this very same model would then be used by other blueprints such as blueprint optimization-threshold to expose the input parameters of each Strategy.
update the Watcher devstack plugin setup to adapt it (remove the auto [watcher_goals] config setup).

Dependencies

None

Testing

Update unit tests in the Watcher Decision Engine
Update Tempest test to run a strategy
Add some Tempest tests to get the list of available goals and for each Goal the list of available strategies

Documentation Impact

Update the page named “Build a new optimization strategy” explaining that when a new goal is added, a new base strategy class should be created.
Update the page named “Configuring Watcher”, removing the “Goals mapping configuration” section.
Update the page named “Watcher User Guide”, providing CLI examples showing how to get the list of goals and how to get the list of available strategies for a given Goal.

References

IRC discussions:

History

None

Allow the developer of a Watcher plugin to add specific parameters

Mon, 23 Oct 2023 00:00:00

https://blueprints.launchpad.net/watcher/+spec/plugins-parameters

In Watcher, it is possible to deploy and configure new strategies, actions, planners, etc. and associate each “plugin” to a Python class, without having to redeploy a new version of Watcher.

However, it should be possible to pass some extra parameters depending of the configuration of the OpenStack cluster or some behaviour (e.g. timeout, weights, etc.) on which Watcher is operating.

This is the reason why, we should give the possibility to the developer to add some parameters depending on the configuration of OpenStack cluster on the plugins.

Problem description

As of now, all plugins that a developer implements are loaded as plain old Python classes before being instantiated. Hence, this means that Watcher does not pass down anything more than hardcoded parameters. So if a developer currently wants to define a set of configuration options specific to its own plugin, he/she would have to manually declare them. This way of doing things causes the following problems:

No standard way of declaring configuration options increases the difficulty for an administrator to configure Watcher:
- The generated configuration file would miss out these parameters which will further increase the configuration complexity.
- How to find where these parameters are without looking at the code?
Every plugin developer has to code the same logic declare and fetch these options.

Hence, Watcher should provide a way to ease the process of declaring and fetching extra options depending on the configuration of the OpenStack cluster or some behaviour (e.g. timeout, weights, workers, etc.) on which Watcher is operating whilst keeping things simple from an administrator viewpoint.

This is the reason why we should provide to the developer a standard way to both declare and fetch configuration options so that whenever the administrator generates the Watcher configuration sample file, it contains the configuration options of the plugins that are currently available.

Use Cases

As a developer, I want to be able to specify a set of configuration options for my plugin So that I can allow an administrator to adapt/tune the behaviour of my plugin given the configuration of an OpenStack cluster.

As an administrator, I want to be able to access and modify the configuration options of all available plugins So that I can adapt/tune their behaviour the requirements of my OpenStack cluster.

As an administrator, I want to be able to generate a configuration file that includes the options provided by each available plugin So that I can understand how to use each one of them without having to look at the codebase.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Here below is the class diagram outlining the changes that will have to be made in order to support the addition of configuration options:

Moreover, all plugins are currently instantiated by the DefaultLoader when its load() is being called. This method should be improved so it also fetch the configuration provided by the plugin using the get_config_opts() abstract class method that every plugin class should implement. The latter method needs to be an class method so that when Watcher will collect the configuration of each plugin, there will be no need to instantiate them.

In order to expose these plugin parameters to the administrator, we also have to auto-discover them when we use the configuration file generator which is triggered either during the generation of the Watcher documentation or manually with the tox -e config command:

In order to be able to achieve the process described in the above sequence diagram, here below is a recap of the changes that will have to be made:

Make each plugin provide a class method named get_config_opts() that will define all its configuration options.
Update the list_opts() function so that it also collects the configuration options from all the available plugins so that generating the configuration file via the tox -e config command includes them in its configuration sample file.

The administrator would then have adapt this configuration sample file to the requirements of any given OpenStack cluster.

Alternatives

Instead of using a Loadable abstract class, we do it by convention. Indeed, the main solution relies on a Loadable abstract base class that defines the get_config_opts() method. But instead, we could use the duck typing to say that by convention, every plugin should implement a get_config_opts() without having to enforce it codewise. This alternative would mean less code but would also be more error-prone for the plugin developer.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

New configuration sections and options will be added to the Watcher configuration file. Hence, the administrator should be aware of them. This also mean that the documentation should be updated to encourage the plugin developer to provide a detailed description of every single configuration option that gets declared.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: vincent-francoise
Other contributors:: david-tardivel

Work Items

Implement the Loadable abstract class
- Add an abstract class method named get_config_opts() with should return a list of oslo_config.cfg.Opt instances.
Make the following classes inherit from Loadable abstract class:
- BaseStrategy
- BaseAction
- BasePlanner
- BaseWorkflowFlowEngine
For each of the above, provide a default implementation of get_config_opts() that returns an empty list which should be the default behaviour.
Enhance the load() method of the DefaultLoader class to now also load the configuration options that relates to the plugin we want to load and inject them into the loaded plugin upon instantiating it.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/get-goal-from-strategy

Testing

Add needed unit tests to cover the new functionality introduced by this blueprint
Update the broken unit tests

Documentation Impact

Update all plugin implementation documentations to now mention that configuration options can be specified via the get_config_opts() method.

References

None

History

None

Provide a scoring module for Watcher

Mon, 23 Oct 2023 00:00:00

https://blueprints.launchpad.net/watcher/+spec/scoring-module

Watcher Decision Engine currently allows to define multiple Goals and implement several Strategies to reach them. However, in some more advanced scenarios, Strategies might use machine learning models, which can be trained and evaluated using external machine learning engines.

This blueprint aims at providing a generic scoring engine module, which will standarize interactions with scoring engines through the common API. Also, it will be possible to use the scoring engine by different Strategies, which will improve the code and data model re-use.

The scoring module will be independent and optional. There is no need for any Strategy to use it.

Problem description

Today, the Strategy implementation is free to use any algorithm or any external framework in order to achieve a given Goal. There is no standard interface or API which would help to share these algorithms or frameworks between different Strategies.

It should be possible to implement scoring engines independently from Strategies. Once implemented prediction or classifier could be shared and re-used allowing to implement more advanced Strategies, which could even use multiple scoring engines.

Use Cases

As a developer. I want to be able to list the available Scoring Engines. So that I can quickly identify them and reuse the available predicted results in my strategy, e.g. prediction energy consumption of the VMs, predicted CPU of the VMs, etc.

As a developer. I want to be able to create and use Scoring Engines in the optimization Strategies. So that I can quickly switch the Scoring Engine or implement multiple versions of the same Scoring Engine (e.g. created using a different data set for learning).

As a developer. I want to be able to configure multiple Scoring Engines without needing an upgrade of Watcher.

As a developer. I want to be able to provide a dynamic list of Scoring Engines in a single plug-in. So that I can register/unregister similar types of Scoring Engines without restarting any Watcher service.

As a developer. When implementing a new Strategy I want to work with different Scoring Engines (possibly from different vendors) in a similar way, using similar API.

As a developer. I want to get metadata information about a given Scoring Engine. The metadata might contain an important information how to work with a specific Scoring Engine, for example: number of input parameters (e.g. number of features), the response format (e.g. label order used by classifier).

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

The proposal is to define an abstraction layer in the Decision Engine, which will provide an abstract ScoringEngine class, which will have to be implemented by all Scoring Engines and a ScoringFactory class, which will be used by a Strategy implementation to select a Scoring Engine to work with.

Usage scenarios

The most basic scenario is presented on the following diagram:

It’s important to notice that Scoring Engines might have different requirements and the implementations might vary. Some of them might be implemented as plain Python classes executing (possibly heavy) calculations. In these cases it makes a sense to delegate the execution to Watcher Scoring module, which will be a new Watcher service, similar to Watcher Decision Engine or Watcher Applier:

Some other Scoring Engines might be implemented using external frameworks or even live entirely in the cloud, exposing only some API to work with them. In this scenario, the abstraction layer will simply delegate work to these external systems (e.g. using some HTTP client libraries), as illustrated on the diagram below:

Implementation details

This above an impact on the design of the Scoring Module and its implementation. The goal of the abstraction layer proposed above is to allow for both:

Simple usage of any Scoring Engine by any Strategy.
Implementation freedom: if a Scoring Engine performs some heavy calculations, its implementation could be moved to external process, which will not affect the Watcher Decision Engine module.

That said, the following changes are going to be implemented:

In the watcher/common package:

ScoringEngine class defining an abstract base class for all Scoring Engine implementations. The abstract class will include the following abstract methods:
get_engine_id:

Method will return a unique string identifier of the Scoring Engine. This ID will be used by factory classes and Strategies wanting to use a specific Scoring Engine

Input:

none

Result:

unique string ID (must be unique across all Scoring Engines)

get_model_metadata:
Method will return a map with metadata information about the data model. This might include informations about used algorithm, labels for data returned by the score method (useful for interpreting the results)
Input:

none

Result:
dictionary with metadata, both keys and values as strings

For example, the metadata can contain the following information (real world example):

scoring engine is a classifier, which is based on the learning data with these column labels (last column is the result used for learning): [MEM_USAGE, PROC_USAGE, PCI_USAGE, POWER_CONSUMPTION, CLASSIFICATION_ID]

during the learning process, the machine learning decides that it actually only needs these columns to calculate the expected CLASSIFICATION_ID: [MEM_USAGE, PROC_USAGE]

because the scoring result is a list of doubles, we need to know what it means, e.g. 0.0 == CLASSIFICATION_ID_2, 1.0 == CLASSIFICATION_ID_1, etc.

there is no guarantee of the order of the columns or even the existence of them in input/output list

this information must be passed as metadata, so the user of the scoring engine is able to “understand” the results

in addition, the metadata might provide some insights like what was the algorithm used for learning or how many training records were used
calculate_score:

Method responsible for performing the actual scoring, such as classifying or predicting data

Input:

list of float numbers (e.g. feature values)

Result:

list of float numbers (e.g. classified values, predicted results)
In the Watcher Decision Engine:
- New scoring package containing:
  - ScoringFactory class defining a factory for Scoring Engine implementations.
  - engines subpackage containing implementations of the Scoring Engines. The Scoring Engines must all extend the ScoringEngine base class.
  - Two sample Scoring Engine implementations:
    - simple Scoring Engine working within Decision Engine module.
    - simple Scoring Engine using the Scoring Module, which will demonstrate how to defer the heavy calculations to the external Python process.
- In Strategies: two sample Strategies using the above Scoring Engines.
In the Watcher API:
- New REST resource URLs to expose list of Scoring Engines and their metadata (read-only)
  - GET /v1/scoring_engines/
  - GET /v1/scoring_engines/(scoring_engine_uuid)
In the Watcher CLI:
- Expose new API in the command line
New Watcher Scoring Module:
- New top level scoring_engine directory inside watcher directory with Watcher source code.
- A new service: watcher-scoring.
- A sample Scoring Engine (not using any external dependencies).

Deployment

The deployment model for Scoring Engine implementations will use the Stevedore pluggability model. There will be entry points defined for the abstraction layer and for the Watcher Scoring module as well. The abstraction layer part will be required to implement, whether the Watcher Scoring module part will be optional (it’s not needed for example when using external analytics platforms running in the cloud).

In addition, it will be possible to register multiple Scoring Engines from a single plug-in. The Scoring Engine list will also be dynamic, meaning that it will be possible to register and unregister a Scoring Engine without any need to restart Watcher services.

Scoring Engine versioning

The rules similar to API versioning should apply to Scoring Engine versioning. Scoring Engines will be identified using their unique ID. A new version of the Scoring Engine should have a different ID, so it doesn’t break the existing usage. Of course it’s possible that the Scoring Engine developer will decide to update the existing Scoring Engine (so ID of the updated version will stay the same), but then she/he should take the full responsibility for that and understand the fact, that it might change the other use cases. The recommendation is to update Scoring Engines only for small bug fixing and give a new ID to the Scoring Engines using different ML algorithm or trained using different learning data.

Alternatives

Each developer could implement a new Strategy using a custom integration with machine learning frameworks. Data Models and Scoring Engines are relatively difficult and time consuming to create, so it would be a big loss if they are not available for wider usage.

Data model impact

None

REST API impact

None

Security impact

There will be a new Watcher Scoring Module service, which means an additional network port open, which is always increasing the security impact.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

When delivering a new Scoring Engine, the operator will deploy the following softwares:

Required:

the main Python class implementing the Scoring Engine
all additional resources or classes required by the new Scoring Engine implementation (for example client code to communicate with external service if a Scoring Engine is implemented and hosted in the cloud)

Optional:

Strategy implementation, which is using the new Scoring Engine (details are out of scope of this document)
Action handlers if they are required by the Strategy (details are out of scope of this document)

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: tkaczynski

Work Items

The list of foreseen work items:

Review this BluePrint, improve it based on feedback received
Implement generic Watcher Scoring Module
Implement Scoring Engine loader
Implement a sample Scoring Engine to demonstrate Scoring Module design and provide a guidance how to use it (no external dependencies)
Implement a sample Strategy using sample Scoring Engine from previous point
Provide documentation of the new Scoring Module
- Update glossary with terms related to Watcher Scoring
- Provide a guidance / tutorial how to implement a Scoring Engine plugin

Dependencies

There are no direct dependencies.

However, in the long run Watcher should provide a flexible plugin model, which would allow to easily integrate Scoring Engines, Strategies and Actions with Watcher without needing to reinstall or upgrade it. The ideal scenario would be that the third party developers would provide implementations in a separate repository, which could then be included in one of the Watcher configuration files. Solving this problem is not in scope of this document.

Testing

Unit tests will be needed for the code in the new Scoring Module. Implementing this module will be transparent for the existing Watcher code base, so no existing tests or functionality will be affected.

Documentation Impact

The documentation will have to be updated, especially the glossary, in order to explain the new concepts regarding Scoring Module definition and Scoring Engine implementations.

The API documentation and Watcher User-Guide will have to be updated to demonstrate how to get information about available Scoring Engines and their metadata.

The architecture description will also need to be updated because there will be a new Watcher component available.

The documentation regarding Watcher installation and configuration will also need to be updated in order to explain:

howto deploy new Scoring Engines into Watcher
howto integrate Strategies with existing Scoring Engines

References

None

History

None

Victoria Project Priorities

Thu, 02 Jul 2020 00:00:00

List of priorities the Watcher drivers team is prioritizing in Victoria.

Priority	Owner
Rollback Mechanism	su zhengwei
Kubernetes helm charts and docker images	Corne Lukken
time series framework	Corne Lukken
the community-wide goals	Li Canwei

Rollback Mechanism

After every audit, there would be one actionplan to execute. Sometimes, the users want to rollback the actionplan or part of the actionplan after one audit. For host maintenance, it will migrate all instances from the maintaining host to others. Aften the host maintenance and active again, there is no mechanism to migrate the instances automatically back to the maintaining host.

Kubernetes helm charts and docker images

Blueprint to track all activities related to being able to deploy Watcher on K8s using helm charts. Also requires that the appropriate docker images are created and uploaded to a registry.

time series framework

Strategies are currently limited to obtain information about metrics in an aggregated form for the most recent measurements. This limits what strategies can achieve as they are unable to retrieve information about past occurrences or information about periodic patterns. A time series framework will allow datasources to provide metrics over specific periods without aggregation, allowing strategies to detect periodic patterns such as a weekly contention and resolve these accordingly.

the community-wide goals

https://governance.openstack.org/tc/goals/selected/victoria/index.html

Time Series Framework

Sun, 14 Jun 2020 00:00:00

https://blueprints.launchpad.net/watcher/+spec/time-series-framework

Strategies are currently limited to obtain information about metrics in an aggregated form for the most recent measurements. This limits what strategies can achieve as they are unable to retrieve information about past occurrences or information about periodic patterns. Currently, an audit will have to be launched at the same time a specific situation is occurring in order for the strategy to be able to identify this. A time series framework will allow datasources to provide metrics over specific periods without aggregation, allowing strategies to detect periodic patterns such as a weekly contention and resolve these accordingly.

Problem description

Strategies can only obtain metrics about the current state and not retrieve any metrics over specific periods, in addition, strategies can only obtain an aggregated value over the entire period instead of obtaining a time series. Effectively, this limits the usability of strategies as they are unable to detect any periodic patterns or resolves issues which occur sporadically. Combined with that it generally takes a long time to run an audit this complicates detecting problems.

Use Cases

As a user I want strategies to detect current and past problems and optimize the infrastructure accordingly.
As a developer I want to make more effective strategies that can detect periodic patterns and make more informed decisions.
As a developer I want to develop an external machine learning component to integrate with Watcher but still use already existing components of Watcher effectively.

Proposed change

To enable time series the datasource base class will need a new method. This method will allow metrics to be retrieved over a specific period with a specific granularity. This method will return a list of values according to the supplied parameters and will subsequently be implemented by all current datasources. An example of how the interface of such a method could look is shown below.

@abc.abstractmethod
def statistic_series(self, resource, resource_type, meter_name, start_time,
                      end_time, granularity):
    """Retrieve metrics based on the specified parameters of a period of time

    :param resource: The object returned by clients such as Server or
    Hypersivor when calling nova.servers.get or nova.hypervisors.get
    :param resource_type: The Type of the resource object selected from
    RESOURCE_TYPES.
    :param start_time: The datetime to start retrieving metrics for
    :type start_time: datetime.datetime
    :param end_time: The datetime to limit the retrieval of metrics to
    :type end_time: datetime.datetime
    :param granularity: Interval between collected data in seconds.
    :return: Dictionary of key value pairs with timestamps and metric values
    """

In addition, a new class will be added that can utilize a datasource to perform time series analysis. Features will include, decomposing trends, periodic variations (daily, weekly, seasonal) and irregular variations (residuals). As well as determining the stationarity. Any machine learning aspects such as predicting future values through models like ARIMA will not be part of this time series class.

A best effort should be made to find a lightweight library that implements the desired time series functionality as libraries such as numpy or scikit-learn are particularly large and offer substantially more functionality than is required. If no suitable library can be found the functionality will be implemented without the use of a third-party library.

Alternatives

The time series analysis along with any potential machine learning features could in its entirety be developed as an external service that integrates with Watcher. This would allow the use of libraries such as numpy and scikit-learn as they are very likely desired dependencies in a machine learning service. The time series method for datasources will still have to be implemented but the class for time series decomposition can be removed.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Collecting lists of values instead of single values per node and or instance can significantly increase the memory consumption of Watcher. Especially, when audits are run for large collections of nodes and instances. Additionally, the transmission of all this data could potentially be a bottleneck if executed sequentially. For this the time series class could utilize the decision-engine threadpool to execute multiple requests in parallel.

Other deployer impact

None

Developer impact

This change will expose a new method in datasources which could potentially be used to enable machine learning features in external projects that integrate with Watcher.

Implementation

Assignee(s)

Primary assignee:: <dantalion>

Work Items

Introduce new method to datasource baseclass
Implement new base method in all datasources
Evaluate available lightweight time series libraries
Implement simple time series class

Dependencies

Potentially a new lightweight time series library will be added as dependency.

Testing

Unit tests for the new method such as validating the specified period
Unit tests to verify the time series decompositions using known dummy data
Unit tests to verify stationarity using known dummy data.
Possible integration tests for testing the retrieval of time series metrics from datasources.

Documentation Impact

No additional documentation is required apart from documenting newly introduced methods and classes.

References

History

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Mon, 27 Apr 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Mon, 27 Apr 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Victoria	Introduced

Example Spec - The title of your blueprint

Mon, 27 Apr 2020 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Victoria	Introduced

Event-driven optimization based

Fri, 14 Feb 2020 00:00:00

https://blueprints.launchpad.net/watcher/+spec/event-driven-optimization-based

Problem description

Watcher has provided ONESHOT and CONTINUOUS audit. ONESHOT audit executes the optimization only once after being launched. CONTINUOUS audit executes the optimization periodly. But there is no way to react to the changes of system, for example an alarm event.

Use Cases

As a user of Watcher, I want to execute optimization as soon as possible when there are alarms in the system.

Proposed change

Many monitor system, such as aodh, can set webhook as alarm action, a POST request will be sent if the alarm is trigered. About the authentication of the POST request, Aodh’s alarm action can set prefix trust to add a X-Auth-Token header to the request url. Watcher add a new config option enable_webhooks_auth to decide whether to enable webhook authentication, the default is True. Watcher add the mechanism for receiving and handling alarm events.

Here is the summary processing: User creates an event based audit, a new audit type, named EVENT, is needed. Creating alarm with Watcher webhook and audit uuid in the monitor system such as aodh. When the alarm is trigered, Watcher will receive the event with audit uuid. Watcher executes the audit and creates actionplan.

Alternatives

CONTINUOUS audit can be used to satisfy the purpose but this will increase the workload.

Data model impact

None

REST API impact

Add a new webhook API

/v1/webhooks/{audit_ident}
- Triggers an event based audit, audit_ident is uuid or name of the audit. Does not perform verification of authentication token.
- Method type: POST
- Normal http response code: 202(Accepted)
- Expected error http response code: 400, 404(Not Found)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Add a new audit type, named EVENT
Add a new webhook API
processing for event based audit

Dependencies

None

Testing

Unit tests for the new feature

Documentation Impact

Add documentation about how to use the new feature

References

https://docs.openstack.org/aodh/latest/

History

Revisions
Release Name	Description
Ussuri	Introduced

Ussuri Project Priorities

Fri, 29 Nov 2019 00:00:00

List of priorities the Watcher drivers team is prioritizing in Ussuri.

Priority	Owner
General purpose decision engine threadpool	Corne Lukken
Kubernetes helm charts and docker images	Corne Lukken
Rollback Mechanism	su zhengwei
Provide scenario jobs for each strategy	Li Canwei
Event-driven optimization based	Li Canwei
the community-wide goals	TBD

General purpose decision engine threadpool

Watcher spends a large portion of time waiting for I/O operations such as writing to disc or waiting for responses to network requests. The time it takes to perform such operations can typically be reduced significantly by implementing parallelism. The general purpose threadpool for the decision engine will allow arbitrary methods to be executed in parallel. The amount of workers in the threadpool should be user configurable.

Kubernetes helm charts and docker images

Blueprint to track all activities related to being able to deploy Watcher on K8s using helm charts. Also requires that the appropriate docker images are created and uploaded to a registry.

Rollback Mechanism

Provide scenario jobs for each strategy

We have provided some tempest tests in Train release, and will continue to improve the scenario tests for strategies.

Event-driven optimization based

We propose an event-driven optimization-based audit control. We wants to select among a list of events which may trigger the audit :

React to a predicted situation.
React to a critical situations and changes in system (e.g: threshold)
A new compute node has been added to the cluster
A compute node has been removed from the cluster
A new virtual machine has been created

The events can be created by AODH, congress, ceilometer, scoring engine (long term).

the community-wide goals

https://governance.openstack.org/tc/goals/ussuri/index.html

Rollback Mechanism

Tue, 29 Oct 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/rollback-mechanism

Problem description

After every audit, there would be one actionplan to execute. The actionplan is a group of actions, such as:

migrations

change nova service state

change node power state

…

Sometimes, the users want to rollback one actionplan even after the actionplan successfully executed.

To tester, they usually make the env unbalance, so that there will be instances migration during audit. After that they need to migrate the instances back to the source host for next test.

Moreover, for host maintenance, it will migrate all instances from the maintaining host to others. After the host maintenance and active again, there is no mechanism to migrate the instances automatically back to the maintaining host.

Use Cases

As a QA of Watcher, I want to rollback the actionplan, so that I can easily restore the environment for next test.

As an openstack operator, I want to migrate the instances back to the source node after the node having been executed “host maintenance” strategy.

Proposed change

In watcher-api, we can add ‘rollback’ action to ActionPlan API. When the api recieves the rollback request and then distributes the request to watcher-decision-engine module.

In watcher-decision-engine, it retrives the actionplan and records every rollback of the actions which belong to the actionplan, then return the rollback result to watcher-api.

Because the compute data model changes by time, the actionplan is possible to rollback failed. To make the rollback mechanism to work as far as possible, only recent(default, recent in one hour) actionplan is allowed to rollback.

Alternatives

None

Data model impact

None

REST API impact

Rollback Action Plan

/v1/action_plans/{actionplan_ident}/rollback
- Method type: POST
- actionplan_ident：UUID of the Action Plan
- Normal http response code(200)
- Expected error http response code(400,404)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <sue>

Work Items

ActionPlan rollback action API
RollBack mechanism for one actionplan

Dependencies

None

Testing

Unit tests

Documentation Impact

Add documentation about how to use the new feature

References

None

History

Revisions
Release Name	Description
Ussuri	Introduced

Rollback Mechanism

Tue, 29 Oct 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/rollback-mechanism

Problem description

After every audit, there would be one actionplan to execute. The actionplan is a group of actions, such as:

migrations

change nova service state

change node power state

…

Sometimes, the users want to rollback one actionplan even after the actionplan successfully executed.

To tester, they usually make the env unbalance, so that there will be instances migration during audit. After that they need to migrate the instances back to the source host for next test.

Use Cases

As a QA of Watcher, I want to rollback the actionplan, so that I can easily restore the environment for next test.

As an openstack operator, I want to migrate the instances back to the source node after the node having been executed “host maintenance” strategy.

Proposed change

In watcher-api, we can add ‘rollback’ action to ActionPlan API. When the api recieves the rollback request and then distributes the request to watcher-decision-engine module.

In watcher-decision-engine, it retrives the actionplan and records every rollback of the actions which belong to the actionplan, then return the rollback result to watcher-api.

Alternatives

None

Data model impact

None

REST API impact

Rollback Action Plan

/v1/action_plans/{actionplan_ident}/rollback
- Method type: POST
- actionplan_ident：UUID of the Action Plan
- Normal http response code(200)
- Expected error http response code(400,404)

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <sue>

Work Items

ActionPlan rollback action API
RollBack mechanism for one actionplan

Dependencies

None

Testing

Unit tests

Documentation Impact

Add documentation about how to use the new feature

References

None

History

Revisions
Release Name	Description
Ussuri	Introduced

General purpose decision engine threadpool

Thu, 17 Oct 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/general-purpose-decision-engine-threadpool

Many I/O operations such as writing to disk or waiting for network responses take a long amount of time. By leveraging parallelism the time taken to perform such operations can be drastically reduced. The general purpose threadpool for the decision engine will allow to perform methods in parallel, thereby reducing the time required to perform these operations.

Problem description

The general purpose threadpool will reduce the time it takes to perform certain operations of the decision engine. Currently, the gathering of metrics or the building of the data model can take large amounts of time proportionally for example.

Use Cases

As a user I want the execution of an audit to be performed as quickly as possible.

Proposed change

Introduce a general purpose threadpool using the futurist library for which end users can configure the amount of threads. This will introduce a new class that uses a singular pattern so it can be used throughout the decision engine. The threadpool will use the GreenThreadPoolExecutor as it does not conflict with other threadpools used in the decision engine. The singleton will contain several methods to simplify the submission of tasks and waiting for their completion.

With the threadpool the building of the data model will be parallelized which is done in three steps. First the calls to retrieve information about aggregates and availability zones are executed in parallel. The information is used to submit tasks to gather the information about each compute node. Finally, using futures will allow to immediately submit a task to retrieve the information about instances for a given compute node.

Alternatives

A purpose build threadpool specific for building the data model. However, this would mean that introducing parallelism in other parts of the decision engine would again require the design and implementation of yet another threadpool.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Overall the time take to perform audits should significantly decrease, however, a large amount of threads could negatively impact external services. Because of this the default amount of threads should be a relatively safe value for most OpenStack infrastructures.

Other deployer impact

Deployers should evaluate the amount of threads they want Watcher to use based on the scale of their infrastructure.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <dantalion>

Work Items

Implement the threadpool singleton based on previous proof of concepts.
Modify the method to build the data model to use the three step parallelism as described.
Implement relevant test cases to test the threadpool and the building of the data model.

Dependencies

futurist, A library maintained and developed by OpenStack.

Testing

Both the behavior of the building of the data model as well as the threadpool should be evaluated.

The verifying of behavior for helper methods in the threadpool.
The verifying of the completeness of the data model when build with parallelization.

Documentation Impact

The documentation will be updated to include usage examples that indicate to other developers how to best use the threadpool. Additionally, typical use cases in what situations this threadpool will be most useful will be provided.

References

History

Revisions
Release Name	Description
Ussuri	Introduced

Example Spec - The title of your blueprint

Thu, 17 Oct 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Ussuri	Introduced

Example Spec - The title of your blueprint

Thu, 17 Oct 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Ussuri	Introduced

Example Spec - The title of your blueprint

Thu, 17 Oct 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Ussuri	Introduced

Energy Saving Strategy

Thu, 19 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/strategy-to-trigger-power-on-and-power-off-actions

This spec describes a new strategy to trigger “power on” and “power off” actions in watcher, which can save energy for those data centers who use watcher to do the resource optimization.

Problem description

For a data center with large amount of VMs and physical hosts,the total power consumption is tremendous. When workload is not heavy, Watcher can be used to reduce power consumption by triggering a request to power off some idle hosts without VMs. And when the workload increases Watcher will trigger a “power on” request to fulfill the service requirements.

This “energy saving” goal can be fulfilled by adding the following subfeatures in Watcher:

Add new actions “power on” and “power off” in Watcher.
Implement a new strategy to generate ‘power on’ and ‘power off’ actions.

This spec implements the second feature: Implement a new strategy to generate “power on” and “power off” actions.

Use Cases

As an openstack operator, I want to keep my data center energy-efficient. When the workload is low, shut down some idle hosts to save energy, and when the workload goes high, power them on to ensure the QoS.

As a developer,I need a strategy to trigger the “power on” and “power off” actions in watcher.

Proposed change

Add one new goal - “energy saving”
Add one new strategy - “Energy Saving Strategy”
The detailed policy in “energy saving strategy” can be described as the followings:

In this policy, a number (min_standby_nodes_num) of standby nodes is given or calculated. The standby nodes refer to those nodes unused but still powered on to deal with boom of new instances.

If the number of unused nodes(in poweron state) is larger than the given number, randomly select the redundant nodes and power off them; If the number of unused nodes(in poweron state) is smaller than the given number and there are spare unused nodes(in poweroff state), randomly select some nodes(unused,poweroff) and power on them.

In this policy, in order to calculate the given number (min_standby_nodes_num), users must provide two parameters:
- One parameter(“standby_nodes_int”) is a number of standby nodes. This number should be int type and larger than zero.
- Another parameter(“standby_used_percent”) is a percentage number, which describes the quotient of number_of_standby_nodes/number_of_nodes_with_VMs, where number_of_nodes_with_VMs is the number of nodes with VMs running on it. Given this percentage and number_of_nodes_with_VMs, we can easily get number_of_standby_nodes. The nodes with VMs refer to those nodes with VMs running on it.
Then chose the larger one as min_standby_nodes_num: min_standby_nodes_num = max(standby_nodes_int, number_of_standby_nodes)

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <li-canwei2>,<yumeng-bao>

Work Items

Add one new goal - “energy saving”
Add one new strategy - “Energy Saving Strategy”
Complete the main functions policy in “energy saving strategy”
1. Complete get_hosts_pool function to Classify all the nodes into three types and count their numbers: nodes_with_vms, unused_poweron, unused__poweroff.
2. calculate the “min_standby_nodes_num” with two given parameters: “standby_used_percent” and “standby_used_int”
3. Write the execute function to compare the number of “unused_poweron nodes” with “min_standby_nodes_num” and decide which nodes should be power on/off.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/add-power-on-off

Testing

Unit and functional test are needed.

Documentation Impact

Add docs on how to use this strategy.

References

None

History

None

Add force field to Audit

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/add-force-field-to-audit

Problem description

Use Cases

As a Watcher user, I want to launch audit despite of ongoing actionplan.

Proposed change

We need to add a new field force in audit table. If force is True, Watcher will execute the audit even other actionplan is ongoing.

def pre_execute(self, audit, request_context):
    LOG.debug("Trigger audit %s", audit.uuid)
    if audit.force is False:
        self.check_ongoing_action_plans(request_context)
    # Write hostname that will execute this audit.
    audit.hostname = CONF.host
    # change state of the audit to ONGOING
    self.update_audit_state(audit, objects.audit.State.ONGOING)

The default value is False. We also need to add an option in the post audit API. So that User can set the value of the force field. CLI example for creating audit with force:

$ watcher audit create -g dummy --force

Alternatives

None

Data model impact

Add new force field in the audit table

REST API impact

Need to add new force parameter in Audits APIs.

Security impact

None

Notifications impact

Add force to AuditPayload

audit.create

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "PENDING",
      "updated_at": null,
      "deleted_at": null,
      "interval": null,
      "scope": [],
      "force": False,
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
     },
    "watcher_object.name": "AuditCreatePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.create",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

Other end user impact

Need to add force to CLI ‘watcher audit create’
Also need to update watcher-dashboard

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Update architecture doc to add the new field
Add new force field in the audit table
Check the value of force before running audit
Update Watcher API version
Update notification AuditPayload
Update python-watcherclient
Update watcher-dashboard

Dependencies

None

Testing

Unittest for all changes

Documentation Impact

Update architecture doc to add the new field.

References

None

History

Revisions
Release Name	Description
Train	Introduced

File based Metric Map

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/file-base-metric-map

Problem description

Watcher supports loading metrics for various datasources however the metrics it uses to retrieve data are stored directly in the codebase and do not allow flexibly changing these names. And customers/admin/user are forced to use a naming scheme expected by watcher. Also it hinders Watcher ease of integration if a particular metric is already in use under a different name.

Use Cases

As a Watcher user, I want that Watcher should be allow choosing a metric name as defined in current datasource being used.

As a Watcher user, I want that Watcher to be able to change the metrics names as required.

Proposed change

An option of metric_map_path should be added to watcher_decision_engine section of watcher.conf

Allow Watcher to load a yaml file which contains a map of internal metric names (as expected by Watcher) to real metrics names in use in the datasources.

The file is optional and needs to contain only those metrics names that the user expects to override.

A method in DataSourceManager class could be called in its __init__ method.

For example:

[watcher_decision_engine]
# ...
metric_map_path = /etc/watcher/metricmap.yaml
# ...

monasca:
  - instance_cpu_usage: VM_CPU
gnocchi:
  - instance_cpu_usage: cpu_vm_util

class DataSourceManager(object):
    def __init__(self):
        #...
        # 1. Initial loading is necessary to initialize the correct defaults
        self.metric_map = {
            mon.MonascaHelper.NAME: mon.MonascaHelper.METRIC_MAP,
            gnoc.GnocchiHelper.NAME: gnoc.GnocchiHelper.METRIC_MAP,
            ceil.CeilometerHelper.NAME: ceil.CeilometerHelper.METRIC_MAP
        }
        new_metric_map = self.metrics_from_file():
        # 2. overide the loaded default by using yaml
        # update self.metric_map recursively using new_metric_map

    def get_backend(self, metrics):
        #...
        if not no_metric:
            ds = getattr(self, datasource)
            # 3. Pass the re-loaded metric map to the datasource
            ds.METRIP_MAP.update(self.metric_map[ds.NAME])
            return ds

    def metrics_from_file(self):
        """Load metrics from the config.metric_map_path"""
        if not os.path.exists(config.metric_map_path or ''):
            return {}
        with open(config.metric_map_path, 'r') as f:
            try:
                return yaml.safe_load(f.readall())
            except yaml.YAMLERROR as e:
                log.info('Could not load %(s): %s' % (
                    config.metric_map_path, str(e)))
                return {}

Comments about implementation:

Step 1: This loading can be avoided if the defaults are to be never stored in python files. But that is outside the scope of this spec as that will make the yaml file mandatory.

Step 2: self.metric_map is a nested dict so will need special handling for recursive update to datasources.

Step 3: This loading can be moved to the individual getter’s we have already defined an “API for metric_map communication” by making METRIC_MAP dictionary a class variable.

Alternatives

Instead of updating the METRIC_MAP directly in the datasource, the map could be passed to each datasoruce classes, but that spills the change over to other classes and files and also this delegation of responsibility is not contributing to any reasonable advantage.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

A configuration option should be added to the ‘watcher_decision_engine’ section, however this is not a requirement, but rather a good-to-have as this can lead to potential name conflict.

A user/admin can configure a new parameter in the config, but this is a non-binding config.

Performance Impact

None

Other deployer impact

No impact as the file creation/existence is optional

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <sumitjami>

Work Items

Add a configuration option ‘metric_map_file’
Add new method to DataSourceManager class
Load the file and update the metric_map variable
update the METRIC_MAP dict of datasource in get_backend method

Dependencies

None

Testing

Add unit test for file existance error check.
Add unit test for checking if file contents are loaded correctly.

Documentation Impact

Update Watcher developer documents.

References

None

History

Revisions
Release Name	Description
Train	Introduced

Formal Datasource Interface

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/formal-datasource-interface

Different strategies pass different values to parameters of datasource methods while these should be unified as implied by the datasource base class. The formalizing of the datasource base class will ensure that all strategies can work regardless of the underlying datasource. Additionally, the return types of methods should also be unified.

Problem description

Watcher strategies rely on specific datasources to work while strategies should work regardless of the underlying datasource. The incompatibilities are due to different types of values being passed to the parameters of methods. Most of these problems are artifacts of back when there was no datasource base class.

Use Cases

As end users I want strategies to work regardless of the datasource I have deployed.

As developer I want to more easily develop or improve new strategies while minimizing how much I should worry about the underlying datasource.

Proposed change

The datasource base class is changed and comments are added to describe expected values and units for parameters. For some parameters were the value can be one of a set of options the list of options is maintained in a list in the base class.

The new lists will be AGGREGATES and RESOURCE_TYPES. AGGREGATES contains all the possible options to pass to the aggregate parameter.

The possible options for AGGREGATES are mean, min and max. Mean will replace every instance of avg and is preferred as it is less ambiguous.

AGGREGATES = ['mean', 'min', 'max', 'count']

Many different parameters are shared across different methods and they share the same types of values. These parameters are

resource_id
period
granularity
aggregate

The resource_id is renamed to resource and should be the object retrieved for the specific resource but more context about the resource should be passed to the datasource. The RESOURCE_TYPES list addresses this needed context. It contains all possible types of resources this will allow the datasource to access the object attribute it needs for data retrieval.

RESOURCE_TYPES = ['compute_node', 'instance', 'bare_metal', 'storage']

The resource types is required to access the correct attributes of the resource object as the names of attributes vary between objects.

"""Demonstration accessing the same data from different objects"""
if resource_type == 'compute_node':
  hostname = resource.hypervisor_hostname
elif resource_type == 'instance':
  hostname = resource.name

raw_kwargs = dict(
  name=meter_name,
  start_time=start_timestamp,
  end_time=end_timestamp,
  dimensions={'hostname': hostname},
)

The names of the resource are based on names in the helper classes to simplify accessing these APIs. The method names in the ironic_helper are will be changed to better fit the names in other helpers such as cinder_helper.

The period is be used to specify the amount of time in seconds that is used to aggregate metrics.

The granularity is be used to specify the interval between the time series in seconds.

The aggregate parameter is already covered as the values are chosen from AGGREGATES.

The statistic_aggregation method no longer uses the meter_name values from the METRIC_MAP dictionary as identifier but will use the keys instead. The datasources can still use the values of the METRIC_MAP to determine how to retrieve metrics from their databases. The parameter aggregation is renamed to aggregate to match the names used by other methods. The parameters group_by and dimensions are removed.

The definitions off most methods in DataSourceBase will now look as detailed below.

def statistic_aggregation(self, resource=None, resource_type,
                          meter_name=None, period=300,
                          granularity=300, aggregate='mean')

def get_host_cpu_usage(self, resource, resource_type, period, aggregate,
                       granularity=None)

def get_host_memory_usage(self, resource, resource_type, period, aggregate,
                          granularity=None)

The expected values and return types will be documented in the DataSourceBase using code blocks.

@abc.abstractmethod
def get_host_cpu_usage(self, resource, resource_type, period, aggregate,
                      granularity=None):
    """ Get the amount of cpu usage for the host

    :param resource: The object returned by clients such as Server or
    Hypersivor when calling nova.servers.get or nova.hypervisors.get
    :param resource_type: The Type of the resource object selected from
    RESOURCE_TYPES.
    :param period: The amount of seconds back in time metrics are aggregated
    over.
    :param aggregate: The method to aggregate data selected from AGGREGATES.
    :param granularity: Interval between collected data in seconds.
    :return: Percentage of total cpu usage represented by float between 0-100
    """

Not all datasources will be able to implement all these different options. As example some datasource do not support granularity and most do not support the count aggregate. These incompatibilities should be met with reasonable alternatives and warnings but throwing errors should be avoided.

Finally, the list_metrics and check_availability methods are used by the API to return information on strategies when executing the state call.

Alternatives

Accepting that certain strategies only work with specific datasources.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

All strategies and some datasource will have to be adapted to be compatible with the new DataSourceBase and this will require development effort.

Implementation

Assignee(s)

Primary assignee:: <Dantali0n>

Work Items

Change DataSourceBase according to new interface specification.
Write comments in DataSourceBase to document interface.
Adapt existing datasources to work with new interface.
Change unit tests to work with new interface.

Dependencies

None

Testing

Current datasource tests have to be adopted to work with the new datasource base class. With the removal of the dimensions parameter the Monasca test cases will require the most changes.

In addition to unit tests the correct functionality of the datasources are examined in a working environment such as devstack.

Documentation Impact

None

References

None

History

None

Config global datasource preference

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/global-datasource-preference

Currently, datasources need to be configured on a per strategy basis. Configuring the datasources per strategy is error prone and time consuming. By being able to configure a global datasource preference the time to configure Watcher is reduced and there is a single point for configuring the datasources used by Watcher.

Problem description

The datasources available to Watcher differs per OpenStack deployment so there can not be a standard preference that will work for all deployments. Configuring which datasources are preferred is essential to correctly deploying Watcher, however, the datasource preference needs to be defined per strategy. As a result the datasource preference will have to be configured many times which can lead to errors and costs more time.

Use Cases

As a user of Watcher, I want to configure the datasources only once.

As a user of Watcher, I want to configure the datasources only once but have specific exceptions for some strategies.

Proposed change

Allow specifying of global datasource order by exposing new configuration parameter. This parameter will be part of a new group labeled watcher_datasource. The datasource parameter for strategies remains available and overrides the globally configured datasources.

[watcher_datasource]
datasources = ceilometer, gnocchi, monasca

Alternatives

Try datasources but ensure that if they are not properly configured an error is raised and an alternative datasource is tried. Major downside of this approach is that it will create a lot of errors in some scenarios.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <dantalion>

Work Items

Add configuration parameter to specify an order of preferred datasources.
Read global datasource preference from config on selecting backend.
Allow specific strategy datasource configurations to preference.
On error attempt the next datasource.

Dependencies

None

Testing

Determine if global preference is used correctly.
Determine if strategies can override the global preference.

Documentation Impact

Documentation on configuration parameters will need to be updated to reflect the new parameter.

References

https://review.opendev.org/#/c/645294/

History

None

Grafana proxy datasource

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/grafana-proxy-datasaource

Watcher requires metrics from compute nodes and instances to perform the resource optimization. Metrics are retrieved using several different datasources such as Gnocchi and Monasca, however, not every OpenStack cloud might have the available datasources deployed. Grafana can be implemented as an additional datasource that can be used to retrieve metrics from several different databases that the Grafana endpoint is configured to use.

Problem description

Not every OpenStack cloud might use the currently available datasources limiting the use of Watcher. By offering a widely used monitoring platform as datasource with flexible configurations options more OpenStack clouds could start using Watcher.

Use Cases

As a service operator I want Watcher to integrate with currently deployed monitoring solutions.

As a service operator I want to limit the amount of external credentials I have to configure for Watcher.

Proposed change

The new Grafana datasource will be able to query for different metrics depending on what is configured by the end user. Flexible configuration options will allow Grafana to work for each user’s configuration. Some of these options will be configurable per metric using key value pairs in a dictionary. These options are called maps as they map the value to a specific metric. Configuration options include:

Endpoint url
Authorization token
Project id map
Attribute map
Database map
Translator map
Query map

A configuration example configured for the host_cpu_usage and instance_cpu_usage metrics could look as follows:

[grafana_client]
token = uyyNKUJOZiLW7AVKRF7XAAAAQQDzoXbnS6cOxxcqJfS8ZEQyxgakF0bSUo0D==
base_url = https://grafana.ch/api/datasources/proxy/
project_id_map = host_cpu_usage:1337,instance_cpu_usage:4337
metric_db_map = host_cpu_usage:production_db,instance_cpu_usage:production_db
attribute_map = host_cpu_usage:hostname,instance_cpu_usage:human_id
translator_map = host_cpu_usage:influxdb,instance_cpu_usage:influxdb
query_map = host_cpu_usage:SELECT 100-{0}("{0}_value") FROM {3}.cpu_percent
            WHERE ("host" =~ /^{1}$/ AND "type_instance" =~/^idle$/ AND
            time > now()-{2}m),instance_cpu_usage:SELECT 100-{0}("{0}_value")
            FROM {3}.cpu_percent WHERE ("host" =~ /^{1}$/ AND "type_instance"
            =~/^idle$/ AND time > now()-{2}m)

Grafana uses project ids to proxy to different databases each of these projects could contain a different type of database. The term project will be used throughout this document to prevent possible confusion. All the desired metrics can be collected from multiple project or a single one depending on how the monitoring is configured but is limited to a single metric per project. This is because there is no method to aggregate a single metric across multiple projects.

The way queries have to be sent to the endpoint and how to interpret the retrieved data will depend upon the project behind Grafana. To account for these differences between projects specific translators will be developed. The influxdb translator will be developed first. The translator map is used to perform the correct translations per metric depending on the type of project.

Projects could contain one or more databases similar to schema’s in MySQL. The database map allows to define a specific database per metric.

Similar to the project map and the database map is the query map. This map contains the queries that will be send to the project to retrieve metrics. Queries are depended on the type of project and in the case of influxdb they are similar to SQL statements.

The attribute map is used to select specific attribute from the resource objects. This is necessary because the attribute used as identifier in projects can differ per deployment and per metric.

From the query map the entries will be formatted so that essential information for retrieving the specific metric for the desired host can be achieved.

query = 'SELECT "{0}_value" FROM cpu_util WHERE host =~ /^{1}$/ AND time > '
        'now() - {2}m'
query.format(aggregate, attribute, period, translator_specific)

The format options can be extended overtime in case other specific projects such as elastic search require different parameters to successfully build a query.

The initial format options will be:

{0} = aggregate
{1} = attribute
{2} = period
{3} = { influxdb: retention_period, }

Because the amount of metrics available to Grafana depends on user configuration some minor changes are made to the datasource manager to build the metric list for Grafana at runtime.

Instead of configuration many parameters using the default configuration file the metric yaml can used to set the configuration but the expected parameters differ from other datasources because of the large amount of parameters.

grafana:
  host_cpu_usage:
    project: 1337
    db: production_db
    attribute: hostname
    translator: influxdb
    query: SELECT 100-{0}("{0}_value") FROM {3}.cpu_percent
            WHERE ("host" =~ /^{1}$/ AND "type_instance" =~/^idle$/ AND
            time > now()-{2}m)

Alternatives

Datasources for individual projects that Grafana integrates with could be developed but this would be significantly more development effort and possibly complicate authorization as it would have to be configured per database.

Data model impact

None

REST API impact

None

Security impact

The configuration file will need to contain the Grafana authorization token which provides read access to the databases Grafana is configured for. The configuration file already contains other important credentials.

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Dantali0n

Work Items

Configuration options
General Grafana datasource
Translator interface
InfluxDB translator
Unit tests for Grafana
Unit tests for translators

Dependencies

The communication with Grafana is realized using the requests library

Testing

Unit tests for both the datasource itself as well as the translator base class and any subsequent translators will be created.

Documentation Impact

A page containing documentation on how end user’s can configure the options to successfully use Grafana as a datasource will be created.

References

History

Revisions
Release Name	Description
Train	Introduced

Improve Compute Data Model

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/improve-compute-data-model

Problem description

The fields(vcpus, memory and disk_capacity) in the Watcher ComputeNode correspond to the fields(vcpus, memory_mb and local_gb) in Nova API list-hypervisors-details. Unfortunately, these fields do not take allocation ratios used for overcommit into account so there may be disparity between this and the used count.

Use Cases

As a Watcher user, I wish Watcher can work correctly in the case of overcommit resources(cpu, memory and disk).

Proposed change

Now Watcher had added Placement helper. Watcher can get resource information such as total, allocation ratio and reserved information from API list resource provider inventories.

Watcher ComputeNode

We need to add some fields to the Watcher ComputeNode:

vcpu_reserved: The amount of cpu a node has reserved for its own use.
vcpu_ratio: CPU allocation ratio.
memory_mb_reserved: The amount of memory a node has reserved for its own use.
memory_ratio: Memory allocation ratio.
disk_gb_reserved: The amount of disk a node has reserved for its own use.
disk_ratio: Disk allocation ratio.

We can calculate the total resource capacity through the total, reserved and allocation ratio.

The formula: vcpu capacity = (vcpus-vcpu_reserved)*vcpu_ratio

For example, for vcpu resource with:

vcpus = 8
vcpu_reserved = 2
vcpu_ratio = 5.0

The vcpu capacity is (8 - 2) * 5.0 = 30.

We also add some new propeties to the Watcher ComputeNode:

vcpu_capacity: The amount of vcpu, take allocation ratio into account, but do not include reserved.
memory_mb_capacity: The amount of memory, take allocation ratio into account, but do not include reserved.
disk_gb_capacity: The amount of disk, take allocation ratio into account, but do not include reserved.

Compute Data Model

CDM(Compute Data Model) will be built when creating audit and updated when receiving Nova notifications. So when building and updating CDM we need to get resource information from Placement API.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Add new fields to Watcher ComputeNode
Update compute collector
Update the processing about Nova notifications

Dependencies

None

Testing

Add unit test

Documentation Impact

None

References

https://developer.openstack.org/api-ref/compute/

https://developer.openstack.org/api-ref/placement/

History

Revisions
Release Name	Description
Train	Introduced

Node Resource Consolidation

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/node-resource-consolidation

Problem description

In the edge computing system, the edge nodes are distributed in different locations. Although the total number of edge nodes may be large, the number of edge nodes in each location is limited, so the resources(VCPU, memory) are limited. An application scenario is to dynamically create VM(application) processing service. After the processing is complete, the VM is deleted and the resource is released. During this process(creating and deleting VM on different nodes), resource fragments are generated. For example, if two nodes each have two free VCPUs, even if the total VCPUs is enough, creating a VM with four VCPUs will still fail.

Use Cases

As a Watcher user, I wish Watcher provides a strategy which can eliminate resource fragmentation by consolidating resources on nodes.

Proposed change

Add a new strategy: Node Resource Consolidation
One input parameter: host_choice(specify/auto). This parameter determines how to select the server migration destination node. The value auto means that Nova schedular selects the destination node, and specify means the strategy specifies the destination. The default value is auto.
The algorithm of this strategy:
- Caculating the used resources of compute nodes
- Sorting compute nodes by the used resources percent
- Dividing compute nodes into source group and destination group. The process: For sorted compute nodes, from the one that uses least resources, if all servers can be migrated to other compute nodes, we put this node into source group, and other nodes into destination group. repeating this process until all compute nodes are placed in the source or destination group. For continuous audit, if there is server that had failed during previous actionplan, we should put this compute node to the destination group.
- Creating strategy solution: If the parameter host_choice is auto, creating a migration action for each VM in the source group and compute nodes in source should be disabled before server migration and be enabled after finishing the migration. If the parameter host_choice is specify, the nodes in the destination group are sorted by free resources. Then, for each VM on the source node, select a node that is most suitable for the VM from the destination group as the destination node of the migration. Repeat this process (reordering the destination group by free resources each time), until all VMs have destination or there is insufficient resources.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Add a new parameter audit to the method do_execute of strategy.
Add the new strategy.

Dependencies

None

Testing

Unit and functional test are needed.

Documentation Impact

Add docs on how to use this strategy.

References

None

History

Revisions
Release Name	Description
Train	Introduced

Support keystoneclient option

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/support-keystoneclient-option

Currently, users can modify the default parameters through the configuration file when creating novaclient and cinderclient, but it is not allowed when creating keystoneclient.

Problem description

The type of endpoint and region name can not be set when creating keystoneclient. Users may want to modify these parameters in different demand scenarios.

Use Cases

As a user of Watcher, I want to specify the type of endpoint and region name when creating keystoneclient by the configuration file.

As a user of Watcher, I want to specify the type of endpoint as internal when creating keystoneclient by the configuration file.

As a user of Watcher, I want to specify the value of region name as RegionTwo when creating keystoneclient by the configuration file.

Proposed change

Allow specifying the type of endpoint and region name when creating keystoneclient by exposing new configuration parameter. These parameters will be part of a new group labeled keystone_client.

[keystone_client]
interface = internal
region_name = RegionOne

The interface will default to admin since that is what is used today:

https://github.com/openstack/python-keystoneclient/blob/3.15.0/keystoneclient/httpclient.py#L251

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <chenker>

Work Items

Add the definition and register of keystone_client group.
Add the definition and register of keystone_client option.
Update the default access method of keystone_client option.

Dependencies

None

Testing

Determine if keystone_client option is used correctly.

Documentation Impact

Documentation on configuration parameters will need to be updated to reflect the new parameters.

References

https://review.opendev.org/#/c/658102/

History

Revisions
Release Name	Description
Train	Introduced

Support Placement API

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/support-placement-api

Problem description

Placement provides a service for managing, selecting, and claiming available resources in a cloud. It was introduced in the Newton release within the nova repository and extracted to the placement repository in the Stein release. Now Watcher gets data directly from Nova for building data model and strategies processing. But some data, such as overcommit allocation ratio for VCPU and Ram, can’t get from Nova. The knowledge available to strategies to make informed decisions is currently limited due to the placement api being unavailable.

Use Cases

As a Watcher developer, I want to get data from Placement.

Proposed change

In this spec we just add Placement helper to Watcher. We plan to improve the data model and strategies in the future specs. The functions in the helper as below:

List resource providers
List resource classes
List resource provider inventories
List resource provider traits
List resource provider allocations
List resource provider usages
List allocation candidates

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Add a new config section [placement_client].

Some config options:

api_version: The minimum version restricted to a given Major API.
interface: The default interface for URL discovery.
region_name: The default region_name for URL discovery.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Add Placement helper
Add unit test

Dependencies

None

Testing

Unittest for Placement helper

Documentation Impact

None

References

https://developer.openstack.org/api-ref/placement/

History

Revisions
Release Name	Description
Train	Introduced

Watcher Planner Selector

Tue, 17 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/watcher-planner-selector

Watcher Planner takes the set of Actions generated by a Strategy and builds the design of a workflow which defines how-to schedule in time those different Actions and for each Action what are the prerequisite conditions. It is only possible to choose only one (default) Planner for all strategies now.

Custom Planners could be added to Watcher, thereby users can adapt Watcher workflow to their needs. As one of the ways for customization, the option could be added to set concrete Planner for specific Strategy that takes features of users’ workflow into account.

Problem description

Currently there is no opportunity to set unique Planner to specific Strategy. User can set default Planner in config file that would be used alongside every Audit.

If using different Planner is the case, user should change default value to desired one. By doing this we restrict ourselves to use this new Planner with all of the Strategies, what is not desired because some of Solutions generated by Strategies could be not compatible with the schedule made by this Planner.

Use Cases

As an Strategy developer, I want to choose what Planner would be applied to the Solution generated by my Strategy.
As an Strategy developer, I expect that if I provide no Planner to my Strategy, it will use default Planner that is set in base Strategy..

Proposed change

We propose to add property to base Strategy class, say ‘planner’, and set default value to this option in concrete Strategies where appropriate (for example, ‘workload_stabilization’ Planner as default for Strategy with the same name).

Planner should be load in AuditHandler. Currently it’s done with the help of PlannerManager, that loads sole planner for every Strategy which set by CONF.watcher_planner.planner value. Definitely, PlannerManager do only loading job, so it’s better to load Planner without this intermediate layer. Thus we propose to exchange AuditHandler._planner_manager with DefaultPlannerLoader. As a result, PlannerManager could be removed from Watcher.

Therefore, AuditHandler now should pass desired Planner name to Loader. We propose to get this name from property of the Strategy used in current Audit. In this case we ought to get Strategy object associated with current Audit.

For this propose, we need to be able to get Strategy prior to AuditHandler._planner_loader.load() method call. It could be be done with the help of StrategyContext class that loads Strategy based on names of Goal and Audit. The problem is that this loading is done during do_execute_strategy() what’s a bad design. We propose to move selecting of strategy to distinct method, say ‘select_strategy’:

def select_strategy(audit, request_context):
    osc = clients.OpenStackClients()
    goal = objects.Goal.get_by_id(request_context, audit.goal_id)

    strategy_name = None
    if audit.strategy_id:
        strategy = objects.Strategy.get_by_id(
            request_context, audit.strategy_id)
        strategy_name = strategy.name

    strategy_selector = default.DefaultStrategySelector(
        goal_name=goal.name,
        strategy_name=strategy_name,
        osc=osc)
    return strategy_selector.select()

Now we could call this method not only in do_execute_strategy() to pick strategy only for execution, but also call it in AuditHandler.

Another benefit to do it in this manner is that if Audit was created without specified Strategy, StrategyContext will take care to retrieve default Strategy for this Goal.

By design, whole AuditHandler class passes current Audit as an argument of many of its methods. It would be convenient to have current Audit as a field of AuditHandler class. Also, it is good design decision, because it’s true OOP paradigm. To sum up, we propose to remove all these arguments and replace them with class field.

As a result, we can get Planner for current Strategy without adding new argument to AuditHandler.planner() or storing name of Strategy in additional field.

Alternatives

Alternatively, we could pass Planner name as argument during creation of AuditTemplate.

As an alternative to changing signatures of AuditHandler methods we could add a new field only for Strategy name.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Signature of PlannerManager.load() method changes. AuditHandler methods change their signatures, ‘audit’ argument would be replaced with same class field.

Implementation

Assignee(s)

Primary assignee:: Egor Panfilov <erakli>

Work Items

Refactor DefaultStrategyContext.do_execute_strategy() and add select_strategy() method
Add ‘planner’ property to BaseStrategy
Replace PlannerManager with PlannerLoader in AuditHandler
Rewrite AuditHandler methods so that ‘audit’ argument would be replaced with class’ field

Dependencies

https://blueprints.launchpad.net/watcher/+spec/planner-storage-action-plan

Testing

Tests with different Planners assigned to different Strategies should be implemented.

Documentation Impact

Documentation should be updated to explain how to set ‘planner’ option.

References

None

History

None

General purpose decision engine threadpool

Tue, 03 Sep 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/general-purpose-decision-engine-threadpool

Problem description

Use Cases

As a user I want the execution of an audit to be performed as quickly as possible.

Proposed change

Alternatives

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Other deployer impact

Deployers should evaluate the amount of threads they want Watcher to use based on the scale of their infrastructure.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <dantalion>

Work Items

Implement the threadpool singleton based on previous proof of concepts.
Modify the method to build the data model to use the three step parallelism as described.
Implement relevant test cases to test the threadpool and the building of the data model.

Dependencies

futurist, A library maintained and developed by OpenStack.

Testing

Both the behavior of the building of the data model as well as the threadpool should be evaluated.

The verifying of behavior for helper methods in the threadpool.
The verifying of the completeness of the data model when build with parallelization.

Documentation Impact

References

History

Revisions
Release Name	Description
Ussuri	Introduced

Purge soft-deleted Watcher objects

Sun, 12 May 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/db-purge-engine

Watcher never deletes data from its DBs and only uses soft deletion (There is special column in each table indicating whether or not an entry is deleted). We keep these data for potential future ML processing in order to improve Watcher auditing recommendations. However, we shall provide a purge mechanism to Cloud Administrator in order to limit the size of this database and make sure it does not increase too much.

Problem description

As of now, Watcher does not provide anything to facilitate the removal of soft deleted objects. So the only way to destroy objects into the Watcher database is to connect directly onto the database backend server and to execute DB commands (or script) to remove objects in tables. Moreover, some objects can be linked (e.g. a set of Action objects are linked to a same Action Plan object, the latter being linked to a unique Audit object). Remove manually such objects can create orphan (e.g. an Action oject with no more linked Action Plan object) in the database. Finally, you could introduce inconsistencies into the database by removing a subset of objects having a same parent (e.g. remove half of action objects linked to a same action plan object will break this action plan integrity).

Watcher database size is not unlimited and queries performance can be degraded due to a huge quantity of soft-deleted objects remained into the Db storage.

Finally, usage of soft-deleted objects is not clearly defined today.

Use Cases

As a Cloud Administrator, I want to manage the size of the Watcher database by purging all soft-deleted objects.

As a Cloud Administrator, I want to manage the size of the Watcher database by purging all soft-deleted objects, apart from orphans objects.

As a Cloud Administrator, I should be able to purge a subset of the Watcher objects from the database by providing a validity interval as an argument (in days) which shall not be destroyed. (e.g. I want to delete objects older than 30 days).

As a Cloud Administrator, I should be able to purge all objects linked to a particular Audit Template, by providing, as argument, the Audit Template UUID.

As a Cloud Administrator, I would like to be notified about the number of objects to be deleted (this number should be computed by Watcher), before validating or not this purge operation.

As a Cloud Administrator, I would like to get, as result, a detailed summary of the purge operation (i.e. the total number of deleted objects, the number of deleted Audit Templates, the number of deleted Audits, the number of deleted Action Plans, the number of deleted Actions).

As a Cloud Administrator, I would like to execute purge command in dry-run mode, for which objects selected for deletion will be not purged.

As a Developer, when an Audit template object is removed, all linked Audit objects shall be removed.

As a Developer, when an Audit object is removed, all linked Action plans objects shall be removed.

As a Developer, when an Action Plan object is removed, all linked Actions objects should be removed.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Add an option to watcher-db-manage command tool to be able to purge the Watcher database.

Alternatives

Do nothing and let operators handle this on their own.

Data model impact

None

REST API impact

None

Security impact

The watcher-db-manage command is only available to administrator. Obviously any entry point to deleting data permanently is dangerous but this spec assumes the deployer has taken the necessary security precautions to lock down access to the watcher-db-manage command already.

Notifications impact

None

Other end user impact

None

Performance Impact

There could be some impact when doing a large purge, so as part of the command implementation there will be a –max-number option. Note that the object integrity must be respected: We shall not remove a part of objects associated in the same parent object. For example, removing of the half of Actions recommended by an Action Plan (because we reach the max number of purged object) does not make any sense, and the integrity of this action plan will be violated.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: None

Work Items

Update DBCommand class by adding new option ‘purge’
Implement purge feature
Update ‘Watcher Manual Pages’ documentation

Dependencies

A lot of OpenStack projects already propose purge mechanism. It would be pertinent to have a look on their mechanism implementation, if needed.

glance : https://review.opendev.org/#/c/216782/
cinder : https://review.opendev.org/#/c/146766/
nova : https://review.opendev.org/#/c/203751/
ospurge : https://github.com/openstack/ospurge

Testing

Add these functional tests :

I purge all objects marked as DELETED.
I purge all objects marked as DELETED, apart from orphan ones.
I purge all objects marked as DELETED, in a maximum limit of M objects.
I purge all objects marked as DELETED and linked to a dedicated Audit Template
I purge all objects marked as DELETED and linked to a dedicated Audit Template, in a maximum limit of M objects.
I purge all objects marked as DELETED and older than N days.
I purge all objects marked as DELETED and older than N days, in a maximum limit of M objects.
I purge all objects marked as DELETED, older than N days and linked to a dedicated Audit Template.
I purge all objects marked as DELETED, older than N days and linked to a dedicated Audit Template, in a maximum limit of M objects.
I run a dry-run purge to check which objects would be purged, according to my settings.

For all of these tests, we should be able to validate integrity of remaining objects (no orphans, no partial group of objects linked to a same parent).

Documentation Impact

Add a new page into ‘Watcher Manual Pages’.

References

None

History

None

Add new actions “power on” and “power off” in Watcher

Sun, 12 May 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/add-power-on-off

For a data center with large amount of VMs and physical hosts,the total power consumption is tremendous. When workload is not heavy, Watcher can be used to reduce power consumption by triggering a request to power off some idle hosts without VMs. And when the workload increases watcher will trigger a “power on” request to fulfill the service requirements.

This feature includes four sub-features:

Build a new baremetal data model in Watcher.
Add new actions “power on” and “power off” in Watcher.
Implement a new strategy based on ‘baremetal’ and ‘compute’ data models, which could trigger ‘compute’ and ‘baremetal’ actions.
Update bare metal data model by ironic notifications.

This spec implements the second sub-feature:Add new actions “power on” and “power off” in Watcher.

Problem description

The telecommunication network traffic tidal [1] is an actual phenomenon in the telecommunication. For example, during the daytime, large amount of users gather in the CBD[2] area and the workload of network traffic system is high. When they get off work, the number of users declines rapidly and so does the network traffic workload. The problem is how to fulfill user network service requirements with minimal energy consumption given the changing network traffic workloads.

As a real case from ZTE corporation, this problem is addressed by the following way. Firstly, a boundary value is set to describe the maximum percentage of idle hosts in a cluster, which is configurable by User. When the ratio of idle hosts gets higher than the boundary value, we power off some idle hosts until the ratio reduces to the boundary value. Correspondingly, when this ratio becomes lower than the preset value, we power on some hosts until the ration increases to the boundary value.

The feature in this spec will implement only the power on and power off actions, the feature of deciding which hosts will be powered off/on according to a boundary value will be left for future work.

Use Cases

As a deployer, I want to have a “power off” action in watcher to help reduce energy consumption by automatically cutting down the number of running hosts when workload is low.

As a deployer, I want to have a “power off” action in watcher to automatically increase the number of running hosts when workload increases and requires more resources.

Proposed change

Given the target hosts, we use Ironic Client to call IPMI tools to do the “power on” and “power off” actions through the follwing Ironic API: https://developer.openstack.org/api-ref/baremetal/?expanded=change-node-power-state-detail

Host state verification before executing the power on/off audit.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Test this new action using unit test. Or as an alternative, developer can use an actuator strategy to allow excplicit action execution. This actuator strategy is described in the following patchset: https://review.opendev.org/#/c/425110/.

Implementation

Assignee(s)

Primary assignee:: <li-canwei2>
Other contributors:: <alexchadin>,<yumeng-bao>

Work Items

Acquire target hosts from Watcher baremetal data model. Implement action:’change_node_power_state’,where the input parameters of this action is ‘on’ or ‘off’.

Dependencies

This feature is in dependent with the following blueprint: https://blueprints.launchpad.net/watcher/+spec/build-baremetal-data-model-in-watcher

Testing

Unit tests
Tempest test

Documentation Impact

None

References

[1]http://ieeexplore.ieee.org/abstract/document/7179335/ [2]https://en.wikipedia.org/wiki/Central_business_district [3]https://developer.openstack.org/api-ref/baremetal/?expanded=show-node-details-detail#Response

History

None

Revisions
Release Name	Description
Pike	Introduced

Support Description For Dynamic Action

Sun, 12 May 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/dynamic-action-description

Add description property for dynamic action. Admin can see detail information of any specify action.

Problem description

Currently, a cloud admin has no way to see the description of planned actions before running an action plan. This literal description is important when the cloud admin wants to see detailed information about a recommended action plan.

Use Cases

As a cloud administrator, I want to be able to see the description of planned actions before running an action plan.

Proposed change

Allow RESTful API(GET /v1/actions/(action_uuid)) to get description information of a specified action.

Define a mapping of action_type and description
- action_type: ‘migrate’
  
  description: ‘Moving a VM instance from source_node to destination_node’
- action_type: ‘change_nova_service_state’
  
  description: ‘Disables or enables the nova-compute service.
  A disabled nova-compute service can not be selected by the nova for future deployment of new server.’
- action_type: ‘resize’
  
  description: ‘Resize a server with specified flavor.’
- action_type: ‘sleep’
  
  description: ‘Wait for a given interval in seconds.’
- action_type: ‘nop’
  
  description: ‘Logging a NOP message’
Add a new table to save the mapping
- This table includes action_type and action description.
Show description of a specified action
- Add the logic of computing action_description to “get_one” method in watcher/api/controllers/v1/action.py

Alternatives

Implement 2 notifications:

In order to share the description of all the action plugins installed alongside the Applier so that we wouldn’t need to also install these plugins on the API side
- the 1st one the API side that requests the action descriptions and the 2nd one on the Applier side that emits all the actions, including their descriptions (both notifications will have to be versioned).
- The latter will also have to be emitted upon starting the Applier to notify the API of any change of description.

action_info.request

{
  "priority": "INFO",
  "payload": {
    "watcher_object.namespace": "watcher",
    "watcher_object.version": "1.0",
    "watcher_object.name": "ActionInfoPayload",
    "watcher_object.data": {
      "description": "moving a VM instance",
      "action_type": "migrate",
    },
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2017-01-04 16:31:36.264673",
  "event_type": "action_info.request",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

action_info.emit

{
  "priority": "INFO",
  "payload": {
    "watcher_object.namespace": "watcher"
    "watcher_object.version": "1.0",
    "watcher_object.name": "ActionInfoPayload",
    "watcher_object.data": {
      "action_type": "migrate",
      "description": "moving a VM instance",
    },
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2017-01-04 16:31:36.264673",
  "event_type": "action_info.emit",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

The implementation is as follows: https://review.opendev.org/#/c/454638/ But in my test, The number of received notifications is often less than the number of notifications sent.

Data model impact

Add a new table named ‘action_descriptions’. This table includes action_type and action description.

REST API impact

URLs with /v1/actions/ and that uses the type Action:

GET /v1/actions/(action_uuid)
GET /v1/actions/detail

There will be a new action description field in the REST URLs respond message.

Security impact

None

Notifications impact

None

Other end user impact

Aside from the API, here are there other ways a user will interact with this feature:

impact on python-watcherclient:
- Display the “description” field when displaying details about an action.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei, hanrong

Work Items

Add a new table to save the mapping Add logic to update the table when action loading Add logic to show the action description

Dependencies

None

Testing

Unit tests should be updated.

Documentation Impact

The documentation will have to be updated, especially the glossary, in order to explain the new concepts regarding “action” definition.

References

https://docs.openstack.org/watcher/latest/#action https://docs.openstack.org/watcher/latest/#action-plan

History

None

Respecting Nova Scheduler Filters

Sun, 12 May 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/scheduler-filters

Problem description

This specification continues ideas of nova-policies blueprint. One of global business requirements to Watcher is a requirement to respect Nova Scheduler filters at least in context of affinity/anti-affinity filters. Many corporate customers build HA architecture for their cloud applications and they want to respect virtual machine’s policies, which are represented as Nova Scheduler filters. There are the following typical filters which can be used in the VMs booting process: * To hold some VMs on different hosts (to achieve HA compatibility). * To place VMs inside of specified host aggregates or availability zones. * To place VMs only on enabled and core-compatible hosts.

As of now, Nova doesn’t expose an open API to Nova Scheduler’s select_destinations method. This method is available for internal Nova’s usage and it isn’t good architectural practice to access this method from a third-party service like Watcher.

Use Cases

As an openstack operator, I want to execute audit’s with taking into account Nova Scheduler filters.

Proposed change

The main proposition is to add Watcher Filters pluggable system, that would use set of Nova-compatible filters in auditing process. We can’t import them directly (cause OpenStack projects can import only project clients to interact to each other), so we need to implement each filter in the way Nova does. Nova Scheduler use internal objects host_state and spec_obj that represents host and instance attributes respectively. We can replace these objects with appropriate nodes from CDM. CDM consumes notifications via notification bus system. Extending attributes in CDM nodes will depend on specific filter. If we can consume notifications of changing availability zones or host aggregates, we can add them to CDM as host attributes. Otherwise, we can get them by API calls.

Primary logic should be saved to save compatibility with appropriate Nova’s filters. Let’s take a look at Nova’s DifferentHost filter:

def host_passes(self, host_state, spec_obj):
    affinity_uuids = spec_obj.get_scheduler_hint('different_host')
    if affinity_uuids:
        overlap = utils.instance_uuids_overlap(host_state, affinity_uuids)
        return not overlap
    # With no different_host key
    return True

There should be created filter subfolder in decision_engine folder. It will contain manager.py and filters/ folder with filter plugins. Filter Manager will load enabled plugins and pass virtual machine along with destination host through the list of the loaded plugins. Instance of FilterManager class should be added to BaseStrategy class as class property to let each strategy filtering VMs. Strategies should be reworked to get support for filters. Method self.verify_destination(instance, host) can be added to BaseStrategy class to simplify its usage for other strategies. There cannot be common point for using filters in strategies because of unique workflow of each one.

In the current situation it’s a trade-off between architectural design and business requirements. I recommend to take it as temporary solution until we find the better one.

Alternatives

Continue to use Watcher scope to segregate resources of cloud cluster.

Another solution is enhance-watcher-applier-engine blueprint. Instead of making exact action, we make all the probable actions and give the decision to Nova Scheduler.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Watcher config file should contain filters option with dict of enabled filters by strategies like:

{"basic": "AvailabilityZoneFilter, DifferentHostFilter",
 "zone_migration": "AvailabilityZoneFilter"}.

By default there should be AllHostsFilter associated with every strategy to pass every virtual machine without limitations.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexchadin

Work Items

Add documentation page regarding filters
Update documentation for strategy developers
Add filter manager to watcher/decision_engine/manager.py
Add affinity/anti-affinity, host aggregates and availability zone filters
Add unit and functional tests
Update configuration file to support filters option.

Dependencies

None

Testing

Appropriate unit and functional tests should be added.

Documentation Impact

Add documentation page regarding filters
Update documentation for strategy developers

References

None

History

None

Support Watcher HA active-active Mode

Sun, 12 May 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/support-watcher-ha-active-active-mode

Watcher Decision Engine and Applier currently don’t support HA active-active mode.

Problem description

There is often more than one controller in the real environment. Watcher Decision Engine and Applier are deployed on the controller. Now there is only one active Watcher-decision-engine or Applier. Otherwise Watcher Decision Engines will make duplicate actionplans for CONTINUOUS audit. And we don’t know which Applier is doing the actionplan. Another problem is how to sync Data Models of Decision Engines to each other. Now Decision Engine updates its Data Model based on notifications from Nova. If there are many Decision Engines, we need to find a way to broadcast notifications. One solution don’t depend on notifications is to update CDM before each audit.

Use Cases

As an operator, I want to enable all Watcher-decision-engine and Applier on all controllers.

Proposed change

If we enable more than one Watcher Decision Engine, the question is how to know the relation between Audit and Watcher Decision Engine. We can add a new ‘host’ field to the Audit table to solve this question. When we update the audit state from PENDING to ONGOING, we set the host field to record the Watcher Decision Engine hostname.

So, the changes will include:

Watcher Decision Engine CDM should consume notifications from Nova in broadcast mode to make CDMs synced.
Add a new ‘host’ field to the audit table
Add a new ‘host’ field to the actionplan table
Record the host when updating the audit’s state from PENDING to ONGOING
Record the host when updating the actionplan’s state from PENDING to ONGOING
For CONTINUOUS audit, we need to check the host that running decision engine and recorded host value in audit table. If they are different, just skip the audit.
When starting Applier process, if there are ONGOING actionplan with the same host, cancel these stale actionplans. Actions of these stale actionplans should also be marked as CANCELLED.
Action Plan includes different resources which are represented in Actions. If user wants to run more than one action plan at the same time, Watcher should check whether new action plan’s resources overlaps with already running one or not. If so, Watcher should prevent running of new Action Plan by raising appropriate error.

Alternatives

None

Data model impact

Add a new ‘host’ filed in the audit table
Add a new ‘host’ filed in the actionplan table

REST API impact

None

Security impact

None

Notifications impact

Add ‘host’ field to AuditPayload, ActionPlanPayload

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexchadin
Other contributors:: licanwei

Work Items

Add broadcast notifications for DE
Update database to add a new field ‘host’
Record host when changing state from PENDING to ONGOING
Add checking host for CONTINUOUS audit
Check stale actonplans when starting Applier

Dependencies

None

Testing

Unittest for each change.

Documentation Impact

Appropriate documentation should be added with new HA section.

References

None

History

None

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/watcher/devref/mitaka.blueprints.html#when-is-a-blueprint-needed
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Project Priority

Does this blueprint fit under one of the Mitaka Project Priorities? If so which one and how?

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Liberty	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Train	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Train	Introduced

Example Spec - The title of your blueprint

Mon, 29 Apr 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/example

Some notes about the watcher-spec and blueprint process:

Not all blueprints need a spec. For more information see http://docs.openstack.org/developer/nova/blueprints.html#specs
The aim of this document is first to define the problem we need to solve, and second agree the overall approach to solve that problem.
This is not intended to be extensive documentation for a new feature. For example, there is no need to specify the exact configuration changes,

nor the exact details of any DB model changes. But you should still define that such changes are required, and be clear on how that will affect upgrades.
You should aim to get your spec approved before writing your code. While you are free to write prototypes and code before getting your spec approved, its possible that the outcome of the spec review process leads you towards a fundamentally different solution than you first envisaged.
But, API changes are held to a much higher level of scrutiny. As soon as an API change merges, we must assume it could be in production somewhere, and as such, we then need to support that API change forever. To avoid getting that wrong, we do want lots of details about API changes upfront.

Some notes about using this template:

Your spec should be in ReSTructured text, like this template.
Please wrap text at 79 columns.
The filename in the git repository should match the launchpad URL, for example a URL of: https://blueprints.launchpad.net/watcher/+spec/awesome-thing should be named awesome-thing.rst
Please do not delete any of the sections in this template. If you have nothing to say for a whole section, just write: None
For help with syntax, see http://sphinx-doc.org/rest.html
To test out your formatting, build the docs using tox and see the generated HTML file in doc/build/html/specs/<path_of_your_file>
If you would like to provide a diagram with your spec, ascii diagrams are required. http://asciiflow.com/ is a very nice tool to assist with making ascii diagrams. The reason for this is that the tool used to review specs is based purely on plain text. Plain text will allow review to proceed without having to look at additional files which can not be viewed in gerrit. It will also allow inline feedback on the diagram itself.
If your specification proposes any changes to the Watcher REST API such as changing parameters which can be returned or accepted, or even the semantics of what happens when a client calls into the API, then you should add the APIImpact flag to the commit message. Specifications with the APIImpact flag can be found with the following query:

https://review.opendev.org/#/q/status:open+project:openstack/watcher-specs+message:apiimpact,n,z

Problem description

A detailed description of the problem. What problem is this blueprint addressing?

Use Cases

What use cases does this address? What impact on actors does this change have? Ensure you are clear about the actors in each use case: Developer, End User, Deployer etc.

Proposed change

Here is where you cover the change you propose to make in detail. How do you propose to solve this problem?

If this is one part of a larger effort make it clear where this piece ends. In other words, what’s the scope of this effort?

Alternatives

Data model impact

Questions which need to be addressed by this section include:

What new data objects and/or database schema changes is this going to require?
What database migrations will accompany this change.
How will the initial set of new data objects be generated, for example if you need to take into account existing instances, or modify other existing data describe how that will work.

REST API impact

Each API method which is either added or changed should have the following

Specification for the method
- A description of what the method does suitable for use in user documentation
- Method type (POST/PUT/GET/DELETE)
- Normal http response code(s)
- Expected error http response code(s)
  - A description for each possible error code should be included describing semantic errors which can cause it such as inconsistent parameters supplied to the method, or when an instance is not in an appropriate state for the request to succeed. Errors caused by syntactic problems covered by the JSON schema definition do not need to be included.
- URL for the resource
- Parameters which can be passed via the url
- JSON schema definition for the body data if allowed
- JSON schema definition for the response data if any
Example use case including typical API samples for both data supplied by the caller and the response
Discuss any policy changes, and discuss what things a deployer needs to think about when defining their policy.

Reuse of existing predefined parameter types such as regexps for passwords and user defined names is highly encouraged.

Security impact

Describe any potential security impact on the system. Some of the items to consider include:

Does this change touch sensitive data such as tokens, keys, or user data?
Does this change alter the API in a way that may impact security, such as a new way to access sensitive information or a new way to login?
Does this change involve cryptography or hashing?
Does this change require the use of sudo or any elevated privileges?
Does this change involve using or parsing user-provided data? This could be directly at the API level or indirectly such as changes to a cache layer.
Can this change enable a resource exhaustion attack, such as allowing a single API interaction to consume significant server resources? Some examples of this include launching subprocesses for each connection, or entity expansion attacks in XML.

Notifications impact

Please specify any changes to notifications. Be that an extra notification, changes to an existing notification, or removing a notification.

Other end user impact

Aside from the API, are there other ways a user will interact with this feature?

Does this change have an impact on python-watcherclient? What does the user interface there look like?

Performance Impact

Describe any potential performance impact on the system, for example how often will new code be called, and is there a major change to the calling pattern of existing code.

Examples of things to consider here include:

A periodic task might look like a small addition but if it calls conductor or another service the load is multiplied by the number of nodes in the system.
Scheduler filters get called once per host for every instance being created, so any latency they introduce is linear with the size of the system.
A small change in a utility function or a commonly used decorator can have a large impacts on performance.
Calls which result in a database queries (whether direct or via conductor) can have a profound impact on performance when called in critical sections of the code.
Will the change include any locking, and if so what considerations are there on holding the lock?

Other deployer impact

Discuss things that will affect how you deploy and configure OpenStack that have not already been mentioned, such as:

What config options are being added? Are the default values ones which will work well in real deployments?
Is this a change that takes immediate effect after its merged, or is it something that has to be explicitly enabled?
If this change is a new binary, how would it be deployed?
Please state anything that those doing continuous deployment, or those upgrading from the previous release, need to be aware of. Also describe any plans to deprecate configuration values or features. For example, if we change the directory name that instances are stored in, how do we handle instance directories created before the change landed? Do we move them? Do we have a special case in the code? Do we assume that the operator will recreate all the instances in their cloud?

Developer impact

Discuss things that will affect other developers working on OpenStack.

Implementation

Assignee(s)

Who is leading the writing of the code? Or is this a blueprint where you’re throwing it out there to see who picks it up?

If more than one person is working on the implementation, please designate the primary author and contact.

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Dependencies

Include specific references to specs and/or blueprints in Watcher, or in other projects, that this one either depends on or is related to.
If this requires functionality of another project that is not currently used by Watcher (such as the glance v2 API when we previously only required v1), document that fact.
Does this feature require any new library dependencies or code otherwise not included in OpenStack? Or does it depend on a specific version of library?

Testing

Is this untestable in gate given current limitations (specific hardware / software configurations available)? If so, are there mitigation plans (3rd party testing, gate enhancements, etc).

Documentation Impact

References

Links to mailing list or IRC discussions
Links to notes from a summit session
Links to relevant research, if appropriate
Related specifications as appropriate (e.g. if it’s an EC2 thing, link the EC2 docs)
Anything else you feel it is worthwhile to refer to

History

Revisions
Release Name	Description
Train	Introduced

Train Project Priorities

Mon, 08 Apr 2019 00:00:00

List of priorities the Watcher drivers team is prioritizing in Train.

Priority	Owner
File based Metric Map	Sumit Dilip Jamgade
Global datasource preference	Corne Lukken
Add resource_name in action input parameter	chen ke
Add Show Datamodel API	chen ke
Add force field to Audit	Li Canwei
Support Placement API	Li Canwei

File based Metric Map

Allow watcher to load a yaml file which contains a map of internal metrics names (as expected by watcher) to real metrics names in use in the datasources.

Global datasource preference

Some clouds might only have a single datasource available setting the datasource preference for every individual strategy is error prone and time consuming. A global datasource preference prevents errors and simplifies configuring watcher.

Add resource_name in action input parameter

Currently watcher has only UUIDs for users to distinguish different actions which is not friendly. This BP will add a resource_name field in action input parameters which is more user friendly.

Add Show Datamodel API

Add Show Datamodel API to see the information of the instance in the specified scope.

Add force field to Audit

As now, Watcher doesn’t allow to launch a new audit when there is actionplan ongoing. This is because if the new audit has the same data model as the ongoing actionplan, the new audit may create a wrong actionplan. But if there are different data model scope, we should allow the new audit to run. We want to give the choice to User, if User set force to True when launching audit, Watcher will execute the audit even other actionplan is ongoing.

Support Placement API

Placement provides a service for managing, selecting, and claiming available resources in a cloud. Many Watcher strategies need to select a target host for VM migrating. Watcher can improve the process by the help of placement.

Add the start and end time for CONTINUOUS audit

Fri, 22 Mar 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/add-start-end-time-for-continuous-audit

Currently we can only set audit execution interval, but we can not set audit start and end time. We need to add the audit start and end time for CONTINUOUS audit.

Problem description

For CONTINUOUS audit, it will run periodically after launch. There is no way to control the start and end time of CONTINUOUS audit yet.

Use Cases

As a Watcher user, I want to set the start and end time for CONTINUOUS audit.

Proposed change

Add new start_time and end_time fields in the audit table
For CONTINUOUS audit, if its state is PENDING or ONGOING, launch the audit only if the current time is between start_time and end_time

The user can specify the start and end time or any one, If user doesn’t specify start and end time, the audit’s behavior is the same as before. If no start time, audit starts after creating. If no end time, audit runs until its state is SUSPENDED or CANCELLED. If end time is provided, the audit will be set from ONGOING to SUCCEEDED after end time and not running again. The datetime format is ISO 8601, such as: YYYY-MM-DD hh:mm:ss

Here are some CLI examples:

create audit with start and end time:

$ watcher audit create -g dummy -t CONTINUOUS -i 300 \
  --start-time '2018-04-01 08:00:00' --end-time '2018-04-03 08:00:00'

update audit start or end time:

$ watcher audit update 64aa6c03-b676-4904-9d6a-855d1d6f9200 \
  replace start_time='2018-04-02 20:30:00'

$ watcher audit update 64aa6c03-b676-4904-9d6a-855d1d6f9200 \
  replace end_time='2018-04-04 20:30:00'

Alternatives

May reuse ‘interval’ field instead of the new start_time and end_time fields. But it will cause complication and not easy to understand.

Data model impact

Add new start_time and end_time fields in the audit table

REST API impact

Need to add new ‘start_time’ and ‘end_time’ parameters in Audits APIs. Their values are local time.

Security impact

None

Notifications impact

Add ‘start_time’ and ‘end_time’ to AuditPayload

Other end user impact

Need to add ‘start_time’ and ‘end_time’ to CLI ‘watcher audit create’
Also need to update watcher-dashboard

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Update architecture doc to add the new fields
Add new start_time and end_time fields in the audit table
Check start and end time before running CONTINUOUS audit
Update notification AuditPayload
Update python-watcherclient
Update watcher-dashboard

Dependencies

None

Testing

Unittest for all changes

Documentation Impact

Update architecture doc to add the new fields.

References

None

History

None

Enhance Watcher Applier Engine

Fri, 22 Mar 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/enhance-watcher-applier-engine

Taskflow is the default workflow engine for Watcher Applier. In Watcher the flow pattern for linked actions is Graph flow. For parent action A and child action B, this means B depends on A and that the execution of B must wait until A finished. Taskflow has a callback function to decide at runtime whether B should be allowed to execute. The callback function returns a boolean True to allow B execution or False to not. Currently the callback function always returns True in Watcher Applier.

Problem description

For example, we want to live migrate a VM, if there are two potential target nodes that meet the requirements. However, we don’t know which one will definitely be able to migrate successfully. The existing strategy will select one to create a Migrate action and the action may fail if there are no enough resource in the destination node. A solution is that we can make two linked actions for the two potential targets. If first action succeeded, then the second action will be ignored. Otherwise continue to execute the second one. For this purpose, We need the callback function returns True or False depending on the result of previous action A execution. Action B should be executed unless action A failed. If action A executes success, the execution of action B should be ignored.

Use Cases

As a Watcher user, I want to decide an action whether to be executed depending on the result of a previous action.

Proposed change

Now the callback function, named decider, always returns True. We propose to add a new config option ‘action_execution_rule’ which is a dict type. Its key field is strategy name and the value is ‘ALWAYS’ or ‘ANY’. ‘ALWAYS’ means the callback function returns True as usual. ‘ANY’ means the return depends on the result of previous action execution. The callback returns True if previous action gets failed, and the engine continues to run the next action. If previous action executes success, the callback returns False then the next action will be ignored. For strategies that aren’t in ‘action_execution_rule’, the callback always returns True.

If exception is throwing out during the action execution, reverting will be triggered by taskflow. To continue executing the next action, we return False instead of throwing an exception.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

The default value of ‘action_execution_rule’ doesn’t change the behavior of Applier.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: licanwei

Work Items

Add a new config option ‘action_execution_rule’
Implement the callback function

Dependencies

None

Testing

Unittest for all changes

Documentation Impact

None

References

None

History

None

API Microversions for Watcher

Mon, 18 Mar 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/api-microversioning

Problem description

As Watcher evolves and obtains new features which are also represented in API changes, it’s necessary to support new versions of Watcher without breaking current infrastructure.

Use Cases

As an OpenStack operator, I want to make a request of resource from client v1.1 to Watcher API server with latest version of 1.2 and get an appropriate response with resource of version 1.1.

As an OpenStack operator, I want to make a request of resource without specifying API version and get a response with resource of default version.

As an OpenStack operator, I want to make a request of resource with specifying API version of 1.2 and get a response with resource of 1.2 version.

Proposed change

Watcher API should receive the API version from client that represents the list resources and their GET/POST/PATCH attributes it can work with. If user makes a request with wrong API microversion, HTTP 406 Not Acceptable should be returned.

Since Watcher API uses Pecan + WSME bundle, Watcher microversion validation should be done in set of special methods which adapt resources to specified API version. For example, if patch set adds new attributes to resource in version 1.2 then version 1.1 should exclude these attributes from response. We will use some parts of Ironic microversion spec and implementation since it has Pecan + WSME bundle too. Let’s take a look at possible use cases of communication between Watcher API and Watcher Client. We will use the term “old Watcher” to refer to a version of Watcher that predates microversions and has no knowledge of them.

Communication without specifying API microversion (old Watcher)

The client makes a connection to Watcher, not specifying the HTTP header OpenStack-API-Version.
Watcher does not see the OpenStack-API-Version HTTP header
Watcher communicates using v1.0 of the REST API, and all communication with the client uses that version of the interface.

Communication between old client and new Watcher API

The client makes a connection to Watcher, specifying the HTTP header OpenStack-API-Version with version 1.0
Watcher communicates using v1.0 of the REST API, and all communication with the client uses that version of the interface.

Communication between new client and new Watcher API

The client makes a connection to Watcher, specifying the HTTP header OpenStack-API-Version with version 1.1.
Watcher communicates using v1.1 of the REST API, and all communication with the client uses that version of the interface. Response comes along with new HTTP header OpenStack-API-Version.

Communication between new client and new Watcher API (with latest microvers)

The client makes a connection to Watcher, supplying ‘latest’ as the version in the OpenStack-API-Version HTTP header
Watcher responds by using the latest API version it supports, and includes this in the OpenStack-API-Version header, along with the -Min- and -Max- headers.

Negotiation between new client/new API without version specifying

Watcher client supports versions 1.1 to 1.3, Watcher API supports versions 1.1 to 1.2.

The user has not specified a version to the client
The client makes a connection to Watcher, supplying 1.3 as the microversion since this is the latest microversion that the client supports.
Watcher responds with a 406 Not Acceptable, along with the -Min- and -Max- headers that it can support (in this case 1.1 and 1.2)
The client should transparently proceed, having negotiated that both client and server will use v1.2. The client should also cache this microversion, so that subsequent attempts do not need to renegotiate microversions.

Negotiation between new client/new API with version specifying

Watcher client supports versions 1.1 to 1.3, Watcher API supports versions 1.1 to 1.2.

The user specifies a particular microversion (e.g. 1.3) that the client should use
The client makes a connection to Watcher, supplying 1.3 as the microversion
Watcher responds with a 406 Not Acceptable, along with the -Min- and -Max- headers that it can support (in this case 1.1 and 1.2)
The client reports this to the user and exits

Communication between new client/new API with unsupported version

The client makes a connection to Watcher, supplying 1.3 as the requested microversion.
Watcher responds with a 406 Not Acceptable, along with the -Min- and -Max- headers that it can support (in this case 1.1 and 1.2)
The client reports this error to the user

To use microservices, client should add OpenStack-API-Version header to HTTP request before sending requests. python-watcherclient has already supplied its requests with OpenStack-API-Version header. As of now, the version is 1.0 and I’d propose to supply ‘latest’ that would always ask for the latest API version of Watcher.

What changes requires a new Microversion:

Changes in query parameters Example: adding new parameter, removing old one, changing parameter type.
Changes in resources Example: adding new resource, removing old one.
Changes in request body Example: new field ‘audit_description’ in POST request body.

This spec also relies on Microversion Specification as a base for the proposed changes.

Alternatives

Leave it as is, when user updates Watcher along with new OpenStack release. It may lead to potential incapability to serve request from legacy clients.

Data model impact

None

REST API impact

Watcher API should accept OpenStack-API-Version HTTP header.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Developer impact

Any future changes to Watcher’s REST API (whether that be in the request or any response) must result in a microversion update, and guarded in the code appropriately.

Implementation

Assignee(s)

Primary assignee:: alexchadin (aschadin@sbcloud.ru)

Work Items

Push OpenStack-API-Version header to API layer.
Add special filtering methods.
Implement versions.py module with history of API changes.

Dependencies

None

Testing

Appropriate unit and functional tests should be added.

Documentation Impact

None

References

http://specs.openstack.org/openstack/api-wg/guidelines/microversion_specification.html

History

None

Adding scope for Watcher data model

Mon, 18 Mar 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/scope-for-watcher-datamodel

Problem description

For a large cloud infrastructure, such as CERN, there are more than 10k servers, retrieving data from Nova to build Watcher compute data model may take a long time. If the audit is just for a subset of all nodes, it’s better to get the data from the nodes that audit needs.

Use Cases

As a Watcher user, I want that Watcher doesn’t build compute data model before creating audit. As a Watcher user, I want that Watcher build compute data model according to the scope of audit.

Proposed change

As now, Watcher builds compute data model when starting the Decision Engine. And there is a periodic task to rebuild the data model. To avoid building the data model before creating audit, we need to check a flag before building the data model. for example:

def execute(self):
    """Build the compute cluster data model"""
    if self._audit_scope_handler is None:
        LOG.debug("No audit, Don't Build compute data model")
        return

    builder = ModelBuilder(self.osc)
    return builder.execute(self._data_model_scope)

Audit scope is a optional parameter when creating audit. If user don’t set a scope ,the default scope is empty, it means this audit used for all nodes. An example of the audit scope:

{"compute":
    [{"host_aggregates": [
                          {"id": 1},
                          {"id": 2},
                          {"id": 3}]},
     {"availability_zones": [
                          {"name": "AZ1"},
                          {"name": "AZ2"}]},
}

When building the data model according to audit scope, there are some cases need to be considered:

no data model, audit scope is empty

It’s the first time to build the data model. Because audit scope is empty, the data model should include all the nodes.

no data model, audit scope is not empty

It’s the first time to build the data model according to audit scope.

Existing data model, new audit scope is empty

If the data model has included all nodes, it will not be rebuilt.
If the data model doesn’t include all nodes, it will be rebuilt.

existing data model, new audit scope is not empty

If the nodes specified in scope are already included in the data model, it will not be rebuilt.
If the nodes specified in the scope aren’t included in the data model, it will be rebuilt.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

This will reduce the impact on system performance, especially for large cloud infrastructure.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <licanwei>

Work Items

Add a check to the audit to create before building the data model.
Add the model scope to record the audit scope in the ModelBuilder class.
Update the data model according to the audit scope.

Dependencies

None

Testing

Add unit tests.

Documentation Impact

Update Watcher developer documents.

References

None

History

Revisions
Release Name	Description
Stein	Introduced

Update DataModel by Nova Notifications

Mon, 18 Mar 2019 00:00:00

https://blueprints.launchpad.net/watcher/+spec/update-datamodel-by-nova-notifications

Problem description

Watcher has a Compute DataModel(CDM) which includes all the info about instances and nodes. Nova will emit notifications when instances or nodes change their states. So Watcher can consume Nova notifications to update its CDM. Now Watcher just consider several notifications, there are more notifications need to be used, such as instance poweron/poweroff.

Use Cases

As a Watcher developer, I want to consume Nova notifications to update CDM.

Proposed change

We define a functional dict to map the notification and CDM. for example:

def InstanceDelete(payload):
    pass

def InstanceUpdate(payload):
    pass

funcdict = {
    'instance.delete.end': InstanceDelete,
    'instance.update': InstanceUpdate,
    }

def updateCDM(instancetype, payload):
    func = funcdict.get(instancetype)
    if func:
        func(payload)

We can classify notifications based on the impact on CDM:

Add or update instance in CDM:
- instance.create.end
- instance.lock
- instance.unlock
- instance.pause.end
- instance.power_off.end
- instance.power_on.end
- instance.resize_confirm.end
- instance.restore.end
- instance.resume.end
- instance.shelve.end
- instance.shutdown.end
- instance.suspend.end
- instance.unpause.end
- instance.unrescue.end
- instance.unshelve.end
- instance.rebuild.end
- instance.rescue.end
- instance.update
- instance.live_migration_force_complete.end
- instance.live_migration_post_dest.end
Remove instance from CDM
- instance.delete.end
- instance.soft_delete.end
Add or update node in CDM
- service.create
- service.update
Remove node from CDM
- service.delete

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Watcher will start to consume more notifications from Nova.

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <licanwei>

Work Items

List Nova notifications that Watcher consumes to upate CDM.
Implement the code that update CDM from notification payload.

Dependencies

None

Testing

Unit tests should be added.

Documentation Impact

None

References

https://docs.openstack.org/nova/latest/reference/notifications.html#existing-versioned-notifications

History

Revisions
Release Name	Description
Stein	Introduced

Define the scope of an Audit as a pool of resources

Tue, 22 Jan 2019 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/define-the-audit-scope

Watcher aims at providing a flexible solution for auditing any pool of resources and optimizing it.

This blueprint aims at providing a generic way to define the scope of an Audit.

The set of audited resources will be called “Audit scope” and will be defined in each Audit Template (which contains the Audit settings).

Problem description

So far, when an Audit is launched, the whole OpenStack cluster is audited and there is no way for the admin to provide a logical subset of the infrastructure (host aggregate, availability zone, network, Hadoop cluster, storage volume, …).

It should be possible to define the set of resources that needs to be optimized independently from the goal of the audit and from the underlying OpenStack services (Nova, Neutron, Sahara, …).

Use Cases

As an operator with an admin role I want to be able to define in an Audit Template the scope of the Audit so that I can optimize some subset of the infrastructure

As an operator with an admin role I want to be notified when a resource UUID that I have added to an Audit scope does not exist or can not be handled by any optimization strategy so that I know whether my audit scope is valid or not

As a developer I want to be able to define the scope of an Audit with a generic description mechanism so that I can develop optimization strategies for any kind of OpenStack resource

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

The Audit Template should have a new attribute, named “scope”, which contains the list of resources to optimize during the audit. If this attribute is not defined, the whole default availability zone is taken by default.

The scope is defined as an array of resource types and for each resource type an array of resource uuid(s):

"scope": [
          {"host_aggregates": [
            {"id": 1},
            {"id": 2},
            {"id": 3}
          ]},
          {"availability_zones": [
            {"name": "AZ1"},
            {"name": "AZ2"}
          ]},
          {"exclude": [
            {"instances": [
              {"uuid": "9766dff1-3c81-4de2-92ae-19e0d9adcec6"},
              {"uuid": "777541bb-ce4f-4bf3-8320-d5792d5cdf6e"}
            ]},
            {"compute_nodes": [
              {"name": "compute1"}
            ]}
          ]}
        ]

We can exclude instances and compute nodes from listed AZ and HA to prevent them from migrations caused by Watcher.

It should be possible to indicate all resources of a given type by using some wildcard character, instead of an array of specific UUIDs. For example, to select all Nova host aggregates and all availability zones, the scope would be defined like this:

"scope": [
          {"host_aggregate": [
            {"id": "*"}
          ]},
          {"availability_zone": [
            {"name": "*"}
          ]}
        ]

When a scope is created or updated, all the resources such as host_aggregates and availability_zones are verified in order to make sure that the resource exists. The following module is used for defining audit scope:

Scope default module (decision_engine/scope/default.py). This module should contain the method that will check whether scope is valid or not. If scope is valid, the latest cluster data model should be copied and modified according to the scope’s content. The resulting copy is a subset of the original cluster data model that can be used in Watcher strategy afterwards. If some of provided resources are not available, the warning message should be added to the logs.

Estimated changes are going to be in the following places:

in the Watcher Decision Engine, the Strategies will receive the audit scope as an input parameter
in the Watcher API, the audit template object must be updated with a new scope field and some verifications must be added regarding the existence of resource types and UUIDs added to the audit scope.

Alternatives

None

Data model impact

The following data object will be impacted:

AuditTemplate:
- We should be able to store in the database the scope defined as an array of resource types and for each resource type an array of resource uuid(s).

REST API impact

There will be an impact on every REST resource URLs that starts with /v1/audit_template/ and that uses the type AuditTemplate:

GET /v1/audit_template
GET /v1/audit_template/(audit_template_uuid)
POST /v1/audit_template
PATCH /v1/audit_template
GET /v1/audit_template/detail

The type AuditTemplate will contain a new Scope field with an array of resource types and for each resource type, an array of resource uuid(s).

Here is a sample of the new JSON payload for an audit template which includes the Scope json composed of three Nova Host Aggregates, two Availability Zones and some objects to exclude:

{
    "created_at": "2016-01-07T13:23:52.761933",
    "deleted_at": null,
    "description": "Description of my audit template",
    "extra": {
        "automatic": true
    },
    "goal": "MINIMIZE_ENERGY",
    "scope": [{
        "host_aggregates": [{
            "id": 1
        }, {
            "id": 2
        }, {
            "id": 3
        }]
    }, {
        "availability_zones": [{
            "name": "AZ1"
        }, {
            "name": "AZ2"
        }]
    }, {
        "exclude": [{
            "instances": [{
                "uuid": "9766dff1-3c81-4de2-92ae-19e0d9adcec6"
            }, {
                "uuid": "777541bb-ce4f-4bf3-8320-d5792d5cdf6e"
            }]
        }, {
            "compute_nodes": [{
                "name": "compute1"
            }]
        }]
    }],
    "links": [{
        "href": "http://localhost:9322/v1/audit_templates/27e3153e-d5bf-4b7e-b517-fb518e17f34c",
        "rel": "self"
    }, {
        "href": "http://localhost:9322/audit_templates/27e3153e-d5bf-4b7e-b517-fb518e17f34c",
        "rel": "bookmark"
    }],
    "name": "My Audit Template",
    "updated_at": "2016-01-07T13:23:52.761937",
    "uuid": "27e3153e-d5bf-4b7e-b517-fb518e17f34c"
}

Security impact

None

Notifications impact

Warning message should be sent when resources are not available. The message should describes which resource is not available.

Other end user impact

The python-watcherclient audit-template-create command will be updated to handle the scope definition provided in a JSON file
The python-watcherclient audit-template-show command will be updated to display the audit scope JSON definition
The python-watcherclient audit-template-list –detail command will be updated to display every audit scope JSON definition

Performance Impact

None

Other deployer impact

The operator may have to configure the frequency of the periodic task that regularly checks the validity of all the audit scopes.

The operator may specify not only Host Aggregates IDs, but Host Aggregate names too (will be provided in future).

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: alexchadin
Other contributors:: jed56 vincent-francoise david-tardivel

Work Items

Here is the list of foreseen work items:

Add a scope field to the watcher/objects/audit_template.py class
Update the AuditTemplate class in watcher/db/sqlalchemy/models.py
Create watcher/decision_engine/scope folder, where default handler and base handler class will be placed.
To apply scope to the CDM we need to add instance of scope default handler to the watcher/decision_engine/strategy/strategies/base.py before strategy execution.
Update the watcher/api/controllers/v1/audit_template.py class to handle the new scope field and the verifications of resource types and UUIDs.
Update unit tests and integration tests (Tempest scenarios)
Provide the database migration script

Dependencies

This blueprint is related to the following blueprint:

Today, the Audit Template has to be populated by hand. In this blueprint: https://blueprints.launchpad.net/watcher/+spec/query-list-of-auditable-resource-types-for-a-goal we would like to add some helpers which enable the admin to get the list of auditable resources for a given goal, depending on which Strategies are installed on the Watcher Decision Engine.

Each strategy will be able to return the list of auditable resource types and therefore it will be possible to get a list of auditable resource UUIDs from Nova, Neutron, …

The admin will just need to select the auditable ressources he/she wants to add to the Audit Template, just like a customer would add products in a basket.

Testing

Of course, the unit tests will have to be updated.

Need to update existing tempest scenarios which create an Audit Template and define an Audit scope in it.

It would be interesting to create a test scenario which creates two Host Aggregates in the test environment and make sure that the Audit only affects the resources of the Host Aggregate which belongs to the Audit scope and ignores the resources of the other Host Aggregate.

Another test should be written handling the case when a resource UUID used in an audit scope does not exist any more.

Documentation Impact

need to define the concept of Audit scope in the Watcher glossary
need to update the REST API documentation and show the JSON payload for defining an audit scope
need to update the user’s guide to show how to define the audit scope in a json file and create an Audit template with this file provided as input parameter for the scope.

References

Links to IRC discussions:
- http://eavesdrop.openstack.org/meetings/watcher/2016/watcher.2016-01-27-14.00.log.html
- http://eavesdrop.openstack.org/irclogs/%23openstack-watcher/%23openstack-watcher.2016-08-10.log.html

History

None

Stein Project Priorities

Thu, 20 Sep 2018 00:00:00

List of priorities the Watcher drivers team is prioritizing in Stein.

Priority	Owner
API Microversioning	Alexander Chadin
Watcher Planner Selector	Alexander Chadin
Define grammar for workload characterization	Alexander Chadin
Bare metal node N+1 redundancy	Dao Cong Tien
Ironic notifications for bare metal DM	Alexander Chadin
Update Data Model by Nova notifications	Li Canwei
Display Data Model of specified Audit	chenke
Add the start/end time for CONTINUOUS audit	Li Canwei
Audit scoper for baremetal data model	Yumeng Bao
Enhance watcher Applier engine	Li Canwei

API Microversioning

We have to provide backward compatibility by adding API microversioning since Watcher get new resources and their attributes during development cycles.

Watcher Planner Selector

This component is responsible for selecting an appropriate planner for a given strategy depending on several factors (e.g list of actions used by the strategy or user request)

Define grammar for workload characterization

As we run several workloads in a cloud, we should be able to characterize such workloads as input to watcher for ensuring Application QoS, placements and consolidation. An example of workload characterization is a weighted combination of CPU, Memory or any other resource attributes like High IOPs, Network latency etc.

Bare metal node N+1 redundancy

This proposes high availability/reliability feature to support Bare Metal Node N+1 Redundancy based on data from Ceilometer.

When a bare metal node is failed due to hardware problem or is likely to be failed due to a sign of hardware failure, this function allows to switch over the node safely to another bare metal node in short time if the failed node is booted from a volume.

Ironic notifications for bare metal DM

Update the bare metal data model by ironic notifications

Update Data Model by Nova notifications

Watcher consumes Nova notifications to update its Data Model. So far only several notifications are used, there are many notifications need to be consider, such as instance poweron/poweroff, they all change the state of instance. Watcher should update the state of instance in the Data Model by consuming appropriate Nova notifications that influence the state of instance or compute.

Display Data Model of specified Audit

This blueprint will allow to display structure of Audit’s Data Model. Data Model stucture should be useful for operators who would like to know the set of resources, which are also known as scope, that are used by Audit. The response can be represent as table in case of CLI and as graph model in case of Watcher Dashboard.

Add the start/end time for CONTINUOUS audit

Currently we can only set audit execution interval, but we can not set audit start and end time. We need to increase the audit start and end time for CONTINUOUS audit.

Audit scoper for baremetal data model

Since baremetal data model was added, we need audit scoper for baremetal data model as compute data model has.

Enhance watcher Applier engine

Currently watcher’s Applier can only run actions one by one or parallel. We need to decide whether the next action is executed based on the result of the previous action, so we need to enhance the applier.

Exclude Project by Audit Scope

Wed, 12 Sep 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/audit-scope-exclude-project

Problem description

Watcher using audit scope can exclude instances, compute nodes, host aggregates, instance metadata from Compute CDM but as of now it can not exclude project from compute CDM.

Use Cases

As a Cloud Administrator sometimes I want to exclude one or more projects out of audit scope. There can be many reasons to exclude some projects out of audit scope e.g SLA requirement, projects running some critical applications etc.

Proposed change

This spec proposes to add feature exclude project from compute CDM. To implement this feature the following things need to be done

Add project_id in compute CDM, project_id will be added in Instance element of compute CDM. example xml refernce of Instance element of compute CDM:

<Instance state=”active” human_id=”” uuid=”INSTANCE_0” vcpus=”10”
disk=”20” disk_capacity=”20” memory=”2” metadata=’{“optimize”: true,”top”: “floor”, “nested”: {“x”: “y”}}’/>

Add exclude project logic in compute CDM

Alternatives

None.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

python-watcherclient help message will be updated. watcher-dashboard should also be updated.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <adi-sky17>
Other contributors:: <nakamura-h>

Work Items

Add project_id field in Instance element of Compute CDM watcher/decision_engine/model/element/instance.py
Get instance project_id information while building instance model watcher/decision_engine/model/collector/nova.py
Add logic for excluding instances with specified project id from audit scope in compute scope watcher/decision_engine/scope/compute.py
Update help message in python-watcherclient

Dependencies

None

Testing

Unit test will be added.

Documentation Impact

None

References

None

History

None

Replace cold migration to use Nova migration API

Wed, 12 Sep 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/replace-cold-migrate-to-use-nova-migration-api

Problem description

Currently watcher implements instance cold migration by it’s own for specifying destination node. The implementation includes instance creation. Since administrator runs watcher API, instance migrates to service project.

Use Cases

Migrate action executes instance cold migration if migration_type is cold.

Proposed change

Since v2.56, Nova migrate Server(migrate Action) API has host option. By replacing watcher_non_live_migrate_instance method in common/nova_helper.py to use the API, we can simply solve the problem.

Alternatives

It may be the alternative to create instance by temporary user in the project where instance is in. But this solution remains quota problem. During temporary user lives and reached quota limit, new user can not be created.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

We may expect cold migration becomes faster because current implementation calls many Nova and Neutron API.

Other deployer impact

Administrator can see migration by watcher using Nova List Migrations API.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>
Other contributors:: <None>

Work Items

Replace watcher_non_live_migrate_instance method in common/nova_helper.py to use Nova migrate Server(migrate Action) API.

Dependencies

None

Testing

We should expect current test code can test this replacement.

Documentation Impact

None

References

Cron-based continuous audits

Sat, 21 Apr 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cron-based-continuous-audits

Problem description

Watcher currently implements a user specified interval when running continuous audits. This may cause some collisions when the Watcher services are down, because the next audit time isn’t stored in the database. Using a time interval isn’t a flexible or user-friendly solution, instead, we should use an industry standard like cron syntax that provides an elastic way to schedule and execute an audit.

Use Cases

As an administrator, I want to use cron syntax to schedule audit execute.
As an administrator, I want be sure that Watcher will set up next time executing of audits properly.

Proposed change

Watcher API already has ‘interval’ property in Audit object so we can use it to store cron like format. API is to check type of incoming interval and if it is string type then Watcher uses croniter module to set next run time. The second proposition is to provide persistence for current ‘interval’ column. To do it, we need to properly update next_run_time since its value may be less than current time. In this case, we may use the following algorithm:

delta = timedelta(seconds=
    (interval - (current_time-old_next_run_time).seconds % interval))
next_run_time = current_time + delta

For example, Watcher had to run audit at 7:30 am, but services went down and we have restored it only at 9:00 am. Using interval with 3600 value, Watcher has to set next run time at 9:30 am.

To make this spec, we need to be sure that Watcher can store continuous jobs and its services, which may work in HA mode, takes only tag-assigned jobs. Blueprint background-jobs-ha takes responsibility to implement job store mechanism.

Alternatives

Continue to use already provided interval.
Delete continuous audits after each maintenance mode and create new ones.

Data model impact

Type of interval column of Audit table should be changed to string type.
New column next_run_time has to be added to Audit table.

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Alexander Chadin <alexchadin>

Work Items

Add next_run_time column to Audit table.
Use croniter module to support cron-like syntax in Watcher.
Update python-watcherclient to support cron-like syntax.
Add logic to calculate values for next_run_time field.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/continuously-optimization https://blueprints.launchpad.net/watcher/+spec/background-jobs-ha

Testing

Unit tests should be updated.

Documentation Impact

Update documentation to mention cron and interval.

References

None

History

None

Noisy Neighbor Strategy

Sat, 21 Apr 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/noisy-neighbor-strategy

This spec describes a new strategy that identifies and migrates a Noisy Neighbor, which is a lower priority VM that negatively affects Last Level Cache (LLC) of a higher priority VM when located on the same host.

Problem description

Noisy neighbor is a big problem in multi-tenant clouds. Some lower priority VMs can negatively affect performance on higher priority VMs by monopolizing CPU, I/O, and network bandwidth. There are already strategies to cover some of them like CPU utilization as well as algorithm using a threshold for cache occupancy as an indicator to identify a noisy neighbor. These algorithms are too simple and do not take into consideration the interaction between VMs including the performance impact if any as well as the VMs’ relative priority to each other. For example a VM could use 50% cache occupancy and it could be fine. It all depends on the VM’s priority. This algortihm utilizes IPC, LLC and VMs’ relative priorities.

LLC(Last Level Cache), or L3 cache in X86, is critical and limit system level resource shared by all apps or VMs on the node. If one VM occupy most of L3 cache, other VMs on the node likely starve without enough L3 cache thus poor performance.

Use Cases

As an openstack operator, I need to find noisy neighbor VMs that affect my high priority VMs, and migrate them to another node that has no noisy negihbor issues.

Proposed change

Add one new goal - “Noisy Neighbor migration”
Extend base strategy classes to add one new strategy - “Noisy Neighbor solution”
Use Ceilometer client to get following metrics for detecting LLC noisy neighbor:
- cpu_l3_cache – LLC occupancy of a VM
Algorithm to detect Noisy Neighbor:
- Monitor L3 cache of all VMs in order of their “watcher-priority”, that is set in VM metadata. Example: L3 cache of VM with watcher-priority as “1” will be monitored before VM with watcher-priority as “8”.
- If L3 cache of a VM goes down by more than the threshold, mark it as high priority. Then start monitoring L3 cache of VMs in reverse order of their “watcher-priority”
- If L3 cache of a VM (VMs to be searched in reverse order of “watcher-priority”) increases by the threshold, then the VM is a noisy neighbor.
- Look for a node with no Noisy Neighbor issue and migrate the noisy VM there.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Querying metrics will probably cause small performance drop when running the strategy.

Other deployer impact

To enable LLC metric, latest Intel server with CMT support is required. Also collectd is required to gather LLC metric data.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <prudhvi-rao-shedimbi>

Work Items

Define proper threshold for IPC and LLC
Write the execute function to locate the noisy neighbor
Choose target node for live migration.

Dependencies

https://pypi.org/project/libvirt-python

Testing

Unit and functional test are needed.

Documentation Impact

Add docs on how to use this strategy.

References

http://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html

History

None

Define grammar for workload characterization

Sat, 21 Apr 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/workload-characterization-grammar

Problem description

When applications are deployed in cloud, they compete for resources and it becomes a challenge for “cloud operators to ensure” and “tenants to get” desired performance and quality of service (QoS) for the workloads.

When we run several workloads in cloud, we need to characterize these workloads as input to watcher for profiling, fingerprinting, application QoS, placements and consolidation. Audit can refer to these grammars and implement QoS Strategy

Also, from a cloud infrastructure standpoint, these workload characteristics can also be used to gauge SLAs and decisions for killing or rebalancing workloads elsewhere.

An example of workload characterization is a weighted combination of CPU, Memory or any other resource attributes like High IOPs, Network latency etc.

Scope of this blueprint is -

A definition of a grammar for characterizing workload specific targeting at composite metrics for a VM.
A lightweight solution for parsing the defined grammar using the TOSCA-Parser library.
A solution for persisting the grammar for VMs leveraging VM metadata.

Use Cases

As an OpenStack administrator or user, I want to be able to create a TOSCA structured workload character. As an OpenStack administrator or user, I want to be able to build an optimization strategy using the defined workload character grammar.

Project Priority

Pike

Proposed change

Workload characterization is important for QoS to achieve performance goals and SLAs. Building optimization strategies or workload placement based on the defined workload character.

Defining a workload character

A workload character will constitute one or more Performance Units.

A Performance Unit will have a resource name (Example - vcpus, cpu_util, memory, memory.usage, disk.device.iops), Value (Number of CPU Cores, Memory in MB, Number of IOPs), Unit, an Operator (>, <, =), and Weight (priority ranging from -20 to 19 for highest and lowest priorities respectively). Weights are used to create a threshold to optimally run the workload.

Resource name and units should be in compliance with Nova and any strategy supported metric services.

Example of a performance unit:

vcpus = 2 Cores
cpu_util > 90 %
memory.usage > 2000 MB
disk.device.iops > 1500 counts/s

Examples of Workload Characters

cpu_bound_workload: (vcpus = 2 Cores OR cpu_util > 90)
big_data_workload = (memory.usage > 2000 MB AND disk.device.iops < 1500 counts/s)

Represent workload character using TOSCA

The above can be represented in a TOSCA structure

Example :

cpu_bound_workload =
    watcher:
      derived_from: tosca.nodes.Root
        description: '{vcpus} or {cpu_util}'
          properties:
            vcpus:
              required: true
              weight: -20
              type: integer
              constraints:
                - equal: 2
            cpu_util:
              required: true
              weight: 19
              type: integer
              constraints:
                - greater_than: 90


disk_io_workload=
    watcher:
      derived_from: tosca.nodes.Root
        properties:
          disk_iops:
            required: true
            weight: 5
            type: integer
            constraints:
              - greater_than: 1200

Parse and validate workload grammar

Use TOSCA-Parser module to parse a TOSCA serialized workload grammar and validate it by applying values from telemetry or other metrics.

Snippet to parse a grammar -

from toscaparser.nodetemplate import NodeTemplate

custom_snippet = '''
watcher:
  derived_from: tosca.nodes.Root
  description: '{vcpus} or {cpu_util}'
  properties:
    vcpus:
      required: true
      weight: -20
      type: integer
      constraints:
        - equal: 2
    cpu_util:
      required: true
      weight: 19
      type: integer
      constraints:
        - greater_than: 90
'''
custom_def = yamlparser.simple_parse(custom_snippet)
equation = custom_def['watcher']['description']
data = '''
server:
  type: watcher
  properties:
    vcpus: 2
    cpu_util: 89
'''
parsed_data = yamlparser.simple_parse(data)
nodetemplate = NodeTemplate('server', parsed_data, custom_def)
p_names = {}
for p in nodetemplate.get_properties_objects():
    try:
        p.validate()
        p_names[p.name] = True
    except:
        p_names[p.name] = False

print(equation.format(**p_names))

Output -

True

Here is more examples and documentation - https://pypi.org/project/tosca-parser/0.7.0

Store Grammar in VM Metadata

The grammar should be added as VM Metadata. Watcher will then poll this data and make it available in the cluster data model for Audit and Strategy to refer. This metadata addition will be watcher specific and should have an identifiable key like “watcher-workload-grammar”.

Example to store grammar in VM Metadata -

{
"meta": {
  "workload_character":
  "watcher:
     derived_from: tosca.nodes.Root
     description: '{vcpus} or {cpu_util}'
     properties:
       vcpus:
         required: true
         weight: -20
         type: integer
         constraints:
           - equal: 2
       cpu_util:
         required: true
         weight: 19
         type: integer
         constraints:
           - greater_than: 90"
 }
}


curl -H 'Content-Type: application/json' -H "X-Auth-Token: token" -X PUT -d '{
"meta": {
  "workload_character":
  "watcher:
     derived_from: tosca.nodes.Root
     description: '{vcpus} or {cpu_util}'
     properties:
       vcpus:
         required: true
         weight: -20
         type: integer
         constraints:
           - equal: 2
       cpu_util:
         required: true
         weight: 19
         type: integer
         constraints:
           - greater_than: 90"
 }
}'
https://openstack_controller:8774/v2.1/tenant_id/servers/9b4daf23-c01c-44ad-a
f7-4c20a7b382d6/metadata/workload_character

Alternatives

No Alternatives

Data model impact

Modifications will be made to the cluster data model to read and store VM metadata in memory for strategies and audits to reference.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Intel and Walmart are working together on this blueprint.

Primary assignee:

Chris Spencer(chrisspencer) Prashanth Hari (hvprash) Susanne Balle (sballe)

Other contributors:

Work Items

Create module for parsing the grammar
Modify cluster data model to read VM Metadata and store in memory for strategies and audits to refer
Refactor the code to populate workload grammar from vm_metadata in cluster data model - Model Root - https://git.io/vXSbA - Nova cluster data model - https://git.io/vXS9N

Dependencies

None.

Testing

None.

Documentation Impact

None.

References

None.

History

Rocky Project Priorities

Mon, 12 Mar 2018 00:00:00

List of priorities the Watcher drivers team is prioritizing in Rocky.

Priority	Owner
Watcher API validation using JSON	Aditi Sharma
Migrate to Zuulv3	Alexander Chadin
Compute CDM include all instances	suzhengwei
Watcher Planner Selector	Alexander Chadin
Watcher Strategy Selector	Aditi Sharma
Exclude project by audit scope	Aditi Sharma
Define grammar for workload characterization	Alexander Chadin
Replace cold migration to use Nova API	Hidekazu Nakamura
JSONschema to validate efficacy indicators	Yumeng Bao
Support Watcher HA active-active mode	Li Canwei
Ironic notifications for bare metal DM	Yumeng Bao
Cluster maintenance	suzhengwei
Add name for audit in watcher dashboard	Yumeng Bao
Add the start/end time for CONTINUOUS audit	Li Canwei
Audit scoper for baremetal data model	Yumeng Bao
Enhance watcher Applier engine	Li Canwei

Watcher API validation using JSON

Cureently Watcher uses different methods to validate API, which causes many bugs and few operations are possible which should not be allowed like a cloud admin can delete “ongoing” actionplan and audit. To have more cleaner and same approach for all operations we should have a unified way of validating the API, which can be done using JSON.

Migrate to Zuulv3

For all of the jobs specific to a particular project, teams should move the auto-converted legacy- jobs to their own repos and rework them to stop using the legacy interfaces. There are two fundamental steps:

Move the jobs to your repo
Rework the jobs to be native v3 jobs

Compute CDM include all instances

When building compute CDM, we will exclude the instances excluded in the scope. It has terrible impact to Watcher.

To some strategies, it would get incorrectly workload of the compute nodes, because the excluded instances was not calculated in.
To server consolidation, it would disable the nodes which has excluded instances running.

Proposal is to include all instances in the scope when build compute CDM. But exclude the instances excluded in the scope when migrations or simulate migrations.

Watcher Planner Selector

This component is responsible for selecting an appropriate planner for a given strategy depending on several factors (e.g list of actions used by the strategy or user request)

Watcher Strategy Selector

There may be several strategies applying for a given optimization goal. Currently , if the admin didn’t specify a strategy watcher select the first strategy available in the list. If Watcher intends to be used in real infrastructure it need a more robust way to select the strategy for a given goal. The strategy selector component will enable watcher to automatically decide which strategy to use. The typical use case for this blueprint is :

The admin selects a goal from the set of available goals.
The strategy selector select the strategy which maximize the strategy “objective function” depending on several factors.
Once a strategy has been selected, it triggers the Watcher Optimizer (DefaultStrategyContext) with it.

Exclude project by audit scope

As an administrator, I want to exclude instances of a specific project from Watcher optimization. This bp proposes to add exclude project feature to audit scope. We need the following tasks:

Add tenant_id to Compute CDM
Add exclude project logic in audit scope

Define grammar for workload characterization

Replace cold migration to use Nova API

As of Now Watcher implements cold migrate in migrate action by not using Nova migration API, since Nova migration API for cold migration could not specify target host. In Queens cycle, Nova has implemented to specify target host for cold migration.

JSONschema to validate efficacy indicators

In this blueprint, we will replace voplutuous with JSON-schema to validate efficacy indicator. Since in watcher we want to remove voluptuous and use JSONSchema as our only JSON validation tool to keep consistency.

Support Watcher HA active-active mode

Only one Decision engine can consume notification from nova and we need to get DEs synced. It can be solved by invoking method on all DEs hosts.

Ironic notifications for bare metal DM

Update the bare metal data model by ironic notifications

Cluster maintenance

Sometimes we need to maintain compute nodes, update hardware and software, and so on. But we don’t want user’s application to be interrupted. This issue imports one goal and strategy for manually maintaining without user’s application interruption.

Add name for audit in watcher dashboard

It is not easy to tell the audits apart just by uuid for end users. If we add a name for an audit, it is more friendly to end users. This bp implements adding name for an audit in watcher dashboard.

Add the start/end time for CONTINUOUS audit

Currently we can only set audit execution interval, but we can not set audit start and end time. We need to increase the audit start and end time for CONTINUOUS audit.

Audit scoper for baremetal data model

Since baremetal data model was added, we need audit scoper for baremetal data model as compute data model has.

Enhance watcher Applier engine

Queens Project Priorities

Sun, 25 Feb 2018 00:00:00

List of priorities the Watcher drivers team is prioritizing in Queens.

Priority	Owner
Baremetal data model in Watcher	Li Canwei
Check the strategy requirements	Alexander Chadin
Notifications for action plan cancel	Aditi Sharma
Watcher Planner Selector	Alexander Chadin
Watcher Strategy Selector	Aditi Sharma
Workload Characterization and QoS	Susanne Balle
Extend node status	suzhengwei
Exclude project by audit scope	Aditi Sharma
Audit scoper for storage data model	Hidekazu Nakamura
Define a tailored Scoper for each CDM	Hidekazu Nakamura
Register and Document Policy in Code	Yumeng Bao
Workload characterization grammar	Susanne Balle
Zone migration strategy	Hidekazu Nakamura
Watcher API validation using JSON	Aditi Sharma
Compute CDM include all instances	suzhengwei
Display input parameters of strategy	Alexander Chadin
Add name for audit	suzhengwei
Cluster maintance strategy	suzhengwei
Multiple datasources for strategies	Aditi Sharma
Replace voplutuous with JSON Schema	Yumeng Bao

Baremetal data model in Watcher

Check the strategy requirements

Running of strategy requires different type of data resources to compute a solution. During the strategies loading, the decision engine should automatically check if all requirements are achieved.

Notifications for action plan cancel

Notifications needs to be added to action and action plan for new operation action plan cancel.

Watcher Planner Selector

This component is responsible for selecting an appropriate planner for a given strategy depending on several factors (e.g list of actions used by the strategy or user request).

Watcher Strategy Selector

There may be several strategies applying for a given optimization goal. Currently , if the admin didn’t specify a strategy Watcher select the first strategy available in the list. If Watcher intends to be used in real infrastructure it need a more robust way to select the strategy for a given goal. The strategy selector component will enable watcher to automatically decide which strategy to use.

Workload Characterization and QoS

Based on the defined workload characteristics we should be able to apply Quality of Services to applications. An example would be leveraging technologies like Intel RDT.

This opens up several application optimization possibilities (use cases like NFV etc.) and also ensures efficient use of cloud resources. Scope of this blueprint is to build a QoS strategy using Intel RDT and workload grammar.

Extend node status

We can get a node status through CDM (cluster data model) in watcher. Most of strategies rely on the node’s status. But the existing status just meets existing strategies. We need to extend nodes status description for new strategies.

Exclude project by audit scope

As of Now, Watcher can exclude instances, compute_nodes, host_aggregates and instance_metadata by audit scope. As an administrator, I want to exclude instances of a specific project from Watcher optimization. This bp proposes to add exclude project feature to audit scope.

Audit scoper for storage data model

Since storage data model was added in Pike cycle, we need audit scoper for storage data model as compute data model has.

Define a tailored Scoper for each CDM

Storage cluster data model was introduced in Pike cycle. Since the model is different from compute data model, current single CDM scoper does not work for the model.

Register and Document Policy in Code

This blueprint tracks the work for moving default policies and documentation into code. This is OpenStack-wide goals for Queens release. https://governance.openstack.org/tc/goals/queens/policy-in-code.html

Storage workload balance

As of now, Watcher optimizes only compute nodes. Storage optimization is also important feature for centralized storage (non distributed storage). This spec will add Storage Workload Balance Strategy to balance the storage resource. And we can use existing goal (workload_balancing) and action(volume_migrate) for storage workload balance.

Workload characterization grammar

As we run several workloads in cloud, we should be able to characterize such workloads as input to watcher for ensuring Application QoS, placements and consolidation.

An example of workload characterization is a weighted combination of CPU, Memory or any other resource attributes like High IOPs, Network latency etc.

Scope of this blueprint is to come up with a grammar structure for defining workload character.

Zone migration strategy

There are thousands of physical servers and storage running various kinds of workloads in the Cloud system. Administrator have to migrate instances and block storage for hardware maintenance once a quarter or so. It requires operators manually to watch workloads, choose instances to migrate and migrate for each instances and block storage for efficiently, with minimum downtime. Watcher can be used to do this task automatically.

Watcher API validation using JSON

Cureently watcher uses different methods to validate api, which causes many bugs and few operations are possible which should not be allowed like a cloud admin can delete “ongoing” actionplan and audit. To have more cleaner and same approach for all operations we should have a unified way of validating the api, which can be done using JSON.

Compute CDM include all instances

When building compute CDM, Watcher excludes the instances excluded in the scope. It has negative impact to Watcher.

To some strategies, it get incorrectly workload of the compute nodes, because the excluded instances was not calculated in.
To server consolidation, it disables the nodes which has excluded instances running.

Display input parameters of strategy

As of now in watcher, it is difficult for users to know what input-parameters are required by each strategy when creating an audit. This blueprint will add a column to display input parameters of each strategy in results of “watcher strategy list”.

Add name for audit

Adding name for audit entity would be more friendly to end users to interact with audit. This blueprint suggests to add name property to audit entity, so it will be easy for users to retrieve an andit by name.

Cluster maintance strategy

Multiple datasources for strategies

We need an abstraction layer with a minimal subset of metrics needed by existing strategies (it will deal with gnocchi & monasca). If we have multiple metrics collection backend deployed, the strategy will need to give the back-end it wants to use / alternatively we could define a backend priority in configuration. We will need a map between metrics names used by strategies and correspond names in backends.

Multiple global efficacy indicator

As of now global efficacy indicator is a single entity. It is useful for the strategies which optimizes only one resource like server consolidation works only with instances. There can be some future strategies which optimizes many resources (volume, instance, network) for them it is necessary to calculate global efficacy for each resource. This blueprint will implement multiple global efficacy indicator.

Replace voplutuous with JSON Schema

In this blueprint, we will replace voplutuous with JSON-schema to validate efficacy indicator. Since we want to remove voluptuous and use JSONSchema as our only JSON validation tool to keep consistency in Watcher.

Load consolidation strategy

Thu, 15 Feb 2018 00:00:00

This specification relates to blueprint: https://blueprints.launchpad.net/watcher/+spec/basic-cloud-consolidation-integration

Problem description

Watcher is a framework which provides support for more energy efficient OpenStack operations. It does this by providing access to system state information and a set of available actions which can be performed on an OpenStack installation. It is specifically designed to provide support for different approaches to realizing energy efficient operations: consequently, interested parties are encouraged to provide their own energy efficiency approaches and integrate them with Watcher. This specification focuses on integration of the rudimentary load consolidation mechanism developed at ICCLab cloud computing research lab at Zürcher Hochschule für Angewandte Wissenschaften (ZHAW) with Watcher.

The original code implementing this algorithm (not in the context of Watcher) was published here: https://github.com/icclab/cloud-consolidation

Use Cases

The use case is one in which the Administrator wants to perform a load consolidation on the resources to reduce the amount of underutilized servers. The Administrator invokes Watcher with the Goal of “VM_WORKLOAD_CONSOLIDATION”. Watcher then executes the Strategy “VM_WORKLOAD_CONSOLIDATION_STRATEGY”. It then presents a set of Actions to the Administrator. The Administrator then approves the recommended action plan - typically VM live-migration actions - and instructs Watcher to perform the actions.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

The proposed change is to add a new Goal and a new Strategy to Watcher. The new Goal is “VM_WORKLOAD_CONSOLIDATION” and the new Strategy is “VM_WORKLOAD_CONSOLIDATION_STRATEGY”. The new Strategy is designed to be a lightweight consolidation mechanism which can be tuned based on experience; it also operates quickly. The purpose of the strategy is to move the aggregate operating point of the Cluster to increase the number of servers with moderate to high load and minimize the number of servers with low load. This can be used in conjunction with a server management mechanism to reduce overall energy consumption.

The new Strategy will leverage a modified first-fit algorithm to achieve increased server CPU and memory utilization which ultimately leads to freeing some of the hosts that can be powered down to save energy. It comprises of two phases, one focused on identifying server with high load and reducing their load and one focused on identifying servers which have spare underutilized capacity. Each of these operates as a first-fit algorithm with utilization ordered in different ways as input to each.

This Strategy will consider compute host’s CPU utilization and memory constraints. These upper utilization thresholds can be set relative to resource capacity and hence will provide simple resource overbooking management if needed. This strategy will not deal with any other limitations such as actual VM memory change rate, network constraints, etc. and relies upon a robust live migration mechanism.

In order to be able to predict host resources utilization the following utilization estimation model is used. A host resource utilization equals to a sum of the resource utilizations of the hosted workloads (VMs). Considering hosts H1, H2 with a workload W running on H1, moving the workload W from H1 to H2 will result in predicted resource utilization as follows: H1 = H1 - W and H2 = H2 + W with the metrics relating to the VM taken from telemetry and those pertaining to the host available via nova metrics.

The strategy will work in two phases. The first phase handles decreasingly sorted hosts (by their CPU utilization) whose CPU utilization is exceeding defined threshold and offloads their workload (VM) to the first suitable less loaded host which is able to accommodate the workload without violating any of the constraints described. This host offloading process is repeated for all overloaded hosts until the host’s CPU utilization is predicted to be under the threshold. Doing so for all overloaded servers outputs in a system without overloaded servers. In this phase the workloads (VM) are handled sorted increasingly by its CPU utilization. The second phase then iterates through the servers in reversed order (sorted increasingly by their CPU utilization and thus starting with the least loaded servers) and looks for a smallest possible space where to accommodate its remaining workloads starting with the largest workload and the most loaded hosts. This process is repeated until there is no workload (VM) left on the host in which case this host can be deactivated. This continues again for the next hosts in the same manner until the source and the destination host becomes the same. In this phase the workloads (VM) are handled sorted decreasingly by its CPU utilization.

Both phases result in a solution whose execution leads to a consolidated system with no overloaded hosts.

This change will not affect any existing Strategies and will not affect Watcher performance.

Concretely, the new Strategy will be implemented as a new Strategy called VMWorkloadConsolidationStrategy inheriting from BaseStrategy. The implementation will be very much based on the BasicConsolidation example in the current Watcher codebase.

Alternatives

The alternatives to this approach are to use different Goals and associated Strategies defined in Watcher.

Data model impact

None expected.

Having reviewed the data models for both information available to the different Strategies as well as the data models for the Actions, we believe that no modifications are necessary to implement this Strategy.

REST API impact

There is no impact on the REST API.

Security impact

As the strategy only computes a new VM placement and doesn’t deal with placement itself, no security impact is envisaged.

Notifications impact

No specific notifications associated with executing a specific Strategy are envisaged. (Notifications could arise from the resulting actions, but these are presumably handled in other parts of Watcher).

Other end user impact

This capability will not have any specific impact on the API. It will have a small impact in how it is used via the python-watcherclient as a new option will now be available for goal parameter in an Audit Template.

Performance Impact

No specific performance impact is envisaged. The Strategy has been designed to operate over hundreds of servers in the order of a few seconds.

Other deployer impact

No specific deployer impact is envisaged.

Developer impact

This will not impact other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:: Seán Murphy <murp>
Other contributors:: Bruno Grazioli <bwg-bruno> Vojtech Cima <cima-vojtech>

Work Items

This task can be considered atomic. It just requires the development and test of a single class.

Dependencies

No dependencies.

Testing

Several unit tests will be provided to test various scenarios using a fake mock models (mock model collector and mock metrics collector) including edge scenarios such as a consolidation of an empty cluster, a consolidation of randomly generated clusters or consolidation of an overloaded cluster.

Testing approaches similar to the basic consolidation strategy will be used, comprising of unit tests and integration tests in which a specific input is given and compared against the expected output.

Documentation Impact

It will be necessary to add new content relating to this new Goal and Strategy to the documentation.

References

No references.

History

No history.

Watcher Overload standard deviation algorithm

Thu, 15 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/watcher-overload-sd

Problem description

Cluster overload is a complex problem, that can be solved by migrating VMs from one node to another. The decision to migrate is based on VM parameters such as CPU utilization or memory utilization. We propose a strategy for Watcher, that monitors if there is a higher load on some hosts compared to other hosts in the cluster and re-balances the work across hosts to minimize the standard deviation of the loads in a cluster.

The main purpose of the strategy is to choose the pair VM:dest_host that minimizes the standard deviation in a cluster best.

This specification relates to blueprint: https://blueprints.launchpad.net/watcher/+spec/watcher-overload-sd

The original code implementing this algorithm (not in the context of Watcher) was published here: https://github.com/joker946/nova/commits/drs

Use Cases

As the Administrator I want to reduce load on cluster by triggering a Watcher optimization with the Workload Stabilization goal to create action plan with the list of recommended actions. This strategy can be configured by using watcher input parameters (e.g. thresholds, weights, metrics). After then I approve proposed action plan, which contains set of VM live-migration actions.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

The new Workload Stabilization Strategy allows to determine if there is an outlier in the load for the hosts in a cluster and respond to it by migrating VMs to average the load across the cluster. Input: It is possible to get information about workload of every VM using Ceilometer. In order to do that the means of CPU and RAM utilization for the last 10 minutes are used (configurable). Then the hypervisor workload of each node can be calculated by summing up workload of all VMs on each node. Knowing total amount of CPU and RAM on every node allows to normalise CPU and RAM load. After that the means of CPU and RAM loads for the cluster are calculated. All this information makes it possible to calculate the standard deviation of CPU or RAM. After standard deviation is calclulated for CPU and RAM, the strategy compares sd values to cpu_threshold and ram_threshold (these are provided by watcher input parameters). If one of the values is greater than the threshold value, the strategy runs cluster stabilization process to reduce load on cluster by performing live-migration of VMs. Pseudo code:

load_for_node = {'compute1': {'cpu', 'ram'}, 'compute2': ...}
for all vm in node:
    load_for_vm = {'uuid', 'cpu_util', 'memory.resident', 'vcpus'}
    load_for_vm[cpu_util] = transform_vm_cpu_util_to_host_cpu_util()
    load_for_node += load_for_vm
load_for_cluster = normalize load_for_node

for all node in cluster:
    compute deviation for each node against load_for_cluster.
    Then choose the src vm and dest host.

The implementation of transform_vm_cpu_util_to_host_cpu_util function is described in a Cluster stabilization process paragraph.

Cluster stabilization process: For each active VM we simulate migration to another node of the cluster by recalculating workload of source node and destination node. The following code shows a general case of simulating:

new_hosts[src_hp_id][metric] -= vm_load[metric]
new_hosts[dst_hp_id][metric] += vm_load[metric]

Meanwhile, the shown code isn’t acceptable for cpu_util metric because of probable difference in the number of node vcpus. Therefore we transform VM CPU utilization to overall host CPU utilization. The following pseudocode shows the right way of transforming:

\(cpu_{impact} = cpu_{util} * {vm_vcpus}/{host_vcpus}\)

After that the load values of nodes are normalised and the new standard deviation is calculated with them. The total standard deviation is calculated as weighted arithmetic mean:

\(\sum weight_{metric}*sd_{metric}\), where:

weight_metric is the weight of metric. The value of weight is to be in range(0, 1). Each weight of metric is set by watcher input parameters.
sd_metric is a calculated standard deviation of metric in cluster.

From all options of moving VM to other nodes we choose the one that minimizes the standard deviation in a cluster best. That option is added to the map of migration options, that afterwards gets sorted by standard deviation in ascending order. From the resulting list we iteratively take a host/vm pair and place it into solution with action “MIGRATION”. We compare the resulting deviations of CPU and RAM to the threshold values. If the resulting deviations are less than the threshold values, the cycle stops and the resulting action plan is sent to Watcher Applier.

Alternatives

The alternatives to this approach are to use different Goals and associated Strategies defined in Watcher.

Data model impact

None expected.

REST API impact

There is no impact on the REST API.

Security impact

None expected.

Notifications impact

Other end user impact

This capability will not have any specific impact on either the API or python-watcherclient.

Performance Impact

No specific performance impact is expected.

Other deployer impact

No specific deployer impact is envisaged.

Developer impact

This will not impact other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:: Alexander Chadin <alexchadin>
Other contributors:: Alexander Stavitskiy <alexstav>

Work Items

This task can be considered atomic. It just requires the development and test of a single class. The main WorkloadStabilization class of this strategy is inherited from BaseStrategy class.

Dependencies

There is a dependency with BP optimization threshold.

Testing

Several unit tests will be provided to test various scenarios.

Documentation Impact

It will be necessary to add new content relating to this new Goal and Strategy to the documentation.

References

No references.

History

No history.

Add a name for an audit

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/add-name-for-audit

Problem description

It is not easy to tell the audits apart just by uuid for end users. If we add a name for an audit, it is more friendly to end users.

Use Cases

As an end user, I want to retrieve an audit by name.

Proposed change

This spec is mainly to add a name for an audit, so it will be easy for users to retrieve an audit.

To watcherclient/watcher-api: User can show/delete/update an audit by name. Create an audit with a name. The audit name is not mandatory. If audit name was not provided, use a default name. The default name is made of strategy or audittemplate or goal name with current time.
To DB: Add a new field in audit table to store the audit’s name. New field in audit table,with unique attribute: name = Column(String(63), nullable=True)
To notifications: Add audit name to AuditPayload, ActionPlanPayload

Alternatives

None

Data model impact

Yes, add a new field ‘name’ in audit table to store the audit’s name.

REST API impact

The following API’s will be changed. The parameter ‘audit’ now can be used to GET, DELETE or PATCH one audit either by UUID or by name of the audit. * GET /v1/audits * DELETE /v1/audits * PATCH /v1/audits

The return value to Audit APIs will be changed. Add ‘name’ field into audit object of below return value types. * AuditCollection * Audit

Security impact

None

Notifications impact

Yes, it will add audit name to AuditPayload, ActionPlanPayload

Other end user impact

It may impact watcher-dashboard.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: sue

Work Items

Add name for audit, changes for python-watcherclient
Add name for audit, changes for watcher api/db
Add name for audit, update audit notifications and API references

Dependencies

None

Testing

Unit test

Documentation Impact

Yes, two docs will be updated.

Notifications in Watcher https://docs.openstack.org/developer/watcher/dev/notifications.html
API References https://docs.openstack.org/developer/watcher/#api-references

References

None

History

None

Audit scoper for storage data model

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/audit-scoper-for-storage-data-model

Problem description

Storage cluster data model was introduced in Pike cycle. Since the model is different from compute data model, we need CDM scoper for storage cluster data model.

Use Cases

As a Watcher user, I want to restrict scope of storage cluster data model.

Proposed change

This spec adds storage cluster data model scoper which can restrict the followings.

storage availability zone
volume type
exclude
- volume
- storage pools
- volumes of project

For adding it, we will implement as compute cluster data model scoper is implemented.

Add JSON schema in cinder cluster data model collector.
Add storage audit scope handler which overrides BaseScope class and implements get_scoped_model method.

Alternatives

Strategy developer can restrict storage cluster data model in a strategy respectively.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

End user can define audit scope for storage cluster data model in audit template by specifying yaml or json file with –scope option in the same manner for compute cluster data model.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <adi-sky17>
Other contributors:: <nakamura-h>

Work Items

Add JSON schema in cinder cluster data model collector.
Add storage audit scope handler which overrides BaseScope class and implements get_scoped_model method.

Dependencies

None

Testing

Unit test will be added.

Documentation Impact

Update the help message in python-watcherclient, provide details how to add scope for storage data model.

References

None

History

None

build baremetal data model in watcher

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/build-baremetal-data-model-in-watcher

This “energy saving” goal can be fulfilled by adding four features in Watcher: 1) Build a new baremetal data model within current Watcher Cluster Data Model 2) Add new actions “power on” and “power off” in Watcher 3) Implement a new strategy based on ‘baremetal’ and ‘compute’ data models, which could trigger ‘compute’ and ‘baremetal’ actions. 4) Update the bare metal data model by ironic notifications

This spec implements the first feature: Build a new baremetal (ironic) data model independent from current Watcher Cluster Data Model.

Problem description

In the “power on” and “power off” actions, Watcher sends request to power on/off physical machines managed by nova using ironic (ironic calling IPMI tools). So watcher should know the hypervisor_id and node_uuid of the target host(s), as well as their mapping list. In this spec, we will build a baremetal data model in watcher to describe this mapping list between hypervisor_id and node_uuid

Let’s take an example, if we need to power off host 3 and 4 as shown in the following list. But watcher doesn’t know their node_uuid.

[root@host_192_51_151_2 ~(keystone_admin)]# nova hypervisor-list +—-+———————+——-+———+ | ID | Hypervisor hostname | State | Status | +—-+———————+——-+———+ | 3 | host_192_51_151_3 | up | enabled | | 4 | host_192_51_151_5 | up | enabled | | 5 | host_192_51_151_7 | up | enabled | | 7 | host_192_51_151_8 | up | enabled | | 9 | host_192_51_151_9 | up | enabled | | 11 | host_192_51_151_10 | up | enabled | +—-+———————+——-+———+

The “Extra” field is served as an extensible field.

Above all, from a watcher user’s perspective, the whole procedure starts from getting compute.node.id by ‘nova hypervisor-list’, then create ironic node manually using ‘ironic node create’ as described in the proposed change part. In the second step, the ironic_node_id is generated and the correlation is built. Given all these id and correlation information, now we can create a “power on and off” audit in watcher to do the optimization.

Use Cases

As a developer, I want to have a new baremetal data model independant from compute CDM in watcher to get the mapping list between hypervisor_id and ironic node_uuid.

Proposed change

Add “compute.node.id” info into Ironic node “extra” field when create nodes or update nodes info. Just like the following two ways:

If we can not get compute.node.id before creating ironic node. 1) ironic node-create 2) nova hypervisor-list 3) ironic node-update
If we can get compute.node.id before creating ironic node. 1) nova hypervisor-list 2) ironic node-create

ironic node-create [-c <chassis>]: -d <driver> [-i <key=value>] [-p <key=value>] [-e <key=value>] [-u <uuid>] [-n <name>] [–network-interface <network_interface>] [–resource-class <resource_class>]
ironic node-create -d your_driver agent_ipmitool: -i ipmi_address=<ipmi_ip> -i ipmi_username=<ipmi_username> -i ipmi_password=<ipmi_password> -e compute_node=compute.node.id -u ironic_node_uuid

compute.node.id is the compute node ID saved in CCDM (Compute Cluster Data Model)

2) On watcher, build the Baremetal Cluster Data Model (BCDM) by periodically requesting Ironic service. The Baremetal Cluster Data Model structure is shown as followings

{: “uuid”: “4f37180e-c310-4327-a286-d5ab9ffc6497”, “power_state”: “power on”, “maintenance”: false, “maintenance_reason”: null, “extra”: {“compute_node_id”: 1} }

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Watcher consumes some Ironic notifications. We leave this for future work, which is described in the following bp: https://blueprints.launchpad.net/watcher/+spec/update-bare-metal-data-model-by-ironic-notifications

Other end user impact

Watcher may report a stale representation of the Ironic resources and to reduce this staleness, they can reduce the syncing interval in the configuration.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <li-canwei2>
Other contributors:: <alexchadin>,<yumeng-bao>

Work Items

Add baremetal data model.

Dependencies

None

Testing

Unit test on the Watcher Decision Engine. Tempest.

Documentation Impact

System Architecture doc will be updated, since model drivers accesses only nova and glance as of now.

References

None

History

Revisions
Release Name	Description
Pike	Introduced

Define a tailored Scoper for each CDM

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cdm-scoping

Problem description

Storage cluster data model was introduced in Pike cycle. Since the model is different from compute data model, current single CDM scoper does not work for the model.

Use Cases

As a Watcher user, I want to restrict scope of storage cluster data model.

Proposed change

Add _audit_scope instance attribute to the BaseClusterDataModelCollector and initialize when instantiated.
Remove audit_scope_handler from the current BaseStrategy and move it within the BaseClusterDataModelCollector.
Add audit_scope argument to CollectorManager’s get_cluster_model_collector method.
Change compute_model method to use Collector’s audit_scope_handler.

Change original DefaultScope.

get_scoped_model simply returns cluster_model.

For representing multi data model for JSON shema, we change DEFAULT_SCHEMA as the following:

DEFAULT_SCHEMA = {
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "properties": {
        "compute": {
            # current JSON schema moved to NovaClusterDataModelCollector
            # retrieved from NovaClusterDataModelCollector
        },
        "storage": {
            # retrieved from CinderClusterDatamodelCollector
        },
        .......
    }
}

Properties are dynamically retrieved from collector plugins.

Update validate method in AuditTemplatePostType in accordance with the above change.

Rename the DefaultScope as ComputeScope and override the audit_scope_handler property in the Compute Collector.
Change audit template help message in python-watcherclient.

Alternatives

None

Data model impact

None

REST API impact

Scope JSON schema definition for the audit template POST data from the request body will be changed in accordance with DEFAULT_SCHEMA change.

Security impact

None

Notifications impact

None

Other end user impact

End user defines audit scope in each audit template by specifying yaml or json file with –scope option. The file format will be changed according to JSON schema change.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>
Other contributors:: <None>

Work Items

Add or remove audit_scope and audit_scope_handler from/to base classes.
Change original DefaultScope.
Change Compute Collector to use ComputeScope instead of DefaultScope.
Change python-watcherclient.

Dependencies

None

Testing

Unit test will be updated.

Documentation Impact

None, but it is preferable to adding more.

References

https://blueprints.launchpad.net/watcher/+spec/cinder-model-integration

Check the strategy requirements

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/check-strategy-requirements

Problem description

Running of strategy requires different type of data resources to compute a solution. First launch of specified strategy causes an error in most cases. Admin has to take a look at logs to get more information about error. It takes a lot of time and is not informative way.

Use Cases

As an OpenStack administrator, I want to be able to check strategy requirements before strategy’s launching.

As an OpenStack administrator, I want to be able to see list of strategy’s requirements along with thier states. This list should get me a conclusion if I can execute selected strategy.

Proposed change

There are some requirements which are to be accomplished:

Metering service should be reachable. (mandatory)
One or more metrics associated with strategy should be presented in metering service. (optional)
Cluster Data Model used by the strategy should be loaded. (mandatory)

Checking of strategy can be called by using command ‘watcher strategy state {name_of_strategy}’. It will show the following table:

Requirement type	Mandatory	State	Comment
Metering Service	yes	Gnocchi: available
Metrics	no	cpu_util: available memory.used: not available	cpu_util is out of bounds (144.2 > 100)
Cluster Data Model	yes	Compute: available

Metering service can be verified by calling one of API resources (response should return HTTP status code 200):

GET /v1/status for Gnocchi
GET /v2/resources for Ceilometer API
GET /v2.0/metrics for Monasca API

API versions are set via conf/{metering_serice}_client.py

Metering service for each strategy is defined in watcher.conf file. If chosen metering service isn’t available, state of metering service should be tagged as ‘not available’. Otherwise, state would be ‘available’.

Watcher should know which metrics are used by requested strategy. This can be reached by using METRIC_NAMES dictionary. This dict contains datasources as keys and their subdicts of strategy_metric_name:datasource_metric_name pairs. This form of used strategy metrics is already used by basic_consolidation, outlet_temp_control, uniform airflow, vm_workload_consolidation strategies. Other strategies should be adapted to use METRIC_NAMES. I would also propose to set min and max bounds for metrics to be sure that requested metric will come in the form of expected unit (i.e. 0.00-100.00 for cpu_util).

Availability of metrics can be verified by sending appropriate requests to the metering service. For example, gnocchi will return list of metrics in response to GET /v1/metric request. It isn’t required to have all strategy metrics in metering service cause some strategies allows to work with some metrics, not all.

Cluster Data Model will be loaded in _collectors struct of CollectorManager class if appropriate collector is defined in collector_plugins option. By default, there is only compute CDM.

Alternatives

None.

Data model impact

None.

REST API impact

GET /v1/strategies/(strategy)/state

Parameters:

strategy (unicode) - UUID or name of the strategy.

Return: List of strategy’s requirements and their states.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: alexchadin <a.chadin@servionica.ru>

Work Items

There are the following steps to be done:

Add new command to strategy API that will check out requirements.
Adapt existing strategies to use METRIC_NAMES dictionary.
Update python-watcherclient to support new command.

Dependencies

None.

Testing

New unit tests should be added along with updating old ones.

Documentation Impact

References

extend node status in compute CDM

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/extend-node-status

Problem description

We can get a node status through CDM(cluster data model) in watcher. Most of the strategies rely on the node’s status. But the existing status just meets existing strategies. We need to extend nodes status description for new strategies. Moreover, there are some potential problems in usage. For example, to activate one compute node which is disabled not by watcher, it will conflict.

Use Cases

As an end user, I want watcher to enable a compute node which is disabled by watcher not by others.

Proposed change

Now we just use for words(up/down/enabled/disabled) to describe one compute node. That is not enough.

This spec will add ‘disabled_reason’ field into ‘ComputeNode’ resource. It will avoid potential usage problems, if we distinguish which nodes are disabled by Watcher and which are not by Watcher. If a node is “disabled” by Watcher(with “disabled_reason” “watcher_disabled”), then it is in the optimize scope. If a node is “disabled” not by Watcher(without specific “disabled_reason”), then it is out of the optimize scope, Wather shouldn’t do any optimization to it.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: sue

Work Items

add ‘disabled_reason’ filed into ‘ComputeNode’ resource, to distinguish which nodes are disabled by Watcher and which are not by Watcher.

Dependencies

Testing

Unit tests

Documentation Impact

None

References

None

History

None

listen to all necessary notifications for updating Compute CDM

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/listen-all-necessary-notifications

Problem description

In Watcher, it firstly builds compute CDM by collecting nova compute resource, later updates the compute CDM by listening to nova notifications. It has listened to following nova events,

service.update
instance.update
instance.delete.end
compute.instance.update
compute.instance.create.end
compute.instance.delete.end
compute.instance.live_migration.post.dest.end

In one cloud env, there would be many solutions, which would make the compute resource strongly relocated, such as ‘auto-scaling’ solution and ‘compute-ha’ solution. If Watcher doesn’t listen to all the notifications which represent that nova compute resource changes, the compute CDM will become stale easily and watcher would not provide efficient solution.

Use Cases

As an OpenStack administrator, I want Watcher still to work steady when other projects make the compute resources changed.

Proposed change

Following events also represent that nova compute resource changes. Watcher needs to listen to the following notifications and update the compute model.

compute.instance.rebuild.end
compute.instance.resize.confirm.end

Notification ‘compute.instance.rebuild.end’ represents the ‘evacuate’ action of one instance is finished, while notification ‘compute.instance.resize.confirm.end’ represents the ‘resize’ action of one instance is finished and confirmed by the user. Once Watcher listens to such notifications, it needs to update the instances mapping in the compute model.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: sue

Work Items

listen to ‘compute.instance.rebuild.end’ event
listen to ‘compute.instance.resize.confirm.end’ event

Dependencies

None

Testing

Unit test

Documentation Impact

None

References

None

History

None

Multiple global efficacy indicator

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/multiple-global-efficacy-indicator

Problem description

Watcher calculates global-efficacy as a single value with available efficacy indicators. It is useful for those strategies which optimizes only one openstack resource. For strategies that optimizes multiple resources, multiple global efficacy indicator is required, as each global efficacy indicator will represent one resource. e.g. There are two strategies which optimizes storage and compute both, cloud admin checks the value of global efficacy for both strategy and decides to choose which is higher. cloud admin actually wants to choose strategy which has better compute optimization, since global efficacy is single entity it will not be able to reflect only compute optimization. There is a possibility that user may choose higher global effiacy strategy but at reality its value is higher because of storage efficacy. so its better to have multiple global efficacy indicator one for each resource.

Use Cases

As a Watcher user, For better selection of strategies I want to have multiple global efficacy indicating each resource.

Proposed change

Initialize global efficacy as list in Efficacy class.

Return global efficacy in following format from EfficacySpecification. It will be a list of objects of efficacy.Indicator class:

[(efficacy.Indicator(name='', description='', unit='', value='')),
 (efficacy.Indicator(name='', description='', unit='', value='')),
]

Change the parsing of format_global_efficacy method in python-watcherclient
Change ActionPlan Data model, as global efficacy was a JSONEncodedDict before and now it will be JSONEncodedList .

Alternatives

We can keep global efficacy as single entity,let user calculate multiple values based on resource using efficacy indicators.

Data model impact

None

REST API impact

global efficacy JSON schema will change. It will affect follwing API’s:

GET /v1/action_plans/(action_plan_uuid)
GET /v1/action_plans/detail
GET /v1/action_plans

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <adi-sky>
Other contributors:: <None>

Work Items

Initialize global_efficacy as list in watcher.decision_engine.solution.efficacy.Efficacy
Update ActionPlan data model.
Update the method get_global_efficacy_indicator in watcher.decision_engine.goal.efficacy.specs.ServerConsolidation
Update in python-watcherclient.

Dependencies

None

Testing

Unit test will be updated.

Documentation Impact

Update the contributor guide for goal plugin. https://docs.openstack.org/watcher/latest/contributor/plugin/goal-plugin.html#implementation

References

https://docs.openstack.org/watcher/latest/glossary.html#efficacy-indicator

action plan cancel notifications

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/notifications-actionplan-cancel

A new operation watcher actionplan cancel is implemented in watcher, This spec propses to implement notifications for watcher action plan cancel.

Problem description

As of now, there is no notification implemented for “watcher actionplan cancel” so no way to know for event based plugins, when action plan cancel started and completed.

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher about actionplan cancel.

As an OpenStack developer, I want to be notified whenever:

an actionplan cancel started
an actionplan cancel completed
an actionplan cancel failed

As an OpenStack developer, I would also want to be notified whenever:

action abort (cancel) started
action abort (cancel) completed
action abort (cancel) failed

Proposed change

In order to implement the above use cases, many different notifications will be needed:

actionplan.cancel.start whenever an actionplan cancellation starts.
actionplan.cancel.end whenever an actionplan cancellation ends.
actionplan.cancel.error whenever an actionplan cancellation failed.
action.cancel.start whenever an action is aborted.
action.cancel.end whenever an action abort end.
action.cancel.error whenever an action abort failed

actionplan.cancel.start

{
  "event_type": "actionvent_type": "action_plan.cancel.start",
  "payload": {
    "watcher_object.namespace": "watcher",
    "watcher_object.name": "ActionPlanCancelPayload",
    "watcher_object.version": "1.0",
    "watcher_object.data": {
      "created_at": "2016-10-18T09:52:05Z",
      "deleted_at": null,
      "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
      "audit": {
        "watcher_object.namespace": "watcher",
        "watcher_object.name": "TerseAuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.data": {
          "created_at": "2016-10-18T09:52:05Z",
          "deleted_at": null,
          "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "scope": [],
          "audit_type": "ONESHOT",
          "state": "SUCCEEDED",
          "parameters": {},
          "interval": null,
          "updated_at": null
        }
      },
      "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
      "fault": null,
      "state": "CANCELLING",
      "global_efficacy": {},
      "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
      "strategy": {
         "watcher_object.namespace": "watcher",
         "watcher_object.name": "StrategyPayload",
         "watcher_object.version": "1.0",
         "watcher_object.data": {
           "created_at": "2016-10-18T09:52:05Z",
           "deleted_at": null,
           "name": "TEST",
           "uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
           "parameters_spec": {},
           "display_name": "test strategy",
           "updated_at": null
         }
       },
       "updated_at": null
      }
    },
    "priority": "INFO",
    "message_id": "3984dc2b-8aef-462b-a220-8ae04237a56e",
    "timestamp": "2016-10-18 09:52:05.219414",
    "publisher_id": "infra-optim:node0"
  }

actionplan.cancel.end

{
  "event_type": "action_plan.execution.end",
  "payload": {
    "watcher_object.namespace": "watcher",
    "watcher_object.name": "ActionPlanCancelPayload",
    "watcher_object.version": "1.0",
    "watcher_object.data": {
      "created_at": "2016-10-18T09:52:05Z",
      "deleted_at": null,
      "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
      "audit": {
        "watcher_object.namespace": "watcher",
        "watcher_object.name": "TerseAuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.data": {
          "created_at": "2016-10-18T09:52:05Z",
          "deleted_at": null,
          "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "scope": [],
          "audit_type": "ONESHOT",
          "state": "SUCCEEDED",
          "parameters": {},
          "interval": null,
          "updated_at": null
        }
      },
      "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
      "fault": null,
      "state": "CANCELLED",
      "global_efficacy": {},
      "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
      "strategy": {
        "watcher_object.namespace": "watcher",
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.data": {
          "created_at": "2016-10-18T09:52:05Z",
          "deleted_at": null,
          "name": "TEST",
          "uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
          "parameters_spec": {},
          "display_name": "test strategy",
          "updated_at": null
        }
      },
      "updated_at": null
    }
  },
  "priority": "INFO",
  "message_id": "3984dc2b-8aef-462b-a220-8ae04237a56e",
  "timestamp": "2016-10-18 09:52:05.219414",
  "publisher_id": "infra-optim:node0"
}

actionplan.cancel.error

   {
     "event_type": "action_plan.cancel.error",
     "publisher_id": "infra-optim:node0",
     "priority": "ERROR",
     "message_id": "9a45c5ae-0e21-4300-8fa0-5555d52a66d9",
     "payload": {
       "watcher_object.version": "1.0",
       "watcher_object.namespace": "watcher",
       "watcher_object.name": "ActionPlanCancelPayload",
       "watcher_object.data": {
         "fault": {
           "watcher_object.version": "1.0",
           "watcher_object.namespace": "watcher",
           "watcher_object.name": "ExceptionPayload",
           "watcher_object.data": {
           "exception_message": "TEST",
           "module_name": "watcher.tests.notifications.test_action_plan_notification",
           "function_name": "test_send_action_plan_cancel_with_error",
           "exception": "WatcherException"
         }
       },
       "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
       "created_at": "2016-10-18T09:52:05Z",
       "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
       "strategy": {
         "watcher_object.version": "1.0",
         "watcher_object.namespace": "watcher",
         "watcher_object.name": "StrategyPayload",
         "watcher_object.data": {
           "uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
           "created_at": "2016-10-18T09:52:05Z",
           "name": "TEST",
           "updated_at": null,
           "display_name": "test strategy",
           "parameters_spec": {},
          "deleted_at": null
        }
      },
      "updated_at": null,
      "deleted_at": null,
      "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
      "audit": {
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher",
        "watcher_object.name": "TerseAuditPayload",
        "watcher_object.data": {
          "parameters": {},
          "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "created_at": "2016-10-18T09:52:05Z",
          "scope": [],
          "updated_at": null,
          "audit_type": "ONESHOT",
          "interval": null,
          "deleted_at": null,
          "state": "SUCCEEDED"
        }
      },
      "global_efficacy": {},
      "state": "CANCELLING"
    }
  },
  "timestamp": "2016-10-18 09:52:05.219414"
}

action.cancel.start

{
  "priority": "INFO",
  "payload": {
    "watcher_object.namespace": "watcher",
    "watcher_object.version": "1.0",
    "watcher_object.name": "ActionCancelPayload",
    "watcher_object.data": {
      "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
      "input_parameters": {
        "param2": 2,
        "param1": 1
      },
      "fault": null,
      "created_at": "2016-10-18T09:52:05Z",
      "updated_at": null,
      "state": "CANCELLING",
      "action_plan": {
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0",
        "watcher_object.name": "TerseActionPlanPayload",
        "watcher_object.data": {
          "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
          "global_efficacy": {},
          "created_at": "2016-10-18T09:52:05Z",
          "updated_at": null,
          "state": "CANCELLING",
          "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
          "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
          "deleted_at": null
        }
      },
      "parents": [],
      "action_type": "nop",
      "deleted_at": null
    }
  },
  "event_type": "action.cancel.start",
  "publisher_id": "infra-optim:node0",
  "timestamp": "2017-01-01 00:00:00.000000",
  "message_id": "530b409c-9b6b-459b-8f08-f93dbfeb4d41"
}

action.cancel.end

{
  "priority": "INFO",
  "payload": {
    "watcher_object.namespace": "watcher",
    "watcher_object.version": "1.0",
    "watcher_object.name": "ActionCancelPayload",
    "watcher_object.data": {
      "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
      "input_parameters": {
        "param2": 2,
        "param1": 1
      },
      "fault": null,
      "created_at": "2016-10-18T09:52:05Z",
      "updated_at": null,
      "state": "CANCELLED",
      "action_plan": {
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0",
        "watcher_object.name": "TerseActionPlanPayload",
        "watcher_object.data": {
          "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
          "global_efficacy": {},
          "created_at": "2016-10-18T09:52:05Z",
          "updated_at": null,
          "state": "CANCELLING",
          "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
          "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
          "deleted_at": null
        }
      },
      "parents": [],
      "action_type": "nop",
      "deleted_at": null
    }
  },
  "event_type": "action.cancel.end",
  "publisher_id": "infra-optim:node0",
  "timestamp": "2017-01-01 00:00:00.000000",
  "message_id": "530b409c-9b6b-459b-8f08-f93dbfeb4d41"
}

action.cancel.error

 {
   "priority": "ERROR",
   "payload": {
     "watcher_object.namespace": "watcher",
     "watcher_object.version": "1.0",
     "watcher_object.name": "ActionCancelPayload",
     "watcher_object.data": {
       "uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
       "input_parameters": {
         "param2": 2,
         "param1": 1
       },
       "fault": {
       "watcher_object.namespace": "watcher",
       "watcher_object.version": "1.0",
       "watcher_object.name": "ExceptionPayload",
       "watcher_object.data": {
         "module_name": "watcher.tests.notifications.test_action_notification",
         "exception": "WatcherException",
         "exception_message": "TEST",
         "function_name": "test_send_action_cancel_with_error"
       }
     },
     "created_at": "2016-10-18T09:52:05Z",
     "updated_at": null,
     "state": "FAILED",
     "action_plan": {
       "watcher_object.namespace": "watcher",
       "watcher_object.version": "1.0",
       "watcher_object.name": "TerseActionPlanPayload",
       "watcher_object.data": {
         "uuid": "76be87bd-3422-43f9-93a0-e85a577e3061",
         "global_efficacy": {},
         "created_at": "2016-10-18T09:52:05Z",
         "updated_at": null,
         "state": "CANCELLING",
         "audit_uuid": "10a47dd1-4874-4298-91cf-eff046dbdb8d",
         "strategy_uuid": "cb3d0b58-4415-4d90-b75b-1e96878730e3",
         "deleted_at": null
       }
     },
     "parents": [],
     "action_type": "nop",
     "deleted_at": null
   }
 },
 "event_type": "action.cancel.error",
 "publisher_id": "infra-optim:node0",
 "timestamp": "2017-01-01 00:00:00.000000",
 "message_id": "530b409c-9b6b-459b-8f08-f93dbfeb4d41"
}

Alternatives

we can work with actionplan.update and action.update notification, but actionplan cancel is a complete operation not just status update so defining new notifications is better approach.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

This blueprint will implement the following notifications:

actionplan.cancel.start
actionplan.cancel.end
actionplan.cancel.error
action.cancel.start
action.cancel.end
action.cancel.error

Other end user impact

None.

Performance Impact

None.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: adisky <aditi.s@nectechnologies.in>

Work Items

Implement the Notfications as described in “Proposed change” section.

Dependencies

None.

Testing

unit tests should be added for new notifications.

Documentation Impact

A notification sample should be added in watcher documentation.

References

Storage Capacity Balance Strategy

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/storage-capacity-balance

As of now, Watcher optimizes only compute nodes. Storage optimization is also an important feature for centralized storage (non distributed storage).

This spec will add Storage Capacity Balance Strategy to balance the storage capacity, which can be also considered a way to balance the storage workload. And we can use existing goal(workload_balancing) and action(volume_migrate) for this storage capacity balance.

Problem description

In current Data Center, the capacity of storage back-end may be not balanced, some are extremely high, some are idle. This situation will degrade the performance of I/O Read/Write, which will finally affect the QoS. This problem can be solved by storage capacity balance strategy.

This strategy migrates volumes based on the capacity utilization of the cinder pools. It makes decision to migrate a volume whenever a pool’s capacity utilization % is higher than the specified threshold. The migration of a volume should make the capacity utilization of the pool where it locates lower than the storage capacity utilization threshold.

Use Cases

As an administrator, I want to be able to trigger an audit that controls the storage capacity utilization below a certain threshold.

Proposed change

Extend base strategy classes to add one new strategy - “ Storage Capacity Balance Strategy”
Use Cinder client to get all volumes with status in available or in-use and no snapshots, and to get all pools except the pools listed as exclude_pools in the configuration file.

Group volume pools into two categories: underload or overload pools according to threshold:

under_pools = list(filter(lambda p: float(p.total_capacity_gb) -
float(p.free_capacity_gb) < float(p.total_capacity_gb) * threshold, pools))

over_pools = list(filter(lambda p: float(p.total_capacity_gb) -
float(p.free_capacity_gb) >= float(p.total_capacity_gb) * threshold, pools))

Determine migrate_volumes, source pools and destination pools based on some factors:
- whether a volume is mounted to a VM
- volume size,
- whether a volume is a mirrored volume
- priority

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <yumeng-bao>,<li-canwei2>

Work Items

Define proper threshold
Write the execute function to locate pool overloaded
Function to generate actions:volume_migrate or volume_retype.

Dependencies

None

Testing

Unit and functional test are needed.

Documentation Impact

Add docs on how to use this strategy.

References

None

History

None

Support multi datasource in watcher

Mon, 12 Feb 2018 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/watcher-multi-datasource

Problem description

Today, Watcher supports Telemetry, Monasca and gnocchi to collect metrics from the cluster. There is a helper class for each respective datasource type implements query and provides the results. We need to create datasource abstract class which will define methods for all the metrics capability methods. These methods have to be implemented by respective datasource helper class.

Use Cases

As administrator, I want to run strategies to specific datasource backend.

Proposed change

Define a common abstraction layer for all datasources. Define get methods for all metrics used by current strategies. It also contains metric map which maps strategy metric requirements to datasource metric name.

class DatasourceBase(object):

    METRIC_MAP=dict(datasource=dict(metric_name_watcher=
                                    metric_name_datasource,),)

    @abc.abstractmethod
    def get_host_cpu_usage(self, resource_id, period, aggregate,
                           granularity=None):
        raise NotImplementedError

    @abc.abstractmethod
    def get_instance_cpu_usage(self, resource_id, period, aggregate,
                               granularity=None):
        raise NotImplementedError

Update all the existing datsource helper classes as the child class of DatasourceBase class. Implement abstract methods of DatasourceBase class in ceilometer, monasca and gnocchi classes.
Support metric data collection from multiple datasource. User can define datasource priority in watcher.conf, we can use the existing configuration by changing it type to list.:
```
[watcher_strategies.basic]
datasource = ceilometer, gnocchi, monasca
```
This can be done in two ways
1. calculate the datasource_backend value on strategy basis. check, if meters used by the strategy available in first priority datasource, choose that backend for all subsequent requests for that audit request.
2. calculate the datasource_backend on metric basic, ie each the strategy request to get some metric data, calculate the value of datasource_backend on the basis or priority set in conf file and availablity of the metric in choosen datasource.
option second has overhead of choosing datasource each time the request arrives, but has advantage that if a metric is not available in one datasource it can check for other. with current watcher strategy requirements option first will suffice.

Define a datasource manager class to calculate the datasource backend.

class DatasourceManager(object):

    def get_backend(self, metrics):
    ''' returns: object of DatasourceBase child
        class '''

Update existing strategies to use new datasource classes, Make the datasource_backend property abstract in strategy base class.

class BasicConsolidation(base.ServerConsolidationBaseStrategy):

    @property
    def datasource_backend(self):
        self.datasource_backend = manager.DataSourceManager(
            config=self.config,
            osc=self.osc
            ).get_backend(self.METRIC_NAMES)
        return self._datasource_backend

   def get_node_cpu_usage(self, node):
       resource_id = "%s_%s" % (node.uuid, node.hostname)
       return self.datasource_backend.get_host_cpu_usage(
           resource_id = resource_id,
           period=self.period,
           aggregate = 'avg',
           granularity=None
           )

Alternatives

If we keep the existing implementation, each strategy has to use the specific interfaces of the data source driver provides. It makes the implementation of strategies highly coupled with a specific data source.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Using Multiple Datasource can degrade the overall openstack cluster performance.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: adi-sky17

Work Items

Implement base class for all datasource drivers.
Define abstract get methods for all meters used by existing strategies.
Implement abstract methods in ceilometer, monasca and gnocchi classes.
Update existing strategies to use the new format.

Dependencies

None

Testing

Unit tests need to be updated

Documentation Impact

Plugin documents should be updated. https://docs.openstack.org/watcher/latest/contributor/plugin/strategy-plugin.html#querying-metrics

References

None

History

None

Zone migration strategy

Mon, 12 Feb 2018 00:00:00

https://blueprints.launchpad.net/watcher/+spec/zone-migration-strategy

There are thousands of physical servers and storage running various kinds of workloads in the cloud system. Operator have to migrate instances and volumes for hardware maintenance once a quarter or so. It requires operator to watch workloads, choose instances and volumes and migrate them efficiently with minimum downtime. Watcher can be used to do this task automatically.

Problem description

It is hard for operator to migrate many instances and volumes efficiently with minimum downtime for hardware maintenance.

Use Cases

As an operator, I want Watcher to migrate many instances and volumes efficiently with minimum downtime automatically.

Proposed change

Add Goal “Hardware maintenance”. The goal is to migrate instances and volumes on a set(called “Zone” in this spec) of compute nodes and storage.

Add eight Efficacy Indicator.

LiveInstanceMigrateCount

The number of instances should be live migrated
PlannedLiveInstanceMigrateCount

The number of instances planned to live migrate
ColdInstanceMigrateCount

The number of instances should be cold migrated
PlannedColdInstanceMigrateCount

The number of instances planned to cold migrate
VolumeMigrateCount

The number of detached volumes should be migrated
PlannedVolumeMigrateCount

The number of detached volumes planned to migrate
VolumeUpdateCount

The number of attached volumes should be updated
PlannedVolumeUpdateCount

The number of attached volumes planned to update

Add Efficacy Specification associated with the goal. The efficacy specification has four global efficacy indicators.

live_instance_migrate_ratio

Ratio of planned live migrate instances to instances should be live migrated. The result of PlannedLiveInstanceMigrateCount / LiveInstanceMigrateCount
cold_instance_migrate_ratio

Ratio of planned cold migrate instances to instances should be cold migrated. The result of PlannedColdInstanceMigrateCount / ColdInstanceMigrateCount
volume_migrate_ratio

Ratio of planned detached volumes to volumes should be migrated. migrate. The result of PlannedVolumeMigrateCount / VolumeMigrateCount
volume_update_ratio

Ratio of planned attached volumes to volumes should be updated. The result of PlannedVolumeUpdateCount / PlannedVolumeUpdateCount

Add Strategy “Zone migration”. The strategy gets compute node and storage pool names given by input parameter, and gets instances and volumes on them from cluster data model. After that, the strategy prioritizes instances and volumes respectively by the following list given by input parameter.

project
compute_node
storage_pool
compute

The strategy chooses instances in descending order of one of the following:
- vcpu_num
- mem_size
- disk_size
- created_at
storage

The strategy chooses volumes in descending order of one of the following:
- size
- created_at

For example, If the following input parameters is given:

"priority": {
    "project": ["pj1"],
    "compute_node": ["compute1", "compute2"],
    "compute": ["cpu_num"],
    "storage_pool": ["pool1", "pool2"],
    "storage": ["size"]
}

And we have list of instances and volumes as the followings.

instances:

{"project": "pj1", "node":"compute2", "cpu_num": 1, "memory_size": 4}
{"project": "pj2", "node":"compute1", "cpu_num": 1, "memory_size": 2}
{"project": "pj1", "node":"compute1", "cpu_num": 2, "memory_size": 3}
{"project": "pj2", "node":"compute1", "cpu_num": 1, "memory_size": 1}

volumes:

{"project": "pj1", "node":"pool1", "size": 3, "created_at": 2017-02-25}
{"project": "pj1", "node":"pool1", "size": 3, "created_at": 2017-02-26}
{"project": "pj2", "node":"pool1", "size": 5, "created_at": 2017-02-25}
{"project": "pj1", "node":"pool2", "size": 3, "created_at": 2017-02-25}

Instances are prioritized as the following:

{"project": "pj1", "node":"compute1", "cpu_num": 2, "memory_size": 3}
{"project": "pj1", "node":"compute2", "cpu_num": 1, "memory_size": 4}
{"project": "pj2", "node":"compute1", "cpu_num": 1, "memory_size": 2}
{"project": "pj2", "node":"compute1", "cpu_num": 1, "memory_size": 1}

Volumes are prioritized as the following:

{"project": "pj1", "node":"pool1", "size": 3, "created_at": 2017-02-25}
{"project": "pj1", "node":"pool1", "size": 3, "created_at": 2017-02-26}
{"project": "pj2", "node":"pool1", "size": 5, "created_at": 2017-02-25}
{"project": "pj1", "node":"pool2", "size": 3, "created_at": 2017-02-25}

The strategy migrates all chosen volumes first by volume_migrate action. Then the strategy migrates all chosen instances by migrate action. Destination can be given by input parameter.

If volume is attached to an instance, the instance can be migrated just after attached volume is migrated. Because they should be near place for performance reason. This behavior is configurable.

The strategy uses weights planner that is the planner by default which has the number of actions to be run in parallel on a per action type basis. In addition to that, the strategy has the number of actions to be run in parallel per node or pool. The number is given by the input parameter.

The strategy gets volumes and instances from prioritized ones and migrates them which are limited to the number of volumes and instances to each number of parallelization per host and the number of parallelization per action in weight planner.

The input parameters are the followings:

compute_nodes": [
    {
        "src_node": "cen-cmp02",
        "dst_node": "cen-cmp01"
    },
    ......
],
"storage_pools": [
    {
        "src_pool": "cen-cmp02@lvm#afa",
        "dst_pool": "cen-cmp01@lvm#afa",
        "src_volume_type": "afa",
        "dst_volume_type": "afa"
    },
    ........
],
"parallel_per_node": 2,
"parallel_per_pool": 2,
"priority": {
    "project": ["pj1", "pj2"],
    "compute_node": ["compute1", "compute2"],
    "storage_pool": ["pool1", "pool2"],
    "compute": ["cpu_num", "memory_size"],
    "storage": ["size"]
}
"with_attached_volume": false

Alternatives

Operator migrates instances and volumes manually one by one.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>
Other contributors:: <adi-sky17>

Work Items

Implement goal “Hardware maintenance”
Implement efficacy indicator
Impement efficacy specification
Implement strategy “Zone migration”
- Filters for prioritizing volumes and instances to be migrated
- Parallel number controller
- Migrating volumes and instances by actions logic

Dependencies

Testing

Unit and tempest tests are added.

Documentation Impact

Strategy documentation is added.

References

https://www.youtube.com/watch?v=_6kB1NTob8o

History

None

Pike Project Priorities

Thu, 08 Feb 2018 00:00:00

List of priorities the Watcher drivers team is prioritizing in Pike.

Priority	Owner
Cinder Model Intergration	Hidekazu Nakamura
Audit tag in VM Metadata	Prashanth Hari
Support Gnocchi in Watcher	Santhosh Fernandes
Noisy Neighbor Strategy	Prudhvi Rao Shedimbi
Workload characterization grammar	Chris Spencer
Stale the Action Plan	Li Canwei
Workload Characterization and QoS	Susanne Balle
Action versioned notifications	Alexander Chadin
Cancel Action Plan	aditi sharma
Dynamic Action Description	Charlotte Han
Power On and Power Off in Watcher	Li Canwei
Suspended audit state	Hidekazu Nakamura
JSONschema validation	YumengBao
Service versioned notifications	Vladimir Ostroverkhov
Notifications for action plan cancel	aditi sharma
Use cron syntax for CONTINUOUS audits	Alexander Chadin
Event-driven optimization based	Alexander Chadin

Cinder Model Intergration

Extend Watcher Cluster Data Model with Cinder-related data. There should be able following features: To integrate storage info at the model build stage

To consume all the needed Cinder Notifications in order to maintain the consistency of the storage-related part of the model

To easily query/retrieve the storage information from within a strategy via a clear set of methods

Audit tag in VM Metadata

When Watcher runs Audit to achieve a Goal, there should be some way for the application/VM owners to know that their VMs are under audit and is flagged for Action Plan execution. These information could be stored in VM metadata with a timestamp after which action plan will be executed.

Support Gnocchi in Watcher

Today, Watcher uses Telemetry and Monasca to collect metrics from the Cluster. There is need to support Gnocchi as well since Ceilometer v2 API is deprecated.

Noisy Neighbor Strategy

L3 cache is critical and limit system level resource shared by all apps or VMs on one node. If one VM occupies most of L3 cache, other VMs on the node likely starve without enough L3 cache thus poor performance.

This BP adds a new Strategy to detect then migrate such cache greedy VM based on some new cache/memory metrics.

Workload characterization grammar

As we run several workloads in cloud, we should be able to characterize such workloads as input to watcher for ensuring Application QoS, placements and consolidation.

An example of workload characterization is a weighted combination of CPU, Memory or any other resource attributes like High IOPs, Network latency etc.

Scope of this blueprint is to come up with a grammar structure for defining workload character.

Stale the Action Plan

When an audit is created and launched successfully, it generates a new Action Plan with status RECOMMENDED. If the Cluster Data Model has changed by and by, the action plan is still keep the RECOMMENDED state. There is not an expiry date or event that can invalidate the action plan by far.

Workload Characterization and QoS

Based on the defined workload characteristics we should be able to apply Quality of Services to applications. An example would be leveraging technologies like Intel RDT.

Action versioned notifications

As of now, there is no way for any service (Watcher included) to know when an action has been created, modified or deleted. This prevents any form of event-based reaction which may be useful for 3rd party services or plugins.

This blueprint should define the list of Action notifications to be implemented as well as their respective payload structures.

Cancel Action Plan

As of now Administrator can update the action plan state to CANCELLED but there is no action taken by Watcher to cancel the action plan. It only updates the action plan state to CANCELLED.

It should be possible to CANCEL execution of the action plan by Watcher.

Dynamic Action Description

By introducing a new way, for developer, to implement a strategy with new customized actions (blueprint watcher-add-actions-via-conf), we have no more the possibility to have the literal description of an planned Action before to execute it (in the Watcher Applier). This literal description is important when the cloud admin want to see details information about a recommended action plan.

Power On and Power Off in Watcher

Watcher need one strategy which can reduce the power consumption.

A traffic system could be running on many virtual machines. The traffic is busy during day time, so the traffic system would increase virtual machines’ number to satisfy its workload. But during the night, the traffic’s workload decreases obviously, so this traffic system would delete redundant virtual machines. This feature we call “elastic scaling” in telecom.

The telecom operators have their own hardware equipment and sometimes the size of hardware is large. So telecom operators want to use cloud center manager software to reduce the energy consumption of hardware automatically based on “elastic scaling”.

Suspended audit state

As of now Watcher have to delete audit and recreate audit if administrator want to stop creating action plan of audit with continuous mode.

This blueprint adds suspended audit state for stopping creation of action plan related to audit with continuous mode.

JSONschema validation

As of now in Watcher both jsonschema and voluptuous are used to validate JSON payloads. However, the problem with voluptuous is that its structure is not standardized compare to jsonschema which means that we cannot easily expose the validation schema through our API.

Service versioned notifications

This blueprint should define the list of Service notifications to be implemented as well as their respective payload structures.

Notifications for action plan cancel

Notifications needs to be added to action and actionplan for new operation actionplan cancel.

Use cron syntax for CONTINUOUS audits

As of now we use a period in seconds to schedule continuous audits. This works well but does not really give the flexibility that an operator might actually want. Therefore, we should also provide a way to express out scheduling needs via the cron syntax which shall give operators a fine grained control.

This change implies the refactoring of the API so backward compatibility should be guaranteed. On the Watcher dashboard side, we should also provide an easy-to-use form to fill in this cron field.

We should also keep the cron syntax and the creation timestamp in the DB

Event-driven optimization based

We propose an event-driven optimization-based audit control. We wants to select among a list of events which may trigger the audit :

React to a predicted situation.
React to a critical situations and changes in system (e.g: threshold )
A new compute node has been added to the cluster
A compute node has been removed from the cluster
A new virtual machine has been created

Noisy neighbor dashboard

Thu, 28 Sep 2017 00:00:00

https://blueprints.launchpad.net/watcher-dashboard/+spec/noisy-neighbor-dashboard

Following the implementation of the noisy-neighbor-strategy blueprint, Watcher now has all of the necessary prerequisites in order to provide L3 cache and memory metrics to the operator. This blueprint will focus on collecting this resource consumption data and displaying it in a series of graphs for operator monitoring and consumption.

Problem description

There is currently no way for an operator to easily access and visualize L3 cache and memory bandwidth consumption data on a per VM basis recorded by Watcher.

Use Cases

As an OpenStack operator, I need to view the noisy neighbor for L3 cache and memory bandwidth metrics to make sure my nodes have good isolation and performance.

As an OpenStack operator, I want to view IPC vs cache occupancy data.

As an OpenStack operator, I want to view IPC vs memory bandwidth data.

As an OpenStack operator, I want to view the memory bandwidth delta vs VM ID.

As an OpenStack operator, I want to view CPU steal vs VM ID.

Proposed change

A Grafana template will be designed to display the desired data and will integrate with both Gnocchi and Monasca telemetry collection services.

Alternatives

A new watcher-dashboard change is needed to display graphs of the above use cases.

Data model impact

None.

REST API impact

None.

Security impact

None.

Notifications impact

None.

Other end user impact

None.

Performance Impact

Possible minor performance drop when viewing data plotting.

Other deployer impact

None.

Developer impact

None.

Implementation

Assignee(s)

Primary assignee:: christopher-m-spencer

Work Items

New Grafana spec will be designed for noisy neighbor strategy.
Collect cache occupancy and memory bandwidth data and output to plots.

Dependencies

Testing

These graphs will have to be tested mainly via integration tests and visual inspection.

Documentation Impact

None.

References

None.

compute CDM include all instances

Wed, 30 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/compute-cdm-include-all-instances

Problem description

When building compute CDM, we will exclude the instances excluded by the scope. It has terrible impact to Watcher.

To workload balance and server consolidation, the excluded instances in the scope are not added into the workload. So it would get incorrectly workload of the compute nodes, and execute unreasonbale migrations.

To server consolidation, it would disable the nodes which still have excluded instances running.

Use Cases

As an end user, I want Watcher to take excluded instances into account during workload calculations but not migrate excluded instances.

Proposed change

Watcher should have a whole scope of the cluster, include all instances. So it would get the correct workload of the nodes, and make the correct optimization.

Compute CDM in Watcher should has a whole scope, include all instances. It needs to be clear which instances are exclude and which are not.

We can add “watcher_exclude” field to “Instance” resource to distinguish them, as following.

class Instance(compute_resource.ComputeResource):

fields = {
“watcher_exclude”: wfields.BooleanField(default=False), “state”: wfields.StringField(default=InstanceState.ACTIVE.value), …

}

When building compute CDM, set the ‘watcher_exclude’ flag ‘True’ if the instance is excluded by the scope.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: sue

Work Items

Add and identify excluded instances in compute CDM
Adjust workload balance and server consolidation, when calculating workload of the compute nodes and generating solutions

Dependencies

Testing

Unit test

Documentation Impact

None

References

https://docs.openstack.org/watcher/latest/glossary.html#cluster-data-model-definition

History

None

Notifications in Watcher

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/watcher-notifications-ovo

Provide a notification mechanism into Watcher that supports versioning.

Whenever a Watcher object (i.e. Goal, Strategy, AuditTemplate, Audit, ActionPlan, Action, EfficacyIndicator or ScoringEngine) is created, updated or deleted, a versioned notification should, if it’s relevant, be automatically sent to notify in order to allow an event-driven style of architecture within Watcher. Moreover, it will also give other services and/or 3rd party softwares (e.g. monitoring solutions or rules engines) the ability to react to such events.

This blueprint is heavily inspired on similar blueprints made on other projects:

Nova: versioned-notification-api blueprint
Ironic: notifications blueprint

Problem description

As of now, Watcher does not emit any notification whenever performing some actions such as an update of an object. These information should be notified so that Watcher services can inter-operate on an event-based basis instead of having to rely on data polling. Moreover, these notifications may find themselves useful to other OpenStack services which can also make use of them, especially Congress, Vitrage, Monasca or Aodh.

Therefore, the purpose of this blueprint is to give Watcher the ability to send versioned notifications.

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher. I also want to know what is the format of the notifications.

As an OpenStack developer, I want to be able to detect and follow up the changes in the notification format later on.

As an OpenStack operator, I want to be able to disable notifications in Watcher.

As an OpenStack operator, I want to be able to define the topic(s) to which Watcher will send its notifications.

Project Priority

N/A

Proposed change

Based on the oslo.versionedobjects library, the objective is to create versioned objects that will later on be serialized and sent as payload for the notifications, each one of these objects will hence be able to hold a version number that will facilitate retro-compatibility whenever adding new fields.

The basic structure for all notifications will be the same as the one that is used in Nova for the versioned-notification-api blueprint, i.e.:

{
    "priority": "INFO",
    "event_type": "audit.update",
    "timestamp": "2016-10-07 09:31:10.895274",
    "publisher_id": "infra-optim:localhost",
    "message_id": "a13cb7a6-8fae-4e20-8fc8-1c4e851fa6f5",
    "payload": {
      // [...]
    }
}

In order to send the versioned notification, we will use the oslo.messaging library which is a standard OpenStack library. Already used within Watcher to listen to notifications emitted by other projects (e.g. Nova), oslo.messaging will help us declare a notifier that allows us to configure the topics all notifications will be sent to.

Alternatives

Instead of using oslo.versionedobjects, we can define the payload of our notifications without any support for versioning.

Data model impact

The following base objects will be defined:

class NotificationPriorityType(Enum):
    AUDIT = 'audit'
    CRITICAL = 'critical'
    DEBUG = 'debug'
    INFO = 'info'
    ERROR = 'error'
    SAMPLE = 'sample'
    WARN = 'warn'

    ALL = (AUDIT, CRITICAL, DEBUG, INFO, ERROR, SAMPLE, WARN)


class NotificationPhase(BaseWatcherEnum):
    START = 'start'
    END = 'end'
    ERROR = 'error'

    ALL = (START, END, ERROR)


class NotificationAction(BaseWatcherEnum):
    CREATE = 'create'
    UPDATE = 'update'
    EXCEPTION = 'exception'
    DELETE = 'delete'

    ALL = (CREATE, UPDATE, EXCEPTION, DELETE)


class NotificationPriorityField(BaseEnumField):
    AUTO_TYPE = NotificationPriority()


class NotificationPhaseField(BaseEnumField):
    AUTO_TYPE = NotificationPhase()


class NotificationActionField(BaseEnumField):
    AUTO_TYPE = NotificationAction()


@base.WatcherObjectRegistry.register_notification
class EventType(NotificationObject):
    VERSION = '1.0'

    fields = {
        'object': fields.StringField(nullable=False),
        'action': fields.NotificationActionField(nullable=False),
        'phase': fields.NotificationPhaseField(nullable=True),
    }


@base.WatcherObjectRegistry.register_if(False)
class NotificationBase(NotificationObject):
    VERSION = '1.0'

    fields = {
        'priority': fields.NotificationPriorityField(),
        'event_type': fields.ObjectField('EventType'),
        'publisher': fields.ObjectField('NotificationPublisher'),
    }

    def emit(self, context):
        """Send the notification."""

REST API impact

None.

Security impact

None.

Notifications impact

None, although this blueprint introduces the required building blocks necessary to implement any notification.

Other end user impact

None.

Performance Impact

When enabled, code to send the notification will be called each time an event occurs that triggers a notification. This shouldn’t be much of a problem for Watcher itself, but the load on whatever message bus is used should be considered.

Other deployer impact

The following configuration option will be added:

A notification_level string parameter will be added to indicate the minimum priority level for which notifications will be sent. Available options will be DEBUG, INFO, WARN, ERROR, or None to disable notifications. INFO will be the default.

Note that some already existing configuration options coming from oslo.messaging that were auto-generated into the watcher.conf.sample configuration sample will now be taken into account. For more information, refer to the oslo.messaging configuration options documentation.

Developer impact

Developers should adhere to proper versioning guidelines and use the notification base classes when creating new notifications.

Implementation

Assignee(s)

Primary assignee:: vincent-francoise

Work Items

Implement all base objects presented in Data model impact
Add new sphinx directive that will help documenting the Watcher notifications by giving a notification sample.

Dependencies

watcher-versioned-objects

Testing

Unit tests should be added to ascertain the good behavior of the newly implemented notifications.

Documentation Impact

Create a new documentation section in Watcher that will automatically expose all the implemented notifications with a complete notification sample that can be used as a reference by the developers trying to consume them.

References

None.

Audit tag in VM Metadata

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/audit-tag-vm-metadata

Problem description

Watcher optimizes workloads by live migrating VMs. This can be disruptive. There should be a feature for tenants to opt-in/opt-out some or all of their VMs from optimization. The solution we propose is to set a VM metadata “optimize” and watcher strategies should consider only those VMs that has “optimize” metadata enabled. This feature for strategies to check VM metadata should be optional and configurable.

Use Cases

As a tenant, I should opt-in/opt-out optimization of my VMs

As a strategy developer, I should be able to filter VMs that are approved for optimizing

Proposed change

The VMs will have an “optimize” metadata key. The compute cluster data model should also store the VM metadata information. We need to a have a configuration option “check_optimize_metadata” to enable this feature.

If “check_optimize_metadata” configuration option is enabled, strategy base class will include only those VMs that has optimize metadata. This will use the Audit Scope to exclude VMs by metadata.

Example -

"scope": [
     {"host_aggregates": [
       {"id": 1},
       {"id": 2},
       {"id": 3}
     ]},
     {"availability_zones": [
       {"name": "AZ1"},
       {"name": "AZ2"}
     ]},
     {"exclude": [
       {"instances": [
         {"uuid": "9766dff1-3c81-4de2-92ae-19e0d9adcec6"},
         {"uuid": "777541bb-ce4f-4bf3-8320-d5792d5cdf6e"}
       ]},
       {"instance_metadata": [
         {"optimize": "False"}
       ]},
       {"compute_nodes": [
         {"name": "compute1"}
       ]}
     ]}
   ]

By default “check_optimize_metadata” will be disabled and “optimize” metadata will not have any effect on the strategies - meaning the strategies will consider all VMs.

Alternatives

Data model impact

There will be changes to the cluster data model to include VM metadata.

REST API impact

Security impact

There is no security impact.

Notifications impact

Other end user impact

Performance Impact

Other deployer impact

Developer impact

Implementation

Assignee(s)

Primary assignee:: hvprash
Other contributors:: v-francoise pradeep-singh-u

Work Items

Platform owners or automations external to watcher will set the VM Metadata

# nova meta vm_name set optimize=True

Enhance the current compute cluster data model to now include the VM metadata in its representation.
Capability in Audit Scope to exclude by instance_metadata (https://github.com/openstack/watcher/blob/54f0758fc3ac47edb4bc3f6eb5e56bf53d4e02f8/watcher/decision_engine/scope/default.py).
Modify base strategy to filter VMs by metadata based on configuration option

def filter_instances_by_audit_tag(self, instances):
    if not self.config.check_optimize_metadata:
        return instances
    instances_to_migrate = []
    for instance in instances:
        optimize = True
        if instance.metadata:
            try:
                optimize = strutils.bool_from_string(
                    instance.metadata.get('optimize'))
            except ValueError:
                optimize = False
        if optimize:
            instances_to_migrate.append(instance)
    return instances_to_migrate

Dependencies

N/A

Testing

Unit tests on watcher Audit and Strategy
Unit tests for audit scope. Also tempest scenarios to create an Audit Template and define an Audit scope with “instance_metadata”

Documentation Impact

Update documentations and reference to configuration options.

References

N/A

History

N/A

Add support to cancel execution of Action Plan

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cancel-action-plan

Problem description

A Cloud Administrator may want to cancel the action plan execution for any reason. An Action Plan can be in execution from long time and a cloud Administrator may want to execute the actions manually or want to fulfill the optimization requirements by some other methods.

As of now Administrator can update the action plan state to CANCELLED but there is no action taken by Watcher to cancel the action plan. It only updates the action plan state to CANCELLED.

It should be possible to CANCEL execution of the action plan by Watcher.

Use Cases

As a Watcher Administrator, I want Watcher to CANCEL the action plan that is in ONGOING state for a long time.

As a Watcher Administrator, I want to CANCEL any action plan that is accidentally launched.

As a Watcher Administrator, I want to CANCEL the action plan because my optimization requirements have been met.

Proposed change

Add new state CANCELLING for the Action Plan.
- CANCELLING, when watcher-api receives request from cloud admin to cancel the ONGOING action plan. When watcher-applier successfully executed the cancel request it will update the action plan state to CANCELLED. When watcher-applier failed to complete the action plan cancel request, it will update the action plan state to FAILED.
Add new states CANCELLING, CANCELLED to Action.
- CANCELLING, when watcher-applier cancel an ONGOING action.
- CANCELLED, when watcher-applier cancel a PENDING action. when watcher-applier successfully completed cancel of ONGOING action.
Add a new command watcher actionplan cancel in watcher-client to cancel the action plan.
Modify the state machine for action plan to add new actions plan state and transitions.
Modify the state machine for action to add new actions states and transitions.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

https://blueprints.launchpad.net/watcher/+spec/notifications-actionplan-cancel

Other end user impact

A new command watcher actionplan cancel will be added to cancel the action plan.

Performance Impact

None

Other deployer impact

Watcher cannot guarantee about the cluster state after cancelling the ongoing optimizations.

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <adi-sky17>

Work Items

Implementation for action plan cancel will be done by using taskflow. When a running engine receives an exception, it stops execution and calls revert method for all the tasks(actions) added in the flow. In the revert method of action, current action state is checked and based on the action state (PENDING, ONGOING, SUCCEEDED) revert method will execute defined steps.
```
e.g. if action_state is PENDING:
      # update action_state to CANCELLED
     if action_state is ONGOING:
      # call abort_action
     if action state is SUCCEEDED:
      # do nothing
```
Before starting any new action watcher-applier will check for the state of action plan, if its state is updated to CANCELLING it will trigger an exception “ActionPlanCancel” to the taskflow engine and taskflow engine will call revert method of all tasks (actions) in reverse manner, keeping the dependencies intact.

Dependencies

For action plan cancel operation we need to call Abort API for ONGOING actions.

The abort API of live migration is available. https://blueprints.launchpad.net/nova/+spec/abort-live-migration

The abort API for cold migration is under development. https://blueprints.launchpad.net/nova/+spec/abort-cold-migration

Testing

The unit and tempest tests will have to be updated.

Documentation Impact

Need to update the Action Plan State Machine in the architecture documentation.

Need to add the Action State Machine in the architecture documentation.

References

History

None

Remove voluptuous and Use JSONSchema as our only JSON validation tool

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/jsonschema-validation

As of now in Watcher, both JSONSchema and voluptuous are used to validate JSON payloads. However, the structure of voluptuous is not standardized compared to JSONSchema. This problem of voluptuous makes it troublesome to expose the validation schema through our API. So we want to remove existing voluptuous validation and use JSONSchema as our only JSON validation tool to validate JSON payloads.

Problem description

This blueprint will replace existing voluptuous validation with JSONSchema in watcher and use JSONSchema as the only JSON validation tool to validate JSON payloads.

Use Cases

As a developer, I want to have a consistent and standard JSON payload validation system which will make it easier to combine all the validation schemas together and expose them through API later.

Proposed change

For each Watcher action, we need to introduce a new JSONSchema to validate its input parameters when it gets to the Applier.

We need to introduce new JSONSchemas to validate the efficacy indicators.

Alternatives

We can also use voluptuous validation as our only validation tool, however voluptuous is not standardized compared to JSONSchema.

Data model impact

The incoming request is represented as an object, in which case the request object would have the jsonschema validator as an attribute.

This blueprint should not require a database migration or database schema change.

REST API impact

This has an impact in the API because it changes the display format of efficacy specification.

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Yumeng Bao <yumeng+bao>

Work Items

Remove voluptuous schemas

As of now, voluptuous is used to: - validate the input parameters of a Watcher action when it gets to the Applier. - validate the efficacy indicators * Implement jsonschemas to validate the efficacy indicators and the input parameter of action in Watcher Applier. * Implement appropriate unit tests to test various scenarios.

Dependencies

None expected

Testing

Appropriate unit tests will be adapted to new changes.

Documentation Impact

It will be necessary to add new content relating to this change.

References

https://blueprints.launchpad.net/watcher/+spec/jsonschema-validation https://python.libhunt.com/project/jsonschema/vs/voluptuous

History

No history

Versioned notifications for services

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/service-versioned-notifications-api

Following the implementation of the watcher-notifications-ovo blueprint, Watcher now has all the necessary prerequisites in order to provide versioned notifications throughout its codebase. This blueprint will focus on describing the notifications to implement in Watcher concerning the Service object.

Problem description

As of now, there is no way for any service (Watcher included) to know when a service of Watcher has been failed or whether it has been reactivated afterwards. This prevents any form of event-based reaction which may be useful for 3rd party services or plugins.

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher about service.

As an OpenStack developer, I want to know what the format of the service notifications are.

As an OpenStack developer, I want to be notified whenever a service has been failed or whether it has been reactivated afterwards.

Proposed change

In order to implement the above use cases, one notification will be needed:

service.update whenever a service has been failed.
service.update whenever a service has been reactivated afterwards.

Monitoring the status of the services will be performed through the background job, which will be performed periodicaly (60 seconds) by the watcher-api service.

Here below is suggestion of background job structure:

scheduling

class APISchedulingService(scheduling.BackgroundSchedulerService):

    def __init__(self, gconfig=None, **options):
        self.services_status = {}
        gconfig = None or {}
        super(APISchedulingService, self).__init__(gconfig, **options)

    def get_services_status(self, context):
        services = objects.service.Service.list(context)
        for service in services:
            result = self.get_service_status(context, service.name)
            if service.id not in self.services_status.keys():
                self.services_status[service.id] = result
                continue
            if self.services_status[service.id] != result:
                self.services_status[service.id] = result
                notifications.service.send_service_update(context, service,
                                                          state=result)

    def get_service_status(self, context, name):
        service = objects.Service.get(context, id)
        last_heartbeat = (service.last_seen_up or service.updated_at
                          or service.created_at)
        if isinstance(last_heartbeat, six.string_types):
            last_heartbeat = timeutils.parse_strtime(last_heartbeat)
        else:
            last_heartbeat = last_heartbeat.replace(tzinfo=None)
        elapsed = timeutils.delta_seconds(last_heartbeat, timeutils.utcnow())
        is_up = abs(elapsed) <= CONF.service_down_time
        if not is_up:
            LOG.warning(_LW('Seems service %(name)s on host %(host)s is down. '
                            'Last heartbeat was %(lhb)s.'
                            'Elapsed time is %(el)s'),
                        {'name': service.name,
                         'host': service.host,
                         'lhb': str(last_heartbeat), 'el': str(elapsed)})
            return objects.service.ServiceStatus.FAILED

        return objects.service.ServiceStatus.ACTIVE

    def start(self):
        """Start service."""
        context = watcher_context.make_context(is_admin=True)
        self.add_job(self.get_services_status, name='service_status',
                     trigger='interval', jobstore='default', args=[context],
                     next_run_time=datetime.datetime.now(), seconds=60)
        super(APISchedulingService, self).start()

    def stop(self):
        """Stop service."""
        self.shutdown()

    def wait(self):
        """Wait for service to complete."""

    def reset(self):
        """Reset service."""

Moreover, we will rely on oslo.versionedobjects to version the payloads of service-related notifications.

Here below is suggestion of notification structure of the aforementioned events:

service.update

{
  "payload": {
    "watcher_object.name": "ServiceUpdatePayload",
    "watcher_object.namespace": "watcher",
    "watcher_object.data": {
      "status_update": {
        "watcher_object.name": "ServiceStatusUpdatePayload",
        "watcher_object.namespace": "watcher",
        "watcher_object.data": {
          "old_state" :"ACTIVE",
          "state": "FAILED"
        },
        "watcher_object.version": "1.0"
      },
      "last_seen_up": "2016-09-22T08:32:06Z",
      "name": "watcher-service",
      "sevice_host": "controller"
    },
    "watcher_object.version": "1.0"
  },
  "event_type": "service.update",
  "priority": "INFO",
    "message_id": "3984dc2b-8aef-462b-a220-8ae04237a56e",
    "timestamp": "2016-10-18 09:52:05.219414",
    "publisher_id": "infra-optim:node0"
}

Alternatives

Instead of using versioned objects, we can define the payload of our service notifications without any support for versioning.

Data model impact

New versioned objects will be created although none of them are to be persisted as they will be used to structure the content of the notifications.

Here are some of the payloads to be declared:

@base.WatcherObjectRegistry.register_notification
class ServicePayload(notificationbase.NotificationPayloadBase):

    VERSION = '1.0'
    fields = {
        'sevice_host': wfields.StringField(),
        'name': wfields.StringField(),
        'last_seen_up': wfields.DateTimeField(),
    }


@base.WatcherObjectRegistry.register_notification
class ServiceStatusUpdatePayload(notificationbase.NotificationPayloadBase):

   VERSION = '1.0'
    fields = {
        'old_state': wfields.StringField(nullable=True)
        'state': wfields.StringField(nullable=True)
    }


@base.WatcherObjectRegistry.register_notification
class ServiceUpdatePayload(ServicePayload):

    VERSION = '1.0'
    fields = {
        'status_update': wfields.ObjectField('ServiceStatusUpdatePayload'),
    }

REST API impact

None.

Security impact

None.

Notifications impact

This blueprint will implement the following notifications:

service.update

Other end user impact

None.

Performance Impact

Other deployer impact

In order for the notifications to be emitted, the deployer will have to configure the notification topics using oslo.messaging. Other configuration options exposed via oslo.messaging may also be tuned.

Developer impact

Developers should add here proper versioning guidelines and use the notification base classes when creating/updating notifications.

Implementation

Assignee(s)

Primary assignee:: Vladimir Ostroverkhov <ostroverkhov@servionica.ru>

Work Items

Implement service.update

Dependencies

Testing

These notifications will have to be tested mainly via unit testing.

Documentation Impact

A notification sample should be provided and made dynamically available in the online documentation.

The sequence diagrams in the Watcher architecture.

References

None.

Define when an action plan is stale/invalid

Fri, 04 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/stale-action-plan

Problem description

When an audit is created and launched successfully, it generates a new action plan with status RECOMMENDED. If the cluster data model has changed by and by, the action plan still remains in the RECOMMENDED state. There is not an expiry date or event that can invalidate the action plan so far.

Use Cases

As a Watcher Administrator, I want watcher to update the state of any action plan from RECOMMENDED to SUPERSEDED automatically if the action plan has expired (default expiry time: 24 hours).

As a Watcher Administrator, after the recommended action plan is launched successfully, I want watcher to update the state of any action plan based on the same cluster data model from RECOMMENDED to SUPERSEDED automatically.

As a Watcher Administrator, when the cluster data model changes, I want watcher to update the state of action plan based on this cluster data model from RECOMMENDED to SUPERSEDED automatically.

Project Priority

None

Proposed change

Three scenarios need to be considered:

SCENARIO_1 action plan expiry

If an action plan remains in the RECOMMENDED state for a specified amount of time (24 hours by default), the state will automatically be set to SUPERSEDED.

for example:

actionplan_1{created_at:'2017-02-07 01:00:00', state:RECOMMENDED}
actionplan_2{created_at:'2017-02-07 02:00:00', state:RECOMMENDED}

when datetime='2017-02-08 01:01:00'
(datetime-actionplan_1.created_at)>24 hours
(datetime-actionplan_2.created_at)<24 hours
result:
actionplan_1{created_at:'2017-02-07 01:00:00', state:SUPERSEDED}
actionplan_2{created_at:'2017-02-07 02:00:00', state:RECOMMENDED}

when datetime='2017-02-08 02:01:00'
(datetime-actionplan_1.created_at)>24 hours
(datetime-actionplan_2.created_at)>24 hours
result:
actionplan_1{created_at:'2017-02-07 01:00:00', state:SUPERSEDED}
actionplan_2{created_at:'2017-02-07 02:00:00', state:SUPERSEDED}

SCENARIO_2 action plan launch

When a RECOMMENDED action plan is launched successfully, other action plans that are still in the RECOMMENDED state will be set to SUPERSEDED state.

Two cases:

CASE a:

Regardless of the scope of the audit generated action plan, as long as the action plans state are RECOMMENDED, will be set to SUPERSEDED.

for example:

scope_1 = {'availability_zones': [{'name': 'AZ1'}]}
scope_2 = {'availability_zones': [{'name': 'AZ2'}]}
scope_3 = {'availability_zones': [{'name': 'AZ3'}]}

actionplan_1{audit.scope:'scope_1', state:RECOMMENDED}
actionplan_2{audit.scope:'scope_2', state:RECOMMENDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

when launch actionplan_1{audit.scope:'scope_1', state:SUCCEEDED}
result:
actionplan_2{audit.scope:'scope_2', state:SUPERSEDED}
actionplan_3{audit.scope:'scope_3', state:SUPERSEDED}

CASE b:

Consider the scope of the audit that generated the action plan, check the scope of the audit that generated the action plan, and only set the action plan with the same scope to SUPERSEDED.

for example:

scope_1 = [{'availability_zones': [{'name': 'AZ1'}, {'name': 'AZ2'}]}]
scope_2 = [{'availability_zones': [{'name': 'AZ1'}, {'name': 'AZ2'}]}]
scope_3 = [{'availability_zones': [{'name': 'AZ2'}, {'name': 'AZ1'}]}]

actionplan_1{audit.scope:'scope_1', state:RECOMMENDED}
actionplan_2{audit.scope:'scope_2', state:RECOMMENDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

when launch actionplan_1{audit.scope:'scope_1', state:SUCCEEDED},
scope_1==scope_2:True
scope_1==scope_3:False
result:
actionplan_2{audit.scope:'scope_2', state:SUPERSEDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

notice: The audit.scope string value is compared here. So the result of scope_1 and scope_3 comparison is false.

SCENARIO_3 data model change

When the node in the data model changes, the action plan in the RECOMMENDED state is set to SUPERSEDED state.

Two cases:

CASE a:

As long as there is a node change, the action plan in the RECOMMENDED state will be set to SUPERSEDED state.

for example:

scope_1 = {'availability_zones': [{'name': 'AZ1'}]}
scope_2 = {'availability_zones': [{'name': 'AZ2'}]}
scope_3 = {'availability_zones': [{'name': 'AZ3'}]}

actionplan_1{audit.scope:'scope_1', state:RECOMMENDED}
actionplan_2{audit.scope:'scope_2', state:RECOMMENDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

when any node in data model change
result:
actionplan_1{audit.scope:'scope_1', state:SUPERSEDED}
actionplan_2{audit.scope:'scope_2', state:SUPERSEDED}
actionplan_3{audit.scope:'scope_3', state:SUPERSEDED}

CASE b:

Check the audit scope of the generated action plan. Only when the node which in the scope of the audit, the corresponding action plan is set to SUPERSEDED.

for example:

scope_1 = {'availability_zones': [{'name': 'AZ1'}]}
scope_2 = {'availability_zones': [{'name': 'AZ2'}]}
scope_3 = {'availability_zones': [{'name': 'AZ3'}]}

actionplan_1{audit.scope:'scope_1', state:RECOMMENDED}
actionplan_2{audit.scope:'scope_2', state:RECOMMENDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

when node in scope_1 change
result:
actionplan_1{audit.scope:'scope_1', state:SUPERSEDED}
actionplan_2{audit.scope:'scope_2', state:RECOMMENDED}
actionplan_3{audit.scope:'scope_3', state:RECOMMENDED}

Discussion conclusion As below there are three options to choose from. According to the conclusion at the PTG meeting, at first we do the simple one, and take scope into consideration where needed in future work.

1: Simple, just SCENARIO_1

2: Stringent, SCENARIO_1 + SCENARIO_2a + SCENARIO_3a

3: Complex, SCENARIO_1 + SCENARIO_2b + SCENARIO_3b

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

Add new config parameter action_plan_expiry and check_periodic_interval.

cfg.IntOpt('action_plan_expiry',
           default=24,
           help=('An expiry date(hours). We invalidate the action plan'
                  'if its created older than the expiry date.')),
cfg.IntOpt('check_periodic_interval',
           default=30*60,
           help=('Seconds between running periodic tasks.'))

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <licanwei>

Work Items

Here is the list of foreseen work items:

Add config parameter action_plan_expiry and check_periodic_interval (SCENARIO_1).
Modify the DefaultActionPlanHandler class, trigger to check the state of action plan(SCENARIO_2).
Currently, when receives the specific event (such as nova notification event_type:service.update, instance.update, instance.delete.end), the decision engine will update the cluster data model. We need to add the state of action plan updates to this process(SCENARIO_3).
Add a new StateManager class. In this class, we implement the action plan state management(SCENARIO_1, SCENARIO_2, SCENARIO_3).

Dependencies

https://blueprints.launchpad.net/watcher/+spec/define-the-audit-scope

Testing

The unit tests and tempest test will have to be updated.

Documentation Impact

Need to update the Action Plan State Machine in the architecture documentation.

References

None

History

None

Versioned notifications for actions

Thu, 03 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/action-versioned-notifications-api

Problem description

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher about actions.

As an OpenStack developer, I want to know what the format of the action notifications are.

As an OpenStack developer, I want to be notified whenever:

an action has been created
an action is updated
an action has been deleted

As an OpenStack developer, I would also want to be notified whenever:

an action starts its execution
an action finishes its execution
the execution of an action failed

Proposed change

In order to implement the above use cases, many different notifications will be needed:

action.create whenever an action has been created.
action.update whenever an action has been updated. This includes all state updates including the deletion of the action.
action.delete whenever an action has been deleted (soft).
action.execution.start whenever an action starts.
action.execution.end whenever an action ends.
action.execution.error whenever an action fails.

Moreover, we will rely on oslo.versionedobjects to version the payloads of action-related notifications.

Here below is suggestion of notification structure for each one of the aforementioned events:

action.create

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "state": "PENDING",
      "updated_at": null,
      "deleted_at": null,
      "action_plan": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
        },
        "watcher_object.name": "ActionPlanPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionCreatePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "action.create",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

action.update

{
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:51:38.722986   ",
  "payload": {
    "watcher_object.name": "ActionUpdatePayload",
    "watcher_object.data": {
      "strategy": {
        "watcher_object.name": "StrategyPayload",
        "watcher_object.data": {
          "name": "dummy",
          "parameters_spec": {
            "properties": {
              "para2": {
                "default": "hello",
                "type": "string",
                "description": "string parameter example"
              },
              "para1": {
                "maximum": 10.2,
                "default": 3.2,
                "minimum": 1.0,
                "description": "number parameter example",
                "type": "number"
              }
            }
          },
          "updated_at": null,
          "display_name": "Dummy strategy",
          "deleted_at": null,
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "created_at": "2016-11-04T16:25:35Z"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "scope": [],
      "created_at": "2016-11-04T16:51:21Z",
      "uuid": "f1e0d912-afd9-4bf2-91ef-c99cd08cc1ef",
      "goal": {
        "watcher_object.name": "GoalPayload",
        "watcher_object.data": {
          "efficacy_specification": [],
          "updated_at": null,
          "name": "dummy",
          "display_name": "Dummy goal",
          "deleted_at": null,
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "created_at": "2016-11-04T16:25:35Z"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "deleted_at": null,
      "state_update": {
        "watcher_object.name": "ActionStateUpdatePayload",
        "watcher_object.data": {
          "state": "ONGOING",
          "old_state": "PENDING"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "interval": null,
      "updated_at": null,
      "state": "ONGOING",
      "action_type": "NOP"
    },
    "watcher_object.namespace": "watcher",
    "watcher_object.version": "1.0"
  },
  "priority": "INFO",
  "event_type": "action.update",
  "message_id": "697fdf55-7252-4b6c-a2c2-5b9e85f6342c"
}

action.delete

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "action_type": "NOP",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "DELETED",
      "updated_at": null,
      "deleted_at": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionDeletePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "action.delete",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

action.execution.start

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "action_type": "NOP",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "action.execution.start",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

action.execution.end

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "action_type": "NOP",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "action.execution.end",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

action.execution.error

{
  "priority": "ERROR",
  "payload": {
    "watcher_object.data": {
      "action_type": "NOP",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": {
        "watcher_object.data": {
          "exception": "WatcherException",
          "exception_message": "TEST",
          "function_name": "test_send_action_action_with_error",
          "module_name": "watcher.tests.notifications.test_action_notification"
        },
        "watcher_object.name": "ExceptionPayload",
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "action.execution.error",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

Alternatives

Instead of using versioned objects, we can define the payload of our action notifications without any support for versioning.

Data model impact

New versioned objects will be created although none of them are to be persisted as they will be used to structure the content of the notifications.

Here are some of the payloads to be declared:

@base.WatcherObjectRegistry.register_notification
class ActionPayload(notificationbase.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'uuid': fields.UUIDField(),
        'action_type': fields.StringField(),
        'state': fields.StringField(),
        'parameters': fields.FlexibleDictField(nullable=True),
        'interval': fields.IntegerField(nullable=True),
        'goal_uuid': fields.UUIDField(),
        'strategy_uuid': fields.UUIDField(nullable=True),
        'goal': fields.ObjectField('Goal'),
        'strategy': fields.ObjectField('Strategy', nullable=True),
        'created_at': fields.DateTimeField(nullable=True),
        'updated_at': fields.DateTimeField(nullable=True),
        'deleted_at': fields.DateTimeField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class ActionStateUpdatePayload(notificationbase.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'old_state': fields.StringField(nullable=True),
        'state': fields.StringField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class ActionUpdatePayload(ActionPayload):

    VERSION = '1.0'

    fields = {
        'state_update': fields.ObjectField('ActionStateUpdatePayload'),
    }

REST API impact

None.

Security impact

None.

Notifications impact

This blueprint will implement the following notifications:

action.create
action.update
action.delete
action.execution.start
action.execution.end
action.execution.error

Other end user impact

None.

Performance Impact

Other deployer impact

Developer impact

Developers should adhere to proper versioning guidelines and use the notification base classes when creating/updating notifications.

Implementation

Assignee(s)

Primary assignee:: joker946 <a.chadin@servionica.ru>

Work Items

Implement action.create
Implement action.update
Implement action.delete
Implement action.execution.start
Implement action.execution.end
Implement action.execution.error

Dependencies

Testing

These notifications will have to be tested mainly via unit testing.

Documentation Impact

A notification sample should be provided and made dynamically available in the online documentation.

The sequence diagrams in the Watcher architecture.

References

None.

Supporting HA for background jobs in Watcher

Thu, 03 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/background-jobs-ha

Problem description

The main problem with the current implementation of the background scheduling of jobs is if two or more decision-engine services are going to take persisted jobs that are stored in DB then each DE service will take these jobs twice or more.

There are no bindings like ‘job <-> service’ so main goal of this BP is to provide a mechanism that will tag (route) jobs for DEs according to the service instance they have associated with. Another future goal for this feature would be to provide a job requeuing mechanism when the bound ‘service_id’ is marked as failed.

Use Cases

As a Watcher administrator, I want to be sure that all Watcher services using background jobs, which are working in HA-mode, will be synced with scheduled jobs.

Proposed change

APScheduler already has SQLAlchemyJobStore class that create table with the following columns:

id
next_run_time
job_state

Column ‘job_state’ contains dumped object of Job. As we can see, SQLAlchemyJobStore doesn’t have relation between Decision Engine and its jobs. I suggest to add new column ‘service_id’ that will be foreign key for Service.id. This relation should entirely define relation between job and service. There also should be added new column ‘tag’ that will contain dict with keys and values which will identify service uniquely. It usually is to contain ‘host’ and ‘name’ keys. Each type of job should then have its own tag so we can easily perform some triaging/filtering on the list of jobs.

Alternatives

None

Data model impact

New table ‘apscheduler_jobs’ is to be added with the following columns: * id * next_run_time * job_state * service_id * tag

REST API impact

None

Security impact

None

Notifications impact

Since it is internal changes that shouldn’t affect on user experience and common workflow, no new notifications are expected.

Other end user impact

‘apscheduler_jobs’ table should be included in new alembic version for the Watcher DB.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: Alexander Chadin <alexchadin>

Work Items

Inherit new class from SQLAlchemyJobStore that will contain ‘service_id’ column and overwrite appropriate methods.
Update watcher/decision_engine/audit/continuous.py to let it work with new job store.
Implement appropriate unit tests to test various scenarios.

Dependencies

None expected

Testing

Appropriate unit tests will be adapted to new changes.

Documentation Impact

None

References

https://blueprints.launchpad.net/watcher/+spec/background-jobs-ha

History

No history

Integrate storage (cinder) information in the model

Thu, 03 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cinder-model-integration

Problem description

As of now, Watcher only provides compute data model. Watcher needs storage data model for optimizing storage provided by Cinder.

Use Cases

As a developer, I want to develop storage optimization strategy using storage data model.
As as developer, I want to develop zone migration strategy using storage data model.

Proposed change

This spec adds storage data model separate from compute data model. Since Cinder backend has many pools, physical layer composed of backends and it’s pools. Backend information can be retrieved by python-cinderclient Service resource. The model has host, zone, status, and state attributes of Service resource. For using Multi backends feature, volume type is also needed. Volume type information can be retrieved by python-cinderclient VolumeType resource.

Pool information can be retrieved by python-cinderclient Pool resource. The model has name, total_volumes, total_capacity_gb, free_capacity_gb and provisioned_capacity_gb and allocated_capacity_gb attributes of Pool resource. Virtual_fee is also element of the model which is not retrieved by python-cinderclient, but retrieved from cinder notification.

Virtual layer is Cinder volume. Volume information can be retrieved by python-cinderclient Volume resource. The model has id, name, size, status, attachements, snapshot_id, project_id and metadata attributes of Volume resource.

In order to maintain consistency of the model, notification endpoint to consume all needed cinder notifications will be added:

CapacityNotificationEndpoint

This consumes the notification emitted in _publish_service_capabilities method periodically in cinder/manager.py from cinder-scheduler service.

Payload is defined as return value of _usage_from_capacity method in cinder.volume.utils.py.

Filter_rule is publisher_id equals r’capacity.*’ and event_type equals ‘capacity_pool’.

This endpoint updates pool information:

pool element

notification payload

name

name_to_id

total_volumes

None

total_capacity_gb

total

free_capacity_gb

free

provisioned_capacity_gb

provisioned

allocated_capacity_gb

allocated

virtual_free

virtual_free

Total_volumes is not included in notification payload so that it is calculated when updating pool information.

Notifications for updating volume information are emitted in cinder/volume/manager.py from cinder-volume service. Payload is defined as return value of _usage_from_volume method in cinder.volume.utils.py.

VolumeCreateEnd

Filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.create.end’.

This endpoint creates an element of virtual layer under the pool element with the condition that host included in publisher_id equals pool name:

volume element

notification payload

id

volume_id

name

display_name

size

size

status

status

attachements

volume_attachment

snapshot_id

snapshot_id

project_id

tenant_id

metadata

volume_metadata

VolumeDeleteEnd

Filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.delete.end’.

This endpoint deletes the element of virtual layer with the condition that volume_id equals id.

VolumeUpdateEnd

Filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.update.end’.

This endpoint updates the element of virtual layer with the condition that volume_id equals id:

volume element

notification payload

name

display_name

size

size

status

status

attachements

volume_attachment

snapshot_id

snapshot_id

project_id

tenant_id

metadata

volume_metadata

VolumeAttachEnd

Filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.attach.end’.

This endpoint updates the element of virtual layer with the condition that volume_id equals id. The attributes updating are the same as VolumeUpdateEnd.

VolumeDetachEnd

filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.detach.end’.

This endpoint updates the element of virtual layer with the condition that volume_id equals id. The attributes updating are the same as VolumeUpdateEnd.

VolumeResizeEnd

Filter_rule is publisher_id equals r’volume.*’ and event_type equals ‘volume.resize.end’.

This endpoint updates the element of virtual layer with the condition that volume_id equals id. The attributes updating are the same as VolumeUpdateEnd.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

Notification endpoint to consume cinder notifications will be added.

Other end user impact

None

Performance Impact

Watcher stores data model in memory. Watcher-decision-engine may need more physical memory.

Other deployer impact

The following is added in setup.cfg.

[entry_points]
watcher_cluster_data_model_collectors =
    storage = watcher.decision_engine.model.collector.cinder:CinderClusterDataModelCollector

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>
Other contributors:: <None>

Work Items

Add storage data model
Add storage data model entry point in setup.cfg
Add _storage_model instance variable and storage_model method with property decorator to BaseStrategy class
Add notification endpoint for updating storage deta model

Dependencies

None

Testing

Unit tests should be updated.

Documentation Impact

Architecture documentation will be updated.

References

https://blueprints.launchpad.net/watcher/+spec/zone-migration-strategy

History

None

Support gnocchi in watcher

Thu, 03 Aug 2017 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/gnocchi-watcher

Problem description

Today, Watcher uses Telemetry and Monasca to collect metrics from the cluster. We need to support gnocchi as well since ceilometer v2 API is deprecated.

Use Cases

As administrator, I want to run strategies with Gnocchi support as data source for metrics.

Proposed change

We need to add support to data source gnocchi and implement statistic_aggregation method, which will query from gnocchi service for sample data.

Provide granularity[1] as input parameter to strategies.

Alternatives

Monasca

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Gnocchi should boost the overall performances of Watcher as compared to Ceilometer.

Other deployer impact

Gnocchi should be configure as backend for ceilometer.

Developer impact

Strategies developers need to adapt them to use Gnocchi.

Implementation

Assignee(s)

Primary assignee:: <sanfern> santhosh.fernandes@gmail.com <alexchadin> a.chadin@servionica.ru

Work Items

Implement data source for gnocchi.
Update all strategies to use gnocchi datasource as well.
Add gnocchi_client section to watcher.conf file.
Add api_version field under gnocchi_client section.
Enable gnocchi plugin in local.conf.controller for devstack.

Dependencies

python-gnocchiclient needs to be installed.

Testing

Unit tests and tempest tests should be updated.
Update multi node gate job

Documentation Impact

Need to update the gnocchi support in configuration

References

[1]https://docs.openstack.org/developer/gnocchi/glossary.html

History

None

Suspended audit state

Thu, 03 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/suspended-audit-state

Problem description

If an audit with continuous mode needs to be stopped temporarily in order to execute an audit and then restarted, it will have to be deleted and recreated.

Use Cases

As an administrator, I want to suspend an audit with continuous mode to execute audit regularly for maintenance.
As an administrator, I want to resume the audit with continuous mode to execute audit after maintenance.

Proposed change

Currently, scheduler periodically picks up audit of PENDING, ONGOING or SUCCEEDED state. Then audit’s job named execute_audit is added to scheduler. Execute_audit job first checks audit state. If state is CANCELLED, DELETED or FAILED, audit’s job is removed and strategy is not executed.

This spec adds new audit state SUSPENDED and changes execute_audit’s audit state check condition. If state is CANCELLED, DELETED, FAILED or SUSPENDED, audit’s job is removed and strategy is not executed. Otherwise, strategy is executed.

Since SUSPENDED is new audit state, audit state transition check when updating audit state is also changed. ONGOING state can be changed to SUSPENDED. SUSPENDED can be changed to ONGOING in reverse.

Alternatives

Delete the audit and create again.
Currently, ONGOING state can be changed to CANCELLED state, but not be changed in reverse. If we can change from the CANCELLED state to ONGOING state, it is also an alternative.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>

Work Items

Add SUSPENDED in State class of watcher.objects.audit module.
Update ContinuousAuditHandler that SUSPENDED is also inactive state.

Dependencies

None

Testing

Unit tests and tempest tests should be updated.

Documentation Impact

Need to update the Audit State Machine in the architecture documentation.

References

None

History

None

Volume migrate action

Thu, 03 Aug 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/volume-migrate-action

Problem description

As of now, Watcher only provides Nova migrate action. Watcher needs cinder volume migrate action for optimizing storage provided by cinder.

Use Cases

As a developer, I want to develop storage optimization strategy which migrates cinder volume.
As a developer, I want to develop zone migration strategy which migrates cinder volume.

Proposed change

This spec adds storage migrate action.

The schema is like the following:

schema = Schema({
    'resource_id': str,  # should be a UUID
    'migration_type': str,  # choices -> "swap", "cold"
    'destination_node': str,
    'destination_type': str,
})

Resource_id and migration_type are mandatory parameters. Default migration_type is swap.

If migration_type is “cold”, This action verifies that volume is detached and has no snapshots. If false, WatcherException.Invalid exception with the message “Invalid state for cold migration” is raised and the action ends with FAILED state. If true, the action continues.

If destination_node is added, first check if volume type of source node and destination node are the same. If true, this action migrates detached volume of resource_id to destination_node by calling Cinder migration API. If false, WatcherException.Invalid exception with the message “Volume type must be same for migrating” is raised and the action ends with FAILED state.

If destination_type is added, first check if volume type of source node and destination node are different. If true, this action migrates detached volume of resource_id to destination_type by calling Cinder retype API. If false, WatcherException.Invalid exception with the message “Volume type must be different for retyping” is raised and the action ends with FAILED state.

If migration_type is “swap” then destination_type is required. If destination_type has not been provided, WatcherException.Invalid exception with the message “destination type is required when migration type is swap” is raised. Moreover, the action verifies that volume is attached and instance status is active, paused, or resized. If false, WatcherException.Invalid exception with the message “Invalid state for swapping volume” is raised and the action ends with FAILED state. If true, the action swaps attached volume of resource id. This action proceeds the following steps:

create a temporary user in the project which created source volume
create destination volume of destination_type by the temporary user
swap volume from source volume to destination volume by using Nova Update a volume attachment API
delete source volume
delete the temporary user

Temporary user password is determined randomly for each user to reduce security risk as much as possible.

The action execution diagram is below.

if "migration_type" +------------->if "migration_type" +------>ERROR
is 'cold'?              no         is 'swap'?              no
      +                                   +
      | yes                               | yes
      |                                   v
      |                            is "destination_type" +---->ERROR
      |                            given?                  no
      |                                   +
      |                                   |yes
      |                                   v
      v                            is volume attached
is volume detached                 and instance is active
and no snapshots? +----> ERROR     paused, or resized?    +--->ERROR
      +             no                    +                no
      |                                   | yes
      | yes                               v
      |                            5 steps described above
      |
      v
is "destination_node"+------------>is "destination_type" +---->ERROR
given ?                  no        given?                  no
      +                                   +
      | yes                               | yes
      |                                   |
      v                                   v
is volume type                     is volume type
the same?                          different?
      +---------------+                   +------------------+
      |               |                   |                  |
      | yes           | no                | yes              | no
      v               v                   v                  v
call cinder migrate  ERROR         call cinder retype       ERROR

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

If migration_type is swap, temporary user password can be security risk.

Notifications impact

None

Other end user impact

If migration_type is swap, creation of temporary user or destination volume may exceed quota and result in error.

Performance Impact

None

Other deployer impact

Swap migration_type works for compute libvirt driver only.

The following is added in setup.cfg.

[entry_points]
watcher_actions =
    volume_migrate = watcher.applier.actions.storage_migration:VolumeMigrate

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <nakamura-h>

Work Items

Add volume migrate action
Add volume migrate action entry point in setup.cfg

Dependencies

None

Testing

Unit tests should be updated.

Documentation Impact

Architecture documentation will be updated since action driver requests cinder API by this spec.

References

History

None

Host maintenance strategy

Mon, 15 May 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/cluster-maintaining

Problem description

Sometimes we need to maintain compute nodes, update hardware or software, and so on, without interrupting user’s applications.

Use Cases

As an openstack operator, sometimes I want to maintain one compute node without interrupting user’s applications.

Proposed change

There will be a new goal and strategy for cluster-maintenance.

Add one new goal - “Cluster Maintenance”
Add one new strategy for this goal - “Host Maintenance”

The new strategy executes as follows

First, get the compute node which needs maintenance. This input parameter is provided by the administrator. Call change_nova_service_state action to set the maintaining node in “maintaining” state (disabled with disable_reason ‘watcher_maintaining’).
Then, call migrate action to migrate all instances on the maintaining node to other nodes. Migrate active instances use “live-migrate” and others use “cold-migrate”. Calculate free cpus/memory/disk of a node to determine whether one instance or all instances from the maintaining node can migrate to. This strategy just consider how to migrate all instances of the maintaining node, further optimization rely on other strategies. There are two methods to migrate the instances of the maintaining node: Method No.1, migrate all instances on the maintaining node intensively to one unused host.The ‘unused’ host means disable but not power-off node for Watcher. If there are more than one “unused” hosts, choose one from them by random. (This method won’t result in more VMs migration among other hosts.) Method No.2, just migrate all instances on the maintaining node dispersedly to other nodes. Method No.1 is priority. Only if Method No.1 fails, Method No.2 will execute. If both methods fail, this audit fails and raise exception with no solution produced.

After the maintenance finished, the administrator needs to activate the maintaining node by cli ‘nova service-enable’ to change the node’s state from “maintaining” to “enabled” manually, which will make the compute node rejoin into compute resource.

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:sue

Work Items

Add strategy and goal for cluster_maintenance

Update change_nova_service_state action, to make it available to maintain one compute node.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/extend-node-status

Testing

Unit tests

Documentation Impact

A documentation explaining how to use this new optimization strategy.

References

None

History

None

Outlet Temperature Based Strategy

Mon, 20 Feb 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/outlet-temperature-based-strategy

Outlet(Exhaust Air) temperature is a new thermal telemetry which can be used to measure the server’s thermal/workload status.

This spec proposes a new Watcher migration strategy based on the outlet temperature of servers. This strategy makes decisions to migrate workloads to the servers with good thermal condition (lowest outlet temperature) when the outlet temperature of source servers reach a configurable threshold.

Note: “server” in this document means “hypervisor”.

Problem description

In current Data Center infrastructure, the cooling air supply to servers can be different. When a server is overloaded or the supply air is too hot, the outlet temperature telemetry can be used to detect the problem. In order to have the server in a reliable thermal condition, some of the server’s workloads should be migrated to other server with safer thermal conditions.

Use Cases

As an administrator, I want to be able to trigger an audit that controls the temperature and perform workload load balancing.

In order to :

Reduce the total power consumption spent on cooling.
Increase the lifespan of the data center because cooling effectiveness is a first order factor.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Watcher already has its decision framework, so this strategy should be a new class which extend the base strategy class.

Set the threshold in 2 steps : hard coded first, then through the template.
Create a new Python class to extend the “BaseStrategy” class.
Use the Ceilometer client to get Outlet temperature metrics of hypervisors.
Use the Nova objects framework to get free CPU/Memory/Disk of hypervisors.
An algorithm to detect if the threshold of Outlet temperature has been reached and to choose the migration target server. It will filter the viable targets according to the free resource information of hypervisors from previous step.

Alternatives

No alternative

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There used to be some performance issues regarding the query of metrics from the Ceilometer database. This is one of the reason why it was rarely used in production environment. These issues may now be solved thanks to an abstraction layer which enables anybody to change the underlying metrics storage backend easily. There is also a performance issue when you query the Nova DB to get cpu usage metrics.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <junjie-huang>

Work Items

function to use Ceilometer client to get outlet temperature of hypervisors.
function to filter servers by Nova basic metrics(free CPU/Memory/Disk)
Rewrite execute function to add the algorithm to detect if the threshold of outlet T has been reached and choose the target hypervisor, generate action plan.

Dependencies

Testing

Unit tests and functional test, will use a fake metrics set for running functional test.

Documentation Impact

A documentation explaining how to use this new optimization strategy.

References

http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-home.html

History

None

Watcher Decision Engine should query metrics from the Ceilometer V2 API

Mon, 20 Feb 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/telemetry-integration

Watcher Decision Engine must query metrics from the Ceilometer V2 API to allow an easiest integration with OpenStack services (especially DevStack).

Telemetry used to be the initial name of the Ceilometer project. Now, the following projects are managed by the Telemetry team:

Aodh - an alarming service
Ceilometer - a data collection service
Gnocchi - a time-series database and resource indexing service

Therefore, behind the word “Telemetry” we encompass the three projects.

Please, refer to the official page of the Project if you want to know more about it.

Problem description

Today, the watcher Decision Engine is not relying on Ceilometer V2 API to query the metrics necessary for the optimization algorithm.

Use Cases

As a developer. I want to be able, in my optimization strategies, to request metrics collected by Telemetry services. So that I can develop and run audits on an official DevStack release (Kilo and above) just by enabling the Telemetry services.

As an operator with an admin role. I want to be able to launch audits on data-centers where compute and storage resources are managed by any Openstack controller deployed with version Kilo and above. So that I can optimize the resources of the data-center which I am responsible of without the need to install a metering chain which is not in an official release of the core modules.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Estimated changes are going to be in the following places:

Modify the query of metrics on the available strategies (for example, basic_consolidation) to query Ceilometer API V2
May need to develop a helper to simplify some complex queries to the Ceilometer V2 API
Be compliant with the naming of Telemetry measurements

Alternatives

No alternative because Watcher definitely needs to rely on Ceilometer API V2 to make sure it is compatible with any deployed OpenStack cluster.

Data model impact

There is no impact on the Watcher MariaDB database (which stores Watcher objects such as Audit Templates, Audits, Action Plans, …).

But there is an impact on how Watcher will request the metrics in the Decision Engine, especially regarding the measurements names because Watcher used to have its own measurements naming system.

See the full list of available measurements for Telemetry

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

The metrics will now be queried by an optimization strategy using the python Ceilometer client package corresponding to the Ceilometer V2 API. This Ceilometer API will relay the query to an abstract Ceilometer Storage Engine Base Interface (see class defined in ceilometer.storage.base.py).

The Storage Engine may be configured with any available driver for requesting a time-series database implementation (InfluxDB, OpenTSDB, …), providing much better performances than a SQL database. For example, see class ceilosca.ceilometer.storage.impl_monasca.py in the monasca-ceilometer project.

The frequency of the queries will depend on the strategy used to achieve the goal of the audit.

The performances of the query will not only depend on the underlying database implementation but also on how it has been deployed and configured for indexing the metrics:

sharding policy of the TSDB + use of a cluster of nodes to balance the load
tags associated to each metric (host id, instance if, …),
use of pre-defined requests (named “Continuous Queries” in InfluxDB)

Other deployer impact

Telemetry services will have to be installed on the controller nodes and compute nodes of the OpenStack cluster that the admin wants to optimize with Watcher.

The main configuration file watcher.conf will also be impacted, especially the watcher_metrics_collector section.

The V2 of the Ceilometer querying API has been released since the official Kilo release. Therefore, this blueprint should deliver a Watcher version which would be compatible with any deployment using the Kilo release and above.

Developer impact

There may be some change requests to the Telemetry team when some metrics are not available and strongly needed for optimization strategies (Energy, …).

Implementation

Assignee(s)

Primary assignee:: jed56

Work Items

Dependencies

Testing

Unit tests on the Watcher Decision Engine
An admin should be able to launch an Audit with Watcher with a SERVERS_CONSOLIDATION goal on an OpenStack cluster which does not have the Watcher metering chain installed but only the Telemetry services.

Documentation Impact

The documentation explaining howto add new optimization strategies will have to indicate that metrics must be queried using the Ceilometer V2 client API.

References

Telemetry official project page: https://wiki.openstack.org/wiki/Telemetry
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-04.log.html
See Gnocchi Features in https://wiki.openstack.org/wiki/ReleaseNotes/Liberty#OpenStack_Telemetry_.28Ceilometer.29
https://wiki.openstack.org/wiki/Gnocchi
Video presenting Ceilosca=Monasca+Ceilometer : https://www.youtube.com/watch?v=5-IvVwIoCzM
Source code project of Monasca+Ceilometer : https://github.com/openstack/monasca-ceilometer

History

None

Uniform Airflow Migration Strategy

Mon, 20 Feb 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/uniform-airflow-migration-strategy

Airflow (Unit: CFM) is a cooling related telemetry which can be used to measure the cooling status of server.

This spec proposes a new Watcher migration strategy based on the airflow of servers. This strategy makes decisions to migrate VMs to make the airflow uniform.

Note: “server” in this document means “hypervisor”.

Problem description

In current Data Center infrastructure, the cooling air supply to servers can be different , depends on the workload and inlet temperature. When a server is overloaded or the supply air is too hot, the airflow can reach the threshold. In this case, we need to move the VM instance to other servers.

Use Cases

As an administrator, I want to be able to trigger an audit that controls the airflow and perform VM instance load balancing.

In order to:

Reduce the total power consumption spent on cooling.
Increase the lifespan of the data center because cooling effectiveness is a first order factor.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Watcher already has its decision framework, so this strategy should be a new class which extend the base strategy class.

Set the threshold through the template. see: https://blueprints.launchpad.net/watcher/+spec/optimization-threshold
Create a new Python class to extend the “BaseStrategy” class.
Use the Telemetry client to get Airflow, Inlet temperature, System Power metrics of hypervisors.
Use the Nova objects framework to get free CPU/Memory/Disk of hypervisors.
An algorithm to detect if the threshold of Airflow has been reached, by default it uses the average value in 5 minutes to compare with the threshold ,and can be configurable like threshold. Here it needs 3 thresholds: Airflow, Inlet temperature, and System power. When the threshold of Airflow has been reached, it will check:
- Whether the threshold of both Inlet temperature and System power been reached, if so, it will choose the first VM to migrate.
- Whether the inlet temperature and system power both lower than threshold if so, it means there might be someting wrong with the hardware, it will migrate all VMs of the hypervisor.
At the end, it will filter the viable targets according to the free resource information of hypervisors from previous step.

Alternatives

No alternative

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There used to be some performance issues regarding the query of metrics from the Telemetry database. This is one of the reasons why it was rarely used in production environment. These issues may now be solved thanks to an abstraction layer which enables anybody to change the underlying metrics storage backend easily. There is also a performance issue when you query the Nova DB to get CPU usage metrics.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <junjie-huang>

Work Items

function to use Telemetry client to get Airflow,Inlet temperature,System power of hypervisors.
function to filter servers by Nova basic metrics(free CPU/Memory/Disk).
Rewrite execute function to add the algorithm to detect the threshold and to choose the target hypervisors, generate action plans.

Dependencies

Testing

Unit tests and functional test, will use a fake metrics set for running functional test.

Documentation Impact

A documentation explaining how to use this new optimization strategy.

References

http://www.intel.com/content/www/us/en/servers/ipmi/ipmi-home.html

History

None

Versioned notifications for audits

Thu, 26 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/audit-versioned-notifications-api

Problem description

As of now, there is no way for any service (Watcher included) to know when an audit has been created, modified or deleted. This prevents any form of event-based reaction which may be useful for 3rd party services or plugins.

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher about audits.

As an OpenStack developer, I want to know what the format of the audit notifications are.

As an OpenStack developer, I want to be notified whenever:

an audit has been created
an audit has finished

As an OpenStack developer, I would also want to be notified whenever:

the audit starts the execution of its strategy (the strategy is triggered by the decision engine)
the audit finishes the execution of its strategy (the strategy has finished and produced an action plan)

As an OpenStack developer, I would also want to be notified whenever:

The Watcher Planner starts scheduling the solution (produced by the strategy of the audit)
The Watcher Planner finishes the scheduling the solution (produced by the strategy of the audit)

Proposed change

In order to implement the above use cases, many different notifications will be needed:

audit.create whenever an audit has been created.
audit.update whenever an audit has been updated. This includes all state updates including the deletion of the audit.
audit.delete whenever an audit has been deleted (soft).
audit.strategy.start whenever an audit starts.
audit.strategy.end whenever an audit ends.
audit.strategy.error whenever an audit fails.
audit.planner.start whenever planning of the audit solution starts.
audit.planner.end whenever planning of the audit solution ends.
audit.planner.error whenever the planning of an audit solution fails.

Moreover, we will rely on oslo.versionedobjects to version the payloads of audit-related notifications.

Here below is suggestion of notification structure for each one of the aforementioned events:

audit.create

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "PENDING",
      "updated_at": null,
      "deleted_at": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditCreatePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.create",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.update

{
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:51:38.722986   ",
  "payload": {
    "watcher_object.name": "AuditUpdatePayload",
    "watcher_object.data": {
      "strategy": {
        "watcher_object.name": "StrategyPayload",
        "watcher_object.data": {
          "name": "dummy",
          "parameters_spec": {
            "properties": {
              "para2": {
                "default": "hello",
                "type": "string",
                "description": "string parameter example"
              },
              "para1": {
                "maximum": 10.2,
                "default": 3.2,
                "minimum": 1.0,
                "description": "number parameter example",
                "type": "number"
              }
            }
          },
          "updated_at": null,
          "display_name": "Dummy strategy",
          "deleted_at": null,
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "created_at": "2016-11-04T16:25:35Z"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "scope": [],
      "created_at": "2016-11-04T16:51:21Z",
      "uuid": "f1e0d912-afd9-4bf2-91ef-c99cd08cc1ef",
      "goal": {
        "watcher_object.name": "GoalPayload",
        "watcher_object.data": {
          "efficacy_specification": [],
          "updated_at": null,
          "name": "dummy",
          "display_name": "Dummy goal",
          "deleted_at": null,
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "created_at": "2016-11-04T16:25:35Z"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "deleted_at": null,
      "state_update": {
        "watcher_object.name": "AuditStateUpdatePayload",
        "watcher_object.data": {
          "state": "ONGOING",
          "old_state": "PENDING"
        },
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "interval": null,
      "updated_at": null,
      "state": "ONGOING",
      "audit_type": "ONESHOT"
    },
    "watcher_object.namespace": "watcher",
    "watcher_object.version": "1.0"
  },
  "priority": "INFO",
  "event_type": "audit.update",
  "message_id": "697fdf55-7252-4b6c-a2c2-5b9e85f6342c"
}

audit.delete

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "DELETED",
      "updated_at": null,
      "deleted_at": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditDeletePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.delete",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.strategy.start

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.strategy.start",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.strategy.end

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.strategy.end",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.strategy.error

{
  "priority": "ERROR",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": {
        "watcher_object.data": {
          "exception": "WatcherException",
          "exception_message": "TEST",
          "function_name": "test_send_audit_action_with_error",
          "module_name": "watcher.tests.notifications.test_audit_notification"
        },
        "watcher_object.name": "ExceptionPayload",
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.strategy.error",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.planner.start

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.planner.start",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.planner.end

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": null,
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.planner.end",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

audit.planner.error

{
  "priority": "ERROR",
  "payload": {
    "watcher_object.data": {
      "audit_type": "ONESHOT",
      "parameters": {
        "para2": "hello",
        "para1": 3.2
      },
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "fault": {
        "watcher_object.data": {
          "exception": "WatcherException",
          "exception_message": "TEST",
          "function_name": "test_send_audit_action_with_error",
          "module_name": "watcher.tests.notifications.test_audit_notification"
        },
        "watcher_object.name": "ExceptionPayload",
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "goal": {
        "watcher_object.data": {
          "uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "name": "dummy",
          "updated_at": null,
          "deleted_at": null,
          "efficacy_specification": [],
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy goal"
        },
        "watcher_object.name": "GoalPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "interval": null,
      "scope": [],
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "AuditActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673   ",
  "event_type": "audit.planner.error",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

Alternatives

Instead of using versioned objects, we can define the payload of our audit notifications without any support for versioning.

Data model impact

New versioned objects will be created although none of them are to be persisted as they will be used to structure the content of the notifications.

Here are some of the payloads to be declared:

@base.WatcherObjectRegistry.register_notification
class AuditPayload(notificationbase.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'uuid': fields.UUIDField(),
        'audit_type': fields.StringField(),
        'state': fields.StringField(),
        'parameters': fields.FlexibleDictField(nullable=True),
        'interval': fields.IntegerField(nullable=True),
        'goal_uuid': fields.UUIDField(),
        'strategy_uuid': fields.UUIDField(nullable=True),
        'goal': fields.ObjectField('Goal'),
        'strategy': fields.ObjectField('Strategy', nullable=True),
        'created_at': fields.DateTimeField(nullable=True),
        'updated_at': fields.DateTimeField(nullable=True),
        'deleted_at': fields.DateTimeField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class AuditStateUpdatePayload(notificationbase.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'old_state': fields.StringField(nullable=True),
        'state': fields.StringField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class AuditUpdatePayload(AuditPayload):

    VERSION = '1.0'

    fields = {
        'state_update': fields.ObjectField('AuditStateUpdatePayload'),
    }

REST API impact

None.

Security impact

None.

Notifications impact

This blueprint will implement the following notifications:

audit.create
audit.update
audit.delete
audit.strategy.start
audit.strategy.end
audit.strategy.error
audit.planner.start
audit.planner.end
audit.planner.error

Other end user impact

None.

Performance Impact

Other deployer impact

Developer impact

Developers should adhere to proper versioning guidelines and use the notification base classes when creating/updating notifications.

Implementation

Assignee(s)

Primary assignee:: vincent-francoise

Work Items

Implement audit.create
Implement audit.update
Implement audit.delete
Implement audit.strategy.start
Implement audit.strategy.end
Implement audit.strategy.error
Implement audit.planner.start
Implement audit.planner.end
Implement audit.planner.error

Dependencies

Testing

These notifications will have to be tested mainly via unit testing.

Documentation Impact

A notification sample should be provided and made dynamically available in the online documentation.

The sequence diagrams in the Watcher architecture.

References

None.

Watcher automatic triggering audit

Thu, 26 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/automatic-triggering-audit

Problem description

This blueprint goes beyond the “continuously-optimization” blueprint by applying automatically the Action Plans. https://blueprints.launchpad.net/watcher/+spec/continuously-optimization

Watcher needs to continuously optimize the OpenStack cloud for a specific strategies or goals. Watcher can periodically trigger some Audit which generates Action Plan.

Unfortunately, in the current implementation the Administrator have to launch manually the recommended Action Plan in order to apply it on the real system.

This specification relates to blueprint: https://blueprints.launchpad.net/watcher/+spec/automatic-triggering-audit

Use Cases

As administrator, telling audit to trigger the action plan everytime is a a headache to the administator, so here we are implementing automatic triggering functionality in audit which triggers the action plans.

Project Priority

Essential for Ocata-2

Proposed change

The watcher system enables a private/public cloud administrator to launch Audit on an Openstack cluster in order to optimize it in regards of one or several goals.

We need to introduce new attribute in the AUDIT,

auto_trigger - To execute the action plans for the AUDIT.

An Audit is an optimization request but in case of auto_trigger, when system finds auto_trigger=True in the AUDIT, then it triggers the action plan for that audit.

The AutoTriggerActionPlan logic will be implemented under the watcher/decision_engine/audit/base.py.

The AutoTriggerActionPlan class will contain all the logic related to trigger the action plan for the audit. To trigger the action plan from AutoTriggerActionPlan, we will use RPC call launch_action_plan.

At the a Applier level, currently only one action plan will execute. https://github.com/openstack/watcher/blob/master/watcher/applier/manager.py#L30 If new action_plan is proposed while executing current action_plan, we will cancel the new action by proper message “An action plan is currently running” and the action plan state should also be set to SUPERSEDED.

Alternatives

To use a cronjob for audit which automatically triggers the action.

Data model impact

We need to introduce the new attribute in audit auto_trigger.

REST API impact

None expected

Security impact

We could have a security impact as the system could start running action plan continuously and seriously damage the stability of the cluster.

Notifications impact

None expected.

Other end user impact

None expected

Performance Impact

No specific performance impact is expected.

Other deployer impact

No specific deployer impact is envisaged.

Developer impact

This will not impact other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:: Alexander Chadin <alexchadin>
Other contributors:: Digambar Patil <diga> Jean-Emile DARTOIS <jed56>

Work Items

Implement AutoTriggerActionPlanHandler class.
Adapt API to support auto_trigger field as boolean.
Add auto_trigger to audit in python-watcherclient.
Add checkbox for auto_trigger in watcher-dashboard.
Implement appropriate unit tests to test various scenarios.

Dependencies

None expected

Testing

Appropriate unit tests will be adapted to new changes.

Documentation Impact

It will be necessary to add new content relating to this change.

References

https://blueprints.launchpad.net/watcher/+spec/continuously-optimization

History

No history.

Provide a graph model describing virtual & physical elements in a data center

Thu, 26 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/graph-based-cluster-model

To support a rich set of graph analysis methods during strategy determination we want to have a simple graph model. The graph model should describe how VMs are associated to compute hosts, how virtual networks entities are mapped to physical networks devices, objects/block storage objects to physical storage devices. Furthermore this shows how all these entities are interconnected (e.g. showing how two VMs are connected virtually and subsequently through the physical network). This allows for seeing relationships upfront between the entities and hence can be used to identify hot/cold spots in the data center and hence influence a strategy decision.

Problem description

The information that represent the state and topology of a Cluster is derived from different sources. Thus, the relation and association of different pieces of data in order to effectively produce an optimization Strategy is a challenging task. To this end, the Cluster Data Model is a vital source of knowledge that contributes to the decision making. Relational data modeling approaches do not capture efficiently the correlations of resource deployments, network attributes, etc.

In this specification we propose a graph based data model that would aggregate resource information for the Cluster and allow for the effective representation of the topology (landscape) of workload deployments. Through this model graph analysis techniques could be employed in the context of the Strategy definition.

Use Cases

The graph model representation of the virtual and physical resource of a Cluster in Watcher would enhance the Cluster Data Model and allow powerful analysis to be applied during the Strategy definition. For example, a goal strategy developer would use the graph model to decide on a destination hypervisor for a VM, which is not connected to the network of hypervisors supporting the tenant/project VMs. Therefore, the use case that the proposed solution affects the Strategy definition and the development of goals in the context of Watcher. The affected actor in that use case is the Watcher developer who has to suggest the appropriate Strategy for a Goal. He would use the graph model included into the Cluster Data Model in order to acquire information regarding the topology of the VMs, attributes of the physical nodes etc.

Project Priority

Not defined.

Proposed change

The proposed model is a representation of the physical and virtual structure of the resource existing in a data centre. It gives us an overview of where a service is placed in the data centre, the underlying hardware, the network components through which resources communicate, details about the virtual servers and the virtual components that constitute an entire deployment etc.

Retrieval of the graph model will be real-time and the structure will mirror that of the infrastructure topology. The information of the cluster will be gathered by polling the Openstack services (e.g. Nova, Neutron) in a predefined time interval. The implemented service will keep in cache the latest graph in order to minimize the interaction with the Openstack services in continous requests.

Each application stack will be mapped with resources in the virtual layer and each virtual layer resources will be mapped with physical layer nodes. A mesh structure of the service, virtual and physical resource will be captured. Each component will also have attributes attached, for example, CPUs would have an attribute for the chipset family and VMs would have an attribute for the flavor. The nodes in the physical layer will cover the compute, network and storage nodes and will all contain attributes that capture their specification. Similar approach will be followed for the virtual layer with virtual compute, storage and network nodes to capture the deployments on the Cluster.

Below we have presented what we believe is a good level of abstraction and have categorized these components into layers: Physical layer * Machines * Switches * Memory * Disks * CPUs * NICs * Physical Bay * Network link * Router * Storage NAS

Virtual Layer * Virtual Machines * VNICs * VCPUs * Virtual networks * Virtual Storage * Containers * Virtual Bay (used by Magnum)

This is an indicative list of nodes that could populate the graph. The exact type of nodes that will be offered is depended from the platfrom services (e.g. Nova, Neutron, ODL etc.) available in the testbed. Components in the landscape/topology will be represented as nodes and connections to other components will be represented as edges between these nodes. By doing this across all components, a graph outlining the structure of a service will be built, which will visually show how a service in the data centre looks and also allow paths to be easily traced. In addition, each node has relationships with other nodes within the same layer (intra-layer) as well as across the layers (inter-layer). By following such modeling we can capture complex topologies and express various dependencies of the components of the Cluster. Also, heterogenous components can be expressed in each layer and therefore workloads with different requirements at the virtual or physical layers can be captured.

The acquisition of the information regarding the topology of the Data Center resource will be realized through the RESTful APIs exposed by OpenStack such as Nova, Neutron etc.

Additionally using graphs to structure resource information will allow the use of well researched and defined graph algorithms to derive further insights from the graph model, an example of this would be the shortest path algorithm, which could be used to improve performance across a data-center and more immediately a given deployment.

Alternatives

The effectiveness of the proposed solution is based on the graph representation of the underlying resources and infrastructure. Alternative approaches would be the use of simple relational modeling and structures where the relationships of the objects are captures in a separate entity or class. Such approach could be supported with relational database schemes but in large scale Clusters it would result in high complexity and inefficiency.

Data model impact

There is no direct impact of the Watcher Data Model regarding the dependencies of the entities in the Architecture. The graph model will be provided by an independent module upon request and could accompany (be a part of) the Cluster Data Model created by the Watcher Decision Engine.

REST API impact

There is no impact on the core Watcher REST API. The graph cluster model will expose its own APIs for retrieving the graph object.

Security impact

There is no impact on the Security.

Notifications impact

There is no impact on the notifications.

Other end user impact

There is no other impact.

Performance Impact

The generation of the graph model will be performed in realtime and upon request. There is no significant performance impact estimated. In case the on-demand graph creation is slow, the task will be triggered on the background upon Watcher start-up and/or event realization.

Other deployer impact

No specific deployer impact is envisaged.

Developer impact

It will not impact other developers working on OpenStack.

Implementation

Assignee(s)

Intel is leading this work. Main assignee: Kevin Mullery <kmullery> Secondary assignees: Gregory Katsaros <gregory-katsaros>, Thijs Metsch <tmetsch>

Work Items

Workplan:

Finalization of the conceptual methodology
Definition of information to be captured and service APIs.
Implementation of the graph model service:
- Implementation of the graph model server from building the NetworkX model.
- Implementation of the RESTful endpoints and APIs for graph extraction (json) and model acquisition.
Integration and testing.

Dependencies

The cluster-model-objects-wrapper is a potential dependency and the efforts on these two BP should be aligned.

Testing

Several unit tests will be provided to test scenarios using a mock-up cluster models.

Testing approaches comprising of unit tests and integration tests in which a specific input is given and compared against the expected output.

Documentation Impact

It will be necessary to add new content relating to the Cluster Data Model operation of the Watcher Decision Engine.

References

This work is related with research activities in the context of the CloudWave FP7 EU project (www.cloudwave-fp7.eu). See also Apex Lake (http://dl.acm.org/citation.cfm?id=2830016) for further information on the modeling concept. The CloudSim (https://github.com/Cloudslab/cloudsim) and SimGrid (http://simgrid.gforge.inria.fr/) can be cited as related infrastructure models.

History

No history.

Make Applier parallel

Thu, 26 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/parallel-applier

Problem description

Watcher Applier starts Action plan with set of Actions, which are executed in a sequence. After planner-storage-action-plan is implemented, we have new parents field that contains list of action’s uuid child action is connected to. Having parents field Watcher can build Directed Acyclic Graph where each independent Action can be run in parallel while actions with some dependencies should fulfill them to be executed.

Use Cases

As an administrator, I would like to be able to run Action plan that will launch actions for every node (compute, network, storage) in parallel.

Proposed change

We can launch several concurrent actions in accordance with their parents.

Workflow engine of Watcher Applier will still use watcher_flow flow, but edges between nodes should be established in accordance with action parents. Unlinked actions with no parents should be connected directly to the watcher_flow and executed firstly.

Main class to modify is DefaultWorkFlowEngine, that defines workflow of actions executing. Since Watcher Default Planner adds actions to DB in accordance with weights (i.e. all migrate actions will be added firstly, then resize actions and so on), Applier will get ordered by weight actions that can be added to watcher_flow flow and linked in one for loop. In this case algorithm of building directed acyclic graph should have O(n) complexity.

There is estimated pseudocode to show part of default planner’s workflow:

flow = gf.Flow("watcher_flow")
for a in actions:
    task = TaskFlowActionContainer(a, self)
    flow.add(task)
    if a.parents:
        for parent_id in action.parents:
            parent_action = get_action(actions, parent_id)
            flow.link(parent_action, action, decider=self.decider)

e = engines.load(flow)
e.run()

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

Since it is perfomance related patch, it is expected to have perfomance improvements because of parallel executing of actions instead of sequence. In order to reduce the impact on system performance and stability, it is better to set the maximum number of concurrent.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Alexander Chadin <a.chadin@servionica.ru>

Work Items

Modify watcher/applier/workflow_engine/default.py to build new parallel workflow engine.
Update the documentation.
Add appropriate unit tests.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/planner-storage-action-plan

Testing

Unit tests will be added to validate these modifications.

Documentation Impact

Update the Applier documentation in accordance with new changes.

References

History

None

Make Watcher objects versioned

Thu, 26 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/watcher-versioned-objects

As we are making Watcher more stable, we now have to tackle the question of retro-compatibility. Other OpenStack projects already faced this problem and the community came up with a common answer which is object versioning.

Below is a non-exhaustive list of blueprint specifications written on the subject by other projects:

https://specs.openstack.org/openstack/oslo-specs/specs/kilo/adopt-oslo-versionedobjects.html

https://specs.openstack.org/openstack/heat-specs/specs/kilo/versioned-objects.html

Problem description

As of now, Watcher maintains some (but not all) building blocks previously used by other projects such as Nova in order to enable the versioning of objects. As Watcher is now part of the Big Tent, we should start taking into account backwards compatibility. Moreover, we should standardize the code and therefore make use of oslo.versionedobjects.

Use Cases

Whenever existing Watcher objects are created or modified or deleted, the concerned Watcher objects should see their version bumped whilst indicating the changes that occurred in order to ease retro-compatibility handling throughout the codebase.

Proposed change

The focus of this blueprint is to rewrite the Watcher codebase implements its objects to leverage oslo.versionedobjects library to enable the versioning of Watcher objects.

Besides, oslo.versionedobjects brings along many new field types, among which ObjectField can be found. This field can reveal itself useful in subsequent work about building versioned notifications as we will be most certainly keen on notifying a more complete set of information that would also include some data that are directly related to the given Watcher object. This work will hence be tackled within this blueprint.

Alternatives

Keep the current codebase and enhance it to support some form of versioning
Do not support any form of object versioning

Data model impact

Add DB relationships on existing models wherever foreign key fields where already defined:

The Strategy will have one ORM relationship field:
- A goal field reflecting the goal_id ForeignKey field
The AuditTemplate will have two ORM relationship fields:
- A goal field reflecting the goal_id ForeignKey field
- A strategy field reflecting the strategy_id ForeignKey field
The Audit will have two ORM relationship fields:
- A goal field reflecting the goal_id ForeignKey field
- A strategy field reflecting the strategy_id ForeignKey field
The ActionPlan will have two ORM relationship fields:
- A audit field reflecting the audit_id ForeignKey field
- A strategy field reflecting the strategy_id ForeignKey field
The Action will have one ORM relationship field:
- An action_plan field reflecting the action_plan_id ForeignKey field
The EfficacyIndicator will have one ORM relationship field:
- An action_plan field reflecting the action_plan_id ForeignKey field

Moreover, unused DB fields will be removed in order to avoid non retro-compatible changes as much as possible.

REST API impact

The API design will not be impacted by the eager flag. Indeed, API endpoints will not return any eagerly-loaded data.

Security impact

None.

Notifications impact

This work will help building versioned notifications.

Other end user impact

None.

Performance Impact

Loading ObjectField data makes queries slower. This is the reason why the concept of eager loading is introduced alongside this blueprint to limit the associated performance hit to areas where this eager loading is necessary. Also note that this eager loading will not be cascaded which means that eagerly loaded related objects will not be subject to eager loading themselves.

Other deployer impact

None.

Developer impact

As objects are being versioned from now on, any modification of an existing Watcher object should be directly translated into a version bump. Retro-compatible changes should see the minor version of the related object bumped whereas any disrupting changes should be translated by a transition to a new major version.

Implementation

Assignee(s)

Primary assignee:: vincent-francoise

Work Items

Here below are the changes to be made:

Enable database loading of ORM relationships in the case of One-to-One and One-to-Many foreign keys.
Add the possibility to either eager load or not the newly added ORM relationships for performance purposes.
Overhaul the code in watcher/objects/base.py to now use oslo.versionedobjects.
Update all Watcher objects which had foreign key fields to add ObjectField fields that will have to be filled whenever an eager loading of the underlying DB model is requested.

Dependencies

None.

Testing

Apart from updating existing unit test suites, this blueprint does not bring any new end-user functionality. Hence, the current set of Tempest test should remain unchanged and still pass upon completion.

Documentation Impact

The DB schema shown in the architecture will have to be updated.

References

None.

History

None.

Improve the storage of the ActionPlan

Wed, 25 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/planner-storage-action-plan

Problem description

Action plan is built by the default planner of the decision-engine service. Every Action is linked to the next one so we have linked list of Actions as result of default planner’s work. Applier service executes the action plan. Launching actions in a sequence has negative influence on performance, especially if there are a lot of actions. Every action must be completed before next action will be run, so it will take a lot of time to launch migration actions in sequence.

Use Cases

As an administrator, I would like to be able to run an action plan that will launch actions for every node (compute, network, storage) in parallel.

Proposed change

We can launch several concurrent actions with the same action weight in parallel.

This specification defines two different planners with the same input/output dataflow: weight planner and workload stabilization planner. Each planner should use new parents attribute that will store list of actions the action is linked to, but there are different ways to fill up this attribute.

Weight planner is designed to get unified way to parallelize execution of actions. It knows nothing about structure of specified actions and how they are related to each other. The main goal is to build sets of actions ordered by their weights. High weight actions will be planned before the low weight ones. Parallelization is the availability to execute several actions, with the same action type, in parallel. Parallelization factor is limited by another taskflow parameter named max_worker. Administrator can specify each action type’s weight and its parallelization ability in the watcher configuration file. Let’s show example of acyclic directed graph for this planner:

  ---sleep2    ---migration1     ---resize1
 /         \  /             \   /          \
I----sleep1-------migration2------resize2----resize3----E
              \             /
               ---migration3

This graph is built with the following settings:

	Weight	Parallelization
Sleep	70	2
Migration	60	3
Resize	50	2

Since there are three resize actions and only two are allowed to be executed at the same time, the third resize action will be executed only when the previous actions are done.

Weight planner is easy to configure and has to be considered as new default planner.

Workload Stabilization planner respects features of action types by allowing to affect on building of parents. Some action types like resizing the instance should be run only before/after migration of the same instance, not in parallel. There are to be specified some constraints:

Unlinked actions are to be performed in parallel.
Only one action can be performed per instance at one time
If there are more than one action per instance, linked actions must be executed in sequence with respect to action weights.

So we can add new attribute parents that will store list of actions the current action is linked to. It can be shown using the acyclic directed graph:

    ----------------a1(t=dis)-----------
   /                                    \
  |                                      \
  | ----------------a2(t=mig)-------------\
  |/                                       \
I-----a3(t=mig) -- a31(t=res)---------------E
  |\                                       /
  | ----------a4(t=resize)----------------/
  |                                      /
   \                                    /
    a5(resource=123, -- a51(resource=123,
       t=mig)               t=resize)

Here we can see the following links:

a1 action disables the compute node and is not linked with another action. It can be run in parallel with other action (there is no constraint).
a2 action migrates the instance and is not linked with another action. It can be run in parallel with other action (there is no constraint).
a3 action migrates the instance and is parent to a31 action. It can be run in parallel with other action.
a31 action resizes an instance after this last one had been migrated by action a3.
a4 action resizes the instance and is not linked with another action. It can be run in parallel with other action (there is no constraint).
a5 action migrates the instance and is parent to a51 action. It can be run in parallel with other action.
a51 action resizes the same instance after this last one had been migrated by action a5.

As we can see, all actions that are independent to each other can be performed in parallel. Meanwhile, if some actions are linked to the same resource then they are to be performed with respecting to action weights. Currently, we can define the following weights:

End of graph - 0
Disable Compute Node - 1
Resize instance - 2
Migrate instance - 3
Initial point - 4

There is estimated pseudocode to show part of workload stabilization planner’s workflow:

action = [uuid, type, resource_id, metadata]

init = Flow()

action_weights = {
    'turn_host_to_acpi_s3_state': 0,
    'resize': 1,
    'migrate': 2,
    'sleep': 3,
    'change_nova_service_state': 4,
    'nop': 5,
}

actions = sorted_by_weights(descended)

for action in actions:
    a_type = action['action_type']
    if a_type != 'turn_host_to_acpi_s3_state':
        db_action = self._create_action(context, action)
        plugin_action = self.load_child_class(
            db_action.action_type)
        parents = plugin_action.validate_parents(
            resource_action_map, action)
        if parents:
            db_action.parents = parents
            db_action.save()
    else:
        # if we have an action that will make host unreachable, we need to
          complete all actions (resize and migration type) related to the
          host.
        parent_actions = get_actions(metadata=action[metadata][host])
        resize_actions = [x for x in parent_actions if x[type] == resize]
        migration_actions = [x for x in parent_actions if x[type] == mig]
        resize_migration_parents = [x[parents] for x in resize_actions]
        # Since resize actions have less weight than migration, they may
          have migration actions as parents and must be connected to the
          turn_host_to_acpi_s3_state action firstly.
        action_parents = []
        action_parents.extend([x[uuid] for x in resize_actions])
        # Add migrations that aren't linked to resize type actions
        action_parents.extend([x[uuid] for x in migration_actions
                              if [x[uuid]] not in resize_migration_parents)
        db_action = create_action(action, parents=action_parents)

This spec is limited to simple chained list of actions as action plan. The second part of modifying action plan’s executing will contain graph for parallel executing of action plans.

Alternatives

None

Data model impact

next column should be removed from the Action table.
parents column should be added to the Action table. Type: JSON.
first_action_id column should be removed from Action Plan table.
ActionPlan object major version should probably be updated to 2.0

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

New configuration parameters in watcher.conf:

[watcher_planner] planner = weight #planner = workload_stabilization

[watcher_planners.weight] #weights = turn_host_to_acpi_s3_state:10,resize:20,migrate:30,sleep:40, change_nova_service_state:50,nop:60 #parallelization = turn_host_to_acpi_s3_state:2,resize:2,migrate:2,sleep:1, change_nova_service_state:1,nop:1

[watcher_planners.workload_stabilization] #weights = turn_host_to_acpi_s3_state:10,resize:20,migrate:30,sleep:40, change_nova_service_state:50,nop:60

Performance Impact

None

Other deployer impact

We will have 2 new planner extensions. We should reinstall properly watcher by running pip install [-e].

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Alexander Chadin <a.chadin@servionica.ru>

Other contributors: Vincent Francoise <Vincent.FRANCOISE@b-com.com

Work Items

Update data model in accordance with proposed changes (in fact API and objects).
Remove default planner.
Add watcher/decision_engine/planner/weight.py and watcher/decision_engine/planner/workload_stabilization.py
Make weight planner as default.
Update the documentation.
Add appropriate unit tests.

Dependencies

https://blueprints.launchpad.net/watcher/+spec/plugins-parameters

Testing

Unit tests will be added to validate these modifications.

Documentation Impact

Update the defaultplanner documentation in accordance with new changes.

References

History

None

Versioned notifications for action plans

Mon, 16 Jan 2017 00:00:00

https://blueprints.launchpad.net/watcher/+spec/action-plan-versioned-notifications-api

Problem description

As of now, there is no way for any service (Watcher included) to know when an action plan has been created, modified or deleted. This prevents any form of event-based reaction which may be useful for 3rd party services or plugins.

Use Cases

As an OpenStack developer, I want to be able to listen to notifications coming from Watcher about action plans.

As an OpenStack developer, I want to know what the format of the action plan notifications are.

As an OpenStack developer, I want to be notified whenever:

an action plan has been created, updated or deleted
an action plan has finished

As an OpenStack developer, I would also want to be notified whenever:

the action plan starts the execution of its actions (the action plan is triggered by the Applier)
the action plan finishes its execution
the action plan fails with error

Proposed change

In order to implement the above use cases, many different notifications will be needed:

actionplan.create whenever an action plan has been created.
actionplan.update whenever an action plan has been updated. This includes all state updates including the deletion of the action plan.
actionplan.delete whenever an action plan has been deleted (soft).
actionplan.execution.start whenever an action plan starts.
actionplan.execution.end whenever an action plan ends.
actionplan.execution.error whenever an action plan fails.

Moreover, we will rely on oslo.versionedobjects to version the payloads of action plan-related notifications.

Here below is suggestion of notification structure for each one of the aforementioned events:

actionplan.create

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
      "state": "RECOMMENDED",
      "global_efficacy": {
          "description": "Global efficacy",
          "name": "test_global_efficacy",
          "unit": "%",
          "value": 95
      },
      "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
      "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
      "updated_at": null,
      "deleted_at": null,
      "created_at": "2016-11-04T16:29:20Z",
      "audit": {
        "watcher_object.data": {
          "audit_type": "ONESHOT",
          "parameters": {
            "para2": "hello",
            "para1": 3.2
          },
          "state": "SUCCEEDED",
          "updated_at": null,
          "deleted_at": null,
          "fault": null,
          "interval": null,
          "scope": [],
          "created_at": "2016-11-04T16:29:20Z",
          "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
        },
        "watcher_object.name": "AuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      }
    },
    "watcher_object.name": "ActionPlanCreatePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673",
  "event_type": "actionplan.create",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

actionplan.update

  {
    "publisher_id": "infra-optim:localhost",
    "timestamp": "2016-11-04 16:51:38.722986",
    "payload": {
      "watcher_object.name": "ActionPlanUpdatePayload",
      "watcher_object.data": {
        "global_efficacy": {
            "description": "Global efficacy",
            "name": "test_global_efficacy",
            "unit": "%",
            "value": 95
        },
        "strategy": {
          "watcher_object.name": "StrategyPayload",
          "watcher_object.data": {
            "name": "dummy",
            "parameters_spec": {
              "properties": {
                "para2": {
                  "default": "hello",
                  "type": "string",
                  "description": "string parameter example"
                },
                "para1": {
                  "maximum": 10.2,
                  "default": 3.2,
                  "minimum": 1.0,
                  "description": "number parameter example",
                  "type": "number"
                }
              }
            },
            "updated_at": null,
            "display_name": "Dummy strategy",
            "deleted_at": null,
            "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
            "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
            "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
            "created_at": "2016-11-04T16:25:35Z"
          },
          "watcher_object.namespace": "watcher",
          "watcher_object.version": "1.0"
        },
        "audit": {
          "watcher_object.data": {
            "audit_type": "ONESHOT",
            "parameters": {
              "para2": "hello",
              "para1": 3.2
            },
            "state": "SUCCEEDED",
            "updated_at": null,
            "deleted_at": null,
            "fault": null,
            "interval": null,
            "scope": [],
            "created_at": "2016-11-04T16:29:20Z",
            "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
            "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
            "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          },
          "watcher_object.name": "AuditPayload",
          "watcher_object.version": "1.0",
          "watcher_object.namespace": "watcher"
        },
        "created_at": "2016-11-04T16:51:21Z",
        "uuid": "f1e0d912-afd9-4bf2-91ef-c99cd08cc1ef",
        "parameters": {
          "para2": "hello",
          "para1": 3.2
        },
        "deleted_at": null,
        "state_update": {
          "watcher_object.name": "ActionPlanStateUpdatePayload",
          "watcher_object.data": {
            "state": "ONGOING",
            "old_state": "PENDING"
          },
          "watcher_object.namespace": "watcher",
          "watcher_object.version": "1.0"
        },
        "state": "ONGOING",
        "priority": "INFO",
        "event_type": "actionplan.update",
        "message_id": "697fdf55-7252-4b6c-a2c2-5b9e85f6342c"
      }
   }
}

actionplan.delete

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
      "state": "DELETED",
      "global_efficacy": {
          "description": "Global efficacy",
          "name": "test_global_efficacy",
          "unit": "%",
          "value": 95
      },
      "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
      "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
      "updated_at": null,
      "deleted_at": null,
      "created_at": "2016-11-04T16:29:20Z",
      "audit": {
        "watcher_object.data": {
          "audit_type": "ONESHOT",
          "parameters": {
            "para2": "hello",
            "para1": 3.2
          },
          "state": "SUCCEEDED",
          "updated_at": null,
          "deleted_at": null,
          "fault": null,
          "interval": null,
          "scope": [],
          "created_at": "2016-11-04T16:29:20Z",
          "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
        },
        "watcher_object.name": "AuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      }
    },
    "watcher_object.name": "ActionPlanDeletePayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673",
  "event_type": "actionplan.delete",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

actionplan.execution.start

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
      "state": "PENDING",
      "global_efficacy": {
          "description": "Global efficacy",
          "name": "test_global_efficacy",
          "unit": "%",
          "value": 95
      },
      "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
      "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
      "updated_at": null,
      "deleted_at": null,
      "created_at": "2016-11-04T16:29:20Z",
      "audit": {
        "watcher_object.data": {
          "audit_type": "ONESHOT",
          "parameters": {
            "para2": "hello",
            "para1": 3.2
          },
          "state": "SUCCEEDED",
          "updated_at": null,
          "deleted_at": null,
          "fault": null,
          "interval": null,
          "scope": [],
          "created_at": "2016-11-04T16:29:20Z",
          "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
        },
        "watcher_object.name": "AuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      }
    },
    "watcher_object.name": "ActionPlanActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673",
  "event_type": "actionplan.execution.start",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

actionplan.execution.end

{
  "priority": "INFO",
  "payload": {
    "watcher_object.data": {
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
      "state": "SUCCEEDED",
      "global_efficacy": {
          "description": "Global efficacy",
          "name": "test_global_efficacy",
          "unit": "%",
          "value": 95
      },
      "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
      "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
      "updated_at": null,
      "deleted_at": null,
      "created_at": "2016-11-04T16:29:20Z",
      "audit": {
        "watcher_object.data": {
          "audit_type": "ONESHOT",
          "parameters": {
            "para2": "hello",
            "para1": 3.2
          },
          "state": "SUCCEEDED",
          "updated_at": null,
          "deleted_at": null,
          "fault": null,
          "interval": null,
          "scope": [],
          "created_at": "2016-11-04T16:29:20Z",
          "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39"
        },
        "watcher_object.name": "AuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      }
    },
    "watcher_object.name": "ActionPlanActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673",
  "event_type": "actionplan.execution.end",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

actionplan.execution.error

{
  "priority": "ERROR",
  "payload": {
    "watcher_object.data": {
      "state": "ONGOING",
      "updated_at": null,
      "deleted_at": null,
      "audit_uuid": "55bc1fe8-8030-4bd0-b0d0-2e62937f02a0",
      "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a3",
      "global_efficacy": {
          "description": "Global efficacy",
          "name": "test_global_efficacy",
          "unit": "%",
          "value": 95
      },
      "fault": {
        "watcher_object.data": {
          "exception": "WatcherException",
          "exception_message": "TEST",
          "function_name": "test_send_action_plan_action_with_error",
          "module_name": "watcher.tests.notifications.test_action_plan_notification"
        },
        "watcher_object.name": "ExceptionPayload",
        "watcher_object.namespace": "watcher",
        "watcher_object.version": "1.0"
      },
      "audit": {
        "watcher_object.data": {
          "audit_type": "ONESHOT",
          "parameters": {
            "para2": "hello",
            "para1": 3.2
          },
          "state": "SUCCEEDED",
          "updated_at": null,
          "deleted_at": null,
          "fault": null,
          "interval": null,
          "scope": [],
          "created_at": "2016-11-04T16:29:20Z",
          "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6",
          "goal_uuid": "bc830f84-8ae3-4fc6-8bc6-e3dd15e8b49a",
          "strategy_uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39"
        },
        "watcher_object.name": "AuditPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "strategy": {
        "watcher_object.data": {
          "parameters_spec": {
            "properties": {
              "para2": {
                "type": "string",
                "default": "hello",
                "description": "string parameter example"
              },
              "para1": {
                "description": "number parameter example",
                "maximum": 10.2,
                "type": "number",
                "default": 3.2,
                "minimum": 1.0
              }
            }
          },
          "name": "dummy",
          "uuid": "75234dfe-87e3-4f11-a0e0-3c3305d86a39",
          "updated_at": null,
          "deleted_at": null,
          "created_at": "2016-11-04T16:25:35Z",
          "display_name": "Dummy strategy"
        },
        "watcher_object.name": "StrategyPayload",
        "watcher_object.version": "1.0",
        "watcher_object.namespace": "watcher"
      },
      "created_at": "2016-11-04T16:29:20Z",
      "uuid": "4a97b9dd-2023-43dc-b713-815bdd94d4d6"
    },
    "watcher_object.name": "ActionPlanActionPayload",
    "watcher_object.version": "1.0",
    "watcher_object.namespace": "watcher"
  },
  "publisher_id": "infra-optim:localhost",
  "timestamp": "2016-11-04 16:31:36.264673",
  "event_type": "actionplan.execution.error",
  "message_id": "cbcf9f2c-7c53-4b4d-91ec-db49cca024b6"
}

Alternatives

Instead of using versioned objects, we can define the payload of our action plan notifications without any support for versioning.

Data model impact

New versioned objects will be created although none of them are to be persisted as they will be used to structure the content of the notifications.

Here are some of the payloads to be declared:

@base.WatcherObjectRegistry.register_notification
class ActionPlanPayload(base.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'uuid': fields.UUIDField(),
        'state': fields.StringField(),
        'global_efficacy': fields.FlexibleDictField(nullable=True),
        'audit_uuid': fields.UUIDField(),
        'audit': fields.ObjectField('Audit'),
        'strategy_uuid': fields.UUIDField(nullable=True),
        'strategy': fields.ObjectField('Strategy', nullable=True),
        'created_at': fields.DateTimeField(nullable=True),
        'updated_at': fields.DateTimeField(nullable=True),
        'deleted_at': fields.DateTimeField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class ActionPlanStateUpdatePayload(base.NotificationPayloadBase):

    VERSION = '1.0'

    fields = {
        'old_state': fields.StringField(nullable=True),
        'state': fields.StringField(nullable=True),
    }


@base.WatcherObjectRegistry.register_notification
class ActionPlanUpdatePayload(ActionPlanPayload):

    VERSION = '1.0'

    fields = {
        'state_update': fields.ObjectField('ActionPlanStateUpdatePayload'),
    }

REST API impact

None.

Security impact

None.

Notifications impact

This blueprint will implement the following notifications:

actionplan.create
actionplan.update
actionplan.delete
actionplan.execution.start
actionplan.execution.end
actionplan.execution.error

Other end user impact

None.

Performance Impact

Other deployer impact

Developer impact

Developers should adhere to proper versioning guidelines and use the notification base classes when creating/updating notifications.

Implementation

Assignee(s)

Primary assignee:: vincent-francoise

Work Items

Implement actionplan.create
Implement actionplan.update
Implement actionplan.delete
Implement actionplan.execution.start
Implement actionplan.execution.end
Implement actionplan.execution.error

Dependencies

Testing

These notifications will have to be tested mainly via unit testing.

Documentation Impact

A notification sample should be provided and made dynamically available in the online documentation.

The sequence diagrams in the Watcher architecture.

References

None.

Make default Planner generic

Mon, 03 Oct 2016 00:00:00

https://blueprints.launchpad.net/watcher/+spec/configurable-weights-default-planner

Problem description

Watcher provides a very modular architecture that currently allows anyone to provide custom plugins:

Strategy

Planner

Action

Workflow Engine

In most of the above, their related implementation(s) are generic enough so that they can work with any 3rd party plugin out of the box. However, this is currently not the case for the defaultplanner which defines a hardcoded set of priorities for each action type.

Hence, this means that adding a new Action via an action plugin and reference it in a strategy would result in an error and the only solution is to create a planner plugin that would adapt to this new action.

This is not very user friendly for the Watcher users because this means that every time someone adds a new action, the source code of the planner will have to be amended.

Use Cases

As an administrator, I would like to be able to add a new Action without having to amend the source code of the defaultplanner.

Project Priority

Medium

Proposed change

This specification suggests modifying the DefaultPlanner class watcher/decision_engine/planner/default.py in order to avoid having to implement a new planner whenever adding a new action plugin to the mix. A simple solution to this would be to leverage the plugins-parameters blueprint so that we can configure weights via the configuration file.

Currently in watcher, the weights are defined as below:

priorities = {
        'nop': 0,
        'sleep': 1,
        'change_nova_service_state': 2,
        'migrate': 3
}

We have to replace this by the code below :

self.weights_dict = {'nop': 0,
                'sleep': 1,
                'change_nova_service_state': 2,
                'migrate':3}
def get_config_opts(self):
return [
cfg.DictOpt('weights', help="These weights are used to schedule
    the Actions.",
    default=self.weights_dict),
    ]

Alternatives

None

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

The user will have to modify the configurable file and reboot the decision engine.

Performance Impact

None

Other deployer impact

We are adding a new field in the configuration file. So, we should revisit the puppet scripts.

Developer impact

None

Implementation

Assignee(s)

Primary assignee: Jinquan Ni <ni.jinquan@zte.com.cn>

Work Items

Replace the storage of the weights in the default planner
Update the documentation
add unit tests

Dependencies

None

Testing

Unit tests will be added to validate these modifications.

Documentation Impact

Update the defaultplanner documentations to now mention that the weights of the planner can be specified in the configuration file.

References

History

None

Ocata Project Priorities

Mon, 19 Sep 2016 00:00:00

List of priorities the Watcher drivers team is prioritizing in Ocata.

Priority	Owner
Define Audit Scope	Alexander Chadin
Graph Model	Kevin Mullery
Watcher Versionned objects	Vincent Francoise
Automatic triggering Action Plans	Digambar Patil
Limit Concurrent Actions	Joe Cropper
Workload Characterization Grammar	Prashanth Hari
Workload Characterization and QoS	Prashanth Hari
Update Notifications	Vincent Francoise
Notifications for audits	Vincent Francoise
Description For Dynamic Action	Charlotte Han
Stale Action Plan	Li Canwei
Audit tag in VM Metadata	Prashanth Hari

Define Audit Scope

Administrator will be able to provide a logical subset of resources where Watcher will run the optimization. This subset will be based on Host aggregates and Availability Zones.

Graph Model

The graph based data model would aggregate resource information for the Cluster and allow for the effective representation of the topology of workload deployments. Through this model graph analysis techniques could be employed in the context of the Strategy definition.

Watcher Versionned objects

Make all Watcher objects support oslo.versionnedobjects.

Automatic triggering Action Plans

Administrator will be able to choose to run the Action Plan automaticaly right after Watcher has run the Audit.

Limit Concurrent Actions

We propose a capability that limits the number of concurrent Action that can be in flight when a particular Audit is invoked.

Workload Characterization Grammar

We should be able to characterize such workloads as input to Watcher for ensuring Application QoS, placements and consolidation. We need to define a grammar to describe those workloads.

Workload Characterization and QoS

Based on the grammar defined earlier, we should be able to describe workload characteristics and map them to applications. Then it would allow Watcher to build optimization Strategy based on those characteristics.

Update Notifications

We should update Watcher notifications system to be compliant with Nova Notifications.

Notifications for audits

Now that all Watcher object are versionned, we need to update the Audit object logic to handle notifications.

Description For Dynamic Action

We should add the ability to submit a description of each Action that can append in an Action Plan.

Stale Action Plan

We should add a new state for Action Plan to be able to set it as superseded in case the Cluster Data Model has changed or an event happened that invalidate all previous Action Plan.

Audit tag in VM Metadata

As watcher runs audits to achieve a Goal, there should be some way for the application/VM owners to know that their VMs are under Audit and its flagged before Action Plan execution.

Watcher Continuous Optimization

Mon, 29 Aug 2016 00:00:00

https://blueprints.launchpad.net/watcher/+spec/continuously-optimization

Problem description

Cluster can be optimized by different Strategies only when they have been triggered by Administrator. Launching a recommended Action Plan manually is not always suitable since state of cluster is constantly changing. It would be better to have two ways of launching audit: either by triggering it manually or by launching it periodically. We propose to include continuous optimization as continuous type of audit object in Watcher Project.

The main purpose of this change is to design and implement active mode of Watcher’s audit.

This specification relates to blueprint: https://blueprints.launchpad.net/watcher/+spec/continuously-optimization

Use Cases

As an administrator, I would like to create a periodic audit to be able to optimize continuously my cloud infrastructure. I can specify a period with –period parameter (in seconds) to launch an audit every 600 seconds.

As an administrator, I would like to be able to remove a continuous audit.

As an administrator, I would like to be able to update the period of an audit.

Project Priority

Essential for Newton-2

Proposed change

The watcher system enables a private cloud administrator to launch Audit on an Openstack cluster in order to optimize it in regards of one or several goals. An Audit is an optimization request.

There are two types of audits :

ONESHOT : the audit will only be executed once
CONTINUOUS : the audit will be executed regularly with a given frequency.

We propose to use the APScheduler library to schedule the continuous audits. Note: This library is already in the Openstack global requirements.

APScheduler provides several scheduler implementations to schedule jobs with a specific interval. The scheduler which seems to match well our requirements is the BackgroundScheduler. APScheduler provides an example: BackgroundScheduler.

The DecisionEngineManager (watcher/decision_engine/manager.py) class will need to be amended in order to instantiate the new ContinuousAuditManager class.

The ContinuousAuditManager class will contain the BackgroundScheduler but also the logic for managing the continuous audits. We should also create a ContinuousAuditJob class in charge of supervising one Audit. This class will contain the APScheduler job and its associated audit.

We can easily add new audits or remove old ones on the fly with BackgroundScheduler. So, the existing continuous audits should be automatically added by the decision_engine during start.

Then, the ContinuousAuditManager will manage the audits in an even driven fashion. In order to do that, we should then modify the ‘post’, ‘patch’ and ‘delete’ methods in the API source file for sending immediate notification messages. The notifications generated by Watcher are generated in JSON format, and placed on an AMQP queue named watcher.status. This parameter must be configurable.

The ContinuousAuditManager will consume these events in order to update the status of the audits.

Immediate Notification Examples

{
"event_type": "audit.create",
 "timestamp": "2016-03-12 17:01:29.899834",
 "message_id": "1234653e-ce46-4a82-979f-a9286cac5258",
 "priority": "INFO",
 "publisher_id": "<service name >:<the host where the service runs>",
 "payload": {
    "watcher_object.namespace":"watcher",
    "watcher_object.name":"Audit",
    "watcher_object.version":"1.0",
    "watcher_object.data":{
    "audit_uuid": "840eeb3e-3486-11e6-ac61-9e71128cae77",
    "type": "CONTINUOUS",
    "state": "PENDING",
    "period": 3600
   }
 }
}

{
"event_type": "audit.update",
 "timestamp": "2016-03-12 17:01:29.899834",
 "message_id": "1234653e-ce46-4a82-979f-a9286cac5258",
 "priority": "INFO",
 "publisher_id": "<service name >:<the host where the service runs>",
 "payload": {
      "watcher_object.namespace":"watcher",
         "watcher_object.name":"Audit",
    "watcher_object.version":"1.0",
    "watcher_object.data":{
    "audit_uuid": "840eeb3e-3486-11e6-ac61-9e71128cae77",
    "type": "CONTINUOUS",
    "state": "ONGOING",
    "period": 3600
   }
 }
}

{
"event_type": "audit.delete",
 "timestamp": "2016-03-12 17:01:29.899834",
 "message_id": "1234653e-ce46-4a82-979f-a9286cac5258",
 "priority": "INFO",
 "publisher_id": "<service name >:<the host where the service runs>",
 "payload": {
      "watcher_object.namespace":"watcher",
         "watcher_object.name":"Audit",
    "watcher_object.version":"1.0",
    "watcher_object.data":{
    "audit_uuid": "840eeb3e-3486-11e6-ac61-9e71128cae77",
    "type": "CONTINUOUS",
    "state": "SUCCEEDED",
    "period": 3600
   }
 }
}

The notification logic isn’t yet available in Watcher. We will work on this with the watcher-notifications-ovo blueprint. So, for the first implementation of this spec, we will manage the audits by querying periodically in the watcher database in order to update running audits and their periods.

APScheduler give also the possibility to store your jobs in a database. In this way, the jobs will survive decision engine restarts and maintain their state. This feature is interesting, but for the first implementation of the continuous Audit we will use the memory backend.

To keep track of the triggered audit, notification has to be pushed on the message bus every time the audit is re-triggered. When a new action plan is proposed, Watcher should cancel all the previously generated action plans (and actions) with same Audit Template become obsolete and therefore their state should be changed to CANCELLED.

Alternatives

To use Congress to automatically trigger audits when some conditions are met.
To use a cronjob which triggers new audit regularly via python-watcherclient.

Data model impact

There must be new field in Audit model: integer ‘period’. ‘period’ field has 3600 by default.

REST API impact

period’s field has to be added as Audit attribute.

Security impact

None expected.

Notifications impact

None expected.

Other end user impact

Support for ‘period’ field must be added to the python-watcherclient and to the watcher-dashboard.

Performance Impact

No specific performance impact is expected.

Other deployer impact

No specific deployer impact is envisaged.

Developer impact

This will not impact other developers working on OpenStack.

Implementation

Assignee(s)

Primary assignee:: Alexander Chadin <alexchadin>
Other contributors:: Vladimir Ostroverkhov <Ostroverkhov> Jean-Emile DARTOIS <jed56>

Work Items

Part 1

Implement ContinuousAuditManager that use APScheduler.
Implement ContinuousAuditJob class.
Implement the logic to add new audits or remove old ones on the fly with BackgroundScheduler by periodically query the watcher db. Audit.list()
Adapt API to support period field.
Make some changes to python-watcherclient to add support for period argument.
Add changes to watcher-dashboard to support CONTINUOUS type.
Implement appropriate unit tests to test various scenarios.

Part 2

We need to wait that watcher-notifications-ovo is implemented for this part.
Load the registered audits in the watcher database during decision engine start.
Implement the logic to add new audits or remove old ones on the fly with BackgroundScheduler by subscripting to the events.

Dependencies

There is a dependency with watcher-notifications-ovo blueprint.

Testing

Appropriate unit tests will be adapted to new changes.

Documentation Impact

It will be necessary to add new content relating to this change.

References

No references.

History

No history.

Optimization Threshold

Mon, 29 Aug 2016 00:00:00

https://blueprints.launchpad.net/watcher/+spec/optimization-threshold

Allow decision engine to pass strategy parameters, like optimization threshold, to selected strategy, also strategy to provide parameters info to end user.

Problem description

Currently all parameters used in strategy are hard coded, end users can’t modify them unless they change the source code. E.g. THRESHOLD of strategy outlet_temp_control is critical and fixed as 35.0, so we can’t tune it for specific usage case without changing source code.

Use Cases

As an administrator. I can choose a goal to optimize my cluster and Watcher will show me a list of custom parameters for the selected strategy.

As an operator. I want to tune the strategy parameter in an easy way for my cloud. I wish I can change it via watcher command line to achieve the best solution.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Need following to enable strategy parameter:

Add more info into audit-template, so that ‘watcher audit-template-show’ tells end user selected strategy and its parameters. Also append these info to the ‘watcher audit-template-create’ output.
Add new cmd option ‘-p’ to ‘watcher audit-create’ for strategy parameter configuration. End user can specify all strategy parameters, like “-p threshold=45.0 -p repeat=10”.
When ‘watcher audit-create’, decison engine would configure selected strategy with these parameters before executing it.

Alternatives

None

Data model impact

For audit template, need add info of selected strategy and its parameters. For audit, need store user specified strategy parameters by hashmap.

The audit template structure is as follows:

+----------------+-------------------------------------------+
| Property       | Value                                     |
+----------------+-------------------------------------------+
| goal           | DUMMY                                     |
| strategy       | outlet_temp_control                       |
| parameter_info | {'threshold': (float, 'Temperature ...')} |
| description    | None                                      |
| name           | 1th_template                              |
| host_aggregate | None                                      |
| uuid           | 0c49a52c-4a0f-4dea-9e19-471360823aef      |
+----------------+-------------------------------------------+

The audit structure is as follows:

+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| uuid                | b564665b-e77b-444c-a43f-3ae654b9a551 |
| audit_template_name | 1st_template                         |
| parameters          | {'threshold': 35.0}                  |
| created_at          | 2016-01-31T08:51:03+00:00            |
| updated_at          | 2016-01-31T08:51:58+00:00            |
| audit_template_uuid | 06cdba2a-069b-4618-8948-540854d8d911 |
| state               | SUCCEEDED                            |
| deadline            | None                                 |
| deleted_at          | None                                 |
| type                | ONESHOT                              |
+---------------------+--------------------------------------+

REST API impact

Audit creation API: POST /v1/audit is changed, so that extra strategy parameters hashmap is accepted in request body.

E.g. to create one audit based on outlet_temperature template with required parameters, we can just POST following:

{
    "type": "ONESHOT",
    "audit_template_uuid": "outlet_temperature_template",
    "parameters": {
        "threshold": 35.0,
        "repeat": 10
    }
}

Security impact

Strategy parameter controls the executing process, and odd parameter probably leads unexpected behavior. This BP enables easy changing of parameter, thus need check against input parameter to make sure it is in reasonable range.

Notifications impact

None

Other end user impact

python-watcherclient need add extra command line for parameter input. End user needs to be aware of this for strategy parameter tuning.

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: edwin-zhai

Work Items

Refactor the code to move strategy selection from audit creation to audit-template creation. As we have no strategy parameter info until instantiate it.
Extend the audit-template to show the info of selected strategy and related parameters
Add extra command line in python-watcherclient, so that strategy parameter can be specified with ‘watcher audit-create -p xxx=XXX -p yyy=YYY …’
When ‘watcher audit-create’, parameter should be used to configure the selected strategy after validation, where voluptuous can be used just like action parameter checking.
Update Horizon plugin for watcher to make sure it still works with these changes.
Each strategy class with parameter need define a new function, ‘input_parameters’, to set all parameters, e.g. threshold.
Each strategy calss with parameter need define a new function, ‘parameters_info’, to return parameter info, like ‘name’, ‘type’ and ‘description’.

Dependencies

None

Testing

Adds unit and functional test for strategy parameter.

Documentation Impact

Need update the watcher API doc and user guide. In dev doc about strategy implementation, need explain how to add input parameters for strategy.

References

None

History

None

Audit parameters should be persistent in Db

Mon, 29 Aug 2016 00:00:00

https://blueprints.launchpad.net/watcher/+spec/persistent-audit-parameters

Problem description

When you start an audit, a new Audit object is created in Db with a reference to an Audit Template object.

When processing an audit request, parameters taken into account by the Watcher decision engine come from Audit object itself and also from Audit Template ones (goal name, …).

Problems:

If we update the Audit Template parameters or if we remove the Audit Template, we lose important information about the Audit itself.

What about data integrity for soft-deleted audit objects (useful for history) ?

If we introduce parameters for goals (as thresholds, …) , we should be able to update them. So how to keep persistent data for all audits done to satisfy such configurable goals ?

How do we handle periodic audits when an audit template is updated ?

If we create an Audit Template with no strategy but with a Goal there is a strategy selector that will automatically pick the most suited strategy for the goal we specified (watcher-strategy-selector) and since the Audit has no record of which strategy was picked, we lose the information stating the actual name of the strategy and the global efficacy is calculated upon. Also, there can be a different strategy for an Audit simply because the selection criteria changed over time (Ref. blueprint efficacy indicator and watcher-strategy-selector). How do we persist strategy for an audit for these cases ?

Other Audit impacting scenarios handled in different blueprints -

1. Impact on Audit Template and Audit when a Goal is deleted - Addressed in blueprint soft-delete-goals

2. Impact on Audit Template and Audit when a Strategy is deleted - Addressed in blueprint get-goal-from-strategy

Use Cases

As an Administrator I need to ensure data integrity and consistency of an Audit when an Audit Template is updated or removed.

As an Administrator I should be able to purge an`Audit Template`_ without impacting Audit

Project Priority

Essential for Newton-2

Proposed change

Modify audit schema to include strategy_id, goal_id and host_aggregate

Alternatives

Audit Template to be immutable if there is any active Audit referencing it. Any modifications to Audit Template should create a new record, so the existing audits are not impacted.

Data model impact

When an audit template is updated, an audit object should not lose related data. We need to store goal_id and strategy_id in audit object. This ensures data integrity for optimization-threshold parameters and strategy selections (Ref. blueprint efficacy indicator and watcher-strategy-selector)

Current audit structure:

+-------------------+-------------+------+-----+---------+----------------+
| Field             | Type        | Null | Key | Default | Extra          |
+-------------------+-------------+------+-----+---------+----------------+
| created_at        | datetime    | YES  |     | NULL    |                |
| updated_at        | datetime    | YES  |     | NULL    |                |
| deleted_at        | datetime    | YES  |     | NULL    |                |
| deleted           | int(11)     | YES  |     | NULL    |                |
| id                | int(11)     | NO   | PRI | NULL    | auto_increment |
| uuid              | varchar(36) | YES  | UNI | NULL    |                |
| type              | varchar(20) | YES  |     | NULL    |                |
| state             | varchar(20) | YES  |     | NULL    |                |
| deadline          | datetime    | YES  |     | NULL    |                |
| audit_template_id | int(11)     | NO   | MUL | NULL    |                |
+-------------------+-------------+------+-----+---------+----------------+

+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| deadline            | None                                 |
| type                | ONESHOT                              |
| uuid                | ab49ee15-fa2b-4d3d-962f-01fe28b7fd92 |
| audit_template_uuid | 7f1e48ba-2639-498e-8fff-9508cf706665 |
+---------------------+--------------------------------------+

Proposed audit structure:

+-------------------+-------------+------+-----+---------+----------------+
| Field             | Type        | Null | Key | Default | Extra          |
+-------------------+-------------+------+-----+---------+----------------+
| created_at        | datetime    | YES  |     | NULL    |                |
| updated_at        | datetime    | YES  |     | NULL    |                |
| deleted_at        | datetime    | YES  |     | NULL    |                |
| deleted           | int(11)     | YES  |     | NULL    |                |
| id                | int(11)     | NO   | PRI | NULL    | auto_increment |
| uuid              | varchar(36) | YES  | UNI | NULL    |                |
| type              | varchar(20) | YES  |     | NULL    |                |
| state             | varchar(20) | YES  |     | NULL    |                |
| deadline          | datetime    | YES  |     | NULL    |                |
| goal_id           | int(11)     | NO   | MUL | NULL    |                |
| strategy_id       | int(11)     | YES  | MUL | NULL    |                |
| host_aggregate    | int(11)     | YES  |     | NULL    |                |
+-------------------+-------------+------+-----+---------+----------------+

+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| deadline            | None                                 |
| type                | ONESHOT                              |
| uuid                | ab49ee15-fa2b-4d3d-962f-01fe28b7fd92 |
| goal_uuid           | None                                 |
| strategy_uuid       | None                                 |
+---------------------+--------------------------------------+

REST API impact

Impacts following audit REST -

GET /v1/audits
GET /v1/audits/(audit_uuid)
POST /v1/audits
PATCH /v1/audits
GET /v1/audits/detail

Security impact

None

Notifications impact

None

Other end user impact

Impacts openstack watcher command line:

The python-watcherclient audit-create should have options for strategy-uuid, goal-uuid and host_aggregate while creating an audit

The python-watcherclient audit-update should include options for updating strategy-uuid, goal-uuid and host_aggregate for an audit

Performance Impact

None

Other deployer impact

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

Other Contributors: hvprash michaelgugino

Work Items

Update Audit object in /watcher/db/sqlalchemy/models.py to include strategy_id, goal_id and host_aggregate
Synchronize audit_template and audit. Refactor the code to update strategy_id, goal_id and host_aggregate in both audit template and audit during CRUD operations. Following python files will be impacted: - /watcher/objects/audit_template.py - /watcher/objects/audit.py - /watcher/tests/api/v1/test_audit_templates.py - /watcher/tests/api/v1/test_audits.py - /watcher/tests/db/test_audit.py - /watcher/tests/db/test_audit_template.py
Additional validation while soft deleting an audit template and purging. De-reference any active audits refering to an audit template: - /watcher/objects/audit_template.py - /watcher/objects/audit.py - /watcher/db/purge.py
Update unit tests and integration tests (Tempest scenarios)
Changes to database migration script

Dependencies

None

Testing

Documentation Impact

References

IRC discussions:

History

None

Select destinations filter

Wed, 10 Aug 2016 00:00:00

Include the URL of your launchpad blueprint:

https://blueprints.launchpad.net/watcher/+spec/select-destinations-filter

Watcher aims at providing an open solution for auditing any pool of resources and optimizing it through recommended Actions.

This blueprint aims at dealing with the ability to find the best possible target host in case of a live migration as a Watcher action. Today the action of filtering hosts is done inside the strategy but should be done as a common function for all our strategies.

This work will allow a code factorization and an easier maintenance if the Nova method select_destinations(self, context, spec_obj) evolves.

Problem description

Today the filtering of targeted hosts in case of a migration of VMs is done in the strategy but this function should be available as a standalone function in Watcher to allow any strategy developper to rely on only one filtering function. This function will be maintained by the Watcher team and deeply related to the select_destinations(self, context, spec_obj) method in Nova.

Use Cases

This is a pure refactoring effort for cleaning up all the code and move it from the basic consolidation strategy to a generic select_destinations(self, hypervisors) method.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Move the code of the method check_migration(self, model, src_hypervisor, dest_hypervisor, vm_to_mig)) in watcher/decision_engine/strategy/strategies/basic_consolidation.py outside of the strategy implementation to be able to use it from any strategy available in Watcher.

Rename this method as select_destinations(self, hypervisors) and update it to handle a list of possible compute nodes as input that will be filtered according to the available capacity of each compute node.

Provide a convenient way to add new implementations of select_destinations(self, hypervisors).

Alternatives

One alternative is to call directly the Nova select_destinations(self, context, spec_obj) method via RPC but after discussions with the Nova team over IRC [1], it is not a viable solution due to performance issues on Nova side.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

Strategy developers will have to rely on this new method to handle the filtering of targeted hosts in case of migration as an action.

Implementation

Assignee(s)

Primary assignee:: <launchpad-id or None>
Other contributors:: <launchpad-id or None>

Work Items

Move the code of the method score_of_nodes(self, current_model, score) in watcher/decision_engine/strategy/strategies/basic_consolidation.py outside the strategy implementation to be able to use it from any strategy available in Watcher.
Rename the method as select_destinations(self, hypervisors).
Provide a stevedore extension to load dynamically the implementation of select_destinations(self, hypervisors).
Refactor existing strategies to use select_destinations(self, hypervisors) method.
Make it available to every strategy developer.

Dependencies

None

Testing

Existing unit tests should continue to pass after moving the method outside of the basic consolidation strategy.
We should also load the select_destinations(self, hypervisors) from another strategy and verify that hosts filtering is still good.
Unit tests are needed for dynamic loading of select_destinations(self, hypervisors) implementation using stevedore.
Unit tests must also be provided to execute select_destinations(self, hypervisors)

Documentation Impact

Build a new optimization strategy page of the generated documentation must be updated to explain how to use the select_destinations(self, hypervisors) method when implementing a new strategy.

References

History

None

Newton Project Priorities

Tue, 12 Jul 2016 00:00:00

List of priorities the Watcher drivers team is prioritizing in Newton.

Priority	Owner
Cluster Object Wrapper	Vincent Francoise
Persistent Audit Param	Prashanth Hari
Watcher Policies	Charlotte Han
Nova Policies	Jean-Emile Dartois
Define Audit Scope	Alexander Chadin
Scoring module	Tomasz Kaczynski
Auto Triggering Audit	Digambar
Dynamic Action Desc	Charlotte Han
Planner Storage Action	Jinquan Ni
Notifications	Vincent Francoise
Uniform Airflow Strategy	Junjie Huang
Overload Strategy	Alexander Chadin
Plugins parameters	Vincent Francoise
Get Goal from Strategy	Vincent Francoise
Efficacy Indicator	Vincent Francoise
Default Planner Generic	Jinquan Ni
Continuous Optimization	Alexander Chadin

Cluster Object Wrapper

Watcher Decision Engine must provide a way to get cluster objects that can be used directly within a Strategy . These objects will be refreshed periodically.

Persistent Audit Param

Watcher must ensure that the Audit parameters are persistent in Db.

Watcher Policies

Admin can use policies to allow or not users to invoke Watcher API methods.

Nova Policies

Watcher Strategy must conform with the Nova polices.

Define Audit Scope

Watcher python clients must validate resources instead of Heat.

Scoring module

Watcher scoring module is a generic machine learning service. It will provide for example predictions or classifications that can be used by the selected Strategy.

Auto Triggering Audit

Watcher will carry out ‘Action Plans’_ automatically during Continous Optimization.

Dynamic Action Desc

‘Watcher Decision Engine’_ must support Strategy with new customized actions.

Planner Storage Action

Watcher must store ‘Action Plans’_ as a directed graph or Directed Acyclic Graph in its database. The ‘Watcher Applier’_ can then run each independent Action in the graph in parallel when its dependencies have been satisfied.

Notifications

Watcher must be compliant with the new notification logic (versionned payload and one topic).

Action Plan Conso

Watcher must support a consolidation mechanism which combine all ‘Action Plans’_ generated from an Audit into a single flow of Actions to be executed.

Uniform Airflow Strategy

This Strategy trigger migration of VMs based on the airflow of servers. This strategy makes decisions to migrate VMs to make the airflow uniform.

Overload Strategy

This ‘Strategy’_ chooses the pair VM:dest_host that minimizes the standard deviation in a cluster best.

Plugins parameters

Watcher must give the possibility to the developer to add some parameters depending on the configuration of OpenStack cluster on the plugins

Get Goal from Strategy

Achieved goal should be returned by each Strategy

Efficacy Indicator

Provide efficacy indicators

Default Planner Generic

An admin needs to be able to add a new Action without having to amend the source code of the default planner.

Continuous Optimization

Watcher Audit must support an active mode

Workload Based Migration Strategy

Tue, 19 Apr 2016 00:00:00

https://blueprints.launchpad.net/watcher/+spec/workload-balance-migration-strategy

This spec proposes a new Watcher migration strategy based on the VM workloads of hypervisors. This strategy makes decisions to migrate workloads to make the total VM workloads of each hypervisor balanced, when the total VM workloads of hypervisor reaches threshold.

Note: * VM workloads means how much CPUs the VM instance fully used, eg a VM

instance has 4 CPUs, the VM workload range is [0.00 , 4.00].

This strategy is based on that all hosts have the same CPUs.

Problem description

In current Data Center, VM workloads on each hypervisor may not balance, some are extremely high, some are idle, which will reduce the cooling efficiency, this strategy will balance the workloads when the CPU utilization of hypervisor reaches threshold.

Use Cases

As an administrator, I want to be able to trigger an audit that controls the CPU utilization below a certain threshold.

In order to:

balance the workloads on each hypervisor, make it close to average workload value of all hypervisors.

Project Priority

Not relevant because Watcher is not in the big tent so far.

Proposed change

Watcher already has its decision framework, so this strategy should be a new class which extend the base strategy class.

Set the threshold in 2 steps: hard coded first, then through the template. see: https://blueprints.launchpad.net/watcher/+spec/optimization-threshold Threshold is the trigger value to start workload balancing. It can be the percentage of CPU utilization of hypervisor.
Create a new Python class to extend the “BaseStrategy” class.
Use the Nova objects framework to get free CPU/Memory/Disk of hypervisors.
Use the Ceilometer client to get VM “cpu_util” to calculate the workloads. The time window is 5 minutes here, will be configurable as threshold. It uses ceilometer aggregation API to get the average value of “cpu_util”.
An algorithm to detect if the threshold of workloads has been reached, it will figure out the suitable VM to be moved, and it will filter the viable targets according to the free resource information of hypervisors from previous step and choose the one with lowest workloads. It will use select-destinations-filter when it is ready. It will also do some check to avoid the corner case, such as “ping pong”.

Alternatives

No alternative

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

There used to be some performance issues regarding the query of metrics from the Ceilometer database. This is one of the reasons why it was rarely used in production environment. These issues may now be solved thanks to an abstraction layer which enables anybody to change the underlying metrics storage backend easily.

There is a performance issue when you query the Nova DB to get CPU usage metrics.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:: <edwin-zhai>

Work Items

function to calculate the total VM workloads of hypervisors.
function to filter hypervisors by Nova basic metrics(free CPU/Memory/Disk).
Rewrite execute function to add the algorithm to detect the threshold and to pick up the suitable VM to be moved and choose the target hypervisors, generate the solution.

Dependencies

Testing

Unit tests and functional test, will use a fake metrics set for running functional test.

Documentation Impact

A documentation explaining how to use this new optimization strategy.

References

None

History

None

Mitaka Project Priorities

Wed, 10 Feb 2016 00:00:00

List of priorities the Watcher drivers team is prioritizing in Mitaka.

Priority	Owner
DevStack integration	Taylor Peoples
Ceilometer integration	Jean-Emile Dartois
Dynamic strategy load	Jean-Emile Dartois
Tasflow integration	Jean-Emile Dartois
Tempest integration	Vincent Francoise
Scoring module	Tomasz Kaczynski
Cluster objects wrapper	Taylor Peoples
Threshold optimization	Edwin Zhai
Watcher dashboard	David Tardivel

DevStack integration

Watcher needs to be DevStack friendly such that it is easily approachable for folks looking to use and/or contribute to Watcher.

Ceilometer integration

Watcher must rely on Ceilometer to get metrics from the underlying infrastructure.

Dynamic strategy load

Watcher Decision Engine must be able to load different optimization Strategy .

Tasflow integration

Watcher Applier must use taskflow as it will help us a lot to make Action Plan execution easy, consistent, scalable and reliable.

Tempest integration

Watcher must provide a set of integration tests that will be used by Tempest and run as a voting job in the gate.

Scoring module

Watcher scoring module is a generic machine learning service. It will provide for example predictions or classifications that can be used by the selected Strategy.

Cluster objects wrapper

Watcher Decision Engine must provide a way to get cluster objects that can be used directly within a Strategy . These objects will be refreshed periodically.

Threshold optimization

Watcher must allow the admin to pass Strategy parameters, like an optimization threshold, to a selected Strategy .

Watcher dashboard

Watcher must provide a human interface to interact through the OpenStack Horizon dashboard.

pool element	notification payload
name	name_to_id
total_volumes	None
total_capacity_gb	total
free_capacity_gb	free
provisioned_capacity_gb	provisioned
allocated_capacity_gb	allocated
virtual_free	virtual_free

volume element	notification payload
id	volume_id
name	display_name
size	size
status	status
attachements	volume_attachment
snapshot_id	snapshot_id
project_id	tenant_id
metadata	volume_metadata