Push Notifications for Agents¶

Changes to RPC callback mechanism¶

The current issue with the RPC callback mechanism and sending objects as notifications is a lack of server operation ordering guarantees and AMQP message ordering guarantees.

To illustrate the first issue, examine the following order of events that happen when two servers update the same port:

• Server 1 commits update to DB
• Server 2 commits update to DB
• Server 2 sends notification
• Server 1 sends notification

If the agent receives the notifications in the order in which they are delivered to AMQP, it will think the state delivered by Server 1 is the current state when it is actually the state committed by Server 2.

We also have the same issue when oslo messaging doesn’t guarantee message order (e.g. ZeroMQ). Even if Server 1 sends immediately after its commit and before Server 2 commits and sends, one or more of the agents could end up seeing Server 2’s message before Server 1’s.

To handle this, we will add a revision number, implemented as a monotonic counter, to each object. This counter will be incremented on any update so any agent can immediately identify stale messages.

To address deletes arriving before updates, agents will be expected to keep a set of the UUIDs that have been deleted. Upon receiving an update, the agent will check this set for the object’s UUID and ignore the update since deletes are permanent and UUIDs cannot be re-used. If we do make IDs recyclable in the future, this can be replaced with a strategy to confirm ID existence with the server or we can add another internal UUID that cannot be specified.

Note that this doesn’t guaruntee message ordering for the agent because that is a property of the message backend, but it does give the agent the necessary info to re-order messages when it receives them so they can determine which one reflects the more recent state of the DB.

Data Model Impact¶

A ‘revision_number’ column will be added to the standard attr table. This column will just be a simple big integer used as monotonic counter that will be updated whenever the object is updated on the neutron server. This revision number can then be used by the agents to automatically discard any object states that are older than the state they already have.

This revision_number will use the version counter feature which is built-in to SQLAlchemy: http://docs.sqlalchemy.org/en/latest/orm/versioning.html Each time an object is updated, the server will perform a compare-and-swap operation based on the revision number. This ensures that each update must start with the current revision number or it will fail with a StaleDataError. The API layer can catch this error with the current DB retry mechanism and start over with the latest revision number.

While SQLAlchemy will automatically bump the revision for us when the record for an object is updated (e.g. a standard attr description field), it will not update it if it’s a related object changing (e.g. adding an IP address to the port or changing its status). So we will have to manually trigger the revision bump (either via a PRECOMMIT callback or inline code) for any operations that we want to trigger the revision number bump.

What this guarantees:

• An object in a notification is newer (from a DB state perspective) than an object with a lower revision number. So any objects with lower revision numbers can safely be ignored since they represent stale DB state.

What this doesn’t guarantee:

• Message ordering ‘on the wire’. An AMQP listener may end up receiving an older state than a message it has already received. It’s up to the listener to look at the revision number to determine if the message is stale.
• That each intermediary state is transmitted. If a notification mechanism reads the DB to get the full object to send, the DB state may have progressed so it will notify with the latest state than the state that triggered the original notification. This is acceptable for all of our use cases since we only care about the current state of the object to wire up the dataplane. It is also effectively what we have now since the DB state could change between when the agent gets a notification and when it actually asks the server for details.
• Reliability of the notifications themselves. This doesn’t address the issue we currently have where a dropped notification is not detected.

Notifications Impact¶

Making existing notifications significantly more data-rich. The hope here is to eliminate many of the expensive RPC calls that each agent makes and have each agent derive all state from notifications with one sync method for recovery/initialization that we can focus on optimizing.

This will result in more data being sent up front by the server to the messaging layer, but it will eliminate the data that would be sent in response to a call request from the agent in the current pattern. For a single agent, the only gain is the elimination of the notification and call messages; but for multiple agents interested in the same resource, it eliminates extra DB calls and extra messages from the server to fulfill those calls.

This pattern will result in fewer messages sent to oslo messaging because of the elimination of the calls from the agents that would result in the same payload we are preemptively broadcasting once instead of casting multiple times to each requesting agent.

Performance Impact¶

Higher ratio of neutron agents per server afforded by a large reduction in sporadic queries by the agents.

This comes at a cost of effectively serializing operations on an individual object due to the compare and swap operation on the server. For example, if two server threads try to update a single object concurrently and both read the current state of the object at the same time, one will fail on commit with a StaleDataError which will be retried by the API layer. Previously both of these would succeed because the UPDATE statement would have no compare-and-swap WHERE criteria. However, this is a very reasonable performance cost to pay considering that concurrent updates to the same API object are not common.

Other Deployer Impact¶

N/A - upgrade path will maintain normal N-1 backward compatibility on the server so all of the current RPC endpoints will be left untouched for one cycle.

Developer Impact¶

Need to change development guidelines to avoid the implementation of new direct server calls.

The notifications will have to send out oslo versioned objects since notifications don’t have RPC versions. So at a minimum we need to switch to oslo versioned objects in the notification code if we can’t get them fully implemented everywhere else. To do this we can leverage the RPC callbacks mechanism.

Alternatives¶

Maintain the current information retrieval pattern and just adjust the timeout mechanism for everything to include back-offs or use cast/cast instead of calls. This will allow a system to automatically recover from self-induced death by stampede, but it will not make the performance any more predictable.

Assignee(s)¶

Primary assignee:

kevinbenton Ihar Hrachyshka

Work Items¶

• Exponential back-off for timeouts on agents
• Implement ‘revision’ extension to add the revision_number column to the data-model and expose it as a standard attribute.
• Write tests to ensure revisions are incremented as expected
• Write (at least one) test that verifies a StaleDataError is triggered in the event of concurrent updates.
• Update DHCP agent to make use of this new ‘revision’ field to discard stale updates. This will be used as the proof of concept for this approach since the DHCP agent is currently exposed to operating on stale data with out of order messages.
• Replace the use of sync_routers calls on the L3 agents for the most frequent operations (e.g. floating IP associations, etc) with RPC callbacks once the OVO work allows it.
• Stand up grenade partial job to make sure agents using different OVO versions maintain N-1 compatibility
• Update devref for callbacks

Possible Future Work¶

• Switch to cast/cast pattern so agent isn’t blocked waiting on server
• Setup a periodic system based on these revision numbers to have the agents figure out if they have lost updates from the server. (e.g. periodic broadcasts of revision numbers and UUIDs, sums of collections of revisions, etc.).
• Add an ‘RPC pain multiplier’ option that just causes all calls to the neutron server to be duplicated X number of times. That way we can set it to something like 200 for the gate which will force us to make every call reasonably performant.
• Allow the HTTP API to perform compare and swap updates by placing an if-match header with the revision number, which would cause the update to fail if the version changed.

User Documentation¶

N/A

Developer Documentation¶

Devref guidelines on the pattern for getting information to agents and what the acceptability criteria are for calls to the server.

RPC callbacks devref will need to be updated with notification strategy.