This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode

Bulk zone update throttling¶

https://blueprints.launchpad.net/designate/+spec/notify-throttling

Implement a mechanism to throttle the delivery of NOTIFY transactions when a large number of zones are updated at the same time.

Problem description¶

If a large number of zones are updated in a short time this will generate a consequently large amount NOTIFY transaction to be sent to the nameservers with no delay leading to a burst of incoming AXFR requests. This might impact on bottlenecks in MiniDNS and the storage layer in terms of CPU, I/O or network bandwidth.

A typical trigger is the update of an NS record in a Pool containing many zones.

The autonomous refreshing of zones performed by resolvers can also trigger a similar burst of AXFR. This can happen on recently started resolvers, where the refresh timers can share the same values across many zones.

Proposed change¶

Implement a mechanism for enqueuing and delayed delivery of notify transactions at a configurable throttle speed.

Also, implement staggering of zone refresh requests by randomizing the refresh interval.

API Changes¶

Expose the count of zones flagged for delayed notify in the Admin API as “/reports/counts/zones_pending_notify”.

Central Changes¶

Implement support for a new database column “pending_notify” and set it to True every time a Pool NS record is updated.

Storage Changes¶

Add an new boolean database column “pending_notify” on Zones. Implement a migration script to add the column to existing databases, defaulting to False. In future, the column might default to True.

Other Changes¶

Implement a Task in Zone Manager to periodically fetch a set of zones that need to receive a Notify starting with the oldest in term of last update time. The task frequency and the maximum set size can be configured to throttle the amount of outgoing Notify. Zone Manager will reset the “pending_notify” flag once done.

Alternatives¶

N/A

Implementation¶

The throttling queue is implemented as a new database column containing a boolean flag. See Central Changes and Storage Changes.

Also, new zones will be created with an uniformly random refresh time between a minimum and a maximum value.

Design considerations¶

The throttling queue could be implemented outside of the database: - No need to create an extra database column - No increased database I/O

We propose using the database for the following reasons: - Zone Manager is the best candidate to handle the delayed Notify. Currently there are no ways for Central to send a list of Zones to Zone Manager other than through the database - The queue can support delayed Notify for changes other than Pool NS record updates - Ability to monitor the queue size and ETA to inform the user and for debugging - A persistent queue can survive Zone Manager unhandled exceptions or restarts - The increased database load is negligible compared to the existing traffic

Risk analysis¶

Zone Manager fails to run the Notify delivery task. The nameservers will eventually refresh the zone anyways. Impact: slow update propagation. Mitigation: expose the notification queue length to the user through Admin API and by logging.
A big notification queue takes a considerable time to be handled. Impact: potentially prevents more urgent changes to be delivered quickly. Mitigation: encourage users to configure the throttling parameters; Provide sensible default values. Implementing a concept of notification priority seems unnecessary.

Assignee(s)¶

Primary assignee:: Federico Ceratto https://launchpad.net/~federico-ceratto

Milestones¶

Target Milestone for completion:: Liberty-3

Work Items¶

Implement refresh time staggering
Implement Notify throttling
Add throttle parameters to configuration files
Document throttling mechanism
Write unit and functional tests
Test throttling and staggering on devstack

Dependencies¶

N/A

Bulk zone update throttling