cinder db purge utility

https://blueprints.launchpad.net/cinder/+spec/database-purge

This spec adds the ability to sanely and safely purge deleted rows from the cinder database for all relavent tables. Presently, we keep all deleted rows, or archive them to a ‘shadow’ table. I believe this is unmaintainable as we move towards more upgradable releases. Today, most users depend on manual DB queries to delete this data, but this opens up to human errors.

The goal is to have this be an extension to the cinder-manage db command. Similar specs are being submitted to all the various projects that touch a database.

Problem description

Very long lived Openstack installations will carry around database rows for years and years. To date, there is no “mechanism” to programmatically purge the deleted data. The archive rows feature doesn’t solve this.

Use Cases

Operators should have the ability to purge deleted rows, possibily on a schedule (cronjob) or as needed (Before an upgrade, prior to maintenance) The intended use would be to specify a number of days prior to today for deletion, e.g. “cinder-manage db purge 60” would purge deleted rows that have the “deleted_at” column greater than 60 days ago

Project Priority

Low

Proposed change

The proposal is to add a “purge” method to DbCommands in cinder/cinder/cmd/manage.py This will take a number of days argument, and use that for a data_sub match Like: delete from instances where deleted != 0 and deleted_at > data_sub(NOW()…)

Alternatives

Today, this can be accomplished manually with SQL commands, or via script. There is also the archive_deleted_rows method. However, this won’t satisfy certain data destruction policies that may exist at some companies.

Data model impact

None, all tables presently include a “deleted_at” column.

REST API impact

None, this would be run from cinder-manage

Security impact

Low, This only touches already deleted rows.

Notifications impact

None

Other end user impact

None

Performance Impact

This has the potential to improve performance for very large databases. Very long-lived installations can suffer from inefficient operations on large tables.

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

primary author and contact.
Abel Lopez <al592b>
Primary assignee:
<al592b>
Other contributors:
<al592b>

Work Items

Add purge functionality to manage.py db/api.py db/sqlalchemy/api.py Add tests to confirm functionality Add documentation of feature

Dependencies

None

Testing

The test will be written as such. Three rows will be inserted into a test db. Two will be “deleted=1”, one will be “deleted=0” One of the deleted rows will have “deleted_at” be NOW(), the other will be “deleted_at” a few days ago, lets say 10. The test will call the new function with the argument of “7”, to verify that only the row that was deleted at 10 days ago will be purged. The two other rows should remain.

Documentation Impact

will need to add documentation of this feature

References

This was discussed on both the openstack-operators mailing list and the openstack-developers mailing lists with positive feedback from the group.

http://lists.openstack.org/pipermail/openstack-dev/2014-October/049616.html