Relational DB Schema with OSLO.DB

https://blueprints.launchpad.net/freezer/+spec/oslo.db

Taking advantage of the oslo.db library to have a more uniform database backend architecture to other OpenStack projects.

Problem description

Currently Freezer uses Elastic Search (ES) as a database backend, which is a NoSQL database specialized for ranked query results. Elastic Search adds additional complexity to an OpenStack system. Most of the components use a relational database management system (DBMS like MySQL or PostgreSQL) which are more common. It is more familiar how to maintain, troubleshoot and develop on top of relational databases.

Since Freezer related data turned out to be relational, it would be more convenient to use it trough the oslo.db pattern library. Using it, the database mapping would be more uniform to other OpenStack projects. It would be less challenging for new developers to contribute.

Use Cases

  • For new developers, already familiar with OpenStack, it should be less challenging to get familiar with the backend code, since most of the OpenStack projects use relational database backend trough the oslo.db pattern library.

  • For Deployers there would be no longer needed to set up a special DBMS just for Freezer, since it could share the relational DBMS used by the other (core) OpenStack projects, still well isolated in it’s own database.

  • For End User it would be less difficult to maintain, since Freezer would not add additional complexity with a less common component, instead it can take advantage of the DBMS that is already deployed for OpenStack.

Proposed change

Implementing the entities using oslo.db and SQLAlchemy base classes. And expose the new entities trough the REST API.

Alternatives

Oslo.db with SQLAlchemy is the de facto standard for OpenStack projects to implement database backends with relational DBMS. It provides high level ORM mapping and abstracts the different database backends. Therefore we gain compability with multiple relational DBMS just like any other OpenStack component using oslo.db.

Using other alternative would either not be more uniform to other OpenStack project tooling, either we would have to implement low level, directly to a specific database driver (just like now with ES).

Data model impact

Changes which require modifications to the data model often have a wider impact on the system. The community often has strong opinions on how the data model should be evolved, from both a functional and performance perspective. It is therefore important to capture and gain agreement as early as possible on any proposed changes to the data model.

Questions which need to be addressed by this section include:

  • What new data objects and/or database schema changes is this going to require?

  • What database migrations will accompany this change.

  • How will the initial set of new data objects be generated, for example if you need to take into account existing backups/jobs/… , or modify other existing data describe how that will work.

As there will be a brand new relational database schema [MIG1]:

Clients

id (varchar) [p_key]

project_id (uuid)

config_id (varchar)

description (varchar)

uuid (uuid)

Actions

id (uuid) [p_key]

action (varchar)

project_id (uuid)

mode (varchar)

src_file (varchar)

backup_name (varchar)

container (varchar(

restore_abs_path (varchar)

Action_reports

id (uuid) [p_key]

action_id (uuid) [f_key]

action_attachment_id (uuid) [f_key]

project_id (uuid)

result (varchar)

time_elapsed (varchar)

metadata (JSON)

report_date (timestamp)

log (blob) < only on failure

Jobs

id (uuid) [p_key]

project_id (uuid)

scheduling (JSON)

description (varchar)

Action_attachments

id (uuid) [p_key]

action_id (uuid) [f_key]

job_id (uuid) [f_key]

project_id (uuid)

priority (int)

retries (int)

retry_interval (int)

mandatory (bool)

Sessions

id (uuid) [p_key]

project_id (uuid)

scheduling (JSON)

policy (varchar)

Job_attachments

id (uuid) [p_key]

client_id (varchar) [f_key]

job_id (uuid) [f_key]

session_id (uuid) [f_key]

project_id (uuid)

event (varchar)

status (varchar)

current_pid (int)

REST API impact

There should be a new v2 API implemented. TBD.

Security impact

None

Notifications impact

TBD.

Other end user impact

None. TBD.

Performance Impact

None.

Other deployer impact

  • The Elastic Search configurations should be replaced with oslo.db configurations

  • When updating from a previous version there must be a data migration from ES to oslo.db (this will be addressed by a nother spec - TBD).

Developer impact

There will be no longer needed to deploy ES.

Implementation

Assignee(s)

Primary assignee:

neilus

Other contributors:

daemontool

Work Items

Work items or tasks – break the feature up into the things that need to be done to implement it. Those parts might end up being done by different people, but we’re mostly trying to understand the timeline for implementation.

  • implementing the database models

  • create adapter for API v1(?) and v2

  • implementing the CRUD API

  • updating the devStack plugin

  • updating documentation

Dependencies

  • Implementing the database migration script (TBD), which migrates data from ES to oslo.db backend DB.

  • We will be using oslo.db library and SQLAlchemy for iplementation.

Testing

TBD.

Documentation Impact

TBD.

  • Freezer API installation doc

References

MIG1

https://etherpad.openstack.org/p/freezer_mysql_migration