This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode

Asynchronous Zone Import/Export¶

https://blueprints.launchpad.net/designate/+spec/async-import-export

Problem description¶

Large zone imports have the potential to take a lot longer than the typically allowed time for API responses. Parsing large zone files, and inserting the necessary records scales linearly with the amount of records that need to be imported.

Proposed change¶

The API for Zone imports needs to be changed to be asynchronous. This would involve a new database table for zone import “statuses”, and a more robust system for managing the parsing and insertion of the zone data. Exporting zones will also return to the v2 API.

API Changes¶

GET /v2/zones/tasks/imports¶

This allows the user to view the statuses of their zone import requests. If the import has completed successfully, they can get the zone id and a link to the zone.

GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66

{
  "imports": [
      {
        "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
        "zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
        "status": "SUCCESS",
        "links":{
            "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
            "href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
      },
      {
        "id": "addda8f0-f558-11e3-a3ac-0800200c9a66",
        "zone_id": "qgqm44f0-s638-15e3-f4d3-1893572c9a67",
        "status": "SUCCESS",
        "links":{
            "self": "http://127.0.0.1:9001/v2/zones/tasks/import/addda8f0-f558-11e3-a3ac-0800200c9a66",
            "href": "http://127.0.0.1:9001/v2/zones/qgqm44f0-s638-15e3-f4d3-1893572c9a67"
      }
  ]
}

POST /v2/zones/tasks/imports¶

This creates a request to import a zone. The zone data is passed via the request body, and the Content-Type header must be set to ‘text/dns’.

This returns an id for the import request, for the user to query.

POST /v2/zones/tasks/import HTTP/1.1
Accept: application/json
Content-Type: text/dns

$ORIGIN example.com.
example.com. 42 IN SOA ns.example.com. nsadmin.example.com. 42 42 42 42 42
example.com. 42 IN NS ns.example.com.
example.com. 42 IN MX 10 mail.example.com.
ns.example.com. 42 IN A 10.0.0.1
mail.example.com. 42 IN A 10.0.0.2

HTTP/1.1 201 Accepted
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66

{
    "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
    "zone_id": null,
    "status": "PENDING",
    "links":{
        "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66"
    }
}

GET /v2/zones/tasks/imports/<id>¶

This allows the user to view the status of their zone import request. If the import has completed successfully, they can get the zone id and a link to the zone.

GET /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: application/json

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Location: /v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66

{
    "id": "cddda8f0-f558-11e3-a3ac-0800200c9a66",
    "zone_id": "agqm44f0-s638-15e3-f4d3-1893572c9a67",
    "status": "SUCCESS",
    "links":{
        "self": "http://127.0.0.1:9001/v2/zones/tasks/import/cddda8f0-f558-11e3-a3ac-0800200c9a66",
        "href": "http://127.0.0.1:9001/v2/zones/agqm44f0-s638-15e3-f4d3-1893572c9a67"
    }
}

GET /v2/zones/<id>¶

This request, with the header “Accept:text/dns” exports the zone in DNS zonefile format to the user.

GET /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66 HTTP/1.1
Accept: text/dns

HTTP/1.1 200 OK
Content-Type: text/dns; charset=UTF-8
Location: /v2/zones/cddda8f0-f558-11e3-a3ac-0800200c9a66

HTTP/1.1 200 OK
Content-Type: text/dns

$ORIGIN example.com.
$TTL 42
example.com. IN SOA ns.designate.com. nsadmin.example.com. (
    1394213803 ; serial
    3600 ; refresh
    600 ; retry
    86400 ; expire
    3600 ; minimum
)
example.com. IN NS ns.designate.com.
example.com.  IN MX 10 mail.example.com.
ns.example.com.  IN A  10.0.0.1
mail.example.com.  IN A  10.0.0.2

Central Changes¶

create_import_domain(body)¶

Parameter	Description	Required
body	Unserialized request data from the API request	Yes

1. Create an entry in the zone_imports table to track the request with status PENDING. 2. Kicks off a thread to _import_domain with the request body. 3. Returns a zone_import object.

get_import_domain(id)¶

Parameter	Description	Required
import_id	The id of the zone import	Yes

1. Calls storage.find_import to get a specific zone_import record from the zone_imports table. 2. Returns the zone_import object

find_import_domains(context)¶

Parameter	Description	Required
context	Context to be passed to storage.find_imports	Yes

1. Calls off to the storage.find_imports to find all the zone imports for a tenant. 2. Returns the zone_import_list object

delete_import_domain(id)¶

Parameter	Description	Required
import_id	The id of the zone import	Yes

1. Calls off to storage.delete_zone_import to delete the zone import record from the zone_imports table.

_import_domain(body)¶

Parameter	Description	Required
body	Unserialized request data from the API request	Yes
zone_import	zone_import object from the original request	Yes

1. Tries to use dnspython’s from_text method to convert the zone into a dnspython object. If that doesn’t work, updates the zone_import object in storage to ERROR to indicate that the zone could not be imported. 2. Converts the dnspython zone to a Designate domain object. 3. Calls central.create_domain with the converted object. 4. Takes the return object’s id and updates the zone_import object in storage with the new zone_id and updates the status of the import to COMPLETE, or similar.

Storage Changes¶

A new table for zone import records will be added, along with the boilerplate get, set, delete, update methods.

New Table - zone_imports¶

Row	Type	Nullable?	Unique?	Notes
id	uuid	No	Yes	Primary key
zone_id	uuid	Yes	Yes	Zone id when the import completes
status	ENUM	Yes	No	One of [COMPLETE, PENDING, ERROR]
message	VARCHAR	Yes	No	A message letting the user know why their import failed. For example: “Malformed zonefile”, “Complete”

Other Changes¶

New DesignateObjects for the Imports will have be created, ZoneImport, ZoneImportList.

Exporting zones will also return to the v2 API, but should remain relatively unchanged.

Alternatives¶

Instead of creating a new zone_import table and object, it may be possible to create an empty domain object and call central._create_domain_in_storage and make an empty domain that is then updated with the result of the import. Only then could the call to the pool manager be made.

This might make the code simpler. But provides very little in the way of letting the user know that their import failed. You could add a status like MALFORMED_ZONEFILE or something, but that would still require the user to delete the zone before they tried again. Unless you soft-delete the zone when it fails and modify the default find_domains criterion to find zones that have been deleted only if they have that status.

Assignee(s)¶

Primary assignee:: tim-simmons-t

Milestones¶

Target Milestone for completion:: Liberty-1

Work Items¶

Implement the database table, migration.
Implement the storage methods.
Implement the central methods.
Fix the API to use the new code.
Move the APIs back into the v2 namespace

Asynchronous Zone Import/Export