https://blueprints.launchpad.net/fuel/+spec/consume-external-ubuntu
Before using Ubuntu release a cloud operator should specify both Ubuntu and OpenStack mirrors.
Currently, it’s hard to provide Ubuntu’s upstream updates, since we don’t fetch them from official mirrors. Therefore, users are vulnerable for security issues for a long time.
This spec provides a detail plan how to solve this problem.
Set both Ubuntu and OpenStack mirrors per cluster.
Once users have created an environment on Ubuntu, they SHOULD be able to provide both Ubuntu and OpenStack repos as well as additional repos on the “Settings” tab. By default, Canonical repo for Ubuntu and Fuel repo for OpenStack SHOULD be used.
The repos SHOULD be specified using Debian format:
deb http://archive.ubuntu.com/ubuntu/ trusty main universe
deb-src http://pl.archive.ubuntu.com/ubuntu/ trusty main multiverse
All Ubuntu packages SHOULD be removed from the Fuel ISO and provided mirrors SHOULD be used for environment deployment.
Note
It’d be great to have format validation for provided repos on both backend and UI.
Note
We SHOULD NOT use mirror:// protocol for default repos, because we won’t know which repos were used and hence debugging will be painful.
See “Data model impact”, “UX impact” and “Developer impact” for details.
OPTIONAL: Check that provided repos contain all required packages.
We SHOULD check that all required packages are available and therefore successful deployment is possible.
Since Nailgun knows nothing about required packages and this information SHOULD NOT be hardcoded (it’s a dynamic), we obviously HAVE TO do it by executing some asynchronous task. The task SHOULD:
Note
Since we’re moving toward declarative tasks, the task will be executed inside MCollective container. So the container HAVE TO be ready for doing this.
Generate provisioning images per cluster.
Since we’re going to specify mirrors per cluster, different clusters may have different set of mirrors. Therefore, we SHOULD generate provisioning images per cluster.
Provisioning images SHOULD be generated once users start deployment. The task for generating images HAVE TO be started before others deployment tasks. If users don’t use image-based provisioning, the task SHOULD NOT be sent to Astute.
Provisioning images SHOULD NOT be regenerated if they are already exists.
Note
Consider to reuse existing images if there are ones with suitable packages.
See “Data model impact” and “RPC impact” for details.
Check that there is connectivity to provided repos.
We SHOULD check whether repos are reachable or not, and in case it’s not we HAVE TO warn user about it. The check SHOULD be implemented as asynchronous task and SHOULD be performed in two cases:
See “Data model impact” and “RPC impact” for details.
None
Release model SHOULD have a new set of editable attributes that allows us to specify a set of repos and their options (e.g. priority).
Here’s the proposed format for Debian-based distributives:
attributes_metadata:
# ...
editable:
# ...
repo_setup:
metadata:
label: "Repos Configuration"
weight: 123
repos:
type: "custom_repo_configuration"
value:
- type: "deb"
name: "OS"
uri: "http://archive.ubuntu.com/ubuntu/"
suite: "trusty"
section: "main"
priority: 1001
- type: "deb"
name: "Fuel"
uri: "http://mirror.fuel-infra.org/fwm/6.1/ubuntu/"
suite: "mos6.1"
section: "main"
priority: 1002
Here’s the proposed format for RHEL-based distributives:
attributes_metadata:
# ...
editable:
# ...
repo_setup:
metadata:
label: "Repos Configuration"
weight: 123
repos:
type: "custom_repo_configuration"
value:
- type: "rpm"
name: "OS"
uri: "http://mirror.centos.org/centos-6/6/os/x86_64/"
priority: 1
- type: "rpm"
name: "Fuel"
uri: "http://mirror.fuel-infra.org/fwm/6.1/centos/os/x86_64/"
priority: 2
We also SHOULD implement the check_connectivity task.
Once users have created an environment on Ubuntu, they SHOULD be able to provide both Ubuntu and OpenStack repos as well as additional repos on the “Settings” tab. By default, UI controls SHOULD be pre populated with defaults.
Note
Users SHOULD NOT be able to provide custom repos for CentOS environemnts.
Additional repos (extra repos) SHOULD be added on demand by pressing some button (for example, “+”/ “add more”).
There SHOULD BE no way to set repos priorities via Fuel UI. Let’s keep this functionality for RESTful API and python-fueclient. If user change some repo on UI the priority SHOULD NOT be changed.
Note
For both Ubuntu and Fuel repos the priorities SHOULD be kept as they specified in openstack.yaml. For each extra repo the priority SHOULD be the same and SHOULD be retrieved from settings.yaml.
Once an Ubuntu environment is created a check_connectivity task SHOULD be sent to Astute. If provided repos are reachable from the master node - it reports success and the environment becomes ready for deployment. Otherwise - a banner SHOULD be shown on Fuel UI that there’s no connectivity to repos and deployment SHOULD NOT be allowed. The request for starting a check_connectivity task SHOULD be performed by Fuel UI. The API call for creating cluster SHOULD NOT do it implicitly for us.
Note
The task SHOULD NOT be used for Cent OS environments.
If the check_connectivity task was failed and deployment isn’t allowed, the user SHOULD be able either provide custom (reachable) repos or fix connectivity issues, and restart the check through the “Settings” tab.
Note
The task SHOULD NOT be restarted automatically by saving settings. It SHOULD be restarted on demand by pressing a special button on UI.
If the check is passed the environment SHOULD become ready for deployment.
Once an environment is deployed users SHOULD NOT be able to change repos.
The check_connectivity task SHOULD be executed by Astute, and its SHOULD be declarative. Here’s the example of the RPC message:
{
"api_version": "1",
"method": "execute_tasks",
"respond_to": "_respond_to_",
"args": {
"task_uuid": "_task_uuid4_",
"tasks": [
{
"id": "_command_id_",
"uids": ["master"],
"type": "shell",
"parameters": {
"cmd": "_command_to_execute_",
"timeout": 180
}
}
]
}
}
Note
The _command_to_execute_ will be executed inside mcollective container, so the container MUST be ready to do this and MUST pre install all required stuff.
Since the task is executed via RPC, the Nailgun’s receiver SHOULD implement some check_connectivity_resp method in order to handle task’s result.
None.
Since we’re going to introduce priorities for repos, the priority of plugins’ repos SHOULD be higher than priority of Ubuntu/Fuel repos. Why? Because plugin developer MAY want to override some package from the core distro.
A notification SHOULD be sent when provisioning images were built.
None.
Ubuntu deployment time MAY be increased due to the fact that the packages will be retrieved directly from the third-party servers. But when the packages get cached the time should be the same.
Hint
The word “MAY” is used because modern DCs may have network connection faster than HDD.
None
Fuel DevOps team HAVE TO prepare a local Ubuntu mirror.
We SHOULD use the local mirror in tests in order to speed up their passing.
The local mirror SHOULD NOT change in time, since it may lead to accidental deployment fails. All new Ubuntu’s mirrors SHOULD BE saved as separate mirrors. In other words, we SHOULD NOT update mirror in place. Instead, each sync with upstream SHOULD create a new version of the mirror.
Note
Consider to re-use scripts from IT guys.
All mirror’s versions SHOULD live at least a week in order to get debug easy.
The latest mirror version SHOULD be available in Fuel CI through environment variable.
Fuel DevOps team HAVE TO reflect changes in Jenkins.
Primary assignee:
Developers:
Mandatory Design Reviewers:
QA:
DevOps:
The documentation SHOULD cover how the end user workflow has been changed for deploying clusters on Ubuntu.