Support SSL for OpenStack endpoints and Fuel Master node

https://blueprints.launchpad.net/fuel/+spec/ssl-endpoints

OpenStack public endpoints that provide APIs on public networks need to operate over SSL. Also such components on master node like Nginx, Keystone, RabbitMQ needs to operate over SSL.

Problem description

OpenStack services receive requests from public networks that are untrusted area. As the network path between the end-users and the OpenStack services is untrusted, encryption is required to ensure confidentiality.

We achieve this by implementing Secure Sockets Layer as recommended in the OpenStack security guide.

Fuel master node after installation also operates via unsecure protocols only. We can achieve more security using access to Fuel UI via SSL. There will be different certificates for OpenStack endpoints and Fuel Master node.

Proposed change

OpenStack services can be configured to use SSL/TLS libraries. We propose to modify the puppet manifests to be able to use SSL/TLS and HTTP services.

This blueprint will focus on HTTP services that provide API endpoints on public networks. The configuration of HAProxy will also be modified to enable SSL for Horizon. Endpoints used for management and other services like messaging will not be done in this blueprint.

We will use HAProxy and its support of SSL. The endpoints of public URLs will be configured with the IP or the name of the HAProxy. It will handle incoming SSL connections, decryption of the SSL and passing the unencrypted request to the corresponding service. That means that the communication between the proxy and the endpoints will not be encrypted.

Here is a schema of the encryption of the traffic paths:

1. Between client and OpenStack services:

            +-------+    +---------------------+    +------+
Client +----+ HTTPS +----+ public API endpoint +----+ HTTP |
            +-------+    +---------------------+    +--+---+
                                                       |
                                                       v
                                                  OSt service

2. Between OpenStack services:

                +------+   +-----------------------+   +------+
OSt service +---+ HTTP +---+ internal API endpoint +---+ HTTP |
                +------+   +-----------------------+   +--+---+
                                                          |
                                                          v
                                                     OSt service

It is recommended that an OpenStack service communicate to another OpenStack service by using the internal API endpoint. Fuel is following this best practice. If we find a service that is not using the management network to communicate with another one we should file a bug about this and reconfigure it to use the internal API endpoint.

Here are the different steps needed to implement SSL with public API endpoints:

• Install at least the version 1.5.0 of HAProxy because the support of SSL is available since this version. As it is not available in Ubuntu (and probably it is the same for CentOS) we will need to rebuild it. https://bugs.launchpad.net/fuel/+bug/1346365/ https://review.openstack.org/#/c/147858/ https://review.openstack.org/#/c/147860/
• Configure HAProxy to manage SSL connections and to forward the request to the corresponding service. Also configure Keystone over https https://review.openstack.org/#/c/186498/
• When an OpenStack service queries another OpenStack service it must use the internal API endpoint. If it’s not possible, the service needs to be configured with the appropriate SSL certificate.
• We will use a self-signed certificate by default. In terms of encryption it is the same than other certificates. Self-signed certificates aren’t less “secure” but end-users have to assume that they are genuine. If we don’t want to use self-signed certificate generated by Fuel, the customer will have to provide it before deploying the cluster. And to generate the certificate, the customer needs to know the public VIP address (or name). We will use openssl to manage them so that means that we will add a dependency to openssl if it is not already the case. So, we need a field in UI page to which user can put DNS name for CN field in certificate.
• We need self-sign keys generation code https://review.openstack.org/#/c/186035/ https://review.openstack.org/#/c/186015/
• As we are using a self-signed certificate we need to ensure that OpenStack services behave in the same way with a self-signed certificate and a CA certificate. As an example, web browser opens a warning window if you are using a self-signed certificate and not if you are using a certificate issued by a trusted authority (ie a CA that is installed in your browser’s certificate manager).
• Even if encrypted communications with a self-signed certificate are a better solution than plain text ones we can not force a user to use a self-signed certificate. By default we will enable SSL but must give the possibility to the user to disable it. It can be a checkbox in Fuel UI to “disable SSL for public endpoints” that needs to be checked to disable SSL. Fuel Master node should be accessible both https and plain http protocols for legacy compatibility https://review.openstack.org/#/c/186706/
• After cluster deployment we should provide to the user right link to Horizon if it was deployed with https support
• In case of “real” HA where several HAProxy are deployed. We need to be able to distribute the certificate. The self-signed certificate will be generated by astute prehook and stored on the fuel master in /var/lib catalog with other keys generated by astute. Then the certificate will be distributed to all nodes that run HAProxy through a mechanism like upload_file. The certificate is related to a cluster because it is valid for a given virtual IP. So if a controller is added to the cluster we just reuse the certificate already generated at the creation of the cluster and stored on the fuel master node.
• As SSL certificates are sensitive data we should ensure that they are not stored in logs or in a database in case of diagnostic snapshot.
• OSTF tests shoud be fixed to operate over SSL
• In first iteration of SSL no notifications about expired SSL certificates will be added. Self-signed certificates will be created for 3650 days to prevent accidental problems with expiring. In next iterations notification about certificate expire must be provided and automatic regeneration of self-signed certificates can be implemented.

Fuel master node should operate over SSL. To achieve this, we should point UI web server to operate over HTTPS. Also some master components like Nginx, Keystone, RabbitMQ can use SSL by default.

Alternatives

1. Enable SSL for all OpenStack services.

   Cons:

   • API services need to respond quickly and SSL handshake can add a significant overhead. Python services do not work well with direct SSL integration.
   • You need to manage as many certificates as there are services. The problem is even more difficult when services are spread on different nodes as it is often the case.

   Pros:

   • It enables SSL on both, public and internal APIs. That means that we don’t need to be concerned with which endpoint is being used by each OpenStack service.

Data model impact

New fields in DB will be created, as user uploaded certificates should be stored there.

REST API impact

The management of certificate with the REST API is not in the scope of this specification.

Upgrade impact

If the updated environment does not use SSL before the update, it will not use SSL after the update.

In the same way we will not support downgrade from SSL to non-SSL.

Security impact

By using SSL/TLS over HTTP services, we will be able to provide a secure system with authentication (but it is not the case currently since you need a certificate generated by a CA) and confidentiality.

Notifications impact

We need to notify that deployment is done and so you need to open your Horizon at https:// instead of http://

We need to notify that a self-signed certificate is used for SSL encryption.

We also need to notify where the certificate that contains the public key that is used by client can be downloaded. It can be done by showing the link to the certificate in the UI. This certificate can be used by OpenStack clients.

We need to track certificate expiration date and show a pop-up with warning prior 1 month to certificate expiration.

Other end user impact

SSL is disabled by default.

As we are using a self-signed certificate by default, if a user is using a web browser to query OpenStack services there will be a popup to warn that the certificate can not be trusted.

Performance Impact

The SSL-overhead is generally small. The major cost of HTTPS is the SSL handshaking so depending the typical session length and the caching behavior of clients the overhead may be different. For very short sessions you can see performance issue.

The internal communication between services involves many API calls for small tasks and this is why we will not implement SSL endpoints for the management network. OpenStack services need to be configure properly.

Other deployer impact

OpenStack services will only use internal/admin URLs. Thus we must be sure that all services can communicate through the management network.

The work flow is to deploy the cluster with a self-signed certificate that was generated by astute at the creation of the cluster. This certificate must be stored somewhere on the fuel master node in /var/lib/fuel/keys/<env_number>/haproxy/haproxy.{crt,key,pem} with the root:root as an owner and u=rw,go-rwx for files containing provate key and u=rw,go=r for all other files. When the cluster is created, if it is available (it is not in the scope of this specification) the user can download its own certificate for the given VIP. If it cannot download it or don’t want to use another certificate then everything will work fine with the self-signed certificate.

Developer impact

OSTF team (health check team) is going to be affected by changing OpenStack endpoints.

Implementation

Assignee(s)

Primary assignee: - sbogatkin

Feature Lead: - assignee of this blueprint

Mandatory Design Reviewers: - Mike Scherbatov

Developers: - Stanislaw Bogatkin

QA: Alexander Kurenyshev

Work Items

stage 1

• Generate a certificate from Fuel that will be used for authentication. It can be self-signed.

• We need to provide the possibility to the user to deactivate SSL if he doesn’t want to use the one generated by Fuel.

• Install HAProxy and configure it to handle SSL connections and forward requests to the corresponding service.

• Check that the following services are configured to use internal URL for communicating with other OpenStack services.

  • ceilometer
  • ceph
  • cinder
  • glance
  • heat
  • horizon
  • keystone
  • murano
  • neutron
  • nova
  • sahara
  • swift

• If the user wants to use its own certificate we need to give him such opportunity. Also we should provide a document about how he can manually install his own certificate for SSL endpoints.

• Generate SSL keypair for master node nginx and apply it. Plain HTTP mode should be retained for backward compatibility.

stage 2

• Use a PKI described in the following blueprint: https://blueprints.launchpad.net/fuel/+spec/ca-deployment
• Wrap Keystone and RabbitMQ on master over SSL
• Add notification about certificates that will be expired soon

Dependencies

• openssl
• haproxy >= 1.5

Testing

Build a new fuel ISO and test if the deployment corresponds to what is expected. Acceptance criteria: For Fuel Master node: 1. After deploy, UI should be accessible via both plain http on port 8000 and SSL https on port 8443

For OpenStack controllers HAProxy: 1. After deploy with enabled SSL for OpenStack services, all OpenStack services should be accessible via https public endpoints 2. After deploy with enabled SSL for Horizon, it should be accessible both on http and https for public endpoint. 3. CN name in certificate should match to public name passed thru UI if self-signed certificates option selected 4. Keypair data should match to data passed thru UI if user-uploaded certificates option selected

Documentation Impact

As we will generate a certificate to allow the usage of SSL for API public endpoints, we need to document how to get the certificate that contains the public key to identify the service if an end-user want to use a CLI (for example nova CLI) to interact with OpenStack services.

If the user wants to use its own certificate we need to describe where and how it can manually upload its certificate.

We need to add a warning that if she/he uploads a certificate, she/he must manage the expiration date of the certificate.

References

• https://blueprints.launchpad.net/fuel/+spec/ssl-endpoints
• https://blueprints.launchpad.net/fuel/+spec/single-controller-ha
• https://bugs.launchpad.net/fuel/+bug/1346365
• http://docs.openstack.org/security-guide/content/ch020_ssl-everywhere.html
• https://help.ubuntu.com/community/OpenSSL
• http://blog.haproxy.com/2012/09/10/how-to-get-ssl-with-haproxy-getting-rid-of-stunnel-stud-nginx-or-pound/