A generic and extendabled healthcheck middleware
They are no simple and common ways to ensure that an Openstack API endpoints can handle requests. But a deployer that use loadbalancers want to have easy way to known if a API endpoint can handle request or not.
For API endpoint that rely on a backend (like a database), checking that API endpoint does not return a 50x errors on the root url of the endpoint is not sufficient. An application like nova can return 200 on ‘/’ even the database is unreachable.
The idea is to create a wsgi middleware that all openstack components can use:
[filter:healthcheck] paste.filter_factory = oslo.middleware.healthcheck:Healthcheck path = /healthcheck (default) backends = database,disable_by_file (optional, default: empty) # used by the 'disable_by_file' backend disable_by_file_path = /var/run/nova/healthcheck_disable (optional, default: empty)
The middleware will return “200 OK” if everything is OK, or “503 <REASON>” if not with the reason of why this API should not be used.
“backends” will the name of a stevedore extensions in the namespace “oslo.middleware.healthcheck”.
oslo.middleware will also provide a base class for these extensions:
HealthcheckResult = namedtuple('HealthcheckResult', ['available', 'reason'], verbose=True) class HealthcheckBaseExtention(object): def __init__(self, conf): self.conf = conf @abc.abstractmethod def healthcheck(): """method called by the healthcheck middleware return: HealthcheckResult object """ class MyDBHealthcheck(HealthcheckBaseExtention): def healthcheck(): ... return HealthcheckResult(available=False, reason="Fail to connect to the database")
And so the setup.cfg will have entry_point like that:
[entry_points] healthcheck = database = oslo.db:DBHealthcheck disable_by_file = oslo.middleware.healthcheck:DisableByFileHealthcheck
The ‘DisableByFileHealthcheck’ extension will return if the ‘disable_by_file_path’ file is missing:
return HealthcheckResult(available=False, reason="DISABLED BY FILE")
return HealthcheckResult(available=True, reason="")
Also, but not part of this blueprint, oslo.db can provide a generic implementation for database checks.
Some works on different project have already been proposed to do that but never get merged:
A deployer could prepare resources in their cloud and build a HTTP request to query these resources to check that everything works, but this method is not efficient, it need to prepare some resources, the HTTP request will have some credentails that need to stored on the loadbalancer. So this method is a bit heavy when we can just do a simple ‘select now();’ into a database to known if a backend works.
Swift already have this kind of middleware, we must ensure we keep the same behavior:
It’s recommanded to block this url or to randomize the url used for healthcheck, because this feature it’s cleary dedicated for tools used by a deployer like load balancer.
The middleware will be configurable:
And the DisablebyfileHealthcheck with:
Middleware will be covered by the unittest And also have a tempest test for each services that have integrated it.
Target Milestone for completion:
This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode