Copyright 2015 OpenStack Foundation This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode
Host a code search service¶
It is often convenient, for instance when researching certain classes of widespread bugs or looking for implementation examples, to be able to search through all of the OpenStack projects’ source code. As a service to our contributors and downstream communities, we should provide a convenient mechanism to be able to do this.
To host a code search service, we need to decide on the software we’ll use, implement a series of configuration management changes/additions, write accompanying management documentation, and build the resources needed to run it.
Create or reuse an existing Puppet module to deploy the Hound code search engine and serve it via Apache from a dedicated virtual machine under care of the OpenStack Infrastructure Project. A proof-of-concept deployment is running at http://hound.openstack.org/ but can be taken down once this work is underway.
Some alternatives were explored:
- Keep expecting people to clone every repo and loop through them with git grep? It works, but it’s pretty painful.
- A number of hosted code search services exist running on the public Internet, but for improved performance and customizability it’s simpler to run one ourselves. Often those third party services get confused by our development workflow, are missing new repos, end up with stale repos indexed after renames and so on.
- There are quite a few code search engines available, published as free software, besides Hound. The alternative we tried most seriously (via a proof-of-concept demo) is Livegrep, but its indexes were static and had to be replaced via a fairly costly process considering the volume of source code we maintain. Others were ruled out for various similar reasons.
- Primary assignee:
- Infra root shepherds:
- fungi pleia2
Use Gerrit topic “code-search” for all patches related to this spec.
git-review -t code-search
- Initiate a change in
openstack-infra/project-configto create an empty
openstack-infra/puppet-houndGit repository as described in the Repository Creator’s Guide. This should be accompanied by a related change to
openstack/governanceadding the new repo as part of the Infrastructure project.
- Create a new
houndPuppet module in
openstack-infra/puppet-houndwhich enables an opinionated and working but not OpenStack-specific deployment of the service suitable for publication to the Puppet Forge. Existing
openstack-infra/puppet-.*should be examined for inspiration and consistency. A rough change at https://review.openstack.org/178488 covers most of the features needed but is unnecessarily entangled with the
openstack_projectmodule. Feel free to copy that code and credit the original author with “Co-Authored-By: ...” in the commit message.
- Upstream improvement to Hound is needed to support noticing config file changes without shutting down and restarting, filed as https://github.com/etsy/Hound/issues/119 and seems to have the general approval of its developer team.
- Upstream improvement to Hound is needed to change the repo list, as it’s a select box and doesn’t support any namespacing making it particularly unwiedly for us. Typeahead completion would be great.
- A change will be proposed to the
openstack-infra/system-configGit repo lightly documenting deployment and management of the service and adding that document to the
doc/source/index.rstfile. Examples in the
doc/sourcedirectory can be used for inspiration and consistency of style. This same change can also add a
codesearch.openstack.orgnode to the global
manifests/site.ppfile along with (if needed) a
modules/openstack_project/manifests/codesearch.ppclass file and any accompanying custom config files or templates.
- Deployment of the above changes will be performed by a root admin
onto a new
codesearch.openstack.orgserver, and related A, AAAA and PTR resource records will be created in DNS for it.
- If necessary, use of the service can be documented in the
openstack-infra/infra-manualrepo as well.
- Once the service appears stable and working as intended, announce it to the openstack-dev, openstack-infra and openstack-operators mailing lists.
openstack-infra/puppet-hound Git repo will be created as part
of this plan.
codesearch.openstack.org server will be added, and the
hound.openstack.org proof-of-concept demo will be
A, AAAA and PTR resource records will be created for the
The service’s deployment and management will be documented in the
openstack-infra/system-config repo, published to the
http://docs.openstack.org/infra/system-config/ site. Optionally, use
of the service can be documented in the Infrastructure
This is not a trusted service, needs no authentication for normal use, and it runs on its own dedicated virtual machine. HTTPS should not be necessary for this service, so no X.509 certificate will be ordered.
The configuration management for this service will be tested via existing apply/syntax CI jobs.
- None identified.