This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode
Swift Request Tagging for detailed logging/tracing
URL of your blueprint:
None.
To tag a particular request/every ‘x’ requests, which would undergo more detailed logging.
Problem Description
Reasons for detailed logging:
- A Swift user is having problems, which we cannot recreate but could tag this user request for more logging.
- In order to better investigate a cluster for bottlenecks/problems - Internal user (admin/op) wants additional info on some situations where the client is getting inconsistent container listings. With the Swift-inspector, we can tell what node is not returning the correct listings.
Proposed Change
Existing: Swift-Inspector (https://github.com/hurricanerix/swift-inspector ) currently
provides middleware in Proxy and Object servers. Relays info about a request back to the client with the assumption that the client is actively making a decision to tag a request to trigger some action that would not otherwise occur.
Current Inspectors:
- Timing -‘Inspector-Timing’: gives the amount of time it took for the proxy-server to process the request
- Handlers – ‘Inspector-Handlers’: not implemented (meant to return the account/container/object servers that were contacted in the request) ‘Inspector-Handlers-Proxy’: returns the proxy that handled the request
- Nodes - ‘Inspector-Nodes’: returns what account/container/object servers the path resides on ‘Inspector-More-Nodes’: returns extra nodes for handoff.
Changes:
- Add logging inspector to the above inspectors , which would enable detailed logging for tagged requests.
- Add the capability to let the system decide (instead of the client) to tag a request and nice to add rules to trigger actions like extra logging etc.
Possible Tagging criteria: Tagging
- every ‘x’ requests/ a % of all requests.
- based on something in the request/response headers (e.g.if the HTTP method is DELETE, or the response is sending a specific status code back)
- based on a specific account/container/object/feature.
Alternatives
- Logging: log collector/log aggregator like logstash.
Implementation
Work Items
- To add an Inspector– ‘Logging’ to existing inspectors , to enable the logs.
- Add rules to tag decide which requests to be tagged
- Trigger actions like logging.
- Restrict the access of nodes/inventory list displayed to admins/ops only.
- Figure out hmac_key access (Inspector-Sig) and ‘Logging’ work together?
Repositories
Will any new git repositories need to be created? Yes.
Servers
Will any new servers need to be created? No.
What existing servers will be affected? Proxy and Object servers.
DNS Entries
Will any other DNS entries need to be created or updated? No.
Documentation
Will this require a documentation change? Yes , Swift-inspector docs.
Will it impact developer workflow? No.
Will additional communication need to be made? No.