NVMeoF Connection Agent¶
https://blueprints.launchpad.net/cinder/+spec/nvmeof-connection-agent
Daemon that monitors NVMeoF connections and MDRAID arrays created by the new NVMeoF connector. It reports initiator-side events to the storage orchestrator, identifies faulted volume replicas, requests new replicas and replaces faulted replicas with newly assigned ones.
Problem description¶
When the NVMe connector connects a client-replicated volume, OpenStack will see it as one volume, and has no way of monitoring managing and healing the replicas in these MDRAID arrays. This agent will take care of that.
Currently there’s no mechanism to monitor and heal these replicated volumes. We cannot do it only on the Cinder driver side because currently there is no integrated mechanism to detect initiator connection events and carry out replica replacement on the compute node.
For target-side volume replication (traditional approach), it is the storage backend that takes care of monitoring and self healing. The NVMe + MDRAID approach moves the data replication responsibility from the storage backend to the consuming initiator (ie. compute node).
So the monitoring and healing needs to be on the initiator / compute side.
With this approach, the agent will monitor the NVMeoF connections and report changes to the storage orchestrator / provisioner. It will monitor MDRAID arrays and reconcile their physical state on the host with expected state from the volume provisioner, replacing broken legs.
Finally, orchestration decisions / optimizations will be carried by the volume orchestrator / provisioner using reported information from agent monitoring. Though this is outside the scope of the agent (it is storage backend implemented functionality) - it is useful to mention here that it will handle cases such as avoid using faulty replicas during re-attachment scenarios, because in this design approach only the initiator node can detect the replicas’ sync states of its MDRAID arrays.
Use Cases¶
When working with replicated NVMeoF volumes that are attached to an instance for a long time, one of the replicas may go faulty. This agent will detect it and attempt to replace it, i.e., self heal the MDRAID array, without the need to detach and re-attach the entire volume from the instance.
Additionally, the agent will detect and report connection and replica sync state events to the storage orchestrator (or potentially other endpoints that can make use of it) - which is gathered by the storage backend for making storage provisioning / orchestration decisions, as well as for telemetry.
Proposed change¶
Add an agent entry point code to os-brick, such as: os_brick/os-brick/cmd/agent.py
Add an entry_points console_scripts entry in os-brick’s setup.cfg
The agent main function will first initialize the agent by reading access information to the volume orchestor / provisioner from a pre-defined config file (such as /etc/nvme-agent/agent.conf ?)
Vendor specific params will be used and prefixed by the vendor prefix, such as: kioxia_provisioner_ip kioxia_provisioner_port kioxia_provisioner_token kioxia_cert_file
Once initialized, the agent will start a periodic task that will do the following:
Host probe / heartbeat to the storage orchestrator / provisioner
Pull volume metadata (connection and replication state) from provisioner
Monitor NVMeoF devices and MDRAID arrays belonging to it
Detect connection and replication state changes and report to provisioner
Request replacements for faulty replicas
Reconcile replica states from provisioner (carry out the replacements)
Typical self healing flow:
volume replica goes faulty
agent notices faulty replica, reports to provisioner
provisioner marks replica as bad (so it wont be used later unless synced)
agent keeps pulling volume information from provisioner
certain grace period passes, agent sees no state changes of faulty replica from provisioner, so it sends explicit request to replace replica
provisioner replaces replica and updates volume information
agent pulls volume replica information, notices a replica has changed
agent carries out replica replacement
Alternatives¶
Operator could use some own scripts to monitor connections and replicated arrays states, report detected events, and carry out replica replacements manually.
Data model impact¶
None
REST API impact¶
None
Security impact¶
Sudo executions of nvme and mdadm Needs access for reading of root filesystem paths such as: /sys/class/nvme-fabrics/… /sys/class/block/…
Active/Active HA impact¶
None
Notifications impact¶
None
Other end user impact¶
None
Performance Impact¶
If configured to run by the operator, this will be a new process running on the compute node. Though it will spend most of its time sleeping, it will wake up every 30 seconds to do its periodic tasks: probe the storage provisioner and inspect nvme connections and mdraid states.
These tasks are not compute intensive, with time mostly spent waiting for a response from the storage provisioner, and the nvme and mdraid operations will only have time complexity linear to the number of devices under the control of the agent (which can be treated as constant due to a low upper limit per host). And finally, the performance effect on the network will also be small, since it will only be sending/receiving small amounts of (meta)data across the network.
Other deployer impact¶
None
Developer impact¶
To allow multiple vendor implementations, the specific methods / logic for:
probing / heartbeating the storage provisioner
pulling / parsing volume metadata from provisioner
reporting state changes to provisioner
requesting provisioner to replace replica
These all involve communication with and functionality carried out by the storage backend provisioner / orchestrator, will need to be implemented on a per vendor basis.
The architecture is such that the agent will be a generic daemon that will define the interface, and the kioxia implementation will be the first example of a vendor-specific implementation.
Implementation¶
Assignee(s)¶
- Zohar Mamedov
zoharm
Work Items¶
Agent entry point and initialization.
Agent periodic tasks:
Host probe / heart beat
Monitoring (connection and replication event detection)
Report events to storage provisioner
Connection and replication state re-conciliation
Dependencies¶
None
Testing¶
We should be able to accept this with just unit tests.
Documentation Impact¶
Document that with this feature os-brick will be coming with a console-script that is used to launch this agent.
Document how to configure the agent for usage.
References¶
Presentation slides with architectural diagram on slide 2 https://docs.google.com/presentation/d/1lPU8mQ7jJmr9Tybu5gXkbE7NC1ppkMnoBS4cgSFhzWc
