Agent message signing¶
https://blueprints.launchpad.net/murano/+spec/message-signing
Problem description¶
In order to deploy software, Murano sends scripts, playbooks and other files to agents that run on each VM created. Murano uses RabbitMQ message broker and its AMQP protocol to deliver messages to the agents. During first boot each VM gets its RabbitMQ credentials, the queue name where the messages are going to appear as well as instructions on where to put execution results. Typical Murano installation has a dedicated RabbitMQ instance that is used for such communications.
However, since user management is not included in AMQP protocol, Murano requires RabbitMQ credentials to be provided in its config file. Hence all the created VMs share the same credentials and it is only the queue name that distinguishes one machine from another. Since murano agents steady pull new commands from the RabbitMQ, if the queue name gets compromised, anyone could send commands to the VM and take complete control over it. Luckily current AMQP version does not have a function to list existing queues. However, since queue name is generally not considered to be a secret information such function might be added in the future. Also this information might be exposed by the RabbitMQ plugins installed in the system. For example, there is a so called Management plugin that can be used for that purpose. Besides, the queue name can be found in the Heat stack definition for particular Murano environment and anyone who can access it could take it from there. Default Heat policy restricts it to users of that tenant plus the admin user. However, particular OpenStack installations might have it configured differently. Also even with default policy it is possible for users of a single project (tenant) to control each other’s VMs regardless of their role in the tenant.
Proposed change¶
The proposed solution is to implement message signing so that agents can verify that the messages come from the murano-engine. There is a keypair that is shared by all the murano-engine instances in the cloud. When murano creates a VM, it provides it with a public part of the keypair in addition to currently provided RabbitMQ credentials and queue name. Each message sent by the murano-engine is going to be signed with the private key. Agents will verify the signature with the public key they gave and ignore all the messages that are not proven to be coming from the engine.
Here is how the communication workflow is going to become:
Murano engine is provided with the keypair (key file that has both the private and the public parts). The default is ~/.ssh/id_rsa which is a standard RSA SSH keypair of the user that is running the murano-engine. However, different key might be provided with the newly introduced config file setting.
When a VM is created, its config that is passed to it via the user-data is going to include the public part of the key.
Every time murano-engine is about to send message M to the agent with queue name Q it calculates
sign(Q + M, private_key)
i.e. the signature of the message bytes with prepended queue name in latin1 encoding. The queue name is prepended so that the messages sent to one VM could not be resent to other VMs. The signature is then put into a custom AMQP message header so that the message body does not change.Once the message is received by an gent, it performs
verify(signature, Q + M, public_key)
and drops the message if the verification fails. Since the queue name is different for each agent, it will also reject all messages that were signed for other VMs.
For the first version it all the cryptographic parameters are going to be fixed so that no additional configuration is needed:
The keypair must be PEM-encoded RSA without password protection
The padding is PKCS#1 v1.5
Digest function used for signing is SHA256
Engine-agent compatibility¶
Since the agent can be baked into the images Murano and updated independently from the server, Murano must still work in situations where the updated server talks to the old agents or vice versa (old engine, new agents).
For the new engine / old agents case the compatibility is retained because old agents will just ignore the new (and unknown to them) setting in the config file (the public key) as well as custom AMQP header and just won’t verify the signature.
In the old engine / new agents case the server won’t provide the public key to the agents. Agents won’t try to verify the messages in this case.
Thus the message signing is going to help only if both the server and the agents support it. Otherwise everything is going to work the way it currently does.
Replay attack prevention¶
Having queue names prepended to the message bodies prevents messages sent to one VM from being resent to another. However it is still possible to resend the same message to the same VM later.
In order to prevent it, the message body JSON is going to get additional key
called Stamp
. It is a long integer value that is monotonic i.e. the
server guarantees that the number is greater from that of the previous
message. The field is optional (since older engines won’t send it), but if
it is present, the agent must ensure that it exceeds the value of the
previously seen messages and ignores the message if it not the case. Murano
engine is going to derive the value from the current timestamp.
Messages with additional field remain compatible with the agents because they ignore all unknown fields. As a bonus, this feature is going to protect the agent against messages that for some reason got delivered twice which can happen in rare cases due to the message was requeued because of unexpected connectivity loss with RabbitMQ upon its acknowledgement or due to some RabbitMQ cluster inconsistencies.
Alternatives¶
None
Data model impact¶
None
REST API impact¶
None
Versioning impact¶
None
Other end user impact¶
None
Deployer impact¶
There should be generated and provisioned for all the engine instances keypair file. However it is going to be optional in order not to break existing murano deployment playbooks.
Developer impact¶
None
Murano-dashboard / Horizon impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
Stan Lagun (slagun)
Work Items¶
Add keypair to the murano-engine config file schema
Extract the public key from the keypair and inject it into the generated agent config data
Implement message signing on the server
Implement signature verification in the python agent
Implement signature verification in the legacy Windows (PowerShell) C# agent
Add stamp value to the messages
Validate stamp value on the agent (including persistent tracking of the previous stamp value) for both agents
Dependencies¶
There are going to be new dependencies on the cryptography python package that provides all the required crypto functionality. Several such libraries are already present in the global requirements list.
Testing¶
New functionality can be fully tested with unit tests.
Documentation Impact¶
New config setting for the Murano engine should be documented.
References¶
None