Support Console Log migration during Live-migration¶
https://blueprints.launchpad.net/nova/+spec/support-console-log-migration
Implement console log migration during live-migration in the libvirt driver
Problem description¶
Currently, in libvirt driver with a kvm hypervisor, console output is written to console.log. Nova responds to a get-console-log request with the contents of this file and this information is useful for debugging issues during boot process. However, during a live-migration the contents of the file in the source node is discarded.
There are two issues which play a role in this.
The new kvm process in the destination would have already started using an empty console log.
While the migration progresses the VM in the source node will continue to write to the console log.
Proposed change¶
We propose the following in this blueprint to solve this issue without depending on kvm.
Require that VIR_MIGRATE_UNDEFINE_SOURCE is not set. Instead wait for the condition that the instance is shutoff at the source.
During post-live-migration copy the console log be from source node and save in the destination node as console.log.1. If log rotation is implemented, all the rotated files need to be rotated once.
Change get-console-log function such that console.log and console.log.1 are merged in the response (within the MAX_CONSOLE_BYTES limit). It log rotation is implemented then the function needs to read as many files as it takes to fill up the MAX_CONSOLE_BYTES limit.
The source VM would get undefined by the periodic task once the database is updated with the new hostname.
Alternatives¶
Change qemu to move the file content
Stream console output to a shared location
If spec/libvirt-serial-console is implemented we can leverage on that mechanism and trigger a rotation and move it to destination.
Data model impact¶
None
REST API impact¶
None
Security impact¶
None
Notifications impact¶
None
Other end user impact¶
None
Performance Impact¶
There’s a brief window between the time the VM is activated in the destination and before post-live-migration is completed. Any nova console-log requests will return almost empty content during this window.
Other deployer impact¶
If people are using VIR_MIGRATE_UNDEFINE_SOURCE then they need to remove this option to get this feature. If this flag exists we will fallback to not having the console log migrated.
Developer impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
parthipan
Work Items¶
Change live-migration to wait for shutoff state if flag VIR_MIGRATE_UNDEFINE_SOURCE is not set.
Change get_console_log to handle rotated log files
Implement console log migration during post-live-migration
Dependencies¶
None
Testing¶
Tempest tests should be added to test that the console logs are merged in the response and catch other corner-cases.
Documentation Impact¶
We expect to have the following documentation changes:
The migration flag changes to get console logs migrated
Expected empty console log during the VM offline period in the final stages of the migration
