The Problem
System cannot boot-up properly with below logs:
[ OK ] Started File System Check on /dev/xvdd. [ OK ] Started File System Check on /dev/xvdg. [ OK ] Started File System Check on /dev/xvdb. [ OK ] Started File System Check on /dev/xvdc. [ OK ] Started File System Check on /dev/disk/by-label/BOOT. Mounting /boot... [ OK ] Mounted /boot. [FAILED] Failed to start LVM2 PV scan on device 202:65. See 'systemctl status lvm2-pvscan@202:65.service' for details. [FAILED] Failed to start LVM2 PV scan on device 202:81. See 'systemctl status lvm2-pvscan@202:81.service' for details. [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-DBORA.device. [DEPEND] Dependency failed for File System Check on /dev/disk/by-label/DBORA. [DEPEND] Dependency failed for /u01.
It shows that a LVM2 PV scan is failing to start which also timed out waiting for device dev-disk-by\x2dlabel-DBORA.device. After trying to disable lvmetad and rebuilding initramfs and then performed reboot, system boot up is stuck with below logs:
Starting Availability of block devices... [ OK ] Started Availability of block devices. [FAILED] Failed to start Activation of LVM2 logical volumes. See 'systemctl status lvm2-activation-net.service' for details. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Started Cellirqbalance. Welcome to emergency mode! After logging in, typGive root password for maintenance
After entering root password to show the details of systemctl status lvm2-activation-net.service:
lvm2-activation-net.service - Activation of LVM2 logical volumes Loaded: loaded (/etc/lvm/lvm.conf; bad; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2020-06-29 22:10:58 EDT; 2min 55s ago Docs: man:lvm2-activation-generator(8) Process: 6457 ExecStart=/usr/sbin/lvm vgchange -aay --ignoreskippedcluster (code=exited, status=5) Process: 6447 ExecStartPre=/usr/bin/udevadm settle (code=exited, status=0/SUCCESS) Main PID: 6457 (code=exited, status=5) Jun 29 22:10:58 hostname systemd[1]: Starting Activation of LVM2 logical volumes... Jun 29 22:10:58 hostname lvm[6457]: Refusing activation of partial LV VolGroup/LogicalVolume. Use '--activationmode partial' to override. Jun 29 22:10:58 hostname lvm[6457]: 4 logical volume(s) in volume group "VolGroup" now active Jun 29 22:10:58 hostname systemd[1]: lvm2-activation-net.service: main process exited, code=exited, status=5/NOTINSTALLED Jun 29 22:10:58 hostname systemd[1]: Failed to start Activation of LVM2 logical volumes. Jun 29 22:10:58 hostname systemd[1]: Unit lvm2-activation-net.service entered failed state. Jun 29 22:10:58 hostname systemd[1]: lvm2-activation-net.service failed.
The Solution
The logical volume (/dev/VolGroup/LogicalVolume) cannot be activated because of a missing physical volume associated with the volume group. Looking at /etc/lvm/archive/VolGroup.cfg:
pv2 {
id = "s1sKXM-29tw-ArT0-7sKW-C6mE-X8ES-QUbOor"
device = "/dev/xvdf1" # Hint only <<<<<
status = ["ALLOCATABLE"]
flags = ["MISSING"]
dev_size = 83881985 # 39.998 Gigabytes
pe_start = 2048
pe_count = 10239 # 39.9961 Gigabytes
}
}
Physical volume /dev/xvdf1 is marked as missing causing the logical volume VolGroup/LogicalVolume activation issue which resulted to start-up failure of lvm2-activation-net.service.
Using vgextend, restore devices marked as missing. Once a device is marked as missing (such as if it cannot be found during a scan and the volume group is activated in "partial" mode), it may need to be manually restored. The vgextend --restoremissing command can do this.
# vgextend --restoremissing VolGroup /dev/xvdf1 Volume group "VolGroup" successfully extended
Activate Logical volume:
# lvchange -ay /dev/VolGroup/LogicalVolume [ 1552.936304] systemd-fsck[11319]: DBORA: clean, 182121/3932160 files, 10770808/15728640 blocks
Check if lvm2-acivation-net.service can be started without errors:
# systemctl start lvm2-activation-net.service # systemctl status lvm2-activation-net.service
Reboot the system:
# shutdown -r now