The Problem
A faulty disk was replaced in CentOS/RHEL 6 machine. RAID1 was rebuilt and the server was rebooted. After reboot, disk got removed/out of sync from RAID. RAID had to be rebuilt after every reboot.
RAID rebuilt before reboot.
# mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Creation Time : Fri Feb 8 10:20:51 2013 Raid Level : raid1 Array Size : 488279488 (465.66 GiB 500.00 GB) Used Dev Size : 488279488 (465.66 GiB 500.00 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Thu Jul 7 09:15:04 2016 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 1e712c78:0c3dc259:9acdafa7:d4878cfe Events : 0.2962397 Number Major Minor RaidDevice State 0 67 2 0 active sync /dev/sdaw2 1 67 18 1 active sync /dev/sdax2
After reboot /dev/sdaw2 gets removed:
# mdadm --detail /dev/md1 /dev/md1: version : 0.90 creation time : fri feb 8 10:20:51 2013 raid level : raid1 array size : 488279488 (465.66 gib 500.00 gb) used dev size : 488279488 (465.66 gib 500.00 gb) raid devices : 2 total devices : 1 preferred minor : 1 persistence : superblock is persistent update time : fri jul 8 17:53:04 2016 state : active, degraded active devices : 1 working devices : 1 failed devices : 0 spare devices : 0 uuid : 1e712c78:0c3dc259:9acdafa7:d4878cfe events : 0.2976471 number major minor raiddevice state 0 0 0 0 removed 1 67 18 1 active sync /dev/sdax2
# cat /etc/mdadm.conf array /dev/md1 level=raid1 num-devices=2 metadata=0.90 uuid=1e712c78:0c3dc259:9acdafa7:d4878cfe devices=/dev/sdaw2,/dev/sdax2 array /dev/md0 level=raid1 num-devices=2 metadata=0.90 uuid=5e7743f6:b2efa5e2:7a28469a:06cda6cd devices=/dev/sdaw1,/dev/sdax1
The Solution
initramfs was not picking the latest configuration from /etc/mdadm.conf.
Note: initramfs had to be recreated. Also, the procedure requires the system reboot.
1. Check “mdadm –detail /dev/mdX” (mdX will be the RAID device name) and make sure device state is “clean” and disk is in “”active sync”” mode
mdadm --detail /dev/mdX
2. Take backup of existing initramfs:
# mv /boot/initramfs-2.6.39-400.264.13.el6uek.x86_64.img /tmp/ ### example kernel, version may differ
3. Recreate initramfs with dracut:
# dracut -f /boot/initramfs-2.6.39-400.264.13.el6uek.x86_64.img 2.6.39-400.264.13.el6uek.x86_64
4. Reboot the system.
# shutdown -r now