2007/12/10

ext3 became read only for journal corruption

*NOTE: I'm NOT SURE about the result of following operations! please backup all your data before execute it.

-----

The EL4 kernel is wacky when it comes the the I/O scheduler locking up and and causing ext3 to remount RO. Various hardware hiccups can cause it to go RO.

And when it does.. you need to tread lightly or you could lose everything.

If your ext3 filesystem had problems and remounted read-only, I would strongly advise /against/ simply fscking it. Often times when your filesystem has gone RO, it may have been that way for 30 minutes or more. Just rebooting or fscking is a great way to lose everything (i.e. everything being dumped into /lost+found/

Instead, I would recommend:
1) rebooting into a rescue CD environment (not allowing the rescue environment to mount or fsck your filesystems).
2) Nuke the ext3 journal:
tune2fs -O ^has_journal /dev/
(possibly doing the same for other problem partitions)
3) Do a fake fsck to see the extent of damage:
fsck -fn /dev/
(after checking things out.. use "-fy" once you're sure that it's safe)
4) Rebuild the journal w, "tune2fs -j /dev/
(rerun at least once until "clean" result is repeatable)
5) Mount and check things out,
"mkdir /mnt/tmp && mount -t ext3 /dev/ /mnt/tmp"
6) Gracefully umount & reboot:
"umount /mnt/tmp && shutdown -rf now && exit"

No comments: