SANS Digital Forensics and Incident Response Blog

How to Mount Dirty EXT4 File Systems

Hal Pomeranz, Deer Run Associates

As some of you may remember, I've previously written about a technique for mounting EXT3 file system images with the read-only option, even when power was abruptly removed from the system- as is typical during forensic seizure- and the file system is still "dirty". In these cases, my technique involves using an alternate superblock, which will not have the "needs journal recovery" flag set, and using the "-t ext2" option to ignore any entries in the EXT3 file system's journal.

In the last year, however, I've been starting to see cases where I've had to analyze the newer EXT4 file system (hence my recent series of articles on EXT4 internals). It turns out that the technique I developed for mounting EXT3 file systems does not work with EXT4:

# mount -t ext2 -o loop,ro,noexec,sb=131072 ext4-test.img /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so
# dmesg | tail -1
[124897.443002] EXT2-fs: loop0: couldn't mount because of unsupported optional features (240).

So, unfortunately, our trick of using "-t ext2" to get the file system drivers to ignore the journal is not going to work for us here, because the EXT2 drivers don't recognize many of the new file system options in EXT4.

So what can we do to mount our EXT4 file systems? When in doubt, refer to the man page:

# man mount
[...]
-r, --read-only
Mount the filesystem read-only. A synonym is -o ro.

Note that, depending on the filesystem type, state and kernel
behavior, the system may still write to the device. For example,
Ext3 or ext4 will replay its journal if the filesystem is dirty.
To prevent this kind of write access, you may want to mount ext3
or ext4 filesystem with "ro,noload" mount options or set the
block device to read-only mode, see command blockdev(8).
[...]
noload Do not load the ext3 filesystem's journal on mounting.
[...]

The "noload" option looks promising. Let's give it a try:

# file ext4-test.img 
ext4-test.img: Linux rev 1.0 ext4 filesystem data, UUID=... (needs journal recovery) ...
# mount -o loop,ro,noexec,noload ext4-test.img /mnt/test/
# ls /mnt/test
backups crash lib lock lost+found opt spool
cache games local log mail run tmp

You can see the "needs journal recovery" flag in the output of the file command, so our file system is definitely dirty. But happily the "noload" option does indeed allow us to mount the EXT4 file system.

But what about EXT3? The manual page suggests that "noload" will work there as well. Unfortunately, this doesn't appear to be correct:

# file ext3-test.img
ext3-test.img: Linux rev 1.0 ext3 filesystem data, UUID=... (needs journal recovery)
# mount -o loop,ro,noexec,noload ext3-test.img /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

# dmesg | tail -1
[126955.823010] ext3: No journal on filesystem on loop0

It appears that the "noload" option does in fact cause the journal not to be loaded. But the EXT3 drivers apparently regard this as an error condition and refuse to do the mount. And before you ask, adding "-t ext2" to the command line above doesn't work either.

So at this point I was stuck with having one method for mounting EXT4 and a different, painful method for mounting EXT3. But then I got an email from Gebhard Zocher, which pointed out a clever solution:

# mount -t ext4 -o loop,ro,noexec,noload ext3-test.img /mnt/test/
# ls /mnt/test
bin dev home lib mnt proc sbinusr
boot etc initrd lost+found opt root tmpvar

Since the EXT4 drivers are backwards compatible with EXT3 file systems, you can just specify "-t ext4" and then use "noload" to mount your EXT3 file systems without mucking around with alternate superblocks. And that means we now have a consistent solution for mounting both EXT3 and EXT4 file systems. Thanks Gebhard!

Hal Pomeranz is an Independent IT/Security Consultant, a SANS Institute Faculty Fellow, and a GCFA. File systems fear him. Hal will be teaching For508: Advanced Computer Forensic Analysis and Incident Response in Baltimore, Oct 9-14.

12 Comments

Posted December 9, 2011 at 6:04 PM | Permalink | Reply

Josh

Very neat trick. Thanks for sharing!

Posted April 12, 2012 at 12:52 AM | Permalink | Reply

Canturk

Great information and excellent writing!
Thanks a lot for all your time preparing this trilogy:)
-canturk

Posted May 3, 2012 at 8:47 AM | Permalink | Reply

Abhishek Datta

Hi Hal,
Its an excellent article.
But I always get an error: mount: unknown filesystem type ''ext4'.
I am using Ubuntu 10.04.2 LTS
Regards,
Abhishek

Posted May 4, 2012 at 1:28 AM | Permalink | Reply

Hal Pomeranz

Abhishek, that's an odd error message. It makes it sound like your Ubuntu system doesn't have support for EXT4. But I know Ubuntu 10.04 supports EXT4 because I'm typing this response from my Ubuntu 10.04 system, booted off of an EXT4 file system. Have you recompiled your kernel or are you running some sort of special distro?

Posted May 5, 2012 at 5:09 PM | Permalink | Reply

Anonymous Coward

I just realized that the root user does not always respect file permissions, so even if a image is set to be read-only, you can still destroy it by running "echo > image.dd".
Are you confident that mounting an image with the -o=ro option will not alter the image file in any way?
Using "losetup -r /dev/loop0 image.dd" provides an extra layer of write protection. I haven't yet checked how this may impact performance.
Speaking of loopdevices, they are usefull to get access to a disk image through VirtualBox: Set up the loop device like described above, and see http://www.virtualbox.org/manual/ch09.html#rawdisk. Again, I don't know how this may impact performance, but I suspect it will be better than by going through shared folders.

Posted May 5, 2012 at 6:14 PM | Permalink | Reply

Hal Pomeranz

If you're really concerned about accidental damage to your image, consider "chattr +i image.dd" to set the immutable flag on the file (can be removed later with "chattr -i image.dd"). You can also "set -o noclobber" in your .bashrc, which would also prevent the sort of output redirection error that you mention.
But in general, yes, I am confident that "-o ro" works as advertised. Do you have evidence to the contrary?

Posted May 5, 2012 at 8:48 PM | Permalink | Reply

Anonymous Coward

Thanks for taking the time to anwser.
The only evidence I have to the contrary is something I read a while back, but it seems to no longer be an issue (http://www.forensicswiki.org/wiki/Forensic_Linux_Live_CD_issues#Example). Using the flags you recommended in this article with the SIFT Workstation 2.12 works great, and the checksums are the same before and after mounting.
Thanks for the tip on the immutable flag. It seems like a pain to work with, but then I guess that's kind of the point.
It would be interesting to see a discussion on whethe using mount -o ro, losetup -r or chattr i is a viable alternative to using a hardware write blocker. Hardware write blockers can theoretically fail (either pysically or because of user error), they are expensive, and they are impractical when working with e.g. laptops where the disk is not always easily accessible. The problems I see with software blocking methods in linux are user error (like getting the flags wrong) or implementation errors like those described in the forensicswiki article I referred to.

Posted May 11, 2012 at 10:04 PM | Permalink | Reply

Ikiris

Thanks for posting this! I have been trying to setup a process for recovering VMDKs from a linux snapshot and finally got down to the filesystem recovery which used your simple tricks to get past the journaling issue.

Posted December 10, 2012 at 5:34 PM | Permalink | Reply

Frank Thynne

I needed to repair a system installed as alternative boot to Windows. The system refused to boot into Ubuntu. This blog was very helpful, but I could not see how to repair the filesystem.
These steps worked for me:
# Recovering an ext4 filesystem requiring journal recovery.
# Ubuntu was hosted in Windows and grub loader could not load the root file system
# Repair steps
# Boot from an XUbuntu CD
# Start terminal and become root
# Create mount points for host system and broken root filesystem
3 cd /
4 mkdir /root.disk
5 mkdir /windows
# Mount host filesystem
6 mount /dev/sda3 /windows
# Check presence of hosted filesystems
8 cd /windows/ubuntu
9 cd disks
# Many aborted steps removed
# Mount damaged filesystem read-only and ignore journal.
34 mount -o loop,ro,noexec,noload /windows/ubuntu/disks/root.disk /root.disk
# Backup everything to host as found
39 cd /root.disk
40 tar -cvzf /windows/root.disk.tgz .
41 tar -tvzf /windows/root.disk.tgz |less
# Repair filesystem. I think this uses what it can of the journal.
62 fsck.ext4 /windows/ubuntu/disks/root.disk
# Two errors were found and fixed.
# Success! Could now restart normally!

Posted May 27, 2013 at 6:11 PM | Permalink | Reply

Joey Morin

Had great hope that your technique could help me. However, after trying all of the following, still no luck. Any ideas?
JJ
$ losetup -r -o 32256 /dev/loop0 disk.img
$ file -ks /dev/loop0
/dev/loop0: Linux rev 1.0 ext4 filesystem data, UUID=7edffc76-2b43-493d-99eb-5897ab9d47f5 (needs journal recovery) (errors) (extents) (large files) (huge files)
$ mount -o ro,noexec,noload /dev/loop0 /mnt/test
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
$ tail /var/log/kern.log
ext4: No journal on filesystem on loop0
$ mount -t ext4 -o ro,noexec,noload /dev/loop0 /mnt/test
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
$ tail /var/log/kern.log
ext4: No journal on filesystem on loop0
$ mount -o ro,noexec /dev/loop0 /mnt/test
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
$ tail /var/log/kern.log
EXT4-fs: INFO: recovery required on readonly filesystem.
EXT4-fs: write access unavailable, cannot proceed.
$ mount -t ext4 -o ro,noexec /dev/loop0 /mnt/test
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
$ tail /var/log/kern.log
EXT4-fs: INFO: recovery required on readonly filesystem.
EXT4-fs: write access unavailable, cannot proceed.

Posted May 27, 2013 at 10:01 PM | Permalink | Reply

Hal Pomeranz

Just a quick follow-up after talking with Joey. Turns out he was using an older Linux distro with an early version of the EXT4 drivers that didn't properly support the "noload" option. Works fine on recent releases.

Posted December 2, 2013 at 1:50 PM | Permalink | Reply

Benjamin Measures

> Since the EXT4 drivers are backwards compatible with EXT3 file systems, you can just specify