Gfrktyl

Neither HFS Plus nor NTFS does any data integrity checking; aka checks for “bit rot” on data files stored on the system. This is concerning because Time Machine and similar tools cannot detect if they are backing up corrupt data.

Are there tools that can detect corruption and warn me of it?

What is the best consumer strategy for keeping my data integrous? Do I have to go all the way and create a ZFS/Btrfs NAS to store my information?

Update:

After some research I found that there are 2 ports of ZFS to Mac OS X:

• https://openzfsonosx.org/


• http://code.google.com/p/maczfs/

This seems the best approach to gaining data integrity on Mac OS.

I’ve researched dealing with this issue for fairly large—20TB+—enterprise-sized storage systems and the consumer reality is this: ZFS based systems is really the only way to deal with this. If data rot is a real concern, then I would recommend having at least one other hard drive for backups that you backup. Not RAID or anything magical, but simply another external drive that is synced using a tool like rsync if you are comfortable using the command line or Carbon Copy Cloner which is basically an app that performs the same function as rsync but has a nice user interface.

I did do some searching right now and found ZFS on Linux which sounds interesting; an open source implementation of ZFS for Linux systems. If you are comfortable rolling up your sleeves and setting up Linux/Unix stuff, this could be a potential solution for a do-it-yourself NAS. But I do not have direct experience with it, so can’t speak for it’s long term usefulness in a “production” environment.

It's worth noting that Microsoft now has ReFS (resilient file system) on Windows 2012+ and Windows 8.1, which does check integrity. Furthermore, if you run ReFS on a mirrored storage space, it can automatically correct those errors by using bits from the other side of the mirror.

ReFS doesn't support all the features of NTFS, so you'll have to decide if any of the things it's missing are important for the files or workloads you need it for.

chkbit is a lightweight bitrot detection tool (OS X/Linux/Windows).

chkbit cannot repair bitrot, its job is simply to detect it.

You should

• backup regularly.


• run chkbit before each backup.


• check for bitrot on the backup media.


• in case of bitrot restore from a checked backup.

You can always manually compute checksums with md5sum and check them periodically, or you can use btrfs, which has an online checksumming feature. On the other hand, it really is kind of redundant and unnecessary since disk drives already have their own error detecting and correcting codes.

For Windows I've discovered this little program:

"DiskFresh is a simple yet powerful tool that can refresh your hard disk signal without changing its data by reading and writing each sector and hence making your disk more reliable for storage"

It does a full read/write cycle of all the sectors on a disk, so that you can prevent bitrot.

I have some SATA harddisks which I use for archiving purposes, so they are not always connected to the computer. I have them in a plastic enclosure and are kept in a drawer together with some moisture absorbing bags. If and when needed I just slide them in one of those sata disk slots I've installed on the tower. Because they sit for extended periods of time offline, I've some concerns for bitrot on those. Found this utility and tried it out on these disks. Just be prepared that it will take a long time, as it performs a full read-write on the whole disk. I usually use it once a year overnight.