What is bit rot?
April 18, 2017 in digital music by Dan Gravell
Like any pursuit, music library management has its own set of inconvenient truths. Stuff you know is or might be a problem, but you'd rather ignore because they sound too difficult to fix. Here's one I've been thinking about more recently: bit-rot.
Bit-rot is also one of those things that sounds suspiciously illegitimate; the kind of thing that exists in a marketeer's head but never really appear to be a problem. Bits are bits, right? Well, yes. Trouble is, how can you be sure the bits you thought you had are the bits you've got? Time to ground this article in some realities...
Bit-rot is a thing
Unfortunately bit-rot is not one of those fantasies that can be ignored.
Your music library is a collection of files; these are stored as a stream of data on your storage device (your hard drives or SSDs). This data is a stream of bytes, encoding the structure of each audio file and including the audio data that you hear when playing a file. It also includes the metadata I talk about so much on this blog.
If music software like bliss or iTunes or whatever can create or change that data then it stands to reason that there's potential for unintended corruption of the data.
The causes of bit-rot (there're more than you might think)
So that's our first cause. Probably not the first cause you might think about or gets discussed (more on those in a second) but a real one. The gradual chip-chip-chipping away of the validity of music files by software.
However I'm including it in this discussion because it still has the potential to introduce silent corruption into your music library, leading to problems later.
This generally only occurs when the software is errant in some way; either a bug has been introduced that compromises how the audio file is written, or some other environmental issue means the software doesn't function as it should (e.g. multiple pieces of software working on the same file).
Remember it's not just music software which has a hand in editing your files; but other software too; anti virus and security software, lower level operating system code, file synchronisation software... there are many possibilities.
That said, probably the most common cause of bit-rot you see discussed is at a hardware level.
This is real. We are already seeing examples of decay of data stored on physical media such as CDs and the same applies to media stored inside computers; the magnetic or solid state media we use to store our music collections.
What are the effects of bit-rot?
Broadly speaking; corruption. Your music files' data is laid out an structured on your storage devices according to the way your operating system's file system has chosen to store it. Depending on the location of the bit-rot, should it occur, you can have different effects - either the loss of the file in the case of where file system level data is corrupted, or corruption internal to the music file.
Let's concentrate on the latter, as this is a blog about music libraries. Corruption here means the altering of the data in a non-intentional way, and could include:
- Altering of bits in the audio stream, leading to audio artifacts ('clicks' and 'pops').
- Loss of audio file structure, meaning software cannot work on it correctly.
- Corruption of metadata, meaning the metadata is changed or lost.
All this sounds pretty depressing. So how can we tell if we've got a problem, and how do we fix it?
Solutions
Solutions exist at multiple layers. Remember how I was saying corruption could occur in the files, or at a lower level in the file system? Well the solutions follow the same pattern.
At the lower level you have hardware and firmware level solutions that can provide protection. Hard drives themselves have a level of error checking built in, and indeed most of these types of issue are fixed at this level. But not everything gets caught.
One level higher up is the file system. Newer file systems like ZFS, Btrfs and ReFS incorporate integrity checking as a second level of redundancy.
Then you have high level tools, and this is where we begin to discuss music-related concepts. These types of higher level tools become especially useful when dealing with bit-rot caused at the software level.
FLAC, for example, has a checksum facility. This is a small piece of data that is generated from and gets stored with the audio. Software knows that if it generates a new checksum from the data and it doesn't match, something has changed in the audio stream. I wrote up a quick tutorial on testing FLACs here.
Unfortunately such forward-thinking is not shared and the vast majority of file formats do not have checksumming built in. This means you have to rely on separate software and data stores to ensure music file integrity, either by continually testing the files or storing checksums outside the data and continually generating and comparing.
So yay! for FLAC. Again.
A word on backups
I often write how important backups are; how they are your ultimate backstop.
They are important. But their usefulness can be limited in the case of bit-rot.
It only take a moment's thought to realise why; bit-rot and the associated corruption of data could be silently stored as part of your backup process. Your backups themselves could thus be corrupted.
This means to find the "last good" backup could take a long time and make recovery from bit-rot a difficult one. Fortunately, in the case of physical causes of bit-rot, the effects tend to be fairly contained.
I hope this quick primer has been a useful introduction, I've scattered some links in the paragraphs above to provide more information. A feature to check the integrity of music files has been suggested on the bliss ideas forum, please support it by clicking "Vote" if it would be useful for you!
Thanks to sebilden and ppressurewash for the images above.
