Long-term storage and backup of digital music files

Your link broke for me. Below should work.
Cars, Music and Nature: Practical ideas about backing up computer files

In response to your blog, I thought that thread wasn't about "backing up data" per se. My focus in here is looking at archiving, the long term storage for the future. In deference to the opposing view of what constitutes a hardy back up plan for a music/data collection.
Perhaps, I am having a different discussion, and that is frictional.

 

What's causing "friction" is forcing "digital" and "long-term" in the same question.

The CD-R (CD-Recordable) spec itself was first published in 1988. It's not even 30 years old.

People who have some familiarity with the history and shelf-life of digital storage are trying to be helpful, but you keep insisting on an answer to a question that has never been answered: "What digital storage/archive system is reliable FOREVER?"

 

oh, ok. I may be tone deaf. Thanks for the feedback.

 

Solid State Drives aren't good for long term storage. The data storage in sold state drives is based on charges that are trapped in memory cells. When you unplug a SSD or flash drive and put it in storage the charges in the cells will leak over time. When the charges leak enough the data goes poof and is gone.

How long a SSD can hold the charges and keep the data depends on many factors. The quality of the memory cells, how new the drive is (cells that have been used a lot won't hold the charge as long), storage temperature, and other factors. How long a SSD will hold data in storage is not a sure thing. Some people say 10 years, some say less, some say more. In some cases it can be much less. A SSD or solid state flash drive is not something I would trust for long term backups.

Here's some articles about SSDs and long term storage:
Death and the unplugged SSD: How much you really need to worry about data retention
Hard-core data preservation: The best media and methods for archiving your data
SSDs can lose data in as little as 7 days without power - ExtremeTech

 

I have only about 400 CD's nearly all ripped to FLAC and a few hundred more downloads in FLAC all stored on my pc's solid state drive. I bought a hard drive only for music and have them all backed up on it and I also have an older hard drive with most of my music on that too.
Family photos are of more concern to her who must be obeyed who to be honest wouldnt care if all my cds got rot and the hard drives failed as long as the photos are saved! Its a huge issue to us music fans and whether they know it or not to most members of the public who couldnt tell FLAC from fluff but would hate to lose the precious photos.

 

Yeah, I changed the blog article title.

The thread title is

"Long-term storage and backup of digital music files"

I think that my blog post is relevant to this thread. You may be conducting your own separate discussion.

In one of your posts you said

"I am looking thirty years down the road and retirement.
I don't want to live without all the music I have acquired, and won't easily find online.
For some of us, these are hard to find or out of print CD, for others there are one of kind needle drops painstakingly made. For others this is homemade or live rcordings.
And for others, they could be family photos and videos(very unique and personal)."

I have similar needs. I know why I use my backup method. I've evolved my methods over 30+ years and they have been successful in getting me through theft, malware attack and hardware failure.

You seem fixated on finding a neat solution for the next 20-30 years. Several people have tried to tell you that it is not realistic. Technology moves along. Businesses change their offerings and some just plain fail. You don't seem to have a real backup plan. You need to get a plan in place to protect you in case you have a problem tomorrow, next month or next year.

 

I once suffered, what I considered to be, catastrophic data loss when a couple of drives I had failed. I did not have a backup plan in place or any sort of redundancy. Although I did not have a massive flac library at the time, I did have many one of a kind recordings you'd be hard pressed to find anywhere else, friend's bands and such. Much more catastrophic were the countless photos and videos of never see again.

Further compounding my stupidity of not having any redundancy or backup, the 2 drives were linked together in some RAID-0 like configuration although it was some non-standard filesystem proprietary (I think) to the Network Attached enclosure I used. I think some storm knocked it out and the enclosure stopped working.


I knew I still had my data on those drives so I didn't do anything drastic, but I had no way of accessing them however hard I tried. I just put the whole enclosure in a closet and forgot about them. I'd tinker with them on and off every now and then, like when you go to the refrigerator repeatedly hoping something new will have materialized in the time being. No such luck. Those drives held my regularly maintained and migrated digital existence going back to the early 90's. I had to start fresh, a digital newborn with no bits to my name.


8 years later, not sure what I did differently, but in one of these pathetic tinkering sessions I was somehow able to get this thing to actually mount and recognized over a network connection. I sped over to Best Buy for a new USB drive, rushed home, and started dumping the files while I still could, hoping these ancient things wouldn't give out mid-transfer. Many, many hours

I managed to recover every file, with the exception of one or 2 things of no significance which refused to transfer, maybe due to bit rot or whatever. The important thing was that all my documents were back.




Those 2 drives are littered with smart errors now but they're still operating in that enclosure daily as a server, with a mirror of my FLAC library. I figured it was only sitting around doing nothing for all that time and now that my important files are off of it, might as well get some use out of it till it really gives out again. It's been a few months but it's still going, and while the setup is pretty flakey, I suppose it does now count as one extra level of redundancy for my music library specifically, although I do not expect or intend to use it for that purpose. Really I just use it to load up things in Foobar in the next room. That's a non-critical usecase, but I think I finally got my money's worth out of this thing.



The takeaway from this should be that I got lucky. Keep a plan and stick to it. Just keep a backup of your drives (however many for your raid array, preferably formatted to something normal) in cold storage, somewhere else. Keep the backup drives up to date and rotate them regularly. For anyone not dealing with enterprise levels of data it really doesn't need to be more complicated than that.

 

haha, what a feeling of victory that must have been when you could access those drives again Glad that you were able to get those files back.

 

Oh, you have no idea. A huge relief and a total joy to see/hear some things I thought were gone for good.

 

Thanks for responding Robert.

Anomalies would probably be a better term for the experience I had. Synctoy kept telling me there was not enough space to backup my library, whne the disks are identical size and model (Samsung 2Tb).

As I said in my post, I re-formatted the target drive. I also tried a different piece of software, Free FileSync. After 36 hours it came up trumps. When I have time I will compare drives in Free FileSync to check that my my backup is valid.

 

My digital library backup protocol consists of the following:

MAIN MEDIA HD
This is the primary HD that I listed to music off of at home, connected to a Mac Mini that is my music server (iTunes). To confirm data integrity every music file has an MD5 checksum, and once a week the files are scanned and recalculated, and any differences in checksums are reported to me so I can investigate and retrieve a good copy from one of the backups.

MEDIA BACKUP
This HD is connected to the same computer as the main HD, and an incremental backup is run every night at 2am.

MEDIA OFF-SITE BACKUP
This HD sits on my desk at work (and is in use, I listen to music off of this HD while at work). I bring it home every two weeks or so to run an incremental backup. It is only in the same physical location as the other two HDs for one night (but in theory I should take the other backup out of the house while the off-site drive is here).

This is a pretty safe setup, but it does rely on the same type of backup (HD) for all three. While I doubt all three drives would fail at the same time, I do plan on utilizing Amazon's S3 service at some point to create a fourth cloud-based backup as well, as a long-term archive solution.

Now, I have ripped a fair number of my physical CDs but far from all of them, and at the moment the unripped CDs have no backup. I'm hoping to eventually rip the entire collection, but given the size of my collection and the number of discs I rarely listen to (but don't want to get rid of) I'm not sure if I'll ever actually get them all ripped.

 

The record labels all know this. We are moving toward a stream world where the consumer will eventually own nothing tangible. They will have ultimate control on what we can hear, when, and how. The casual listener doesn't care. They are happy with what they can hear today, and don't remember history.

Now, from my experience, the best way to guard against hard drive failure is to spin them up at least every six-to twelve months. keep them cool, and avoid shock and vibration. Unless one is ridiculously wealthy, there's no way to transfer everything you have to 30 i.p.s. magnetic tape and store it in a temperature-controlled, underground warehouse. So, hard drives are the best we have. You could spend the next two years making backups to M-Discs, but i'm wondering about the future availability of the disc drives.

My advice: if you can at all afford the space, don't get rid of your CDs, tapes, and vinyl. If you are like me and keep all of your music on hard drives, make several backups and spin them every few months.

 

Ahoy BrewDrinkRepeat !

What backup software do you use?

 

When is the last time you went shopping for a hard drive?the spinning platter drives are all over the place!

 

Interesting opinions and comments from all!

Grant, I'm pretty sure I saw in another thread some time ago comments along the lines of there being no point in starting up stored backup drives on any kind of schedule. May I ask your thoughts on why they should be spun up?

 

There are so many things wrong with this idea! First, it is cost prohibitive for most people. Second, 7 1/2 is inadequate. Three: you better make several copies and store them off-site.

 

I have/had two hard drives that were all solid state ones and nowadays I avoid to buy these things.
My first one (Fujitsu Siemens from 2009) still runs but I only use it very seldom these days.
The other one is a iOmega hard drive (2TB, bought it around 2010 or so) and the content/partitions died not long after, I don't have any access on this stupid thing anymore since 2013.
Basically every solid state hard drive is a no-go for me today. While the Fujitsu was turning off automatically (when I shut down the computer) I often forgot to turn off the iOmega because it had a switcher instead. Guess I wasn't that innocent after all.

[Edit:] Oh wait. My old hard drives aren't SSDs but external hard drives where you couldn't open the casing. Is there a specific name of these kind of hard drives?

 

A periodically activated drive is better than one that just sits. Parts do wear out faster if they don't move.

I've managed to keep eight and nine-year-old drives spinning simply by powering them up every few months. I keep an internal music drive in my computer tower that is on 24/7, and it has been running faithfully for five years.

Another trick is to use 5400 RPM drives Western Digital green drives. They run cooler.

 

OK, crazy! Most of us have CDs from the early 80s that are as stable as when they were manufactured.

 

There is a difference in manufactured CDs vs. CD-R. Big difference. Manufactured CDs are literally pressed. CD-R are made by a laser etching pits into a chemical dye that changes over time.

You do realize that analog tape is not an economic or practical option for most people, especially those of us with huge music collections, right? The OP wants to know opinions about digital, so mention of tape is just not relevant or viable, and almost amounts to threadcrapping. Now, if you want to distribute some of your wealth to us so we can all backup to 1/2 analog tape, i'll PM you a way to deposit the money into my bank account.

 

I know it's not quite the same thing but I do have some SD cards with veritably ancient data still on them from the early 2000's. Not that I would normally expect that of a flash based storage device, but more that it was a pleasant surprise.

 

Reading your story, I could just feel your joy and stress when it started running again. I can feel you quickly rushing to Best Buy to buy a new drive, hoping that it would still be running when you got home, and the sweat on your forehead waiting for it to copy.

Now, what you need to do is make one more copy!

 

Some file syncing tools need enough free space on the drive to hold both the total size of the files being changed along with the total size of the files being deleted. So if you have 50 GB free on the drive but are copying over 60 GB of changed files the software will think there is not enough room on the drive, even though there is technically enough free space on the drive. One reason for this is that the sync software is doing a "robust" style copy operation. Robust in the sense that if power is lost during the copy operation that no data or files will be lost. Robust in the sense that the software doesn't actually delete the file until it has verified that the changed version of the file has been successfully copied over. Once the software knows the file copy was successful it will delete the old version of the file, or put the old version in the recycle bin.

One way to get around that is to have the software first delete the files on the destination drive that need to be changed/updated and then make sure the recycle bin is empty. That will make enough free space to do the regular sync.

Another way to get around this problem is to do the sync in multiple steps. First sync 25% of the directories, then the next 25%, etc. Do it in smaller chuncks.

 

CD-R is too small, too slow and too expensive to be a viable backup media.

 

How's storing stuff on BluRay? Some say the lifetime of these discs is longer than anything else but I'm not sure about that.