Long-term storage and backup of digital music files

Discussion in 'Audio Hardware' started by frank3si, Feb 7, 2017.

  1. frank3si

    frank3si Forum Resident Thread Starter

    Blackwood NJ USA
    Earlier today I was reading an article in the latest issue of Pro Sound News titled "Audio Archiving: Taking the Long View." It essentially was covering long-term archiving of masters and projects. Its main conclusion is that linear tape - or LTO - is currently the "first choice for archival of digital assets."

    That line was immediately followed by a quote from Greg Parkin, director of digital solutions for Iron Mountain Entertainment. "The problem with hard drives," Parkin said, "is that they don't spin up in 10 years."

    Now I realize that Parkin's intent may have been to send a shiver down the spine of professional media entities and drive them to Iron Mountain for solutions like LTO archiving, but it gave me - an advanced music consumer - a scare as well.

    I have a few TBs worth of audio files - purchased hi-res releases, vinyl rips, collector recordings, CD rips, etc. - pretty much all in FLAC format. I got started with 1- or 2TB drives. I now have 4TB drives connected to my PC, with the original 1- and 2-TB drives now acting as backups to their original content which now resides on the 4TBs. Anything on the 4TBs connected to the PC exists on a 1- or 2TB drive stored both in my office closet as well as offsite. In other words: three copies of everything, stored on and off site.

    To me, this seems like the most rational and cost effective approach to backing all this up. Still, seeing Parkin refer to 10-year-old drives as candidates that "don't spin up" is kind of alarming. I checked my oldest 1TB drives and they go back to 2009. That's coming up on eight years. True, they are now just used as backups, with their content I regularly access now living on the 4TBs, but if you doubt your backups you're not getting the peace of mind that all this backing up should bring :agree:

    I'd be interested to see any thoughts about this, and if there is any known "common wisdom" that a 10-year-old drive should not be expect to spin up. I'd also like to know if others are backing up by a means other than the hard drives so many of us seems to rely on, or if there are new backup technologies (LTO, even?) anyone is aware of that may eventually make their way down to a consumer level to backup FLAC content.

    JimmyCool likes this.
  2. roughdiamondnickel

    roughdiamondnickel Well-Known Member

    Phoenix, AZ
    A backup drive that is not regularly used should spin up in 10 years, especially if it's just sitting there in the closet. I think he's referring to something that is continually plugged in.
  3. whaiyun

    whaiyun Forum Resident

    Well that's good news because most hard drives now are all solid state and don't have spinning platters. Just have multiple backups or RAID two HDDs if you're super paranoid.
    PearlJamNoCode likes this.
  4. MrRom92

    MrRom92 Forum Supermodel

    Long Island, NY
    Solid state drives are even worse for long term archival reliability

    I really wouldn't worry about hard drive failure if you stick to popular and well known reliable drives. In the case that they don't want to function in 10 years, there are professionals who can transplant the platters into a unit with a working motor or do other tricks to get your data back - again, operating under the assumption that the drive is on functional after a decade cold storage.

    Replace offsite backup drives every 10 years, RAID in your active drive setups, and I don't think you'll have a problem.

    There's always M-Discs in a safe or bank vault if you don't mind 4.7 or 25gb media and want something foolproof. Those will outlast everything else so long as you've still got a functional DVD or blu ray drive to read them with.
    Dave112, Mike-48, Grant and 1 other person like this.
  5. Callipleura

    Callipleura New Member

    I think you are doing all you can as a consumer. 3 copies with one off-site.

    I would be more worried with the disk losing polarity over time as it will have trouble reading the magnetic fields which get interpreted as 1's and 0's then anything rusting up... lol... You can write the HDD's at least once a year to strengthen the polarity of all of the alternating fields with something like DiskFresh for long term storage.

    I consider a HDD old after 3 years and give them away to people who don't have one or are too cheap to back-up - excuse to buy bigger HD's for me though. :)
    Backblaze has statics when drive start to fail over time:
    What We've Learned from Running 61,590 Hard Drives in Our Data Center
  6. Cherrycherry

    Cherrycherry Forum Resident

    I have an interest in this thread. I have a large digital library stored on three separate hard drive units. I believe that I have enough redundancy (two of the HDD are actual RAID units, well Beyond RAID. I use DROBO products). One hard drive dying won't kill me and I should be able grow my collection or mature it to the next storage medium comfortably.

    Since I know next to nothing about LTO, what would it cost one of us to back up our 40/50/>60,000 digital collections?
    Doing a little googling and I am seeing prices of 4-6 thousand dollars. Is this even practical for the home audio guy/gal to get into? expected lifespan of the tape is 15-30years, so there is that.
    The FRiNgE and MrRom92 like this.
  7. MrRom92

    MrRom92 Forum Supermodel

    Long Island, NY
    If you don't need to have the latest and greatest, you can get a screaming deal on a retired drive for an older LTO standard. The tapes are compatible by at least 2 generations I believe. I think all you miss out on is capacity per-tape but the tapes themselves tend to be pretty cheap.

    Not worth trading one form of magnetic media for another IMO. I don't see LTO being vastly more reliable, and probably doesn't end up being cost effective unless you have more than 8-10tb of data total.

    My flac library weighs in at 1.5tb. If I were to back them up to 100gb M-disc media, it would take only 15 discs. I haven't done the math but I'm betting at that scale it's still cheaper than LTO (unless you got a reaaallly good deal on an LTO drive) and I'm betting I'll have much better luck finding a working blu ray drive than a working LTO drive if/when I actually need to read them. And that's non-magnetic media that's proofed against basically every disaster that could possibly happen.
  8. SKBubba

    SKBubba Forum Resident

    The op is doing about the best that a home audio enthusiast can do with three copies, one offsite. Every hard drive will fail eventually. Expect to replace them every five years or so. Also RAID alone is not backup.
  9. serendipitydawg

    serendipitydawg Dag nabit!

    Berkshire UK
    How timely that this thread has appeared !

    I am in the process of doing a backup of my music files. Something that instantly takes me out of my "comfort zone", despite having owned a PC for more than 20 years.

    I am using Microsoft SyncToy to synchronise two identical 2Tb drives and was instantly out of my "comfort zone" when I realised I had forgotten how to use SyncToy ( so long since I did the previous backup).

    I got progressively baffled by each re-learning step and eventually re-formatted the target drive (the other 2Tb drive contains my music library) and re-started from scratch, rather than figure out the multitudinous errors that SyncToy was throwing at me.

    Any recommendations for idiot-proof software that I could test on myself?:)
  10. mj_patrick

    mj_patrick Forum Resident

    Elkhart, IN, USA
    I think that man's statements are more about his personal preference than fact. Sounds like he got burned in the past. Maybe his company bought poor quality hard drives. Maybe he didn't have multiple back up solutions. Who knows.

    For a moderately sized library like mine I really think LTO is absolute overkill.

    I've been backing up to multiple hard drives including at least one offsite one. I have been considering looking into cloud storage, but that would be an additional layer of safety to what I'm already doing. And frankly that's probably overkill as well.
  11. BuddhaBob

    BuddhaBob Forum Resident

    Erie, PA, USA
    Any device can and will fail. Redundancy is the key to never losing your data, along with keeping a redundant backup in a separate location. Data recovery can easily cost thousands of dollars. Disks are cheap. Make copies and keep one away from the rest.
    andrewskyDE, johnny q and JimmyCool like this.
  12. SKBubba

    SKBubba Forum Resident

    In addition to local backup (using robocopy, the command line version of synctoy and even more complicated) I use Crashplan for offsite cloud backup.

    Their cloud backup service costs a little, BUT their free backup utility software can be used for local/network backups. The free mode can also be used to setup a private mirror over the internet, so you could, for example, setup reciprocal offsite backups with friends/family.
  13. Erik Tracy

    Erik Tracy Meet me at the Green Dragon for an ale

    San Diego, CA, USA
    Just my 2 cents worth from the Enterprise world, but 10 years for any given device is expecting a lot.

    Not just from a reliability standpoint, but support and overall system operability/compatibility.

    Remember USB 1.0 or even USB2.0? If you bought a USB1.0 drive are you really going to keep that for 10 years?

    In 10 years, will the OS that you are on even support that? Will there be drivers to support that OS? As technology advances, do YOU really want to be stuck with that type of performance?

    "Data Migration" is an constant on-going task in data centers as storage technology races on in density and performance and we were always planning around 5 year cycles to recap storage solutions to save power, cooling, footprint, support issues.

    Even with LTO. We did a data migration from LTO-2 to LTO-4 so as to take advantage of newer drive speeds and densities and not lose support.

    There is much more to consider, even in the home user market, than just how long a device 'can' operate.
  14. Cherrycherry

    Cherrycherry Forum Resident

    MrRom92 likes this.
  15. MrRom92

    MrRom92 Forum Supermodel

    Long Island, NY
    I agree. I really like these. It's the only thing I would consider to be foolproof.
    The only downside is the cost and small density (about $20 for a pack of 5 dvds which are typically 4.7gb a disc, that's about $1 per GB - maybe not the most cost effective although there are the higher capacity Blu Ray discs, still not close to the density or cost/gb for magnetic media)

    The DVDs do need an M-Disc capable burner to write to them (higher powered laser) but these are cheap now. I think all of LG's drives are capable of writing them.
    The Blu Rays are capable of being written in any blu ray burner.
    And the discs are capable of being read in any DVD / Blu Ray drive like any other disc.

    Considering these are practically fireproof on their own, I think a stack of these in a nice safe is about as protected as you could possibly get from irrecoverable data loss.

    I don't have any files on these (yet) or anything but I may decide to use them for photos or truly irreplacable documents. I have a stack of the DVDs and they don't do me any good by going unused. I have used them for re-writing copies of home videos which were originally provided to us by the videographer on basic DVD-R media. The original discs have since become unreadable in some cases. But these have them preserved indefinitely and I can still pop them in the DVD player to watch them.
  16. SamS

    SamS Forum Legend

    Considering LTO for consumer audio file long term storage is crazy talk. LTO was a preferred method for enterprise-level storage 10+ years ago.

    If you really want to augment your HDD (including onsite/offsite) storage for disaster recovery purposes, just use something like Amazon AWS Glacier storage. It's about $5/month for 1TB, and no fees to send up new files, only if you need to retrieve them. And AWS cloud storage is enterprise-grade, and you don't fool around with maintaining LTO equipment. Keep in mind, you're not streaming from the cloud, just using it as disaster-recovery storage.

    And before the folks come in and say "I don't trust the cloud for anything"... keep in mind that all your money in your bank account, retirement, etc. is essentially stored "in the cloud". Your music files will be just fine :)
    Stromung likes this.
  17. Cherrycherry

    Cherrycherry Forum Resident

    AWS, Amazon Web Services, Cloud Archive Pricing – Amazon Glacier – AWS
    If I am reading that right:
    About $4.00/month, $48.00/year for approx. 1TB.
    And if I needed to retrieve the 1TB, about $10.00 fee.(using Standard speed)

    I want to check the fine print to see how/when the pricing might change in the future.
    NOS300B likes this.
  18. SamS

    SamS Forum Legend

    Amazon S3/Glacier pricing goes down over time, not up.

    Oh, and the federal government uses it as well ;)
    Amazon Web Services Achieves FedRAMP High Authorization | AWS Government, Education, & Nonprofits Blog
  19. Ron Scubadiver

    Ron Scubadiver Forum Resident

    Houston TX
    For all my data backups whether music, video or photos I use pairs of drives with the same capacity. These are not raided because with raid if you make a mistake it instantly is carried to the other drive. The biggest threat to digital assets today is cryptolock malware.
  20. RiCat

    RiCat Forum Resident

    CT, USA
    HDD's are cheap. I keep my music on a raid 1 NAS server and a copy on a separate drive as the backup. If any of the 3 drives shows any errors they all get replaced. Simple and for me cost effective. Risk management is the name of the game. I have friends who have lost photo and music data from off site storage services when they went under. Nothing is perfect but with so many easy to implement solution choices we can all find a method to last our lifetimes.
    TonyCzar and BuddhaBob like this.
  21. Erik Tracy

    Erik Tracy Meet me at the Green Dragon for an ale

    San Diego, CA, USA
    Yes, the cloud is great and good, but it still depends on a reliable data link.

    I've already had 1 disconnect this morning and a couple of actual Spectrum outages over the last couple of months.

    My download speed is not nearly fast enough for me to consider depending on a data link for storing/restoring data.

    If it works for you - that's cool. But I refuse to depend on a chain of services which represent a serial set of single points of failure for access to music.
    HotHere, SandAndGlass and BuddhaBob like this.
  22. SamS

    SamS Forum Legend

    What does it matter if it takes a week to download/retrieve your music, after your house burns down?
  23. Bingo Bongo

    Bingo Bongo No music, no Life

    Yep, I keep one at work, and update them every half a year or so (if I've added new music)
    SandAndGlass likes this.
  24. Erik Tracy

    Erik Tracy Meet me at the Green Dragon for an ale

    San Diego, CA, USA
    Oh c'mon man.

    I could paint all sorts of doomsday scenarios for the cloud too...it's all about the level risk you want to assume.
    56GoldTop and Grant like this.
  25. SamS

    SamS Forum Legend

    I wasn't really exploring the doomsday scenarios... rather pointing out the fact that for "long term storage", it shouldn't matter if it takes 2 hours to download from a cloud back up, or 2 weeks. You're only retrieving the files in the event of a serious disaster, it's a once in a lifetime event.

Share This Page