DON'T LOSE YOUR FILES! (like I almost did)

That is the hard drive which came very close to eating 2 months of my work.

Firstly, if you don't wanna read all this text, then just listen to my podcast style version on YT here:

If not, then read on good sir or madam.

The drive  just died on me this morning.....no mounting....no nothing, barely any warning.

Me when confronted with a hard drive that simply "stops existing".

Just a few small read errors in the day leading up to it dying, errors that I assumed were corrupt files, not a corrupt drive.

And just to let you guys out there know what a "non mounting" hard drive usually means....it means all your data is gone.

There are companies that specialize in scavenging files off damaged hard drives.....but we're talking thousands of $$$ and then you're probably going to get given back another hard drive full of a bunch of mish-mashed damaged files that aren't named properly.

Trust me on this....when a drive breaks....just accept that the data on that drive is GONE FOREVER.

So bearing that in mind any of us working on projects we care about on a computer....we need to have a damn good backup system.

How much work do you wanna lose? 1 hour? 1 day? 1 week?

The answer is NONE.

So luckily for me I had learnt my hard lessons...and I had an active backup system in place, so when my traitorous hard drive failed I just switched to my backup drive and began restoring all the lost data (about 1.5 TB).

Me when I have my backup system in place

But for some of you young guns, you may not have experienced the crippling loss of a sudden HDD failure and the realization that months of work have simply STOPPED EXISTING. If you're one of those people, here's my advice.

Get a multi pronged backup system going.

When I used to work for video production companies who had hundreds of TB in video data, they would usually have a massive raid server room. The raid was attended to by some external IT company guy once a month or whenever there were issues (which was often).

What IT guys do in the locked server room

For those who don't know, most companies who deal with lots of data they cant afford to loose usually store it on a RAID array, which is basically a bunch of hard drives connected together using a really fast networking system, the multiple hard drives ensure that even if 1 drive dies (which it will) then there are copies of all that data on other HDD in the RAID, then the system will automatically restore the missing data.

RAID servers work something like this....blah blah blah blah -who cares?

The other benefit of RAIDS is that they read/write data super fast, usually a lot faster than a normal single old fashioned HDD, so when you hook up a bunch of computers to the RAID network it can feed data back and forth at high speeds (good for multiple people editing video files). 

But here's the thing with RAIDS....they suck, they are ridiculously complex and expensive. They are prone to bizarre complicated failures in the way the system is run, the hardware to network them into your computer is also rare and expensive.

You only want a RAID system when you're working with a big team with huge amounts of raw data that's changing constantly. Such as a video studio environment.

Now for gamedev I would never recommend a RAID system unless you're making serious $$$ and can have ur own IT staff and spend triple your normal computer budget on specialized RAID networking hardware. In that case you wont be reading this, you'll be too busy humiliating your poor I.T. staffer.

Yes most of them are like this....it's not their fault.

So back to the life of a solo gamedev or micro team (3 person team or smaller).

STEP 1 -  immediate daily cloning

That's right, cloning is the way of the future.

This is when you copy your entire working HDD to another HDD, so you always have 2 working copies of your data, I used to do this manually folder by folder in an old job that had no IT department (because they were cheap and the company was run by a non-technical person).

But doing this process by hand (literally dragging folders around in file explorer) is super duper dumb. I learnt that the hard way, it's slow and forces the standard OS file handler to copy huge amounts of data, which will lead to erros, unwanted overwrites and pointless duplication of data.

What you need on Windows is a great program called "SyncBackFree" https://www.2brightsparks.com/freeware/freeware-hub.html

Get it, learn how to use it and then with the click of a button you can run backups on folders or entire hard drives, where it will only update files that have been changed. That's the key here, files that have been modified.

Do that at least once a day and you're generally protected from the most likely failure....that of a single hard drive going bad.

STEP 2 - apocalypse protection (the cloud)

So step 1 will protect you in 95% of all instances, the chances of 2 hard drives breaking randomly on the same day is never worth worrying about. But what COULD happen if youre really unlucky is a housefire or burglary, where you might lose both hard drives forever.

In that case you need to have a cloud backup solution as well.

I'm currently using Google Drive with it's desktop syncing app at the moment. https://www.google.com/drive/download/backup-and-sync/

The free version gives you about 12 GB of backup cloud storage.

I've heard good things about Github which I'm looking into and OneDrive and I guess Dropbox could work too.

Now the ease of use will all depend on your internet connection speed, me being in Australia means I'm still having to function on 1990s internet speeds, but I assume for most other countries this isnt such an issue.

Me thinking that it's 2020 and I have ADSL2+ equivalent upload speeds......

So for me I just have to accept that cloud backups are an overnight affair if I have big file changes...but after that it will be incremental so not as much of a problem.

STEP 3 - versional backups

Now this is just as important as the critical backups IMO.

Since you have step 1 & 2 protecting you from the worst case scenarios, let's not forget keeping "snapshots" of your old game versions.

You might need these for reference, for rollbacks if you create unfixable errors in your current game version, or as I usually use them marketing materials where I can show the progress of the game.

I'd recommend completely copying your game folder at least twice a week to a backups folder and just labelling it as "game folder from this date", here's how mine looks.

The entire snapshots backup folder is 2.2GB in size and is continually growing, so you may or may not want to also backup this to the cloud, for me I'm happy just backing up these guys to my local backup drive, as it wouldnt be the end of the world if I lost these guys and the chance that I would with the current system is only down to house fire or burglary.

protection level "extreme"

Guys girls I hope you learnt something and that you never lose your precious work!

Cheers, Martin