So all your data is up in the cloud – Email, Calendars, Documents etc but how do you recover if disaster strikes or a user accidentally deletes something they didn’t mean to?
Google, we hope, are backing up our data, but they will only do restores if the whole service goes down and your data becomes unavailable because of a fault with their systems. They won’t recover an email you deleted or a calendar entry.
And besides, what if Google’s backups were bad or out of date? It makes sense to have a local copy as a fallback should there be a major issue.
This post describes the processes, tools etc I’ve put in place as a start to backing up Google Apps. It’s by no means complete and I very much welcome feedback.
Our main backup server runs Linux and BackupPC so all the tools described are running on Ubuntu 12.04 and the resulting files created are backed up by BackupPC for archiving/revisioning.
If you’re a Google Apps Administrator, you’re probably already familiar with Google Apps Manager (GAM). Well Jay from Ditto also wrote a very handy GMail backup utility called Got Your Back (GYB). It uses the Google Apps Domain OAuth2 domain key to get you access to all your users mailboxes via IMAP, and then dump a copy out to local disc. It also stores an index of which messages it has backed up and what labels are applied to those messages, so that you can only download the differences on subsequent runs.
This approach requires access to Google’s IMAP server from your backup server – which may not be available if your only internet access is via a proxy server.
I wrote a little script that automates GAM to download the user list from Google Apps and then iterate over it calling GYB on each mailbox, and then move any mailboxes that don’t exist any longer to an archive folder. BackupPC then backs up the resulting folder structure and the files are stored for restore later.
GYB can restore mailboxes directly back to Google Apps from the command line. It won’t duplicate messages that are already there (ie haven’t been deleted) and it can optionally add a label to all messages it does restore. This seems to make sense as it’ll let you restore back all deleted messages for a user, allow them to recover what they need, remove the restore label from those messages, and then delete all the other messages that weren’t needed for that restore.
Note that using GYB stores a local plaintext copy of all emails on your local drive. That’s great for systems such as BackupPC that de-duplicate files when they backup since one email sent to all your users will generate a file in each users backup folder. It’s not so great if your backup system doesn’t de-duplicate however.
The best tool I found for this was googlecalendarbackup. It’s a PHP script that uses your Google Apps Administrator account to connect to each of your users accounts and download an ICS file for each of their calendars.
I had to make a couple of tweeks to it to get it working as I wanted, and you need to download a copy of the Zend 1.x framework and put it in the same folder as the script (in a folder called Zend) to make it work. You also need the php5-cli package installed to run the script from the command line.
I wrote a very similar script to the GMail backup script to download a list of users from GAM and call the googlecalendarbackup script for each item. That then creates a folder for each user with a bunch of ICS files – one for each calendar.
If you need to restore a calendar, you can simply import the ICS file directly in to Google Calendar. It won’t duplicate events that are already in the calendar (assuming the ICS file exported came from the same calendar you’re importing events in to).
My version of googlecalendarbackup is available below. You’ll need to modify it to select where you want your users files created. It also expects usernames to be passed in in email address format – unlike the original. I assume it’s OK to distribute this. The file has no licensing information however the project site states it’s GPL v2.
This is next on my list to implement and I’ll update this document when I have something in place. GDataCopier looks promising and I’m working with the main developer at the moment to get a version that uses the full OAuth2 two-legged authentication process to backup all users docs to local files. I may also have a quick hack on php-google-backup to see if I can get access to Docs that way too – as it looks very similar to the calendar code above it may be possible.
After a bit of fiddling I managed to get GDocBackup working as needed. It’s written in .NET but mono runs it nicely on the Linux command line.
The command line options required took a bit of guess work so here’s the command I’m using with it:
/usr/bin/mono /path/to/GDocBackupCMD.exe -mode=backup -destDir=/path/to/myBackups -docF=odt -sprsF=ods -presF=ppt -drawF=png -appsMode=1 -appsdomain=mydomain.com -appsOAuthSecret=myAppsOauthDomainKey -firstname.lastname@example.org -password=adminPassword
That will download docs for all users in the domain and store them in folders in /path/to/myBackups