Archive for the ‘Google Apps’ category

Backup Google Apps for your Organisation

November 23rd, 2012

So all your data is up in the cloud – Email, Calendars, Documents etc but how do you recover if disaster strikes or a user accidentally deletes something they didn’t mean to?

Google, we hope, are backing up our data, but they will only do restores if the whole service goes down and your data becomes unavailable because of a fault with their systems. They won’t recover an email you deleted or a calendar entry.

And besides, what if Google’s backups were bad or out of date? It makes sense to have a local copy as a fallback should there be a major issue.

This post describes the processes, tools etc I’ve put in place as a start to backing up Google Apps. It’s by no means complete and I very much welcome feedback.

Our main backup server runs Linux and BackupPC so all the tools described are running on Ubuntu 12.04 and the resulting files created are backed up by BackupPC for archiving/revisioning.


If you’re a Google Apps Administrator, you’re probably already familiar with Google Apps Manager (GAM). Well Jay from Ditto also wrote a very handy GMail backup utility called Got Your Back (GYB). It uses the Google Apps Domain OAuth2 domain key to get you access to all your users mailboxes via IMAP, and then dump a copy out to local disc. It also stores an index of which messages it has backed up and what labels are applied to those messages, so that you can only download the differences on subsequent runs.

This approach requires access to Google’s IMAP server from your backup server – which may not be available if your only internet access is via a proxy server.

I wrote a little script that automates GAM to download the user list from Google Apps and then iterate over it calling GYB on each mailbox, and then move any mailboxes that don’t exist any longer to an archive folder. BackupPC then backs up the resulting folder structure and the files are stored for restore later.

GYB can restore mailboxes directly back to Google Apps from the command line. It won’t duplicate messages that are already there (ie haven’t been deleted) and it can optionally add a label to all messages it does restore. This seems to make sense as it’ll let you restore back all deleted messages for a user, allow them to recover what they need, remove the restore label from those messages, and then delete all the other messages that weren’t needed for that restore.

Note that using GYB stores a local plaintext copy of all emails on your local drive. That’s great for systems such as BackupPC that de-duplicate files when they backup since one email sent to all your users will generate a file in each users backup folder. It’s not so great if your backup system doesn’t de-duplicate however.

Google Calendar

The best tool I found for this was googlecalendarbackup. It’s a PHP script that uses your Google Apps Administrator account to connect to each of your users accounts and download an ICS file for each of their calendars.

I had to make a couple of tweeks to it to get it working as I wanted, and you need to download a copy of the Zend 1.x framework and put it in the same folder as the script (in a folder called Zend) to make it work. You also need the php5-cli package installed to run the script from the command line.

I wrote a very similar script to the GMail backup script to download a list of users from GAM and call the googlecalendarbackup script for each item. That then creates a folder for each user with a bunch of ICS files – one for each calendar.

If you need to restore a calendar, you can simply import the ICS file directly in to Google Calendar. It won’t duplicate events that are already in the calendar (assuming the ICS file exported came from the same calendar you’re importing events in to).

My version of googlecalendarbackup is available below. You’ll need to modify it to select where you want your users files created. It also expects usernames to be passed in in email address format – unlike the original. I assume it’s OK to distribute this. The file has no licensing information however the project site states it’s GPL v2.


Google Docs

This is next on my list to implement and I’ll update this document when I have something in place.

GDataCopier looks promising and I’m working with the main developer at the moment to get a version that uses the full OAuth2 two-legged authentication process to backup all users docs to local files.

I may also have a quick hack on php-google-backup to see if I can get access to Docs that way too – as it looks very similar to the calendar code above it may be possible.

After a bit of fiddling I managed to get GDocBackup working as needed. It’s written in .NET but mono runs it nicely on the Linux command line.

The command line options required took a bit of guess work so here’s the command I’m using with it:

/usr/bin/mono /path/to/GDocBackupCMD.exe -mode=backup -destDir=/path/to/myBackups -docF=odt -sprsF=ods -presF=ppt -drawF=png -appsMode=1 -appsOAuthSecret=myAppsOauthDomainKey -password=adminPassword

That will download docs for all users in the domain and store them in folders in /path/to/myBackups

Google Apps Email Permissions – Only allow some users to Email a Group

September 17th, 2012

So you have certain groups in Google Apps and Active Directory that you want to protect from the majority of users being able to email those groups.

For example we have groups such as All Students, Everyone in School, All Staff etc. We wanted it setup so that only members of the “All Staff” group could email All Students and members of the group “Can Send Global Emails” could email groups like “All Staff”, “Everyone in School” etc.

Google Groups has a setting where only owners of a group can send email to the group, but by default Active Directory offers only the managedBy attribute and while you can set that to a group, GADS (Google Apps Directory Sync) won’t sync the members of that group up to Google Apps as the members of that group, only the email address of the group itself.

So to work around this, I’ve written ManagedBy2Owners.

It takes the managedBy attribute in AD, and if it’s a group, expands it in to the nonSecurityGroupMember attribute as a list of all the users of that group. You then use GADS to sync the nonSecurityGroupMember field in to Apps as the group owner.

Optionally, Managed2ByOwners can use GAM (Google Apps Manager) to set the permissions on each group where you have a managedBy field to “Owner Only”.

The workflow is as follows:

  • For each group you want to protect, set the Managed By attribute to the user or group who should have access to email that group.
  • Run ManagedBy2Owners. If there’s a large number of groups, disable updating permissions for the first run.
  • Set GADS to use the nonSecurityGroupMember attribute as the Owner Reference Attribute in the Group Search Rule dialogue
  • Sync GADS
  • If you opted not to allow Managed2ByOwners to change your group permissions, manually change your group permissions in Google Apps Control Panel. If you disabled updating permissions but intend to allow it after the initial run, run ManagedBy2Owners again now with permission change enabled.

ManagedBy2Owners is part of MIS2AD but can be used standalone if you wish.

The latest version will always be available here:

It requires the same setup as MIS2AD to run (see this post) and GAM setup and configured if you want to change permissions automatically.


The script uses the nonSecurityMember attribute in AD to store your group owners. Note that Microsoft Exchange uses this field to store non-user distribution list members, so be aware that if you’re still using Exchange on your domain this script is probably not for you. IT WILL WIPE THAT FIELD CLEAN ON ALL GROUP OBJECTS IN YOUR DOMAIN, AND POPULATE IT USING THE MANAGEDBY VALUE, EVEN IF MANAGEDBY IS BLANK. It is your responsibility to understand this and understand the implications before you run the software. I offer you no warranty or promise of support.

Introducing MIS2AD

June 22nd, 2012

The Itch

They say the best free software development comes from itch-scratching. This is probably no exception.

The school I work for has long wished to be able to have in their email system email groups for each class group and all the teachers who teach a specific student. These groups help users target emails to just those who need to read them rather than sending an email en-mass and expecting staff to discard those that don’t interest them.

We’ve been a Frog customer for several years now and have asked on numerous occasions for “teachers of” groups, but despite being rated highly in their old customer suggestions system, still nothing has been implemented.

So tired of waiting, I sat down and in a short afternoon wrote MIS2AD.


This code is something I wrote to solve a specific problem here. It’s offered to you in case you find it useful. It comes with absolutely no warranty. It modifies your Active Directory in potentially destructive ways, so please ensure you have a backup before you run this!

The Scratch

MIS2AD is a tool to extract SIMS timetables for staff and students and create/maintain groups in Active Directory based on that data. It’s written in Python and used the pyad library to connect to and modify Active Directory. It uses the SIMS CommandReporter utility to talk to the SIMS database so using this utility will not invalidate any support agreements with Capita or your LSU.

How it works

The most difficult thing with this integration is accurately mapping students/staff in SIMS to user accounts in your Active Directory structure. For student accounts, MIS2AD can be run in a mode where it will do its best to match students based on their name – and then write their UPN in to the Delivery Office field in AD for use when matching later on.

Where you have multiple students with the same name, you’ll need to manually add their UPNs in to Active Directory.

Staff are matched using their teaching initials on the Initials field in AD and these must be manually entered.

Next you must create an empty OU to contain all the new groups. It’s important that the OU is empty as MIS2AD will (optionally) delete any object in that OU (or sub OU) when it tidies groups. It does this to enable it to delete groups for students that have left or classes that no-longer exist etc

Once the links between SIMS and AD are complete, the tool can be run in one of two modes:

Teachers of

This creates/maintains groups “Teachers of Student Name (Mentor Group) UPN”. Each group contains just the members of staff who teach that student.

Teaching Group

This creates / maintains groups “Teaching Group Class-Code”. Each group contains all the students in that class, plus any teachers or assigned staff.


The MIS2AD utility needs to be run on a workstation/server that has SIMS .net installed and configured, and as a user that has permission to modify Active Directory Accounts.

My intension is to eventually provide an MSI based installer, but for now, you need to install Python 2.7, setuptools, pywin32 and pyad. Then download the source from Launchpad here:

You then need to create a site.cfg file in your installation folder and copy any directive you need to modify from defaults.cfg to site.cfg (under the appropriate headings).

You can then run –help to see the command line options.

You probably want to run match mode first, then you can run with -v -t -c -z to create “Teachers of” groups, “Teaching Group” groups and cleanup any groups not required.

Ultimately once you’re confident it’s doing the right thing, you could set it to run as a scheduled task. It takes a fair amount of time to run on my system so probably only needs to be run once or twice per day.