Backup Google Apps for your Organisation

November 23rd, 2012 by Alex No comments »

So all your data is up in the cloud – Email, Calendars, Documents etc but how do you recover if disaster strikes or a user accidentally deletes something they didn’t mean to?

Google, we hope, are backing up our data, but they will only do restores if the whole service goes down and your data becomes unavailable because of a fault with their systems. They won’t recover an email you deleted or a calendar entry.

And besides, what if Google’s backups were bad or out of date? It makes sense to have a local copy as a fallback should there be a major issue.

This post describes the processes, tools etc I’ve put in place as a start to backing up Google Apps. It’s by no means complete and I very much welcome feedback.

Our main backup server runs Linux and BackupPC so all the tools described are running on Ubuntu 12.04 and the resulting files created are backed up by BackupPC for archiving/revisioning.

GMail

If you’re a Google Apps Administrator, you’re probably already familiar with Google Apps Manager (GAM). Well Jay from Ditto also wrote a very handy GMail backup utility called Got Your Back (GYB). It uses the Google Apps Domain OAuth2 domain key to get you access to all your users mailboxes via IMAP, and then dump a copy out to local disc. It also stores an index of which messages it has backed up and what labels are applied to those messages, so that you can only download the differences on subsequent runs.

This approach requires access to Google’s IMAP server from your backup server – which may not be available if your only internet access is via a proxy server.

I wrote a little script that automates GAM to download the user list from Google Apps and then iterate over it calling GYB on each mailbox, and then move any mailboxes that don’t exist any longer to an archive folder. BackupPC then backs up the resulting folder structure and the files are stored for restore later.

GYB can restore mailboxes directly back to Google Apps from the command line. It won’t duplicate messages that are already there (ie haven’t been deleted) and it can optionally add a label to all messages it does restore. This seems to make sense as it’ll let you restore back all deleted messages for a user, allow them to recover what they need, remove the restore label from those messages, and then delete all the other messages that weren’t needed for that restore.

Note that using GYB stores a local plaintext copy of all emails on your local drive. That’s great for systems such as BackupPC that de-duplicate files when they backup since one email sent to all your users will generate a file in each users backup folder. It’s not so great if your backup system doesn’t de-duplicate however.

Google Calendar

The best tool I found for this was googlecalendarbackup. It’s a PHP script that uses your Google Apps Administrator account to connect to each of your users accounts and download an ICS file for each of their calendars.

I had to make a couple of tweeks to it to get it working as I wanted, and you need to download a copy of the Zend 1.x framework and put it in the same folder as the script (in a folder called Zend) to make it work. You also need the php5-cli package installed to run the script from the command line.

I wrote a very similar script to the GMail backup script to download a list of users from GAM and call the googlecalendarbackup script for each item. That then creates a folder for each user with a bunch of ICS files – one for each calendar.

If you need to restore a calendar, you can simply import the ICS file directly in to Google Calendar. It won’t duplicate events that are already in the calendar (assuming the ICS file exported came from the same calendar you’re importing events in to).

My version of googlecalendarbackup is available below. You’ll need to modify it to select where you want your users files created. It also expects usernames to be passed in in email address format – unlike the original. I assume it’s OK to distribute this. The file has no licensing information however the project site states it’s GPL v2.

google-backup-calendars

Google Docs

This is next on my list to implement and I’ll update this document when I have something in place.

GDataCopier looks promising and I’m working with the main developer at the moment to get a version that uses the full OAuth2 two-legged authentication process to backup all users docs to local files.

I may also have a quick hack on php-google-backup to see if I can get access to Docs that way too – as it looks very similar to the calendar code above it may be possible.

After a bit of fiddling I managed to get GDocBackup working as needed. It’s written in .NET but mono runs it nicely on the Linux command line.

The command line options required took a bit of guess work so here’s the command I’m using with it:

/usr/bin/mono /path/to/GDocBackupCMD.exe -mode=backup -destDir=/path/to/myBackups -docF=odt -sprsF=ods -presF=ppt -drawF=png -appsMode=1 -appsdomain=mydomain.com -appsOAuthSecret=myAppsOauthDomainKey -username=adminuser@mydomain.com -password=adminPassword

That will download docs for all users in the domain and store them in folders in /path/to/myBackups

Google Apps Email Permissions – Only allow some users to Email a Group

September 17th, 2012 by Alex 3 comments »

So you have certain groups in Google Apps and Active Directory that you want to protect from the majority of users being able to email those groups.

For example we have groups such as All Students, Everyone in School, All Staff etc. We wanted it setup so that only members of the “All Staff” group could email All Students and members of the group “Can Send Global Emails” could email groups like “All Staff”, “Everyone in School” etc.

Google Groups has a setting where only owners of a group can send email to the group, but by default Active Directory offers only the managedBy attribute and while you can set that to a group, GADS (Google Apps Directory Sync) won’t sync the members of that group up to Google Apps as the members of that group, only the email address of the group itself.

So to work around this, I’ve written ManagedBy2Owners.

It takes the managedBy attribute in AD, and if it’s a group, expands it in to the nonSecurityGroupMember attribute as a list of all the users of that group. You then use GADS to sync the nonSecurityGroupMember field in to Apps as the group owner.

Optionally, Managed2ByOwners can use GAM (Google Apps Manager) to set the permissions on each group where you have a managedBy field to “Owner Only”.

The workflow is as follows:

  • For each group you want to protect, set the Managed By attribute to the user or group who should have access to email that group.
  • Run ManagedBy2Owners. If there’s a large number of groups, disable updating permissions for the first run.
  • Set GADS to use the nonSecurityGroupMember attribute as the Owner Reference Attribute in the Group Search Rule dialogue
  • Sync GADS
  • If you opted not to allow Managed2ByOwners to change your group permissions, manually change your group permissions in Google Apps Control Panel. If you disabled updating permissions but intend to allow it after the initial run, run ManagedBy2Owners again now with permission change enabled.

ManagedBy2Owners is part of MIS2AD but can be used standalone if you wish.

The latest version will always be available here:
https://code.launchpad.net/mis2ad

It requires the same setup as MIS2AD to run (see this post) and GAM setup and configured if you want to change permissions automatically.

CAUTION

The script uses the nonSecurityMember attribute in AD to store your group owners. Note that Microsoft Exchange uses this field to store non-user distribution list members, so be aware that if you’re still using Exchange on your domain this script is probably not for you. IT WILL WIPE THAT FIELD CLEAN ON ALL GROUP OBJECTS IN YOUR DOMAIN, AND POPULATE IT USING THE MANAGEDBY VALUE, EVEN IF MANAGEDBY IS BLANK. It is your responsibility to understand this and understand the implications before you run the software. I offer you no warranty or promise of support.

Introducing MIS2AD

June 22nd, 2012 by Alex 1 comment »

The Itch

They say the best free software development comes from itch-scratching. This is probably no exception.

The school I work for has long wished to be able to have in their email system email groups for each class group and all the teachers who teach a specific student. These groups help users target emails to just those who need to read them rather than sending an email en-mass and expecting staff to discard those that don’t interest them.

We’ve been a Frog customer for several years now and have asked on numerous occasions for “teachers of” groups, but despite being rated highly in their old customer suggestions system, still nothing has been implemented.

So tired of waiting, I sat down and in a short afternoon wrote MIS2AD.

DISCLAIMER

This code is something I wrote to solve a specific problem here. It’s offered to you in case you find it useful. It comes with absolutely no warranty. It modifies your Active Directory in potentially destructive ways, so please ensure you have a backup before you run this!

The Scratch

MIS2AD is a tool to extract SIMS timetables for staff and students and create/maintain groups in Active Directory based on that data. It’s written in Python and used the pyad library to connect to and modify Active Directory. It uses the SIMS CommandReporter utility to talk to the SIMS database so using this utility will not invalidate any support agreements with Capita or your LSU.

How it works

The most difficult thing with this integration is accurately mapping students/staff in SIMS to user accounts in your Active Directory structure. For student accounts, MIS2AD can be run in a mode where it will do its best to match students based on their name – and then write their UPN in to the Delivery Office field in AD for use when matching later on.

Where you have multiple students with the same name, you’ll need to manually add their UPNs in to Active Directory.

Staff are matched using their teaching initials on the Initials field in AD and these must be manually entered.

Next you must create an empty OU to contain all the new groups. It’s important that the OU is empty as MIS2AD will (optionally) delete any object in that OU (or sub OU) when it tidies groups. It does this to enable it to delete groups for students that have left or classes that no-longer exist etc

Once the links between SIMS and AD are complete, the tool can be run in one of two modes:

Teachers of

This creates/maintains groups “Teachers of Student Name (Mentor Group) UPN”. Each group contains just the members of staff who teach that student.

Teaching Group

This creates / maintains groups “Teaching Group Class-Code”. Each group contains all the students in that class, plus any teachers or assigned staff.

Download/Setup

The MIS2AD utility needs to be run on a workstation/server that has SIMS .net installed and configured, and as a user that has permission to modify Active Directory Accounts.

My intension is to eventually provide an MSI based installer, but for now, you need to install Python 2.7, setuptools, pywin32 and pyad. Then download the source from Launchpad here:

https://code.launchpad.net/mis2ad

You then need to create a site.cfg file in your installation folder and copy any directive you need to modify from defaults.cfg to site.cfg (under the appropriate headings).

You can then run mis2ad.py –help to see the command line options.

You probably want to run match mode first, then you can run with -v -t -c -z to create “Teachers of” groups, “Teaching Group” groups and cleanup any groups not required.

Ultimately once you’re confident it’s doing the right thing, you could set it to run as a scheduled task. It takes a fair amount of time to run on my system so probably only needs to be run once or twice per day.

Sony VPCEJ1Z1E Keyboard Removal

April 18th, 2012 by Alex No comments »

I was asked to replace a keyboard on a Sony VPCEJ series laptop this week and couldn’t find any kind of service manual or dissasembly guide, so here’s my guide on removing the keyboard.

Follow these instructions completely at your own risk. I offer no warranty that this procedure is correct or that it won’t damage your laptop.

1. Start with the laptop upside down on the desk. Remove the battery.
2. Remove the optical drive by removing the screw (ringed in red below) and then sliding the drive out of the chassis to the right.

Remove Optical Drive

Remove Optical Drive

3. Remove the two screws securing the keyboard (ringed in red)

Remove Keyboard Screws

Remove Keyboard Screws

4. Turn the laptop over and open the lid. Through the optical drive bay opening, release the locking tabs (marked red) and use the holes (marked orange) to release the right hand edge of the keyboard. Insert a plastic pry tool and gently work along the top edge of the keyboard releasing the tabs (marked blue). The keyboard will release and fold towards to you get access to the cabling. Note in the image the keyboard has already been removed to better illustrate the locations of the locking tabs and access holes.

Remove Keyboard

Remove Keyboard

5. Remove the securing tape and lift the black ribbon cable connector up to release the keyboard cable.

Disconnect Keyboard

Disconnect Keyboard

6. Reassembly is the reverse of removal.

Is Google Hypocritical for dropping H.264 but not Flash

January 12th, 2011 by Alex 2 comments »

This blog post is a response to a post by John Gruber.

The argument is that Google are hypocritical for not dropping support for Flash at the same time as supporting WebM instead of H.264.

In my view, saying this is hypocritical is complete rubbish.

Whatever your opinion of Flash, it’s widely used on the net at the moment and has some features that cannot be replicated SENSIBLY by HTML5 at the moment. In time that will change, but that’s the reality of the situation at the moment.

Chief amongst those features is access to client side audio and video from webcams. Flash is the only sensible way of accessing these resources and streaming them beyond installing additional plugins (which may or may not be any better than Flash’s effort).

In WebM, Google has a patent free alternative to H.264 (the Apple-preferred codec which may at any time cease to be free for use on websites). The quality of WebM encoded video is arguably very slightly lower than H.264, but immaterial when discussing low bandwidth streaming in a browser. Google’s commitment to integrating open standards understandably sees them implementing WebM in Youtube and Chrome because it’s a sensible and viable alternative to H.264. Sure, right now there’s no hardware acceleration for decoding WebM, but that will come in time. Of the major browsers, Microsoft Internet Explorer, Opera, Firefox and Chrome will all support WebM/VP8 encoded video in HTML5.

There is currently no credible alternative to Flash. If Google were to abandon Flash purely on the basis that it is propitiatory,  they would be cutting off their own nose to spite their face. In the same way that H.264 served them until there was a viable alternative (in WebM/VP8), Flash will continue to serve Chrome users until the shortcomings in HTML5 are addressed.

If, and only if, at that time, Google choose not to remove Flash in favour of the open standard, then they can justifiably be called hypocrites.

FileHunter 0.4

November 10th, 2010 by Alex 1 comment »

One small bug fix to FileHunter today:

  • Ignore symbolic links when searching the haystack

FileHunter-0.4.tar.gz

Hell Freezes over – Is Steve Jobs right?

October 25th, 2010 by Alex No comments »

It’s been widely reported that Steve Jobs thinks that 7″ tablets from RIM and Samsung are “dead on arrival”. The 7″ screen isn’t big enough for a pleasant touch screen experience and in any case they’re too expensive.

I seldom find myself agreeing with Steve Jobs, but I think he might just have a point here.

Today by “IT Buyers Guide” from BT Business Direct dropped in to my pigeon hole and their quoting £509.79 plus VAT for the Samsung Galaxy Tab – that’s a penny short of £600 for a tablet PC. I don’t see how that is a sensible price for a tablet in anyone’s book.

I’m toying with getting a couple of tablets for my staff to use when they’re out and about in classrooms to cut the amount of walking back to the office needed to pick up new jobs – but at £510 each there’s no way I’ll be buying Galaxy Tabs.

Equally I won’t be buying iPads, lovely as the hardware is, because of Apple’s attitude towards app developers and the devices’ junkie-like iTunes habit.

Is 7 inches enough?

Quite frankly I don’t think so – especially if the screen in question is a 16:9 or 16:10 aspect ratio. Tablets look like they should be used in the portrait orientation, as you would a clipboard, yet with a 7″ screen that forces web content to be relatively small. Those extra 3″ coupled with a 4:3 aspect ratio makes a huge difference.

I guess time will tell if Samsung has found a form factor that people like using. Having used a Dell Streak briefly it felt like a large phone (which it essentially is) and I see the Galaxy Tab as a slightly enlarged version of that – although at 7″ you’re going to look alot like Dom Joly using it!

Personally I want a 10″ device, powered by Android or maybe even ChromeOS and I’m hoping that the muted Google Tablet will be very much along those lines.

Direct Email to Schools – Why SchoolEmails.com are scum

July 7th, 2010 by Alex 2 comments »

It’s a fact of life that if you work in Education ICT once someone gets hold of your email address then it’s very hard to turn back the tide of unsolicited “opt-in” email lists that you find yourself subscribed to.

Generally it’s a case of clicking through an unsubscribe process for that company and you don’t hear from them again.

Not so with customers of School Emails. School Emails aka SchoolsRegister seem to be a particularly obnoxious company in the way that they deal with the email addresses they claim you have given them for inclusion in their listing.

In my dealings with them I have requested on 3 separate occasions via email and twice via their web form to be removed from their lists – yet I’m still bombarded by emails from their customers over a month on. They also refuse to tell me how they obtained my contact details and what opportunities I was given to opt-in or out of their service and on what dates.

Of particular note for being ignorant and unhelpful are PromoPrintUK aka Money4school.co.uk who initially sent all their mailings with no unsubscription information, and then tried to tell me they have no control over who they send their bulk emailings to.

Their later mailings do finally carry unsubscription information, however it’s just a link back to SchoolEmails web form which we already know is studiously ignored by the company.

Other companies I’ve encountered using their services include WTA Education Services who are at least helpful in removing email addresses from their bulk mailings.

If you’re a company considering bulk-emailing schools, think long and hard before subjecting your potential customers to emails via SchoolEmails. They clearly don’t care – do you want your company to be associated with that attitude?

Open Letter to Simon Kirby

May 12th, 2010 by Alex 3 comments »

The upshot of the General Election for me was a change in Parliamentary representative in the form of the Conservative MP Simon Kirby.

There follows an open letter to Mr Kirby outlining my suggestions for cuts to the BSF program which would potentially save the country millions of pounds and safeguard jobs in to the bargain. I also mention the need to repeal or heavily rework the Digital Economy Act and to urgently look at Copyright reform.

I intend to publish here any response I might receive.

Dear Mr Kirby

Firstly my congratulations on your recent election success, and your parties subsequent proposed coalition Government with the Liberal Democrats.

As my elected representative, I’m taking a couple of minutes to put to you my point of view on a couple of important issues in the hope that you can take my views on board.

One of the stated aims of the Conservative Party is to begin cutting the budget deficit with immediate effect. May I suggest a thorough review of the Building Schools for the Future scheme and the quango appointed to run it (Partnerships for Schools) would be an excellent point to begin.

BSF in its conception was a revolutionary rework of state-funded secondary schools nationwide, however over time it has become clear that the needs of large businesses have been pushed to the fore and it is now the normal situation in a BSF scheme to spend millions of pounds unnecessarily on consultants, propitiatory software and replacing nearly new equipment like-for-like at each and every school it touches.

Education desperately needs its funding for ICT, but schools that have invested in their ICT services are being penalised by this levelling process that has been imposed by Partnerships for Schools.

Supporters of the BSF program would say that it’s possible for schools to opt out of the Managed ICT Service side of BSF – and it is, theoretically and on paper. In reality it’s a very long and difficult process to demonstrate that existing provision exceeds that offered by an MSP. It falls to the school – which is rightly working towards educating its students – to counter the claims made by dedicated teams from the MSPs. It’s not surprising that almost all schools that embark on that route fail to win their argument.

Please therefore consider cutting the monies spent on the BSF Managed ICT Service such that the remaining funds are spent directly where they are needed by frontline ICT staff, Headteachers and Governors in schools rather than wasted on consultants and replacing systems like-for-like.

The previous Parliament also presided over the farce that was the passing of the Digital Economy Act. The arguments surrounding that act are well publicised and I’m sure you’re aware of the pertainant points.

Nick Clegg made an election promise to repeal the Digital Economy Act if the Liberal Democrats were in power. I hope that the Con-Lib coalition will seriously consider repealing the act or at the very least undertake a major overhaul of the act to address the very serious shortcomings that it has.

And finally please use this period of Conservative-Liberal accord to tackle the issue of Copyright reform. The UK is still applying copyright laws passed decades ago to modern business practises and I’m afraid they just aren’t fit for purpose any longer. Indeed proper copyright reforms would largely address the contentious issues that the Digital Economy Act attempts to legislate around and would probably negate the need for its existence at all.

Yours sincerely

Alex Harrington

Xibo and MythTV

April 14th, 2010 by Alex 2 comments »

W’e've got a new room at work for our older students which has a TV mounted on the wall for Digital Signage.

What we wanted to do was provide a live/recorded TV function for breaks and lunchtime and then an easy way to switch over to our Digital Signage solution (Xibo) at other times.

It turned out to be reasonably simple.

First we installed and configured a new MythBuntu frontend (we already had the myth backend server running from a different project). I then installed the Xibo Python client (1.1.0a21) and made a small modification to the code to allow it to exit by remote control button press (our Media Centre remote sends a backspace character when you press the back button):

if e.scancode == 22:
    log.flush()
    self.parent.downloader.running = False
    self.parent.downloader.collect()
    self.parent.scheduler.running = False
    self.parent.scheduler.collect()

    log.log(5,"info",_("Blocking waiting for Scheduler"))
    self.parent.scheduler.join()
    log.log(5,"info",_("Blocking waiting for DownloadManager"))
    self.parent.downloader.join()
    log.log(5,"info",_("Blocking waiting for Player"))
    self.player.stop()
    os._exit(0)

That block was added after line 2852 appropriately indented of course :D

It’s then a simple enough job to add a Xibo button to the MythTV menu.

As your mythfrontend user (ie the user that automatically logs in to Mythbuntu on boot), do the following:

mkdir ~/.mythtv
cp /usr/share/mythtv/themes/defaultmenu/mainmenu.xml ~/.mythtv

You then edit ~/.mythtv/mainmenu.xml and add in a block like this:

<button>
    <type>SETTINGS_VIDEO</type>
    <text>Xibo Digital Signage</text>
    <action>EXEC /opt/xibo/pyclient/client/python/run.sh</action>
</button>

Then restart mythfrontend. You should have a new icon on the menu system that launches Xibo. Once Xibo is running, the client should quit when you press the back button on your remote control and drop you back in to MythTV.

Here’s a short video to show the system in action!