Archive for the ‘Open Source Schools’ category

Backup Google Apps for your Organisation

November 23rd, 2012

So all your data is up in the cloud – Email, Calendars, Documents etc but how do you recover if disaster strikes or a user accidentally deletes something they didn’t mean to?

Google, we hope, are backing up our data, but they will only do restores if the whole service goes down and your data becomes unavailable because of a fault with their systems. They won’t recover an email you deleted or a calendar entry.

And besides, what if Google’s backups were bad or out of date? It makes sense to have a local copy as a fallback should there be a major issue.

This post describes the processes, tools etc I’ve put in place as a start to backing up Google Apps. It’s by no means complete and I very much welcome feedback.

Our main backup server runs Linux and BackupPC so all the tools described are running on Ubuntu 12.04 and the resulting files created are backed up by BackupPC for archiving/revisioning.

GMail

If you’re a Google Apps Administrator, you’re probably already familiar with Google Apps Manager (GAM). Well Jay from Ditto also wrote a very handy GMail backup utility called Got Your Back (GYB). It uses the Google Apps Domain OAuth2 domain key to get you access to all your users mailboxes via IMAP, and then dump a copy out to local disc. It also stores an index of which messages it has backed up and what labels are applied to those messages, so that you can only download the differences on subsequent runs.

This approach requires access to Google’s IMAP server from your backup server – which may not be available if your only internet access is via a proxy server.

I wrote a little script that automates GAM to download the user list from Google Apps and then iterate over it calling GYB on each mailbox, and then move any mailboxes that don’t exist any longer to an archive folder. BackupPC then backs up the resulting folder structure and the files are stored for restore later.

GYB can restore mailboxes directly back to Google Apps from the command line. It won’t duplicate messages that are already there (ie haven’t been deleted) and it can optionally add a label to all messages it does restore. This seems to make sense as it’ll let you restore back all deleted messages for a user, allow them to recover what they need, remove the restore label from those messages, and then delete all the other messages that weren’t needed for that restore.

Note that using GYB stores a local plaintext copy of all emails on your local drive. That’s great for systems such as BackupPC that de-duplicate files when they backup since one email sent to all your users will generate a file in each users backup folder. It’s not so great if your backup system doesn’t de-duplicate however.

Google Calendar

The best tool I found for this was googlecalendarbackup. It’s a PHP script that uses your Google Apps Administrator account to connect to each of your users accounts and download an ICS file for each of their calendars.

I had to make a couple of tweeks to it to get it working as I wanted, and you need to download a copy of the Zend 1.x framework and put it in the same folder as the script (in a folder called Zend) to make it work. You also need the php5-cli package installed to run the script from the command line.

I wrote a very similar script to the GMail backup script to download a list of users from GAM and call the googlecalendarbackup script for each item. That then creates a folder for each user with a bunch of ICS files – one for each calendar.

If you need to restore a calendar, you can simply import the ICS file directly in to Google Calendar. It won’t duplicate events that are already in the calendar (assuming the ICS file exported came from the same calendar you’re importing events in to).

My version of googlecalendarbackup is available below. You’ll need to modify it to select where you want your users files created. It also expects usernames to be passed in in email address format – unlike the original. I assume it’s OK to distribute this. The file has no licensing information however the project site states it’s GPL v2.

google-backup-calendars

Google Docs

This is next on my list to implement and I’ll update this document when I have something in place.

GDataCopier looks promising and I’m working with the main developer at the moment to get a version that uses the full OAuth2 two-legged authentication process to backup all users docs to local files.

I may also have a quick hack on php-google-backup to see if I can get access to Docs that way too – as it looks very similar to the calendar code above it may be possible.

After a bit of fiddling I managed to get GDocBackup working as needed. It’s written in .NET but mono runs it nicely on the Linux command line.

The command line options required took a bit of guess work so here’s the command I’m using with it:

/usr/bin/mono /path/to/GDocBackupCMD.exe -mode=backup -destDir=/path/to/myBackups -docF=odt -sprsF=ods -presF=ppt -drawF=png -appsMode=1 -appsdomain=mydomain.com -appsOAuthSecret=myAppsOauthDomainKey -username=adminuser@mydomain.com -password=adminPassword

That will download docs for all users in the domain and store them in folders in /path/to/myBackups

Google Apps Email Permissions – Only allow some users to Email a Group

September 17th, 2012

So you have certain groups in Google Apps and Active Directory that you want to protect from the majority of users being able to email those groups.

For example we have groups such as All Students, Everyone in School, All Staff etc. We wanted it setup so that only members of the “All Staff” group could email All Students and members of the group “Can Send Global Emails” could email groups like “All Staff”, “Everyone in School” etc.

Google Groups has a setting where only owners of a group can send email to the group, but by default Active Directory offers only the managedBy attribute and while you can set that to a group, GADS (Google Apps Directory Sync) won’t sync the members of that group up to Google Apps as the members of that group, only the email address of the group itself.

So to work around this, I’ve written ManagedBy2Owners.

It takes the managedBy attribute in AD, and if it’s a group, expands it in to the nonSecurityGroupMember attribute as a list of all the users of that group. You then use GADS to sync the nonSecurityGroupMember field in to Apps as the group owner.

Optionally, Managed2ByOwners can use GAM (Google Apps Manager) to set the permissions on each group where you have a managedBy field to “Owner Only”.

The workflow is as follows:

  • For each group you want to protect, set the Managed By attribute to the user or group who should have access to email that group.
  • Run ManagedBy2Owners. If there’s a large number of groups, disable updating permissions for the first run.
  • Set GADS to use the nonSecurityGroupMember attribute as the Owner Reference Attribute in the Group Search Rule dialogue
  • Sync GADS
  • If you opted not to allow Managed2ByOwners to change your group permissions, manually change your group permissions in Google Apps Control Panel. If you disabled updating permissions but intend to allow it after the initial run, run ManagedBy2Owners again now with permission change enabled.

ManagedBy2Owners is part of MIS2AD but can be used standalone if you wish.

The latest version will always be available here:
https://code.launchpad.net/mis2ad

It requires the same setup as MIS2AD to run (see this post) and GAM setup and configured if you want to change permissions automatically.

CAUTION

The script uses the nonSecurityMember attribute in AD to store your group owners. Note that Microsoft Exchange uses this field to store non-user distribution list members, so be aware that if you’re still using Exchange on your domain this script is probably not for you. IT WILL WIPE THAT FIELD CLEAN ON ALL GROUP OBJECTS IN YOUR DOMAIN, AND POPULATE IT USING THE MANAGEDBY VALUE, EVEN IF MANAGEDBY IS BLANK. It is your responsibility to understand this and understand the implications before you run the software. I offer you no warranty or promise of support.

Introducing MIS2AD

June 22nd, 2012

The Itch

They say the best free software development comes from itch-scratching. This is probably no exception.

The school I work for has long wished to be able to have in their email system email groups for each class group and all the teachers who teach a specific student. These groups help users target emails to just those who need to read them rather than sending an email en-mass and expecting staff to discard those that don’t interest them.

We’ve been a Frog customer for several years now and have asked on numerous occasions for “teachers of” groups, but despite being rated highly in their old customer suggestions system, still nothing has been implemented.

So tired of waiting, I sat down and in a short afternoon wrote MIS2AD.

DISCLAIMER

This code is something I wrote to solve a specific problem here. It’s offered to you in case you find it useful. It comes with absolutely no warranty. It modifies your Active Directory in potentially destructive ways, so please ensure you have a backup before you run this!

The Scratch

MIS2AD is a tool to extract SIMS timetables for staff and students and create/maintain groups in Active Directory based on that data. It’s written in Python and used the pyad library to connect to and modify Active Directory. It uses the SIMS CommandReporter utility to talk to the SIMS database so using this utility will not invalidate any support agreements with Capita or your LSU.

How it works

The most difficult thing with this integration is accurately mapping students/staff in SIMS to user accounts in your Active Directory structure. For student accounts, MIS2AD can be run in a mode where it will do its best to match students based on their name – and then write their UPN in to the Delivery Office field in AD for use when matching later on.

Where you have multiple students with the same name, you’ll need to manually add their UPNs in to Active Directory.

Staff are matched using their teaching initials on the Initials field in AD and these must be manually entered.

Next you must create an empty OU to contain all the new groups. It’s important that the OU is empty as MIS2AD will (optionally) delete any object in that OU (or sub OU) when it tidies groups. It does this to enable it to delete groups for students that have left or classes that no-longer exist etc

Once the links between SIMS and AD are complete, the tool can be run in one of two modes:

Teachers of

This creates/maintains groups “Teachers of Student Name (Mentor Group) UPN”. Each group contains just the members of staff who teach that student.

Teaching Group

This creates / maintains groups “Teaching Group Class-Code”. Each group contains all the students in that class, plus any teachers or assigned staff.

Download/Setup

The MIS2AD utility needs to be run on a workstation/server that has SIMS .net installed and configured, and as a user that has permission to modify Active Directory Accounts.

My intension is to eventually provide an MSI based installer, but for now, you need to install Python 2.7, setuptools, pywin32 and pyad. Then download the source from Launchpad here:

https://code.launchpad.net/mis2ad

You then need to create a site.cfg file in your installation folder and copy any directive you need to modify from defaults.cfg to site.cfg (under the appropriate headings).

You can then run mis2ad.py –help to see the command line options.

You probably want to run match mode first, then you can run with -v -t -c -z to create “Teachers of” groups, “Teaching Group” groups and cleanup any groups not required.

Ultimately once you’re confident it’s doing the right thing, you could set it to run as a scheduled task. It takes a fair amount of time to run on my system so probably only needs to be run once or twice per day.

Hell Freezes over – Is Steve Jobs right?

October 25th, 2010

It’s been widely reported that Steve Jobs thinks that 7″ tablets from RIM and Samsung are “dead on arrival”. The 7″ screen isn’t big enough for a pleasant touch screen experience and in any case they’re too expensive.

I seldom find myself agreeing with Steve Jobs, but I think he might just have a point here.

Today by “IT Buyers Guide” from BT Business Direct dropped in to my pigeon hole and their quoting £509.79 plus VAT for the Samsung Galaxy Tab – that’s a penny short of £600 for a tablet PC. I don’t see how that is a sensible price for a tablet in anyone’s book.

I’m toying with getting a couple of tablets for my staff to use when they’re out and about in classrooms to cut the amount of walking back to the office needed to pick up new jobs – but at £510 each there’s no way I’ll be buying Galaxy Tabs.

Equally I won’t be buying iPads, lovely as the hardware is, because of Apple’s attitude towards app developers and the devices’ junkie-like iTunes habit.

Is 7 inches enough?

Quite frankly I don’t think so – especially if the screen in question is a 16:9 or 16:10 aspect ratio. Tablets look like they should be used in the portrait orientation, as you would a clipboard, yet with a 7″ screen that forces web content to be relatively small. Those extra 3″ coupled with a 4:3 aspect ratio makes a huge difference.

I guess time will tell if Samsung has found a form factor that people like using. Having used a Dell Streak briefly it felt like a large phone (which it essentially is) and I see the Galaxy Tab as a slightly enlarged version of that – although at 7″ you’re going to look alot like Dom Joly using it!

Personally I want a 10″ device, powered by Android or maybe even ChromeOS and I’m hoping that the muted Google Tablet will be very much along those lines.

Open Letter to Simon Kirby

May 12th, 2010

The upshot of the General Election for me was a change in Parliamentary representative in the form of the Conservative MP Simon Kirby.

There follows an open letter to Mr Kirby outlining my suggestions for cuts to the BSF program which would potentially save the country millions of pounds and safeguard jobs in to the bargain. I also mention the need to repeal or heavily rework the Digital Economy Act and to urgently look at Copyright reform.

I intend to publish here any response I might receive.

Dear Mr Kirby

Firstly my congratulations on your recent election success, and your parties subsequent proposed coalition Government with the Liberal Democrats.

As my elected representative, I’m taking a couple of minutes to put to you my point of view on a couple of important issues in the hope that you can take my views on board.

One of the stated aims of the Conservative Party is to begin cutting the budget deficit with immediate effect. May I suggest a thorough review of the Building Schools for the Future scheme and the quango appointed to run it (Partnerships for Schools) would be an excellent point to begin.

BSF in its conception was a revolutionary rework of state-funded secondary schools nationwide, however over time it has become clear that the needs of large businesses have been pushed to the fore and it is now the normal situation in a BSF scheme to spend millions of pounds unnecessarily on consultants, propitiatory software and replacing nearly new equipment like-for-like at each and every school it touches.

Education desperately needs its funding for ICT, but schools that have invested in their ICT services are being penalised by this levelling process that has been imposed by Partnerships for Schools.

Supporters of the BSF program would say that it’s possible for schools to opt out of the Managed ICT Service side of BSF – and it is, theoretically and on paper. In reality it’s a very long and difficult process to demonstrate that existing provision exceeds that offered by an MSP. It falls to the school – which is rightly working towards educating its students – to counter the claims made by dedicated teams from the MSPs. It’s not surprising that almost all schools that embark on that route fail to win their argument.

Please therefore consider cutting the monies spent on the BSF Managed ICT Service such that the remaining funds are spent directly where they are needed by frontline ICT staff, Headteachers and Governors in schools rather than wasted on consultants and replacing systems like-for-like.

The previous Parliament also presided over the farce that was the passing of the Digital Economy Act. The arguments surrounding that act are well publicised and I’m sure you’re aware of the pertainant points.

Nick Clegg made an election promise to repeal the Digital Economy Act if the Liberal Democrats were in power. I hope that the Con-Lib coalition will seriously consider repealing the act or at the very least undertake a major overhaul of the act to address the very serious shortcomings that it has.

And finally please use this period of Conservative-Liberal accord to tackle the issue of Copyright reform. The UK is still applying copyright laws passed decades ago to modern business practises and I’m afraid they just aren’t fit for purpose any longer. Indeed proper copyright reforms would largely address the contentious issues that the Digital Economy Act attempts to legislate around and would probably negate the need for its existence at all.

Yours sincerely

Alex Harrington

Xibo and MythTV

April 14th, 2010

W’e’ve got a new room at work for our older students which has a TV mounted on the wall for Digital Signage.

What we wanted to do was provide a live/recorded TV function for breaks and lunchtime and then an easy way to switch over to our Digital Signage solution (Xibo) at other times.

It turned out to be reasonably simple.

First we installed and configured a new MythBuntu frontend (we already had the myth backend server running from a different project). I then installed the Xibo Python client (1.1.0a21) and made a small modification to the code to allow it to exit by remote control button press (our Media Centre remote sends a backspace character when you press the back button):

if e.scancode == 22:
    log.flush()
    self.parent.downloader.running = False
    self.parent.downloader.collect()
    self.parent.scheduler.running = False
    self.parent.scheduler.collect()

    log.log(5,"info",_("Blocking waiting for Scheduler"))
    self.parent.scheduler.join()
    log.log(5,"info",_("Blocking waiting for DownloadManager"))
    self.parent.downloader.join()
    log.log(5,"info",_("Blocking waiting for Player"))
    self.player.stop()
    os._exit(0)

That block was added after line 2852 appropriately indented of course 😀

It’s then a simple enough job to add a Xibo button to the MythTV menu.

As your mythfrontend user (ie the user that automatically logs in to Mythbuntu on boot), do the following:

mkdir ~/.mythtv
cp /usr/share/mythtv/themes/defaultmenu/mainmenu.xml ~/.mythtv

You then edit ~/.mythtv/mainmenu.xml and add in a block like this:

<button>
    <type>SETTINGS_VIDEO</type>
    <text>Xibo Digital Signage</text>
    <action>EXEC /opt/xibo/pyclient/client/python/run.sh</action>
</button>

Then restart mythfrontend. You should have a new icon on the menu system that launches Xibo. Once Xibo is running, the client should quit when you press the back button on your remote control and drop you back in to MythTV.

Here’s a short video to show the system in action!

Converting Windows VMWare machines to KVM

February 24th, 2010

I had to do some work on a Windows 2003 virtual server running on VMWare this week which is running on our old playground VMWare install (Server 1.0.4!) and it was sooooo slow I decided enough was enough and it was time to move it on to a sensible platform.

It actually turned out to be quite easy to convert from one platform to the other, with a bit of help from my old boss James Lidderdale.

  1. Shutdown the VMWare server and take a full backup
  2. Boot the VMWare server and uninstall VMWare tools
  3. Apply mergeide.reg registry patch to enable windows to boot on KVM virtual hardware. I’ve no idea where this came from. James had it. I’d like to credit the original author though! Rename .txt to .reg and then merge in as normal.
  4. Shutdown the VMWare server
  5. Convert the vmdk disk image to a single pre-allocated monolithic image:
    • vmware-vdiskmanager -r Server.vmdk -t 2 /some/other/folder/Server.vmdk
  6. Copy the resulting Server-flat.vmdk image over to your KVM server
  7. Now optionally convert the disk to a qcow2 file:
    • qemu-img convert Server-flat.vmdk -O qcow2 Server.qcow2
  8. Finally create a suitable virtual machine definition using that file as the main hard drive. If all went well you should see your VMWare machine boot inside KVM.

The first time I tried this it failed miserably. Turned out that the VMWare machine I was working with had snapshots associated with it. In my case, the disk I needed to flatten with vmware-vdiskmanager was actually Server-000001.vmdk. Once I figured that out it worked first time.

School Christmas Dinner Photo

December 18th, 2009

There’s been a literally a submission for the “best school Christmas dinner” crown:

My Christmas School Dinner, Longhill High School

My Christmas School Dinner, Longhill High School

So here’s the first, and to date only entry. From me! Longhill High School by Innovate Ltd. I can honestly say it’s the best school Christmas dinner I’ve had in recent times.

Turkey roll, stuffing ball,  roast and boiled potatoes, carrots, sprouts and parsnips with gravy. Pudding was Christmas pudding and custard.

Well worth a mention were the parsnips. They were so sweet – absolutely delcious. Apparently they’re cooked in oil and honey which explains it.

If you’ve taken a pic of your Christmas Dinner, send it over along with a short review if you like and I’ll add it on!

Best School Christmas Dinner

December 13th, 2009

At work our canteen is due to be enlarged next year as we can’t seat all the students during a lunchbreak.

So next week we’ve got a Christmas dinner marathon – with the festive meal three days on the trott – and I was wondering how these three dinners will stack up against the school dinners available nationwide.

I therefore call all school staff to forward me a photo of your school chrismas dinner. I’ll stick them up here and we can see who’s getting the best deal!

Linux Teacher PC Update

December 11th, 2009

Well we’ve spent a little time on this. Seb’s been helping out and has done the majority of the work so far.

We’ve got the following working:

  • “Domain” logins (against the LDAP database)
  • CCTV Software (via Wine)
  • Themed to look like our standard XP workstations (but not to the extent that you can’t tell the difference. We’re still using the standard Gnome menus). This needs further work.
  • Promethean ActivInspire for Linux (full marks Promethean)