retrospective

Wed, Oct 29, 2014

Sometimes it’s nice to sit down and have a look over what you’ve been up to over the last month, so I brewed up a nice mug of coffee and did just that. It’s been pretty busy with different development projects jostling for space in the old brainbox but with judicious use of Jira (release management) and Stash (git), it’s a pleasurable experience.

Toolkit Image Bank

Languages: Ruby, Javascript
Technologies: Rails, LDAP, Fedora, GSearch, Solr

I started the Toolkit as a Rails app in response to requests from a dept. to provide a repository for resources that could be used in teaching and learning. A sort of bazaar where staff could upload interesting stuff and other staff could find it and make use of it. In this batch of development I upgraded to Rails 4.1.6 and added an image bank, with associated user interface gubbins to make it distinct from the ‘normal’ resources.

One of the biggest issues was to work out how to paginate image search results. The normal resources use DataTables but images are visual, of course, so they need to be handled differently. In the end I plumped for infinite scroll in combination with Freewall. So lots learned in the world of Javascript/JQuery/CSS and how to implement infinite scrolling/page dimming/spinners/toggle switches etc.

Behind all that I needed to express images as a distinct type of resource in the backend Fedora repository but that was a simple addition to the Toolkit ontology (isInImageBank). Lots of Solr schema rebuilding/reindexing and fine tuning of search queries and filters and the job was done.

ActiveMQ

Languages: Java Technologies: Apache ActiveMQ, Apache Camel

While the Toolkit was simmering on the pot (awaiting user testing results) I turned my attention to upgrading ActiveMQ to 5.10.0. This is what powers Matrix, the distributed provisioning system I developed a couple of years ago. We’re looking at getting StuTalk to send messages from SITS to Matrix but it uses raw HTTP POST rather than JMS formatted messages. This is fine but I wanted the latest ActiveMQ as it better supports POSTing messages to topics.

A bit of rummaging around for the Camel config and merging with the production settings and all was good, once I’d recompiled Matrix to work with 5.10.0. The development ActiveMQ now sends an email, courtesy of Camel, when a message is POSTed to a particular topic, to allow StuTalk testing to go ahead.

Matrix

Languages: Java, C# Technologies: .NET, Active Directory, Apache ActiveMQ NMS, Windows Services

While upgrading ActiveMQ I kept getting weird OSGI errors:

NoSuchMethodError org.osgi.framework.FrameworkUtil.getBundle(Ljava/lang/Class;)Lorg/osgi/framework/Bundle;

until I realised the core Matrix library was still the 5.8.0 version. Upping the pom to 5.10.0 for ActiveMQ and Camel sorted that.

Next on the list was an upgrade of the Active Directory integration, to put HE students into specific groups. You’d think this would be easy, you just add the user to the group. It’s not like eDirectory where you have to add the user to the group in both the user object and the group object. In AD you only have to add the user to the group object and it handles the association and cleans up the membership when the user object is deleted.

But there’s a huge gotcha. When I did this for eDirectory, many moons ago, I was connecting to a specific regional server, dictated by the student’s institution location. It was to do with being near their files in their home directory but an upside was the server knew the user existed immediately they were created. In AD I connect to a domain controller via LDAP. No idea which one as AD chooses it. This means the fully constituted user may not (usually isn’t) visible to the other domain controllers. As the groups container lives in a different part of the tree, you can’t work at the top level with objects as you just get referrals back. If I try to add the new user to a group using Principal derivatives, unless I hit the same domain controller the user was created on, I’ll just get a referral back. In effect the controller is saying ‘yes ok I know this user exists but it’s not ready for me to use but I know a controller that can use it’. No use to me though.

The solution lies in underlying objects, tree roots and SIDs


PrincipalContext context = getContext(contextPathWholeTree);
using (GroupPrincipal groupPrincipal = GroupPrincipal.FindByIdentity(context, IdentityType.DistinguishedName, groupDN))
{
  UserPrincipal user = UserPrincipal.FindByIdentity(context, IdentityType.SamAccountName, username);
  DirectoryEntry groupDirectoryEntry = (DirectoryEntry)groupPrincipal.GetUnderlyingObject();

  if (mode == ADD)
  {
    groupDirectoryEntry.Properties["member"].Add(user.DistinguishedName);
  }
  else if (mode == REMOVE)
  {
    groupDirectoryEntry.Properties["member"].Remove(user.DistinguishedName);
  }

  groupDirectoryEntry.CommitChanges();
}

Once I’d nailed that and written the tests for MatrixActiveDirectory.dll I bunged it into MatrixActiveDiretoryService.exe, plumbed it into the development broker and sent messages to it to make sure the group stuff was working.

RFC is in and approved by CAB. Release on Monday.

Stats

Languages: Ruby Technologies: grim looking regexes!

A fairly easy one this, parse the Shibboleth Identity Provider (IdP) logs and bung them in an MSSQL database. This to allow general usage stats to be compiled by those who do that sort of thing. It was a one off to get legacy stats into the database so I didn’t bother with ODBC this time. Just generating a dirty great CSV file which I imported into an MSSQL database. The next version will be ODBC enabled. Or maybe I’ll port it to Java and use JDBC, for sanity’s sake you understand.

SUSHI

Languages: Java Technologies: Apache Axis2, XMLBeans, WSDL, Web Services, SOAP

I do like working with SOAP these days! You don’t often hear that, if ever. The reason is down to a combination of Maven/Axis2 WSDL2Java/XMLBeans. Running this lot against a WSDL gives you a nice jar full of Java classes representing everything in the WSDL. Request/Response documents and associated classes. It’s wonderful. So wonderful in fact, I once provided a patch to Axis2 to provide some missing XMLBeans functionality.

What’s it all for though? To pull out COUNTER v4 reports from JUSP via SUSHI. This to allow the library team to analyse usage stats for the electronic resources. Eventually these reports will end up in another MSSQL database as part of the big reporting push but for now I was doing exploratory work to see what SUSHI was going to divulge.


SushiServiceStub stub = new SushiServiceStub("http://www.jusp.mimas.ac.uk/sushiserverR4/");

ReportRequestDocument reportRequestDocument = ReportRequestDocument.Factory.newInstance();
ReportRequest reportRequest = reportRequestDocument.addNewReportRequest();

Requestor requestor = reportRequest.addNewRequestor();
requestor.setName("Your Name");
requestor.setEmail("your.email@somwhere.com");
requestor.setID("SOME_ID");

CustomerReference customerReference = reportRequest.addNewCustomerReference();
customerReference.setID(Integer.toString(supplierID));

ReportDefinition reportDefinition = reportRequest.addNewReportDefinition();
reportDefinition.setRelease("4");
reportDefinition.setName(reportType);
ReportDefinition.Filters filters = reportDefinition.addNewFilters();
Range usageDateRange = filters.addNewUsageDateRange();

Calendar beginDate = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
beginDate.clear();
beginDate.set(2014, Calendar.AUGUST, 1);

usageDateRange.setBegin(startDate);

Calendar endDate2 = Calendar.getInstance(TimeZone.getTimeZone("UTC"));
endDate2.clear();
endDate2.set(2014, Calendar.AUGUST, 31);

usageDateRange.setEnd(endDate);

ReportResponseDocument reportResponseDocument = stub.getReport(reportRequestDocument);

CounterReportResponse counterReportResponse = reportResponseDocument.getReportResponse();
Reports reportsObject = counterReportResponse.getReport();
Report[] reports = reportsObject.getReportArray();
for (Report report : reports) {
    ReportItem[] reportItems = report.getCustomerArray(0).getReportItemsArray();
    for (ReportItem reportItem : reportItems) {
        Metric[] metrics = reportItem.getItemPerformanceArray();
        for (Metric metric : metrics) {
            // do stuff with metrics
        }
    }
}

Viper

Languages: C#
Technologies: ASP.NET MVC 4, XSD, WSDL, IIS

Just some housekeeping here. As part of our ERASMUS work I’d supervised a student integrating our video conferencing system with our video streaming system. The goal being to allow staff to view and select video conferences they think are useful enough to archive and stream. I’d prepared the way by developing the SOAP integration using wsdl.exe to parse the video conferencing WSDL to C# classes, much in the same vein as the SUSHI Java work, only in .NET. The student had then built a user interface making use of this, with lots of nice interfacy gubbins.

This was just contacting the student who was doing some extra work from Belgium and getting it all committed to git and their Jira issues tidied up.

The next step is getting to grips with IIS. I wanted to try this on a headless Windows server using PowerShell but it’s a pig to work with, especially getting FTP enabled on a site as I’m only going to build one of these things so might as well make it of use to multiple sites.

Having said that, there’s a new initiative about to hit the .NET streets, ASP.NET vNext which seems to do away with the need for IIS for ASP.NET MVC apps and instead lets you run it as a Windows service in its own .NET box. Should I hold out for that? It’s in Visual Studio 2015 I think.

Matrix query app

Languages: Javascript
Technologies: Node.js, LDAP, ODBC, MSSQL

Some more housekeeping, making sure the Node.js app I’d developed to query account provisioning status for the Helpdesk was doing ok. It was an exploratory app to see if Node.js was worth investing time in and as usual, there’s no point knocking up a “Hello World!” app when you want to integrate to quite a depth with the technology. So I’d developed an app that used LDAP to query accounts and ODBC to query the Matrix MSSQL audit database for timings and statuses of messages.

Lessons learned? Make more use of Javascript promises!

Codename ???

Languages: Java, Objective-C, Swift Technologies: Android, iOS, Location, Bluetooth Low Energy (BLE)

This is a project just ramping up although I’ve done some exploratory work on home projects using a Raspberry Pi and BLE dongle to get my iPhone to say hello when I walk into my office.

The plan is to develop an Android app (and possibly an iOS one) initially to provide walk ‘n talk functionality. Extending into BLE services. To this end I’m having a lot of fun dropping these iBeacons around the campus and planning a wee walk ‘n talk on iOS.

The Android version will take a lot more work as iBeacon is basically an Apple protocol and for Android, you need to parse the packets yourself. Luckily there’s a library for that and a good book I’ve bought.

Currently having a think about the agenda for the developer meeting next week in sunny Inverness, if the roads from the west coast are open after the big storm!

Books

Languages: English, Gaelic Technologies: Brain, Life

I occasionally pop into The Pragmatic Bookshelf to see if there’s anything new of interest. It’s my favourite online bookshop, managed by developers for developers who want to buy books written by developers.

And of course there’s the lunchtime read to think about, if I’m not doing my walk up to the stream and back. The college has a good bookshop that I pop into now and again and the other day I picked up a couple of Gaelic short story books. Saorsa, which includes a story by Tim Armstrong who wrote the excellent Air Cuan Dubh Drilseach. The other collection is A’Ghlainne agus Sgeulachdan Eile by Màiri E. NicLeòid. Two cracking books I can dip into over a nice coffee at lunchtime.

Octopress

Languages: Ruby, English! Technologies: Octopress, git

Finally, before writing this post I thought I’d update Octopress. I’d been sweating about this for ages but in the end it turned out to be easy:


# Make sure blog posts are committed
git add .
git commit -m "blog"
# Update Octopress
git pull origin master
# Slowly uncover your eyes
git status
# Do what Octopress wants (add .rvmrc back later)
git rm .rvmrc
git rm Gemfile.lock

# I had a couple of conflicts which wer logical and easily fixed:
#
# Rakefile
# 
#   open(filename, 'w') do |post|
#     post.puts "---"
#     post.puts "layout: post"
#     post.puts "title: \"#{title.gsub(/&/,'&')}\""
# <<<<<<< HEAD
#     post.puts "description: \"\""
#     post.puts "date: #{Time.now.strftime('%Y-%m-%d %H:%M')}"
#     post.puts "comments: false"
# =======
#     post.puts "date: #{Time.now.strftime('%Y-%m-%d %H:%M:%S %z')}"
#     post.puts "comments: true"
# >>>>>>> c6c5b74f4d5dc4cc0a34e97c918e0b1a59954a16
#     post.puts "categories: "
#     post.puts "- __CAT1__"
#     post.puts "tags: "
#     post.puts "- __TAG1__"
#     post.puts "keywords: \"\""
#     post.puts "---"
#   end

# _config.yml
# 
# <<<<<<< HEAD
# url: http://codebrane.com/blog
# title: codeBrane Blog
# subtitle: Software development and philosophical musings
# author: Alistair
# simple_search: http://google.com/search
# description: A blog about software development and philosophical musings
# =======
# url: http://yoursite.com
# title: My Octopress Blog
# subtitle: A blogging framework for hackers.
# author: Your Name
# simple_search: https://www.google.com/search
# description:
# >>>>>>> c6c5b74f4d5dc4cc0a34e97c918e0b1a59954a16

# Resolve the conflits and commit
git commit -i Rakefile
git commit -i _config.yml
# Rebuild gemset
rvm use 1.9.3@octopress-codebrane --rvmrc --create
# Complete upgrade
bundle install
rake update_source
rake update_style

# Fix Build Warning: Layout 'nil' requested in categories/guanxi/atom.xml does not exist.
# change layout: nil to layout: null in these:
# source/_includes/custom/category_feed.xml
# source/atom.xml
# source/robots.txt

So that’s October. Lots of interesting things going on. Lots of brain porridge to stir and lots of wild autumnal weather to cope with on the 20 mile cycle to work on the Isle of Skye.