lightweight integration scenario soa project 1
Thu, Oct 30, 2008
The SOA Project is basically an amalgamation of a couple of projects that were waiting to be taken on, namely linking Blackboard and Elgg to SITS, with the requirement that changes in Blackboard objects have to be replicated in the institutional “data silo”, which isn’t SITS. So what is it? The plan is to rollout Novell Identity Manager (IDM) to handle provisioning of the core systems, eDirectory, NDS and Groupwise, replacing the framework that’s been running for the last six years or so. Whether the rest of the applications that make up a user’s online portfolio should be plumbed in to IDM is a current topic of interest. Novell discourage custom IDM driver development, which to me either means they don’t trust third party developers or they don’t trust their product. I’ve seen this before. Microsoft discouraged third party NT4 driver development as kernel mode drivers are very difficult to develop and lots of people were doing them, badly, giving NT4, which is the best OS Microsoft ever produced, a bad name. So my gut feeling is IDM is either too complicated for the average bod to develop against, or it’s too ropey for anyone other than Novell to handle. A quote from the DirXML documentation sheds some interesting light on the latter:
“… DirXML does take some precautions when calling into a native driver [C++] … All calls into the native driver are protected by a try-catch block that catches all exceptions … If an exception is caught … the driver is shut down … On Win32 this works well. On NetWare however, it does not work at all. At the time of this writing it is unknown how well this works on the Unix platforms” (my emphasis).
So that rules out all C++ driver development. Will we need it? Doubt it but we’ve needed it in the past for Groupwise integration, so it can’t be ruled out in the future. All it takes is for a Windows application to be integrated using COM and we suddenly have a problem plumbing it into IDM. So that would suggest we need a separate “integration engine” that is fed from a reliable data silo. This is where I’ll introduce the first version of what I think could work:
What basically happens is IDM handles the core applications and also, via its native LDAP Driver, populates an ApacheDS instance. This acts as the institutional data silo, providing LDAP authentication services for applications such as Blackboard, Elgg, Sympa and the Shibboleth Identity Provider (IdP). It also provides a rich attribute store for the IdP. Why ApacheDS? Why not openLDAP? ApacheDS has superb trigger support, openLDAP has none. I’ll expand on that.
Throughout the entire life of the current integration framework it’s been beset by cache problems. The cache is required to detect changes in SITS and propagate them to the framework but in the first version of the cache even the slightest network glitch caused the cache to be destroyed and recreated, causing huge dumps of pre-processed data, which swamped the system for days. The second version had user operational problems which more or less resulted in the same effect. In short, caches are big, clunky, cumbersome and prone to failure. They’re fine if you’re using them as a source of data but not for comparisons between data sets. Interestingly, another aspect of the framework did not use a cache to feed the framework and it has not been affected in any way. It’s run quietly in the background with no problems.
LDAP servers do not handle diffs. i.e. they cannot tell you what changed in the last hour. LDIFF is not a comparison mechanism, it’s a Data Interchange Format. It’s for non programmatic LDAP access. So it can only be used as the basis for a cache. i.e. exporting the entire LDAP server’s state as an LDIFF file every hour and comparing each dump. It’s open to network glitches and the same failures as before. But the integration world has moved on since the days of local state caches and we now have event driven messaging systems. In fact, it’s an industry standard Enterprise Integration Pattern.
Hence ApacheDS and its trigger support. Every modification that’s made to the data silo can be configured with a trigger, which can run whatever code you like. These internal events are at the CRUD level of objects but the current thinking on SOA and integration is that these are too low level to be used in an enterprise situation such as this. Instead, higher level messages should be used that are aligned with the business processes that drive the integration. Instead of a trigger causing a Create message to be sent, it would analyse what data had been changed in the silo and issue, perhaps, a UserEnrolled message. I’ll term these Business Events.
These Business Events are sent to the “integration engine” for want of a better term just now, which has an embedded ApacheMQ instance. This is an open source JMS implementation. It provides message brokering services on JMS queues and topics. Applications subscribe to these queues and topics and act accordingly, when messages are available. They also reply based on the results of the execution of the message payloads. e.g. Blackboard would consume the UserEnrolled message and add an existing user to the specified module. It could then reply with a status update.
Should there be separate queues for each Business Event? Should there just be a queue per application? Not sure just now. Say Blackboard has consumed the UserEnrolled message and tries to add the user to the MOD101 module, which doesn’t yet exist. Blackboard could publish that message to its pending queue and periodically consume messages from the pending queue. In this way, institutional lag can be dealt with by the messaging framework. Messages can be marked as persistent, which allows them to live across restarts.
The beauty of ApacheMQ is its support for multiple protocols, to allow non Java applications to produce and consume Business Events. This would mostly be via the STOMP protocol for our purposes. This also allows Nagios integration, to allow queues and topics to be monitored. For example, if an account hasn’t shown up in Blackboard, Nagios could be used to look at the queues to see where the event is in the system.
To satisfy the requirement that Blackboard state be replicated in the data silo, Blackboard could produce Business Events of its own, which the messaging framework can ultimately route to the data silo.
In summary, if we limit IDM to the core applications eDirectory, NDS, Groupwise and ActiveDirectory (for Citrix) and tap a feed from IDM via its LDAP Driver to populate an institutional data silo, we are free to design a framework that is agile enough to respond to the institution’s constantly changing connected applications landscape.