soa provisioning apache style
Wed, Jul 1, 2009
Now that I’ve sorted out ActiveDirectory provisioning I thought I’d look again at the SOA provisioning project. Originally it was to include Novel Identity Manager but that proved to be a problem, so I’m now looking at Apache Camel with embedded Apache ActiveMQ:
The basic idea is something trawls the institutional data store looking for things that have changed. A new account has appeared, an email address has changed, module enrollments have changed etc. That something then generates domain messages from the data events and sends them to a main JMS topic in the Camel instance. At the moment, the “something” is son-of-pliers, which trawls the SITS database looking for changes. It then creates files with the change information and dumps them in a directory. The main provisioning engine picks up these files and acts on them. If the database goes down, changes will be picked up by son-of-pliers when it comes back. If son-of-pliers goes down, the changes in the database aren’t lost, they’re just pulled out when son-of-pliers restarts and if the main provisioning engine goes down, the changes are still waiting in the directory for it to restart. It’s worked well for 6 years so it’s a model I want to continue, rather than direct Observer pattern RPC for example. The current model uses a form of Publish-Subscribe in that son-of-pliers publishes files to a directory and the main engine subscribes to that directory. The new architecture I’ve come up with is based on Apache Camel/MQ and JMS topics.
Each application that needs to be provisioned subscribes to the main topic, perhaps with a message filter to reduce traffic and each has an error topic, where it can store problem accounts.
To provision ActiveDirectory, the domain messages come into the main topic, GADfly picks them up and for any account it can’t create, delete or update, it publishes the original message with the addition of a fault status, to its error topic. For all other messages, it publishes the original message with a success status to its success topic. What I haven’t decided yet is whether to have one success topic in total, to which all applications publish, or one success topic per application. Same with the error channel. One or many? Not sure yet.
The idea is that a harvester can periodically clean out the success topic(s) and persist the messages to database, in case forensics are required. The error messages stay in the error topic(s) indefinitely. Each application can periodically retry accounts in its error topic and Nagios can monitor the error topic(s) using STOMP.
So it should be possible to get the status of an account at any point in the system. If it’s not in the success topic(s), then perhaps it’s been caught in a system failure on ActiveDirectory, in which case it’ll be in the GADfly error topic. Or it might just be waiting to be processed, in which case it’ll be in the main topic. If it’s in none of those places, it might be in the database as a historical record, proving it was processed, when, by what and the result. If it’s nowhere to be found in the system, then the problem is with the main institutional data store and the account hasn’t appeared from it.