auditing provisioning with activemq and mongodb
Fri, Mar 22, 2013
At the moment, Matrix provisioning which is based on Apache ActiveMQ has been trundling along for some time with some basic reporting. So recently I sat down with some quality coffee and some new experience of MongoDB to come up with a way of easily finding out the status of accounts. The current setup for provisioning Blackboard using ActiveMQ is shown below:
The Extractor finds new accounts that have appeared in the student records system, turns them into JMS messages and sends them into the Matrix. Eventually they make their way to the Blackboard topic at the ActiveMQ broker and the subscriber does the work of parsing the messages and creating the accounts in Blackboard. If it can't do that, it sends the message to the Blackboard error topic with a retryCount header added and set to 1. Each time the message is sent back to the error topic the retryCount header is incremented until it reaches 10. After that it's routed to the Blackboard Invalid Message Channel (IMC) where it sits until someone intervenes to see what the problem is. If it's something to do with Blackboard the message can be replayed later by a ruby script as the message is persisted to disk once it reaches the IMC, along with details of the error, where the message came from and where it was going. So the persisted message contains enough information to reinject it into the Matrix.
So how to log all this activity, including successful messages? I chose to use MongoDB. Each topic subscriber extends the Matrix::System module so they all have access to common code and methods, which now includes a messaging system whereby each module will send an Audit message to the Matrix::Audit module. It's here the logging takes place. When Matrix::Audit picks a message up on its topic it parses it and inserts it into a MonogDB instance. This allows some pretty nifty reporting to be done from the mongo commandline:
So that was the proposal and I built and tested the whole shebang, to find out that departmental strategy had shifted to Microsoft for 'backoffice' stuff like institutional systems reporting. But Matrix is modular and the only part that needs to be changed to log to MS SQL is Matrix::Audit. Agile, you gotta love it!