DevOps In Action

Wed, Apr 19, 2017

The modern developer is expected to know how to set up their development server where they put their app through its paces and the process of doing so is known as DevOps. In theory it sounds easy. Just run some Puppet scripts, deploy and get down to testing. In reality it can be a bit gnarly. What if it’s an old server with no Puppet support and everything’s custom installed by hand? Again, in theory, it’s just as easy. You compile and configure what you need and get on with it. That is, until you get a crossover between custom and system. i.e. At some point in the process your custom compiles rely on a system provided package that relies on a system version of an app you’ve custom compiled. Confused? Read on!

I’d been developing Microsoft Active Directory Azure (AAD) integration for a Rails app I’d developed a few years ago. It uses ‘traditional’ LDAP authentication but with the rise of the cloud, it was time to plumb it into AAD for authentication and single sign on and make use of the Graph API for authorisation. That bit was software development. The Dev of DevOps.

The Ops of DevOps first required silencing a security scan by upgrading openssl and apache on the development server. This is trivial. I just compiled a suitable version of openssl and compiled the latest apache, pointing it to the custom version of openssl. The system openssl is still 0.9.8.

Next up was upgrading Ruby and hence Passenger integration with apache. The Ruby compilation was fine and after judicious use of EXTRA flags for passenger, it installed fine too. Then I installed the app’s gems, started up apache and opened it in the browser. Bang went passenger.

...
Passenger core crashed with signal SIGSEGV
...
Premature end of script headers
...

Looking in the passenger crash log there were some suspicious openssl entries:

...
Passenger core[0x815348b]
[0xffffe410]
/usr/local/openssl/lib/libcrypto.so.1.0.0(?? at ??:0; RSA_flags+0xd) [0xb7e13b8d]
...
Passenger 9957 root  mem    REG        8,2  1455676   13282 /usr/lib/libcrypto.so.0.9.8
Passenger 9957 root  mem    REG        8,2   299032   13283 /usr/lib/libssl.so.0.9.8
...

Passenger was using the system openssl-0.9.8. I had compiled it with openssl-1.0.1c so what was going on?

A quick check of Ruby showed it was using the correct version of openssl:

export LD_LIBRARY_PATH=/usr/local/openssl/lib:/usr/local/imagemagick/lib
ruby -ropenssl -e ‘puts OpenSSL::OPENSSL_VERSION’
OpenSSL 1.0.1c 10 May 2012

The reason for the crash was clear but how the system openssl was getting involved was a mystery and clues were thin on the ground. After lots of researching, poking and prodding I drew a blank. It was time for a walk. Time away from a problem often yields results so I went down to the shore and wandered around, admiring the storm clouds coming in over the Atlantic and the wind crashing the waves on the rocky shore. Back at the desk something occurred to me.

I’d recently been reading about the passenger security check on startup and how passenger also required a certain version of curl. I hadn’t compiled a custom curl as the passenger install said the system version was adequate. Was it though? I decided to find out for myself:

ldd /usr/bin/curl
libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0xb7ff7000)

The problem was staring me in the face. Passenger was statically linking against the sytem curl which was linked against the system openssl and the whole lot was crashing on startup when passenger was doing its security check.

The solution was to compile a custom curl, linked against the custom openssl and compile passenger with the custom curl.

So the moral of the story is use something repeatable like Puppet to set up your development environment. If that’s not possible, go for a walk.