using apache qpid with persistence

that's just a quick post about two weeks struggle with a problem which was eventually solved in half an hour.

as at my company we (i) decided to use apache QPID as a message queue framework, we certainly needed to use persistance for a queues and messages. building qpid itself was really easy and straight forward, and for building the persistance module msgstore.so i was following the directions posted at Lahiru Gunathilake's Blog here. i build the whole setup on my local machine and everything was working just perfectly - messages were sent persitently, queues were durable etc.

then i started deploying on the dev servers

the qpid broker started crashing on startup.
the qpid broker started crashing on startup.
the qpid broker started crashing on startup.

and on and on and on

it was easy to detect that the problem appeared when the broker was started with --load-module=msgstore.so and that it seg faulted when attempting to create a berkeley db database. but why??
well...on my local machine i am running fedora, and the dev server rules with CentOs.
my fedora has berkeley db 4.8 and the server has 4.5. this was the first 'ding'.

now, how to trick the qpid configure into using db4.8 instead of db4.5? we couldn't just install the new libs as there might have been incompatibilities with the product, so what to do? we tried modifying configure scripts, playing with sym links etc - i myself don't have much experience with linux so was relying mostly on the admin guy, but he was helpless with this too - seg fault after segfaul, while on my machine the broker hummed silently, transfering persistant messages and maintaining durable queues.

then after a loooot of reading we came to an obvious solution - the qpid configure itself gave it too us, and hadn't i been too shortsighted, this would have been done days ago -

just build berkeley 4.8 with --perfix to place it in a separate directory (say $BDB48), away from the db4.5 libs the product needs, and then, before running qpid configure, run these

export CPP_PATH=$BDB48/include
export LIB_PATH=$BDB48/libs

(i'm writing these by memory, check qpid's configure script with --help to see the correct ones)

after that run configure and it will find and use the berkeley libs you wanted it too!

awesome!
now lets do the same in production where we have RedHad 5. BOOM! here goes the so familiar seg fault that we managed to escape by the clever export trick. why? what is so different on redhat???
another week of research, testing, breaking, hair-tearing followed, this time with no result.
i was even contemplating to run a virtual fedora machine on the redhat server, just and so to have the familiar setup and start the broker with persitancy there.

luckily i didn't have to -
one day i sat with the it manager to explain him the situation and while we were scratching our heads we looked at persistance module's readme.txt where it said that it was tested with berkeley db 4.3.
so what? we have db4.8 and it should be fine, right?
what if we gave it another try, but not with the 4.8, but 4.3 set-up the same way?
this took about10 minutes to setup and when i hit 'enter' for qpidd --load-module=msgstore.so, before my eyes was the beautiful log dump, saying that the module was loaded and so on and so on ....

aaah....rtfm? the thing is we actually tested db4.3 when facing the initial problem of setting up the dev servers. this failed somehow, and so we didn't consider this when fighting with production setup

anyways - this post grew as long as the first two episodes of "game of thrones" that i watched today :D

have a good night and don't give up the fight!

OSSP mm - Shared Memory Allocation

i played a few days with the mm shared memory lib. i must say i liked it!
it is easy to understand and easy to implement into existing code.
of course the lack of new()/delete() is somewhat limiting, but not that difficult to overcome in most cases.
alternatively, on our project we were looking into boost:interprocess, but i was kinda reluctant to use it, mostly because of the syntax (btw i found a boost/ace alternative at POCO, the users and designers say it provides boost/ace-a-like functionality, even if it cannot compare in perfomance, but the best gain is the C#-oriented syntax; probably worth it to examine).

there was one really weird problem with mm though, when i got the impression that mm_lock() wasnt working.
the workflow was a follows:
we have a hash located on the shared memory segment and before i get() from it i mm_lock it and subsequently mm_unlock() it. still i noticed that many times same value was being written to the hash, which was unexpected by design.
i fired our test up to create a few working processes and breaked in each one before a get(). the locks seemed to work, but after a put() suddenly the lock seemed to vanish in the haze?! digging deeper i found out that in put() we of course try to allocad some new memory in the shared mem segment and traceing down into mm_malloc i found out it was trying to lock too, and when done - calling mm_unlock()! piece of bad design or weird side-effect? either way, apart from that my impressions with OSSP mm were great. however it turned out that the boost guys go a few more tricks up in their sleeves, especially the name persisted storage seemed a very attractive idea for our purposes and we eventually decided to use the boost approach.
so...moving on and up again :)

random performace thoughts

just wanted to lay things out so i can see them more clearly. if it doesn't make sense to you, dear readers, please excuse me :)

so, we have many hashes (maps, containing cached data), but due to our implementation with fork() all the data is lost once a particular task is done. i had the idea of storing all those in a berkeley db so that every new forked task gets it from there and doesn't have to fill the hash by itself, but then - what is a task gonna read from the db when it doesn't know what it is gonna need beforehand. we cannot read everything as the data is HUGE and we cant read bit by bit as this slows down performance a lot - we had a test case like this: we tried using memcached as a remote cache, but this slowed us down even if we read from it only once into a local hash and all next calls where local (quick); if we hadn't read into local hash first, the slowdown would be even more devastating.

one possible solution is for the main process to reread all from the stored data and all children would get it via fork/copy-on-write and then put any changes/additions back into the db for the parent to read when a new task requests start.

i have another one around - when if we realise what we need from the cache much earlier than when we need it? we could then start a thread to pre-load the expected data while the main thread goes on processing and when it reached the point when the hash is needed - voila! its all up there! in case there are gaps they can be filled runtime, should be a small percentage of the overall effort

i'm working on a shared memory solution, but all these got me into doubt about the performance hit, similar to hitting the memcached everytime (by the way, i found out about OSSP mm shared memory allocation project at http://www.ossp.org/pkg/lib/mm/, and it looks really good. i dont know how i missed it earlier, guess i was looking for a std-kind-of allocator and got ptu back by some messages that boost had given up on the idea. another btw - is the new std with C++x0 gonna include something like this? from what i read it does include some fancy new stuff related to multithreaded and multicore programming which is awesome. would be great once we get our hands on it)

so where was i....aaah...back to the drawing board, time for another brainstoriming session...tomorrow :D