Database Madness – Day 1

I work at WhatCounts, an WhatCounts, Inc.. We send out email marketing and newsletter campaigns for the corporate-type clients. From time to time, we run into database and other system performance problems. Our database is almost a terabyte in size, so performance is an absolute must when dealing with this much data.

It all started on Wednesday around 10am. Complaints started coming in that the publishing jobs were not running. We checked and found one job was holding up the entire publishing queue. It was running, but very, very slowly. We check the normal things: blocking queries, slow network, etc. All checked out fine. As we dug a deeper, we found this one client’s publishing job retrieved different content for each message from an external site, and the content was conditional, depending on who was getting the email. We have two technologies to retrieve content: one that caches and one that doesn’t. This one was using the one that doesn’t. Fortunately for us, we could just go into their email template and change it to the one that does. That’s what we did, and the system started flowing nicely again.

Little did we know that this was just the beginning of our problem.