So, what's the bottleneck?

Submitted by jay on September 9, 2008 - 9:20am

I recently released some RAID testing I did using the sysbench testing framework.  In light of the recent attention paid to multi-core CPU scalability, I have been working on some related tests trying to identify sources of contention using that same set of tests.

After effectively turning off every single innodb safety setting (like flush_log_at_trx_commit, checksums, doublewrite, etc.), and not seeing any real performance increase, I started to wonder what was going on.  

Surely my test client server wasn't the problem, it had plenty of idle CPU according to top, right?  Wrong.

I've been able to drive more QPS to my mysql test servers by starting up parallel sysbench tests from multiple test servers, but using (more or less) the same number of total test threads.  

It seems like a good way to find the optimal number of testing threads is to watch the reads per second on the innodb engine using my myq_status script in a standard select by primary key test and stopping/starting sysbench with more or less threads and seeing the effect on the reads per second (it's best to wait until the buffer pool is fully loaded and the number of disk reads for the buffer pool stabilizes).  

Very subjectively, it seems like there is an optimal number of test threads per test client machine, and going over that starts to hurt performance.  

I don't have a lot of conclusions to share about this yet, except that I need to figure out a way to do more automated testing across multiple test servers rather than the manual method I'm using now (i.e., start sysbench at the same time on both servers and add the QPS results together).  

My other purpose in sharing this is to warn the other benchmarkers out there to not assume idle CPU on your test client means your server is bottlenecked.  

Trackback URL for this post:

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

More information about formatting options

Mollom CAPTCHA (play audio CAPTCHA)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.


Comment abuse is not tolerated on this site, besides all the comments are moderated, so don't bother posting comments that are not on topic, only for increasing the SEO of your site, or are outright spam.  If you've got something intelligent to contribute, by all means, post a link to your blog.  

About Me

Jay Janssen
Yahoo!, Inc.
jayj at yahoo dash inc dot com
High Availability
Global Load Balancing
View Jay Janssen on Twitter  View Jay Janssen's LinkedIn profile View Jay Janssen's Facebook profile