So, what's the bottleneck?

I recently released some RAID testing I did using the sysbench testing framework.  In light of the recent attention paid to multi-core CPU scalability, I have been working on some related tests trying to identify sources of contention using that same set of tests.

After effectively turning off every single innodb safety setting (like flush_log_at_trx_commit, checksums, doublewrite, etc.), and not seeing any real performance increase, I started to wonder what was going on.  

Surely my test client server wasn't the problem, it had plenty of idle CPU according to top, right?  Wrong.

I've been able to drive more QPS to my mysql test servers by starting up parallel sysbench tests from multiple test servers, but using (more or less) the same number of total test threads.  

Innodb RAID performance on 5.1

I've been doing some benchmarking recently to satisfy the curiosity about 5.1's performance compared with 4.1.  The major question this time revolves around how much additional performance an external RAID array can provide (for us it's typically beyond the 6 drives a Dell 2950 can hold). 

These tests are done on using an MSA-30 drive enclosure with 15k-SCSI drives.  The testing framework is sysbench oltp.  The test names are hopefully fairly obvious:  selects = single selects, reads = range tests, xacts = transaction tests, etc.   Transaction tests are counting individual queries, not transactions.   The "Rdm" tests are using a uniform distribution, whereas the non-'Rdm' tests are 75% of queries are using 10% of the rows.  

Innodb RAID performance testing on 5.1

Innodb RAID performance testing on 5.1

Innodb Multi-core Performance

There's been a lot of rumors floating around internally at Yahoo that it's best to turn off some of your CPU cores when using Innodb, especially if you have a machine with > 4 cores.  At this point there's no question in my mind that Innodb doesn't perform much better when you double your cores from 4 to 8, but I really wanted to know if 8 actually performed worse. 

To test, I used a Dell 2950 with 6 drives and a simple mysqlslap test script.  There's basically no I/O going on here, just a small table in memory being queried a lot.  To be fair, I actually got this test from Venu.  I used maxcpu=4 in my grub.conf to limit the cpus (I also tested with tasksel and it seemed to have the same effect as maxcpu).

About Me

Jay Janssen
Yahoo!, Inc.
jayj at yahoo dash inc dot com
High Availability
Global Load Balancing
View Jay Janssen on Twitter  View Jay Janssen's LinkedIn profile View Jay Janssen's Facebook profile