So you want to talk about Single points of failure, eh?

In reply to Arjen's post about Single points of failure:

Arjen, you are absolutely right.  It doesn't matter how over-engineered a storage solution is (I'm thinking of a giant dual-headed Netapp with redundant everything).  After you've paid a few hundred K for that, you still have a single point of failure.  Is it a highly-unlikely point of failure?  Sure, but it's still a point of failure. 

Let's take it a step further, at Yahoo we're beyond thinking about how to make a single node redundant (be it for storage, networking, or even a simple webserver), we consider entire datacenters to be single points of failure.  What does that mean?  

So, what's the bottleneck?

I recently released some RAID testing I did using the sysbench testing framework.  In light of the recent attention paid to multi-core CPU scalability, I have been working on some related tests trying to identify sources of contention using that same set of tests.

After effectively turning off every single innodb safety setting (like flush_log_at_trx_commit, checksums, doublewrite, etc.), and not seeing any real performance increase, I started to wonder what was going on.  

Surely my test client server wasn't the problem, it had plenty of idle CPU according to top, right?  Wrong.

I've been able to drive more QPS to my mysql test servers by starting up parallel sysbench tests from multiple test servers, but using (more or less) the same number of total test threads.  

Innodb RAID performance on 5.1

I've been doing some benchmarking recently to satisfy the curiosity about 5.1's performance compared with 4.1.  The major question this time revolves around how much additional performance an external RAID array can provide (for us it's typically beyond the 6 drives a Dell 2950 can hold). 

These tests are done on using an MSA-30 drive enclosure with 15k-SCSI drives.  The testing framework is sysbench oltp.  The test names are hopefully fairly obvious:  selects = single selects, reads = range tests, xacts = transaction tests, etc.   Transaction tests are counting individual queries, not transactions.   The "Rdm" tests are using a uniform distribution, whereas the non-'Rdm' tests are 75% of queries are using 10% of the rows.  

Innodb Multi-core Performance

There's been a lot of rumors floating around internally at Yahoo that it's best to turn off some of your CPU cores when using Innodb, especially if you have a machine with > 4 cores.  At this point there's no question in my mind that Innodb doesn't perform much better when you double your cores from 4 to 8, but I really wanted to know if 8 actually performed worse. 

To test, I used a Dell 2950 with 6 drives and a simple mysqlslap test script.  There's basically no I/O going on here, just a small table in memory being queried a lot.  To be fair, I actually got this test from Venu.  I used maxcpu=4 in my grub.conf to limit the cpus (I also tested with tasksel and it seemed to have the same effect as maxcpu).

No conference for me, *sniff*

*Sigh*, well I won't be making it out to the conference this year:  My wife is due any day now with our third child.  Scheduling to fly to California next week just doesn't seem like it falls under the category of "supporting my wife."  

Top ten things I'll miss about the mysql conference 2008:
  1. How replication is dead
  2. Why 5.1 isn't stable yet
  3. How wonderful Sun/Maria/Falcon/Cluster/6.x/Proxy is
  4. Grilling Heikki about Innodb internals and bottlenecks
  5. Wringing more information about high performance MySQL from Paul Tuckfield
  6. The food!
  7. Vendor / O'Reilly Swag
  8. How great MySQL will run on Sun hardware
  9. Conference Day 3 daze -- that part of the conference where your eyes start to glaze over. 
  10. Getting really, really sick of talking about MySQL
Have fun everyone, see you next year!

About Me

Jay Janssen
Yahoo!, Inc.
jayj at yahoo dash inc dot com
High Availability
Global Load Balancing
View Jay Janssen on Twitter  View Jay Janssen's LinkedIn profile View Jay Janssen's Facebook profile