After effectively turning off every single innodb safety setting (like flush_log_at_trx_commit, checksums, doublewrite, etc.), and not seeing any real performance increase, I started to wonder what was going on.  

Surely my test client server wasn't the problem, it had plenty of idle CPU according to top, right?  Wrong.

I've been able to drive more QPS to my mysql test servers by starting up parallel sysbench tests from multiple test servers, but using (more or less) the same number of total test threads.  

It seems like a good way to find the optimal number of testing threads is to watch the reads per second on the innodb engine using my myq_status script in a standard select by primary key test and stopping/starting sysbench with more or less threads and seeing the effect on the reads per second (it's best to wait until the buffer pool is fully loaded and the number of disk reads for the buffer pool stabilizes).  

Very subjectively, it seems like there is an optimal number of test threads per test client machine, and going over that starts to hurt performance.  

I don't have a lot of conclusions to share about this yet, except that I need to figure out a way to do more automated testing across multiple test servers rather than the manual method I'm using now (i.e., start sysbench at the same time on both servers and add the QPS results together).  

My other purpose in sharing this is to warn the other benchmarkers out there to not assume idle CPU on your test client means your server is bottlenecked.