Performance issue (Resolved)

Upgraded the nginx version to 1.0.10 and while testing noticed a very bad timeout with the php fpm backend. Since it was well past midnight and it didnt appear to be causing an issue with the Drupal 6.x sites, I decided to get some sleep and put the task of resolving the issue in the background.

Got up, fumbled around with no results other than making it slightly better, but still the normal sub millisecond response was 1000x worse. Set nginx into debug mode and took a look at the response logs, sometimes would get good results and other times 10+ second delays and/or timeouts. What could be causing this?

Issue seemed to be random. Kept digging and finally got a clue when I logged into the mysql cli. Slow response. Next I used "strace drush -d status" to run a site from the command line. Yep, waiting on the database.

I logged into the db server and ifconfig showed rx errrors. I went ahead and restarted it and it seems to have cleared the issue. Still looking for the root cause, but for now it is back to normal. I suspect it is the port on the gigabit switch. Will have to make a visit to the co-lo with a new/better switch if it happens again.

I did find it interesting that this caused more problems for the Drupal 7.x sites than the Drupal 6.x sites. Maybe the PDO db layer is more picky, but it would be interesting to test it under such a situation to see exactly why it would be slower than what is in Drupal 6.x.