Alright, it's directly related to caching as you said (xcache, memcached). I increased the limits as below and the problem seems to be fixed for now.
Max Connections: 50
lsphp5 memory soft limit: 512M
lsphp5 memory hard limit: 768M
(under security tab settings are same with the above)
lsphp5 process soft limit: 200
lsphp5 process hard limit: 200
(400 and 450 for security tab settings)
Are these reasonable values for a high traffic web site with 8 GB ram installed on the server?
I decreased Per Client Throttling options and noticed my pages with 100+ small gifs loads slower. So my current settings:
Static Requests/second 100
Dynamic Requests/second 50
Outbound Bandwidth (bytes/sec) 0
Inbound Bandwidth (bytes/sec) 0
Connection Soft Limit 50
Connection Hard Limit 100
Grace Period (sec) 15
Banned Period (sec) 300
I know these are a little high for preventing ddos but i want my pages which have tons of gif and flash load really fast.
What i want is reasonable ddos protection and no performance limit. Especially for Google bot crawling many pages at the same time.