LiteSpeed Support Forums - [solve] cpu usage high

LiteSpeed Support Forums (http://www.litespeedtech.com/support/forum/index.php)

- General (http://www.litespeedtech.com/support/forum/forumdisplay.php?f=25)

- - [solve] cpu usage high (http://www.litespeedtech.com/support/forum/showthread.php?t=6305)

[solve] cpu usage high

hi guys

i'm hosting a site that gets 1M+ visits a day in litespeed right now.
the site is a combination of php and cgi and i noticed that my cpu's hover around 20-30 percent w/ load avg around 8-15. server has 32 cpu cores of intel romley btw. litespeed is configured with lsphp + suexec.

the server has a ton of ram (64gb), but it is only utilizing 2-3 percent of it and i was wondering if i haven't configured litespeed correctly for this. disk is SSD w/ disk i/o less than 2 percent;. mysql is hosted on separate server.

is there some setting that i'm not aware of?

i've upped all the sysctl settings, like this:

net.ipv4.netfilter.ip_conntrack_max=524288
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn = 10000

not sure if it's too low or too high.

can i please get some assistance in optimizing our current situation?

here is a snapshot of the top:

top - 13:55:25 up 41 days, 1:55, 2 users, load average: 11.67, 11.76, 11.85
Tasks: 469 total, 12 running, 457 sleeping, 0 stopped, 0 zombie
Cpu0 : 46.7%us, 3.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 16.7%hi, 33.3%si, 0.0%st
Cpu1 : 50.0%us, 15.6%sy, 0.0%ni, 31.2%id, 0.0%wa, 0.0%hi, 3.1%si, 0.0%st
Cpu2 : 63.6%us, 12.1%sy, 0.0%ni, 21.2%id, 0.0%wa, 0.0%hi, 3.0%si, 0.0%st
Cpu3 : 68.8%us, 12.5%sy, 0.0%ni, 18.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 70.0%us, 13.3%sy, 0.0%ni, 16.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 78.1%us, 12.5%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 74.2%us, 12.9%sy, 0.0%ni, 12.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 75.0%us, 12.5%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 3.1%si, 0.0%st
Cpu8 : 62.5%us, 12.5%sy, 0.0%ni, 25.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 67.7%us, 9.7%sy, 0.0%ni, 22.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 61.3%us, 12.9%sy, 0.0%ni, 25.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 40.6%us, 6.2%sy, 0.0%ni, 53.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 35.5%us, 3.2%sy, 0.0%ni, 61.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 12.9%us, 3.2%sy, 0.0%ni, 83.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 15.6%us, 3.1%sy, 0.0%ni, 81.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu17 : 16.7%us, 3.3%sy, 0.0%ni, 80.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu19 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu20 : 9.7%us, 0.0%sy, 0.0%ni, 90.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21 : 6.7%us, 0.0%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu22 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu24 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu25 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu26 : 12.5%us, 9.4%sy, 0.0%ni, 78.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu27 : 75.0%us, 15.6%sy, 0.0%ni, 9.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 51532768k total, 35681456k used, 15851312k free, 554420k buffers
Swap: 7124816k total, 0k used, 7124816k free, 32931908k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4852 someuser 25 0 73480 11m 2328 R 28.8 0.0 0:00.09 index_dl.cgi
4861 someuser 25 0 73472 11m 2328 R 22.4 0.0 0:00.07 index_dl.cgi
5736 nobody 0 -19 124m 21m 6872 S 22.4 0.0 31:23.28 litespeed
4863 someuser 25 0 73424 11m 2328 R 19.2 0.0 0:00.06 index_dl.cgi
4864 someuser 25 0 73376 11m 2332 R 16.0 0.0 0:00.05 index_dl.cgi
4866 someuser 25 0 42720 9816 1484 R 16.0 0.0 0:00.05 index_dl.cgi
4865 someuser 25 0 69200 10m 2304 S 12.8 0.0 0:00.04 index_dl.cgi
4870 someuser 24 0 38708 7568 1452 R 9.6 0.0 0:00.03 index_dl.cgi
4869 someuser 25 0 27400 4548 1424 R 6.4 0.0 0:00.02 index_dl.cgi
4874 someuser 24 0 27380 4528 1424 R 6.4 0.0 0:00.02 index_dl.cgi
2417 someuser 15 0 138m 9868 6284 S 3.2 0.0 0:00.16 lsphp5
2653 root 15 0 13020 1424 828 R 3.2 0.0 0:00.27 top
4701 someuser 15 0 138m 9552 5900 S 3.2 0.0 0:00.04 lsphp5
4791 someuser 18 0 69196 10m 2288 S 3.2 0.0 0:00.05 index_dl.cgi
4875 someuser 24 0 21348 2748 1356 R 3.2 0.0 0:00.01 index_dl.cgi
4878 someuser 24 0 21240 2636 1352 R 3.2 0.0 0:00.01 index_dl.cgi
5718 root 4 -19 5892 440 352 S 3.2 0.0 1:16.52 httpd
1 root 15 0 10364 692 580 S 0.0 0.0 0:06.43 init
2 root RT -5 0 0 0 S 0.0 0.0 0:15.52 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:03.30 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:17.37 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.26 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root RT -5 0 0 0 S 0.0 0.0 0:21.21 migration/2
9 root 34 19 0 0 0 S 0.0 0.0 0:00.24 ksoftirqd/2
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
11 root RT -5 0 0 0 S 0.0 0.0 0:14.93 migration/3
12 root 34 19 0 0 0 S 0.0 0.0 0:00.43 ksoftirqd/3
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
14 root RT -5 0 0 0 S 0.0 0.0 0:28.79 migration/4
15 root 34 19 0 0 0 S 0.0 0.0 0:00.21 ksoftirqd/4
16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4
17 root RT -5 0 0 0 S 0.0 0.0 0:29.10 migration/5
18 root 34 19 0 0 0 S 0.0 0.0 0:00.27 ksoftirqd/5

Quote:

load avg around 8-15.

you have 28 (32 ?) CPU and normal load is 28, now only 25% - 50% load, so not high, there is still room for higher traffic, for example, 2M visits.

yes but when load avg starts to reach 15+ (indicating more traffic), the site starts to slow down and response time of site starts to become greater than 4+ secs, when normally it's 1 second or so.
could i be hitting some max connections settings (either litespeed and/or php, kernel, etc) and it is placed in queue?

so when slow down, capture the screen shot of real-time stats and paste here. it provides many useful info there. most likely can identify where the bottle-neck is.

Hi Nitewave, site is taking a while to load now. here is a screen paste of w, top, and iostat:

// w

-bash-3.2# w
14:30:52 up 6 days, 22:05, 2 users, load average: 20.58, 22.98, 24.17

// top by cpus

top - 14:31:41 up 6 days, 22:06, 2 users, load average: 21.18, 22.81, 24.05
Tasks: 567 total, 17 running, 550 sleeping, 0 stopped, 0 zombie
Cpu0 : 7.7%us, 7.7%sy, 0.0%ni, 0.0%id, 0.0%wa, 30.8%hi, 53.8%si, 0.0%st
Cpu1 : 53.3%us, 6.7%sy, 0.0%ni, 33.3%id, 0.0%wa, 0.0%hi, 6.7%si, 0.0%st
Cpu2 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 50.0%us, 14.3%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu4 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 61.5%us, 7.7%sy, 0.0%ni, 30.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 61.5%us, 7.7%sy, 0.0%ni, 30.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 78.6%us, 14.3%sy, 0.0%ni, 7.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 71.4%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 66.7%us, 8.3%sy, 0.0%ni, 25.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 64.3%us, 14.3%sy, 0.0%ni, 21.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 78.6%us, 14.3%sy, 0.0%ni, 7.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 64.3%us, 7.1%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 28.6%us, 7.1%sy, 0.0%ni, 57.1%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu15 : 64.3%us, 7.1%sy, 0.0%ni, 28.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu16 : 21.4%us, 7.1%sy, 0.0%ni, 64.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu17 : 76.9%us, 15.4%sy, 0.0%ni, 7.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu18 : 21.4%us, 14.3%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 21.4%si, 0.0%st
Cpu19 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu20 : 14.3%us, 14.3%sy, 0.0%ni, 71.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu21 : 61.5%us, 7.7%sy, 0.0%ni, 23.1%id, 0.0%wa, 0.0%hi, 7.7%si, 0.0%st
Cpu22 : 46.2%us, 7.7%sy, 0.0%ni, 46.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu23 : 71.4%us, 7.1%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Cpu24 : 50.0%us, 7.1%sy, 0.0%ni, 42.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu25 : 46.2%us, 7.7%sy, 0.0%ni, 38.5%id, 0.0%wa, 0.0%hi, 7.7%si, 0.0%st
Cpu26 : 76.9%us, 15.4%sy, 0.0%ni, 7.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu27 : 53.8%us, 7.7%sy, 0.0%ni, 38.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu28 : 46.2%us, 7.7%sy, 0.0%ni, 46.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu29 : 64.3%us, 14.3%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 7.1%si, 0.0%st
Mem: 51532700k total, 48579012k used, 2953688k free, 550444k buffers
Swap: 7124816k total, 0k used, 7124816k free, 44709260k cached

// top

top - 14:31:09 up 6 days, 22:06, 2 users, load average: 21.34, 23.01, 24.16
Tasks: 567 total, 20 running, 547 sleeping, 0 stopped, 0 zombie
Cpu(s): 63.0%us, 12.0%sy, 0.0%ni, 20.8%id, 0.0%wa, 1.0%hi, 3.1%si, 0.0%st
Mem: 51532700k total, 48567912k used, 2964788k free, 550444k buffers
Swap: 7124816k total, 0k used, 7124816k free, 44704736k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15041 nobody 0 -19 79736 30m 6952 S 53.1 0.1 8:09.29 litespeed
20394 someuser 25 0 69064 10m 2076 S 9.0 0.0 0:00.27 index_dl.cgi
20756 someuser 25 0 73624 11m 2360 R 8.6 0.0 0:00.26 index_dl.cgi
16314 someuser 15 0 138m 9876 6292 S 4.6 0.0 0:00.48 lsphp5
20634 someuser 25 0 69248 10m 2288 S 4.3 0.0 0:00.13 index_dl.cgi
20800 someuser 25 0 73580 11m 2360 R 4.0 0.0 0:00.12 index_dl.cgi
4757 someuser 15 0 138m 10m 6616 S 3.7 0.0 0:01.70 lsphp5
7801 someuser 15 0 138m 13m 8968 S 3.7 0.0 0:01.49 lsphp5
20808 someuser 25 0 42720 9816 1484 R 3.7 0.0 0:00.11 index_dl.cgi
20803 someuser 25 0 79836 12m 2396 S 3.3 0.0 0:00.10 index_dl.cgi
20807 someuser 18 0 42716 9800 1484 R 3.3 0.0 0:00.10 index_dl.cgi
20812 someuser 25 0 73580 11m 2360 S 3.3 0.0 0:00.10 index_dl.cgi
11907 someuser 15 0 139m 13m 8904 S 3.0 0.0 0:00.85 lsphp5
12113 someuser 16 0 139m 13m 8968 S 3.0 0.0 0:00.90 lsphp5
19864 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
19865 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
19882 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20115 someuser 25 0 69068 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20291 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20354 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20474 someuser 23 0 69200 10m 2272 S 3.0 0.0 0:00.09 index_dl.cgi
20530 someuser 17 0 69196 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20567 someuser 25 0 69068 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20621 someuser 25 0 69248 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20639 someuser 25 0 69248 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20665 someuser 25 0 69196 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
20714 someuser 25 0 69064 10m 2076 S 3.0 0.0 0:00.09 index_dl.cgi
20717 someuser 25 0 69200 10m 2288 S 3.0 0.0 0:00.09 index_dl.cgi
19868 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19915 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19916 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19923 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19937 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20047 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20048 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20072 someuser 25 0 69064 10m 2080 S 2.7 0.0 0:00.08 index_dl.cgi
20078 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20292 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20294 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20301 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20353 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20361 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20410 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20415 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20416 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20468 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20469 someuser 25 0 69068 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20471 someuser 25 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
20819 someuser 19 0 69064 10m 2076 S 2.7 0.0 0:00.08 index_dl.cgi
19944 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20131 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20133 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20171 someuser 25 0 69068 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20296 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20366 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20460 someuser 25 0 69200 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20585 someuser 25 0 69064 10m 2076 S 2.3 0.0 0:00.07 index_dl.cgi
20748 someuser 25 0 69248 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20805 someuser 25 0 69200 10m 2288 S 2.3 0.0 0:00.07 index_dl.cgi
20815 someuser 25 0 69204 10m 2248 S 2.3 0.0 0:00.07 index_dl.cgi
20823 someuser 25 0 42720 9812 1484 R 2.3 0.0 0:00.07 index_dl.cgi
20824 someuser 25 0 42716 9812 1484 R 2.3 0.0 0:00.07 index_dl.cgi
15011 root 6 -19 5892 436 352 S 2.0 0.0 0:19.76 httpd
20239 someuser 25 0 69064 10m 2076 S 2.0 0.0 0:00.06 index_dl.cgi
20271 someuser 25 0 69064 10m 2076 S 2.0 0.0 0:00.06 index_dl.cgi
20588 someuser 25 0 69068 10m 2080 S 2.0 0.0 0:00.06 index_dl.cgi

// iostat
-bash-3.2# iostat
Linux 2.6.18-308.11.1.el5 10/18/2012

avg-cpu: %user %nice %system %iowait %steal %idle
19.63 0.03 5.86 0.01 0.00 74.47

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hda 10.52 9.09 709.88 5434707 424570329
hda1 0.00 0.00 0.00 2707 71
hda2 5.21 0.27 126.23 161635 75498586
hda3 0.00 0.00 0.00 1459 0
hda4 0.00 0.00 0.00 8 0
hda5 5.30 8.81 583.65 5268306 349071672

Please let me know if there is something I can do to optimize.

it seems your cgi scripts take most cpu cycles. so your application is more of cpu bound.

take a look at real-time stats in LiteSpeed Admin Console (Actions=> Real-Time Stats) and paste a screen shot here (as NiteWave requested). It should tell us whether your application processes are piling up and where bottleneck if any, etc.

Is this for busy time? lsws seems to be ok. NO processes in WaitQ which means no delay in processing PHP or CGI. BTW, is your up link 100Mb/s or 1Gb/s?

i don't think we really have a busy time, traffic is constant throughout as we also get lot of international traffic.
uplink is 100mb. do you think we are hitting a bottleneck in network?
i see we use 30mb down and 30-50mb up.