[SOLVED] One of the site suddenly goes offline frequently

Discussion in 'Bug Reports' started by hamityanik, Mar 3, 2018.

  1. hamityanik

    hamityanik Member

    Hello,
    We are using Litespeed Enterprise v5.2.4 on our CentOS 6.9 server. One particular site suddenly goes offline frequently. Website is reachable again when I restarted Litespeed. I checked the logs, there are some logs repeats.

    Code:
    2018-03-03 22:55:28.812417 [WARN] [XX.XX.XX.XX:62093] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
    2018-03-03 22:55:28.816708 [WARN] [XX.XX.XX.XX:62092] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
    2018-03-03 22:55:29.060744 [WARN] [XX.XX.XX.XX:62094] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
    2018-03-03 22:55:29.064484 [WARN] [XX.XX.XX.XX:62095] smProcessReq():  last state: HSPS_HKPT_HTTP_AUTH, current state: HSPS_HANDLER_PROCESSING, ret=-1!
    Site is not reachable when this log repeats:
    Code:
    2018-03-04 01:16:58.922001 [NOTICE] [XX.XX.XX.XX:16114] Content len: 0, Request line: 'GET /xyz1.html HTTP/1.1'
    2018-03-04 01:16:58.922024 [NOTICE] [XX.XX.XX.XX:16114] Redirect: #1, URL: /index.php
    2018-03-04 01:16:59.310905 [NOTICE] [XX.XX.XX.XX:53296] Content len: 0, Request line: 'GET /xyz2.html HTTP/1.1'
    2018-03-04 01:16:59.310941 [NOTICE] [XX.XX.XX.XX:53296] Redirect: #1, URL: /index.php
    2018-03-04 01:16:59.906645 [NOTICE] [XX.XX.XX.XX:28912] Content len: 0, Request line: 'GET /xyz3.html HTTP/1.1'
    2018-03-04 01:16:59.906667 [NOTICE] [XX.XX.XX.XX:28912] Redirect: #1, URL: /index.php
    2018-03-04 01:17:00.320637 [NOTICE] [XX.XX.XX.XX:16115] Content len: 0, Request line: 'GET /xyz4.html HTTP/1.1'
    2018-03-04 01:17:00.320664 [NOTICE] [XX.XX.XX.XX:16115] Redirect: #1, URL: /index.php
    
    Full log:
    [​IMG]

    Any suggestions? I'll be happy if you help.
    Thank you.
     
  2. NiteWave

    NiteWave Administrator

    please try:
    1. rm -rf /dev/shm/lsws
    2. /usr/local/lsws/admin/misc/lsup.sh -f -v 5.2.5
    to see if the issue will be gone
     
  3. hamityanik

    hamityanik Member

    I've updated to v5.2.5 with this way. It happened again.

    stderr.log file:
    Code:
    2018-03-04 16:27:17.035 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:17.636 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:17.736 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:17.836 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:17.936 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:18.036 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:18.136 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:18.236 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    2018-03-04 16:27:18.336 [STDERR] Reached max children process limit: 200, extra: 66, current: 266, please increase LSAPI_CHILDREN.
    error.log file is same:
    Code:
    2018-03-04 16:27:28.635663 [NOTICE] [XX.XX.XX.XX:52747] Content len: 0, Request line: 'GET /xyz.html HTTP/1.1'
    2018-03-04 16:27:28.635687 [NOTICE] [XX.XX.XX.XX:52747] Redirect: #1, URL: /index.php
    2018-03-04 16:27:29.698651 [NOTICE] [XX.XX.XX.XX:10048] Content len: 0, Request line: 'GET /xyz1.html HTTP/1.1'
    2018-03-04 16:27:29.698680 [NOTICE] [XX.XX.XX.XX:10048] Redirect: #1, URL: /index.php
    2018-03-04 16:27:29.698862 [NOTICE] [XX.XX.XX.XX:62349] Content len: 0, Request line: 'GET /xyz2.html HTTP/1.1'
    2018-03-04 16:27:29.698877 [NOTICE] [XX.XX.XX.XX:62349] Redirect: #1, URL: /index.php
    2018-03-04 16:27:30.031875 [NOTICE] [XX.XX.XX.XX:55369] Content len: 0, Request line: 'GET /xyz3.html HTTP/1.1'
    2018-03-04 16:27:30.031902 [NOTICE] [XX.XX.XX.XX:55369] Redirect: #1, URL: /index.php
    
    I'm monitoring the server continuously. Suddenly this is happened:
    [​IMG]

    [​IMG]
     
  4. hamityanik

    hamityanik Member

    Litespeed configuration is attached.
     

    Attached Files:

  5. NiteWave

    NiteWave Administrator

    it is possible the website is being (D)Dos suddenly.
    what's your Server->Security->Per Client Throttling ?
     
  6. hamityanik

    hamityanik Member

    Here you are:
    Screen Shot 2018-03-04 at 18.23.19.png
     
  7. NiteWave

    NiteWave Administrator

    please try following(example only, you can adjust it) :
    Dynamic Requests/second: 1
    Connection Soft Limit:15
    Connection Hard Limit:20
    to see if it can mitigate the attack effectively(of course need restart lsws after change made)
    in Real-Time Stats, check "Anti-DDoS Blocked IP"
     
  8. hamityanik

    hamityanik Member

    Thank you! This helped too much! Problem gone since these settings.
     
    NiteWave likes this.

Share This Page