[solved] LiteSpeed not returning Content-Encoding header for Range requests, breaks gzip

Discussion in 'Bug Reports' started by Sean Kinsey, Aug 1, 2014.

  1. Sean Kinsey

    Sean Kinsey New Member

    When combining `Accept-Encoding: deflate, gzip` and `Range: bytes=0-524281` as request headers, the response, both gzipped and limited to the range, is returned with a missing `Content-Encoding` header.
    This causes Curl and other clients to fail to decode the response.

    oyvind-mbp:~ oyvind$ curl -v -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" -H "Range: bytes=0-524281" --compressed "http://modernpracticality.com/"
    * Adding handle: conn: 0x7fe42c003a00
    * Adding handle: send: 0
    * Adding handle: recv: 0
    * Curl_addHandleToPipeline: length: 1
    * - Conn 0 (0x7fe42c003a00) send_pipe: 1, recv_pipe: 0
    * About to connect() to modernpracticality.com port 80 (#0)
    *   Trying
    * Connected to modernpracticality.com ( port 80 (#0)
    > GET / HTTP/1.1
    > User-Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    > Host: modernpracticality.com
    > Accept: */*
    > Accept-Encoding: deflate, gzip
    > Range: bytes=0-524281
    < HTTP/1.1 206 Partial Content
    < Date: Fri, 01 Aug 2014 04:50:32 GMT
    * Server LiteSpeed is not blacklisted
    < Server: LiteSpeed
    < Accept-Ranges: bytes
    < Connection: Keep-Alive
    < Keep-Alive: timeout=5, max=100
    < Last-Modified: Fri, 01 Aug 2014 04:48:10 GMT
    < Content-Type: text/html
    < X-Pingback: http://modernpracticality.com/xmlrpc.php
    < Vary: Accept-Encoding, Cookie
    < Content-Range: bytes 0-26570/26571
    < Content-Length: 26571
    We have seen this with multiple LiteSpeed powered domains, and so believe this is related to LiteSpeed, or how it interacts with other proxy components.

    (referenced from https://developers.facebook.com/bugs/335400146618351/)
  2. mistwang

    mistwang LiteSpeed Staff

    For Range request handled by LiteSpeed Enterprise Web Server itself, gzip compression is turned off, in other words, LiteSpeed does not apply gzip compressed to range requests.

    If it was compressed, it must be done by something else.

    When I try your test case, the page is served by PHP directly, without responding to the range request.
  3. Sean Kinsey

    Sean Kinsey New Member

    Hey @mistwang, although you might be right about LiteSpeed being self-consistent, this is an issue that we are only seeing with LiteSpeed servers. Attached is another sample:

    $ curl -v --compressed -H 'Connection: close' -H 'Range: bytes=0-524288' -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "http://www.republica.com.uy/" | head
    * Hostname was NOT found in DNS cache
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying
    * Connected to www.republica.com.uy ( port 80 (#0)
    > GET / HTTP/1.1
    > User-Agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
    > Host: www.republica.com.uy
    > Accept: */*
    > Accept-Encoding: deflate, gzip
    > Connection: close
    > Range: bytes=0-524288
    < HTTP/1.1 206 Partial Content
    < Date: Tue, 18 Nov 2014 18:10:02 GMT
    * Server LiteSpeed is not blacklisted
    < Server: LiteSpeed
    < Accept-Ranges: bytes
    < Connection: close
    < Last-Modified: Tue, 18 Nov 2014 18:09:54 GMT
    < Content-Type: text/html; charset=UTF-8
    < X-Pingback: http://www.republica.com.uy/xmlrpc.php
    < Vary: Accept-Encoding, Cookie
    < Pragma: public
    < Cache-Control: public
    < Content-Range: bytes 0-22802/22803
    < Content-Length: 22803
    { [data not shown]
    �Ks#G�.���!�Hv!�"Y�G�:,�%���bY��th����D&���j��Y�屻��ؽf��5�B�1�z3f�2�����H `���f��Ȍ�G�����?���f�����=q���mQ���?/n��;G;�/_��j�"�B�Gn�������Qh�qg�\>;;+�-���Y>z[>G[UT6�V��Yrb���y�onT��J:��=��(T�F!jal'�p��/�V��B�I�k˒�KI�|ֱ�d���T[EeOg��k��b�!O�\��)�r�;Ǐ��b���V.SY_�q�!mU��%_Ž�\j��Ԡ��R���7Xe;��lȶ�u7<�mH7|��Fz^[�T���ʜ@�sQ��T�R*�q��6�bu������:��'�4���3��ucI��_����k�>X�[���P�|֍�i��,�T�0
  4. mistwang

    mistwang LiteSpeed Staff

    Please try the latest 4.2.19 build, we fixed a gzip related issue yesterday.
    If it does not fix it, we may need to login and take a look at it.
  5. Sean Kinsey

    Sean Kinsey New Member

    Hi @mistwang, I'm actually reporting this on behalf of Facebook, so won't be able to upgrade any implementations.
    I will refer bug reporters to this thread though.

    Would you be able to have someone verify if the bug could present itself as described in this thread?
  6. mistwang

    mistwang LiteSpeed Staff

    Sure, we will work with our client to figure it out.
    I probably know what is problem. Need to find a solution.

    Will facebook side takes a "200" response instead of "206 Partial Content" response contains the whole file?
  7. Sean Kinsey

    Sean Kinsey New Member

    Great, and yes, we'll accept a 200 response for the Range request.
  8. mistwang

    mistwang LiteSpeed Staff

    This issue should have been addressed in our latest 4.2.19 build. Any user reporting this issue should upgrade to 4.2.19 with

    /usr/local/lsws/admin/misc/lsup.sh -f -v 4.2.19
  9. My host (VentraIP) is running Litespeed Server API 6.7. I have several wordpress sites on this server. They all have the Wordfence plugin. It has a gzip caching system called Falcon Engine. When that is enabled, I get garbled output on Facebook scrapes. Attached is what Facebook sees after a fresh scrape of one of my websites. Compress content is disabled in cPanel. I have no other caching/gzip functions. Wordfence keep telling me that my content is getting gzipped twice, which it is not.
    Refer https://wordpress.org/support/topic...ine-gzip-on-litespeed?replies=11#post-6272195

    Attached Files:

  10. I just found out from my host that they are running Litespeed 4.2.15. I see this issue has been fixed in 4.2.19 so I will wait until they upgrade.
  11. Update - I just checked with VentraIP and they said they upgraded to 4.2.19 two weeks ago. The problem still remains with my websites. I have kept Wordfence Falcon Engine (gzip caching) enabled on this site mullewashow.com.au if you want to check via https://developers.facebook.com/tools/debug/og/object/
  12. NiteWave

    NiteWave Administrator

    please ask them to do a force-reinstall to ensure the latest 4.2.19
    on command line:
    #/usr/local/lsws/admin/misc/lsup.sh -f -v 4.2.19
  13. This is what VentraIP says:

    "The strange thing here is - the Litespeed version referenced in those posts (4.2.19) is the same as what's already installed. (Our WHM reports 4.2.19 as well)"
  14. mistwang

    mistwang LiteSpeed Staff

    tell them to do a force reinstall to get the latest build of 4.2.19.
  15. Apologies if I am repeating myself, this is VentraIPs reply:

    Could you please let them know that we are already running the latest build of LiteSpeed/4.2.19 Enterprise.

    If they simply request a force install again then please let us know.
  16. NiteWave

    NiteWave Administrator

    yes, just a simple force install
    to compare:
    #ls -l /usr/local/lsws/bin/lshttpd.4.2.19
    to know when they installed lsws 4.2.19 last time.

    then simply do a force re-install
    #/usr/local/lsws/admin/misc/lsup.sh -f -v 4,2,19
    or through lsws web admin
    Actions->Version Manager->4.2.19:Force Reinstal

    then your issue should be resolved.
  17. Just letting you know that force reinstall finally fixed the issue and scraping is now working :D

Share This Page