[Resolved] Problems with large file uploads

Discussion in 'General' started by pythod, Jul 13, 2010.

  1. pythod

    pythod Member

    I am running a file hosting site. I recently switched over to Litespeed on one of my large storage boxes as Nginx+PHP-FPM wasn't handling file uploads properly, but mainly due to lack of support from nginx. I was hoping that LS wouldn't have the same problem, but unfortunately it does :(

    The problem is, large file upload don't work. Problems happen when I try to upload a 300MB+ file that takes around 12 mins to upload. My PHP app receives the file once the web server okays it, but if my app can't find it/LS didn't pass the uploaded file to my app. However, a 100 MB file takes around 3-4 mins to upload and works out just fine. But, when I upload a 330MB file from a slower connection that takes around 80 mins to complete, LS just doesn't accept it. I mean, I can see LS taking in the full upload, but something unknown just happens when the file upload is complete. Although, I have set the connection timeout to the maximum possible, but it just doesn't work. I'm beginning to believe it could be a system/memory cache issues. Any ideas?

    My Litespeed config:
    Connection Timeout: 10000
    Max Request URL Length (bytes) 8192
    Max Request Header Size (bytes) 16380
    Max Request Body Size (bytes) 1G
    Max Dynamic Response Header Size (bytes) 8K
    Max Dynamic Response Body Size (bytes) 2047M

    Swap is on /dev/shm (8GB RAM drive) - File uploads are temporarily stored by LS here.

    Here's my current setup:
    Ubuntu 10.04 server amd64
    1 x Intel Xeon E5405
    16GB RAM
    300 GB SATA 10K RPM
    24 x 1 TB SATA
    Litespeed Web Server Enterprise v4.0.15
    PHP 5.2.11 LS SAPI

    Last edited by a moderator: Jul 14, 2010
  2. mistwang

    mistwang LiteSpeed Staff

    When upload the 300MB file, have you noticed that LSWS storing the request body in the swap directory? If you do, it means LSWS accepts and receives the request.
    You can turn on debug logging and check if LSWS send the request body to PHP successfully or not.
    There might be a problem on PHP side when receiving the request body from LSWS. First thing is memory limit, PHP uses a lot of memory handling a large upload. The memory limit should be set high enough, check error log and stderr.log see if there are memory related errors from PHP.
  3. pythod

    pythod Member

    Yes, I am able to see the request body while the client is POSTing to the server.

    PHP's values are also pretty much the same as pulled out from php.ini:

    Directive Local Value Master Value
    max_execution_time 7200 7200
    max_input_nesting_level 64 64
    max_input_time 7200 7200
    memory_limit 4000000000 4000000000
    post_max_size 2047M 2047M

    You'll notice that these values are more than sufficient to allow a 300MB upload and that doesn't take more than 1 hour!

    All uploads less than 100-150MB are being posted to PHP just fine, but when upload size is 300MB+ and takes more than 20-30 mins, the request body is not being sent to PHP!

    I just checked error.log and there are no memory related errors from PHP. I just enabled stderr Log, so as soon as I have something, I will post it here.

  4. pythod

    pythod Member

    I turned stderror logging on, but there's nothing being written to that file.

    What I noticed is that a large file can be uploaded as soon as the server is restarted, and then after some time (say hour or two later), if I try to upload the same file, php doesn't receive it!
  5. NiteWave

    NiteWave Administrator

    can you try lsws 4.1RC3 and see if any difference?
  6. pythod

    pythod Member

    Well, that's difficult because I already have a lot of traffic hitting on this server. Can I try any other config fix?
  7. pythod

    pythod Member

    Would you like access to my LS's admin to see if all configs are all right?
  8. NiteWave

    NiteWave Administrator

    it should be safe -- the 4.0.15 still there. you can switch between 4.1RC3 and 4.0.15 in admin console safely and easily, just by clicking one button.

    and with 4.1RC3, the upload progress bar scripts like uber-upload is working as expected. (please refer http://www.litespeedtech.com/support/forum/showthread.php?t=4136 for more info)

    I can check your lsws settings after you've tested 4.1RC3.
  9. pythod

    pythod Member

    I can't find the link to 4.1RC3 or the instructions to install it. Could you please help?
  10. pythod

    pythod Member

  11. pythod

    pythod Member

    Alright, so got the 4.1RC3 installed, but the same problem persists! A 100MB file that took around 15 mins to upload was not received by my PHP script. However, a 330MB file that took around 14 mins to upload was received just fine... What shall I do here now?
  12. pythod

    pythod Member

    Something else I should note down here. I have /dev/shm/ls/swap for ls swap and /dev/shm for php temp location. When file uploads are in progress, I can see the same file being written at both the swap location and /dev/shm. Although the swap contains all form elements, while the temp php file (e.g. /dev/shm/phpZJm3V0) contains the actual file that I am uploading. I just recently changed the temp file location in php.ini and previously it was set to /tmp. I was not able to see any php temp files at that location earlier until after I changed the temp file location in php.ini.
  13. NiteWave

    NiteWave Administrator

    the lsphp itself has memory soft/hard limit, just re-read previous posts, not mention it yet.

    it's in Server-->External App-->lsphp5-->Memory Soft Limit (bytes)/Memory Hard Limit (bytes), default is 250M/300M.
  14. pythod

    pythod Member

    Actually I just discovered that the fault, not exactly the fault, but our "upload confirmation" page was set to expire 15 minutes after the upload started, so after changing that value to 3 hours, the upload page finally appears. Therefore, in reality the upload was actually getting completed, but the upload confirmation wasn't showing up. Anyhow, sorry for all this trouble. This issue has now been resolved!

    Thanks again.

Share This Page