[GMG-Devel] debugging a large file upload, EOFError

ayleph ayleph at thisshitistemp.com
Fri Nov 21 14:32:04 EST 2014


My lighttpd config uses /tmp to hold files being uploaded. Check your nginx config to see where it stores files in transfer. Check the available disk space at that location.

Another idea I have is that perhaps you're exhausting system memory trying to load that file in for processing. But I don't understand the processing system well enough to know if that's a potential issue.

On November 21, 2014 11:27:03 AM PST, Alon Levy <alon at pobox.com> wrote:
>Hi,
>
> I see there are some tickets about large file suport (647, 834, 419)
>but I'm not sure if I'm hitting a limitation of gmg or of my webserver
>or the fcgi or firefox perhaps, so I thought I'd give details of my
>setup and ask for help:
>
> Submitting a 1.4GB video file from firefox 33.1 to mediagoblin running
>in the same network (wifi though), firefox shows a message about
>timeout
>(sadly I closed the window) and in mediagoblin I get an EOFError
>exception, details below.
>
>I'm using nginx with fastcgi. I think I got a 499: (just looking now at
>the logs)
>
>10.81.2.1 - - [21/Nov/2014:20:17:27 +0200] "POST /submit/ HTTP/1.1" 499
>0 "http://mediagoblin.tami.org.il/submit/" "Mozilla/5.0 (X11; Linux
>x86_64; rv:33.0) Gecko/20100101 Firefox/33.0" "10.81.2.231"
>
> nginx.conf has 'client_max_body_size 2560m;'
>
>
> Does anyone have a working setup for two gigabyte uploads? If so could
>you describe it? Any ideas why I cannot upload large files? I don't
>know
>what the cutoff is, I guess that would be a next step. I hope someone
>has a better suggestion,
>
>Alon
>p.s. complete stack trace from mediagoblin.log:
>
>Error - <type 'exceptions.EOFError'>:
>URL: http://mediagoblin.tami.org.il/submit/
>File
>'/usr/lib/python2.7/site-packages/paste/exceptions/errormiddleware.py',
>line 144 in __call__
>  app_iter = self.application(environ, sr_checker)
>File '/home/mediagoblin/mediagoblin/mediagoblin/app.py', line 268 in
>__call__
>  return self.call_backend(environ, start_response)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
>line 411 in __call__
>  return self.app(environ, start_response)
>File '/home/mediagoblin/mediagoblin/mediagoblin/app.py', line 188 in
>call_backend
>  request.locale = translate.get_locale_from_request(request)
>File '/home/mediagoblin/mediagoblin/mediagoblin/tools/translate.py',
>line 97 in get_locale_from_request
>  request_args = (request.args, request.form)[request.method=='POST']
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/utils.py',
>line 77 in __get__
>  value = self.func(obj)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wrappers.py',
>line 383 in form
>  self._load_form_data()
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wrappers.py',
>line 318 in _load_form_data
>  data = parser.parse_from_environ(self.environ)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
>line 164 in parse_from_environ
>  return self.parse(stream, mimetype, content_length, options)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
>line 188 in parse
>  content_length, options)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/formparser.py',
>line 100 in wrapper
>  stream.exhaust()
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
>line 787 in exhaust
>  self.read(chunk)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/Werkzeug-0.8.3-py2.7.egg/werkzeug/wsgi.py',
>line 801 in read
>  read = self._read(to_read)
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
>line 161 in read
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
>line 147 in _waitForData
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
>line 696 in process_input
>File
>'/home/mediagoblin/mediagoblin/lib/python2.7/site-packages/flup-1.0.3.dev_20110405-py2.7.egg/flup/server/fcgi_base.py',
>line 474 in read
>EOFError:
>
>
>CGI Variables
>-------------
>  CONTENT_LENGTH: '1500501143'
>  CONTENT_TYPE: 'multipart/form-data;
>boundary=---------------------------4982243468044615281985221407'
>  DOCUMENT_ROOT: '/usr/share/nginx/html'
>  DOCUMENT_URI: '/submit/'
>  GATEWAY_INTERFACE: 'CGI/1.1'
>  HTTP_ACCEPT:
>'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
>  HTTP_ACCEPT_ENCODING: 'gzip, deflate'
>  HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
>  HTTP_CONTENT_LENGTH: '1500501143'
>  HTTP_CONTENT_TYPE: 'multipart/form-data;
>boundary=---------------------------4982243468044615281985221407'
>  HTTP_COOKIE: 'mediagoblin_csrftoken=2178534525516131666;
>MGSession=eyJzdGF5X2xvZ2dlZF9pbiI6dHJ1ZSwibWVzc2FnZXMiOltdLCJ1c2VyX2lkIjoiMSJ9.B04tgg.s1z0ifQ23U1HlKy6Z-6D1Kb7Mjk;
>mediagoblin_csrftoken=2178534525516131666'
>  HTTP_HOST: 'mediagoblin.tami.org.il'
>  HTTP_REFERER: 'http://mediagoblin.tami.org.il/submit/'
>  HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Linux x86_64; rv:33.0)
>Gecko/20100101 Firefox/33.0'
>  HTTP_X_FORWARDED_FOR: '10.81.2.231'
>  PATH_INFO: '/submit/'
>  REDIRECT_STATUS: '200'
>  REMOTE_ADDR: '10.81.2.1'
>  REMOTE_PORT: '54761'
>  REQUEST_METHOD: 'POST'
>  REQUEST_URI: '/submit/'
>  SERVER_ADDR: '10.81.2.247'
>  SERVER_NAME: 'localhost'
>  SERVER_PORT: '80'
>  SERVER_PROTOCOL: 'HTTP/1.1'
>  SERVER_SOFTWARE: 'nginx/1.4.7'
>
>
>WSGI Variables
>--------------
>  application: <mediagoblin.app.MediaGoblinApp object at 0x21a0450>
>  paste.throw_errors: True
>  werkzeug.request: <Request 'http://mediagoblin.tami.org.il/submit/'
>[POST]>
>  wsgi process: 'Multiprocess'
>
>_______________________________________________
>devel mailing list
>devel at mediagoblin.org
>http://lists.mediagoblin.org/listinfo/devel

-- 
ayleph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mediagoblin.org/pipermail/devel/attachments/20141121/f1c61d8c/attachment.html>


More information about the devel mailing list