File Downloads Done Right
Getting your file downloads right is one of the most important parts of your File Management functionality. A poorly implemented download function can make your application painful to use, not just for downloaders, but for everyone else too.
Thankfully it’s also one of the easiest things to get right.
The simple version
For the purposes of this article let’s assume that your application needs to provide access to a large zip file, but that access should be restricted to logged in users.
The first choice we have to make is where to store this file. In this case there’s really only one wrong answer, and that’s to store it in the public folder of your rails application. Every file stored in public will be served by our webserver without the involvement of our rails application. This makes it impossible for us to check that the user has logged in. Unless your files are completely public, you shouldn’t go anywhere near the public folder.
So let’s assume we’ve stored the zip file in:
/home/railsway/downloads/huge.zipNext we need a simple download action to send the file to the user, thankfully rails has this built right in:
before_filter :login_required
def download
send_file '/home/railsway/downloads/huge.zip', :type=>"application/zip"
end
Now when our users click the download link, they’ll be asked to choose a location and then be able to view the file. The bad news is, there’s a catch here. The good news is it’s easy to fix.
What’s the catch?
The problem here is one of scarce resources, and that resource is your rails processes. Whether you’re using mongrel, fastcgi or passenger you have a limited number of rails processes available to handle application requests. When one of your users makes a request, you want to know that you either have a process free to handle the request, or that one will become free in short order. If you don’t, users will face an agonizing wait for pages to load, or see their browser sessions timeout entirely.
When you use the default behaviour of send_file to send the file out to the user, your rails process will read through the entire file, copying the contents of the file to the output stream as it goes. For small files like images this probably isn’t that big of a deal, but for something enormous like a 200M zip file, using send_file will tie up a process for a long time. Users on slow connections will soak up a rails process for correspondingly longer.
If you get a large number of downloads running, you may find all your rails processes taken up by downloaders, with none left to serve any other users. For all intents and purposes your site is down: you’ve instituted a denial of service attack against yourself.
What about threads?
Unfortunately threads in ruby won’t save us. The combination of blocking IO and green threads mean that even though you’re doing the work in a thread, it’s blocking the entire process most of the time anyway. JRuby users may get a performance improvement, but it’s still going to be a noticeable consumption of resources when compared to letting a web server stream the file.
Don’t believe everything you read on the internet, threads and ruby just won’t help you with most of this stuff.
So What’s the Solution?
Thankfully this problem was solved a long time ago by the guys at live journal. They used perl instead of ruby, but had the same problems. Downloading files would block their application processes for too long, and cause other users to have to wait. Their solution was elegant and simple. Instead of making the application processes stream the file to the user, they simply tell the webserver what file to send, and let the web server bother with the details of streaming the file out to the client.
Their particular solution is quite cumbersome to set up and use, but there’s a very similar solution available called X-Sendfile. It’s supported out of the box with later versions of lighttpd, and available as a module for apache.
The way it works is instead of sending the file to our users, our rails application will simply check they’re allowed to download it (using our login_required filter) then write the name of the file into a special response header then render an empty response. Once apache sees that response it will read the file from disk and stream it out to the user. So your headers will look something like:
X-Sendfile: /home/railsway/downloads/huge.zipThe apache module has a slightly annoying default setting that prevents it from sending files outside the public folder, so you’ll need to add the following configuration option:
XSendFileAllowAbove onThankfully for rails users x-sendfile support is built right in to rails, allowing us to make a few minor changes and we’re done.
before_filter :login_required
def download
send_file '/home/railsway/downloads/huge.zip', :type=>"application/zip", :x_sendfile=>true
end
With that, we’re done. Our rails process just make a quick authorization check and render a short response, and apache uses its own optimised file streaming code to send the file down to our users. Meanwhile, the rails process is free to go on to the next request.
Nginx users can use a similar header called X-AccelRedirect. This is a little more fiddly to set up, and requires your application to write a special internal URL to the http response rather than the full path, but in terms of scalability and resource contention, it’s just as great. There’s an intro to the nginx module available if you’re an nginx user. If only uploads were this easy!
Up Next
The next article in the series will cover my experiences when dealing with the storage of your files. Should you use S3? What about blobs, NFS, GFS or MogileFS?
The Rails Way is all about teaching "best practices"
in 
Good strategy, I didn’t fully realize what sending files did. Thanks. Though, this never hindered my applications because I’ve usually used 3rd party storage (Amazon).
Would you say there is another solution besides the one you described, and Amazon?
How do you manage the download if you don’t have the ability to use the apache module (because not possible to compile the module into apache)?
I also would like to read about File Upload Done Right. The problem is that during uploads, the code is trying to hold the file in memory. This really hold server resources. I would like the code or the web server to write the files directly to the disk rather than trying to hold it in a memory.
Thansk
Thank you for this great post. I always wondered how to do protected downloads right and this seems to be it. If any one wonders (like I did), large file uploads does not block rails processes. (Although it requires a lot of memory and cpu usage).
I also would like to hear your thoughts on file uploads – done right. Thanks for all your great articles.
@Daniel: Using S3 is a topic for another article in the series, but in short if you’re redirecting them to the signed url, you’re not tying up any additional resources.
@Brian, @Cory: Uploads done right are the topic of another article in the series, stay tuned, you’ll hopefully be pleasantly surprised.
Great article! I’d heard about the theory of all this but I had no idea it was actually so simple to implement.
I look forward to your uploads done right article. I think uploading as well as downloading is one of those grey areas for most rails developers.
i’ve been using the :x_sendfile parameter in rails for a while but unfortunately i got files with good filename but wrong size (1 byte only).
seems like i’ll have to use the XSendFileAllowAbove option. thanks for this information.
I’m wondering how one would count the number of completed downloads per file. Are there callbacks so you can figure out how many bytes of a file have been downloaded or something?
Or is it just impossible to reliably count the number of downloads for a file. I mean, it’s easy to just +1 a counter column when starting send_file but what if you want to known the amount of people that fully downloaded the complete file?
Great article. I wrote a similar article with information on how to process file downloads with Rails using Nginx as your front web server at http://ramblingsonrails.com/how-to-protect-downloads-but-still-have-nginx-serve-the-files
Thanks for pushing the X-Sendfile solution. We’ll be offering out-of-the-box X-Sendfile support on RailsCluster from this Saturday onwards. @Jaap: you’ll need to use mod_logio.
I like the X-Sendfile option a lot, especially because it’s much easier to test. Asserting against the value of a header is so much more convenient than trying to fake download a file in your tests. Of course, you may want some integration level tests to make sure things are set up properly, but this certainly helps on the unit test level.
Its great, we have been using this feature in our application. Mongrel threads are happy now.
Thanks for the info, that’s really useful.
I’m interested to hear more about S3 and this technique. Thanks for the article!
kfkfk , http://www.simteach.com/wiki/index.php?title=User:Buy_aciphex&oldid=9219#1 drug aciphex, oijh , http://www.simteach.com/wiki/index.php?title=User:Buy_acomplia&oldid=9221#1 buy acomplia, 222 , http://www.simteach.com/wiki/index.php?title=User:Buy_allegra&oldid=9223#1 costa allegra, 098776 , http://www.simteach.com/wiki/index.php?title=User:Buy_allopurinol&oldid=9225#1 allopurinol drug, edvt , http://www.simteach.com/wiki/index.php?title=User:Buy_avapro&oldid=9227#1 150 avapro mg, wvsxoki , http://www.simteach.com/wiki/index.php?title=User:Buy_adalat&oldid=9229#1 adalat, khjg , http://www.simteach.com/wiki/index.php?title=User:Buy_amoxil&oldid=9231#1 amoxil 500mg, 5115 kjj , http://www.simteach.com/wiki/index.php?title=User:Buy_amoxicillin&oldid=9233#1 amoxicillin
If you’re installing mod_xsendfile on a new macbook pro, you’ll probably need to install it like described here: http://iprog.com/posting/2008/04/compiling_mod_xsendfile_for_mac_os_x
And if you’re installing it on Apache2 on Ubuntu, you’ll need something like this -> http://www.qc4blog.com/?p=547
I am wondering if this can be used to what I’m trying to achieve ?
The exe file I want to password protect is on another server. I want to keep the link hidden from the user. Here is the link to the question I asked http://stackoverflow.com/questions/682971/question-about-creating-a-link-to-download
Thanks!
I have a related problem: I have a large proprietary image database that I don’t want to put in my “public” folder/directory—I want to provide access only to logged in users. But I don’t want to use a tool like X-Sendfile; I just want to be able to do something equivalent to .
Any ideas?
Sorry, my pseudocode fragment didn’t render… I want something equivalent to
<img src=”(login-protected location)” />
And here’s the answer:
1. create a function in a suitable login-protected controller:
Class ImagesenderController < ApplicationController before_filter :authorize
def show_image send_file “(path to stored image not in public directory)”, :type => ‘image/jpg’, :disposition => ‘inline’ end
end
[Depending on how you keep track of your images, you can do something like retrieve the appropriate pathname from a database lookup using an id parameter passed to show-image]
2. Then your template can use this function as follows:
<img src=”imagesender/show_image/id”>
Anyone looking at the page source finds only the image accessor function which is only usable by logged-in users to access the image. The actual image files are hidden from unauthorized users.
:
I have trouble getting xsendfile to send anything but 1 Kb files.
Another user stated this is probably due to the: XSendFileAllowAbove on option. Where do I set it?
Im running my application on mod_rails ( phusion passenger ) apache2 with mod_xsendfile installed. I’ve tried creating a .htaccess file in the public folder of my application, but it seems to destroy my views, as if the css is not read.
Very intresting article. Thanks. Does anyone know of an equivalent solution for S3, especially when access must be restricted? Simple re-direction to an S3 URL won’t do if the bucket is private.
Further thoughts… One approach might be to redirect via using a signed URL, i.e. one that can be authenticated as coming from the bucket owner, and expires at an appointed time.
To the other David and those who are getting 1kb downloads.
I opened the 1kb “mp3” file with TextMate and saw that it was the HTML for the Rails 500 error page.
My download action had an issue with a double redirect error (which wasn’t showing up in the browser). As soon as I cleared that up, my downloads worked.
David