The Art of Thoughtful Programming

Rails Paperclip Plugin and the EXT3 File System

Jul/06/2011

Many of the technical problems we face here at the center revolve around or are related to the vast number of headgear pages we have in our database. Often times Google will take several weeks to index even the slightest of changes made to our SEO. Here recently it seems that we've pushed the limit once again, only this time our problem was in the amount of images we have stored on our server.

Our site is implemented in rails 2.3.8 and makes use of many of the fine plugins available for both rails and JQuery. When it comes to handling file uploads, we use a combination of Uploadify (JQuery client-side plugin) coupled together with Jon Yurek's Paperclip plugin. As of this writing, we have exactly 3,166 items published live on our site. Taking into account that we have roughly 7 to 11 photos per item and that each photo has its own folder, it's easy to see how this number could quickly approach the maximum number of 32,000 sub-folders allowed for a single directory in the EXT3 file system that our Ubuntu web server uses.

To make matters worse, the cause of this problem was not immediately visible. When encountering an error during the upload process, the Uploadify plugin spits out a very generic 'http error'. This tells us nothing about the actual cause of the problem; it could be due to a simple file permissions problem, an invalid character in the file name, or (in this case) a much larger problem lurking beneath view. After double checking the file permissions for consistency and still having no luck, I took a peek at the rails production log and found the following error involving paperclip:

"Errno::EMLINK (Too many links - /name/of/photos/directory)"

This was also followed by a long list of errors generated by the ruby fileutils library as it tried (in vain) to create a new sub-folder for storing the uploaded image. If the fileutils errors don't give it away then the 'Too many links' error should. Everything in Linux is a file, even directories. A Linux directory contains a list (with links) of the sub-directories (nodes) that are contained within it. This error message is quite simply complaining that creation of sub-directory failed because there are already too many sub-directories (links) contained inside the photos directory. A few Google searches later and I was able to find a fix. However, this fix is a two step process and could become quite tricky. First, when creating the paperclip plugin the guys over at Thoughtbot must have foreseen this problem; they created an idpartition tag for the url and path options of the hasattachedfile paperclip function. This tag allows for the layering of folders inside one another so that a much larger number of images can be uploaded before reaching the maximum number of sub-folders allowed by the EXT3 file system. Before encountering this problem, none of the models in our site specified the url and path options. This resulted in the following default values being used:

:url => "/:attachment/:id/:style/:basename.:extension"
:path => ":rails_root/public/:attachment/:id/:style/:basename.:extension"

Each of the tags (:attachment, :id, :style, etc) will be replaced with actual values at run time (for an in-depth explanation of these tags, check out this article). Thus, assuming we want to call our attached file 'photo' and that we want to have two sizes (large and medium), we could put the following code in our model (replacing :id with :idpartition):

has_attached_file :photo,
  :path => ":rails_root/public/system/:attachment/:id_partition/:style                 /:basename.:extension",
   :url => "/system/:attachment/:id_partition/:style/:basename.:extension",
   :styles => {  
              :large => "1280x1024>",  
              :medium => "456x281#",
            }

This will ensure that as photos are read from and uploaded to the server, the directory created for each photo will follow the new directory partitioning pattern (e.g /system/photos/000/000/000, system/photos/000/000/001, system/photos/000/000/002, etc). However, this does nothing for the 32,000 existing sub-directories that need to be partitioned to fit this pattern as well. Thankfully, the guys over at Orbital Voice came up with this ruby script to do just that!

  • This script takes an input parameter 'TARGETDIR' specifying the base directory in which your images a stored. For the example above it would be ':railsroot/public/system/photos' where ':railsroot' is of course the root of your rails application.
  • You must specify in this script the correct path to the fileutils library on your server (this is likely to vary from system to system so the default location provided in the script is probably not going to work for you).
  • If your photos or uploads directory is large (ours was just shy of 200gb) you'll probably want to create a backup before running this script. Better safe than sorry.

Even though I’ve only been using rails for less than a year, I really like ease with which I can create file uploads using the Paperclip plugin. It's much easier than having to deal with the PHP $FILES array, but at the same time I do miss the level of control offered by that approach. Hopefully this will go away over time as I adjust to the 'rails' way of doing things.

New Site Launch

Jun/27/2011

This post marks the official launch of my new blog / portfolio site in which to collect my thoughts, release open source projects I'm working on, and (most importantly) to express my opinions on various event taking place in the IT industry.