15 May 2011
Why is Rails startup so slow?
For me, as for a whole lot of other people, Rails take about 30 seconds to start-up, regardless of whether it is starting the server, running the tests or firing up the console. For a professional Rails developer this means starring at a black screen and waiting impatiently many times a day and wasting a lot of time. People have been complaining about this on the web, but the answers people give to this seem to be evasive and do not really hit the issue. Turns out there is a really simple reason why the start-up is so slow and the only question remains why this wasn’t fixed a long time ago.
If you try to profile Rails startup with Ruby-prof, it turns out around 80% of time is spent executing “require” statements. That may seem pretty weird and it indeed is. You can get a bit of understanding why the requires are so slow by doing a “strace -o syscalls rails console” and browsing the results - what you will see is a whole lot of calls to open and stat that result in an ENOENT “No such file or directory”, in my case, 35 000 calls for starting up Rails. It looks somewhat like this (and this is just for one require statement!):
open("[...]/lib/ruby/site_ruby/1.9.1/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/site_ruby/1.9.1/i686-linux/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/site_ruby/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/vendor_ruby/1.9.1/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/vendor_ruby/1.9.1/i686-linux/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/vendor_ruby/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/1.9.1/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/lib/ruby/1.9.1/i686-linux/readline.rb", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/will_paginate-3.0.pre2/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/thin-1.2.11/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/sqlite3-1.3.3/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/ruby-prof-0.10.7/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rspec-rails-2.6.0.rc6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rspec-2.6.0.rc6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rspec-mocks-2.6.0.rc6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rspec-expectations-2.6.0.rc6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rspec-core-2.6.0.rc6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/rails-3.0.6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/railties-3.0.6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/thor-0.14.6/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/pg-0.11.0/lib/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
open("[...]/gems/pg-0.11.0/ext/readline.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT
[ 50 more lines ]
The whole issue in my opinion is caused by bad design of RubyGems and/or Bundler, depending on the way you look at it. Both of those work by augmenting $LOAD_PATH, so that includes all the directories that contain gem files - in my application this makes $LOAD_PATH around 100 items long. If you know how $LOAD_PATH works in Ruby, this means that each time you do a “require”, Ruby will browse through all the items in $LOAD_PATH and test each of them for the presence of the file you are trying to require. If it takes 500 requires to start up your application, it will in the worst case result in 500 * 100 = 50 000 attempts to access the filesystem, most of which are unnecessary.
RubyGems or Bundler could instead maintain a cache that would map the potential parameters to “require” to absolute locations in the filesystem, so that the gem directories are scanned only once. I have attempted to build a solution that would do just this (it is possible to change the way “require” works in ruby”), but it is hard to work out a strategy for building this cache - many “require” statements are ambiguous and you have to map them to exactly the same files RubyGems do to get things working. I have sent an email to Yehuda Katz about this, but so far I haven’t got any reply. Maybe someone more knowledgeable about RubyGems/Bundler could workout a solution or help me with working one out or at least explain why I’m wrong or say why it can’t be done.
Comments ():
Faisal N Jawdat, 15 May 2011, 11:02 pm
https://github.com/rdp/faster_require adds such a caching mechanism. See if that helps.
stiff, 16 May 2011, 7:05 am
I know about faster_require, it is a nasty hack though and it doesn’t work at all at least under Linux, the ENOENTs still occur and the startup takes longer than without it.
Robert Pankowecki, 26 May 2011, 3:33 pm
http://redmine.ruby-lang.org/issues/3924
Keep your eyes on the official bug.
Robert Pankowecki, 30 May 2011, 2:50 pm
Have you tried it : http://rhnh.net/2011/05/28/speeding-up-rails-startup-time ?
stiff, 30 May 2011, 3:14 pm
Thanks for the pointers, good to see some work is being done in this area. I haven’t tried it, I was thinking of patching Ruby on my own, but now that a fix is ongoing I will probably just wait for the next Ruby version. Anyhow, I don’t know how the patch really looks like, but it seems wrong-headed to me that they are only considering it a O(n^2) vs. O(n) algorithm kind of problem. That’s probably part of the issue as well, but another part is the excessive filesystem access that is due to the general $LOAD_PATH lookup strategy and that I described above along with a possible fix. From the discussion or the blog post you linked it doesn’t look like they’re working on fixing this part of the problem.
Robert Pankowecki, 31 May 2011, 6:15 pm
You are absolutely right that they should do more. I think that it could be much faster to get tree of files for every rubygem and then check which one should be loaded using the memory representation of filesystem instead of using those system calls to check.
It won’t work in a cases like adding new file during script execution but who cares ? :-)
Mesothelioma, 27 July 2011, 12:22 pm
$LOAD_PATH is the main function of this.
Saleh, 8 August 2011, 10:06 pm
The better solution would be starting rails from the rails console. But that requires a rails console that is capable of doing the normal tasks that an average user needs to do (vim, emacs, mv, cp, cd)
Formal Dresses, 16 August 2011, 3:03 am
some really interesting information, well written and broadly speaking user pleasant.
abercrombie Milano, 24 August 2011, 11:17 am
I really loved reading your blog. It was very well written and simple to understand. Unlike additional blogs I have read. I also found it very interesting. In fact after reading, I had to go show the spouse and she enjoyed it as well!
web development outsourcing, 27 August 2011, 6:48 am
This is fit with my topic. It helps a lot because the information are so great and it is really so related what I been supporting write now. Thanks. Simply amazing works.
New Style Dr Dre Beats Graffiti Limited Edition Headphones White, 29 August 2011, 3:30 am
As an online business platform mainly retailing,we provide our buyers with an efficient and manageable procurement process covering every phase of the international supply chain and streamlining trade channels. Also welcome wholesaling,feedback now!http://www.cheap-beatsbydr-onsale.com/monster-beats-lamborghini-studio-headphones-limited-edition-p-64.html
New Style Dr Dre Beats Graffiti Limited Edition Headphones White, 29 August 2011, 3:32 am
Very nice article! I can’t agree with you more, I love your blog so much! This blog worth reading, I hope the blog owner keep updating, Thanks for your work! Have a nice day!
http://www.cheap-beatsbydr-onsale.com/dr-dre-beats-studio-ferrari-headphones-limited-edition-all-red-p-67.html
Dr Dre Beats Studio Ferrari Headphones Limited Edition - All Red, 29 August 2011, 3:33 am
Thank you for another fantastic blog. Where else could one get that kind of information written in such a perfect way?http://www.cheap-beatsbydr-onsale.com/monster-beats-studio-diamond-headphones-limited-edition-white-p-58.html
thomas sabo, 2 September 2011, 8:25 am
I am totally agree with your oppinion.this blog post is very encouraging to people who want to know these topics.
North Face Discount, 13 September 2011, 10:57 am
The article is worth reading,Thank you very much! I will keep your new articles.
Buy zithromax, 14 September 2011, 1:59 pm
Your post will be rather good and I’m sure some will find it interesting because it’s about a topic that’s as widely discussed as others. Some may even find it useful.Thanks so much for your post
buy lasix, 15 September 2011, 1:47 pm
Very nice and want knw more about this article ..I really enjoyed reading it..plz keep updating..
Game Of Thrones Season 1 DVD, 16 September 2011, 4:02 am
I feel strongly about information and love learning more on this. If possible, as you gain expertise, It is extremely helpful for me. http://www.dvdbestonline.com
moncler outlet, 16 September 2011, 7:52 pm
Not everyone responded to my e-mail, not everyone agreed to answer the questions, maybe I will also get some answers after I published this, I didn’t have the patience to wait longer, so new things may appear here over time.
buy ultracet, 17 September 2011, 8:47 am
I really like when people are expressing their opinion and thought. So I like the way you are writing.
True Blood Season 3 DVD, 20 September 2011, 9:55 am
I easily get nice & updated information for research purposes… I’d definitely appreciate the work of the said blog owner…
drupal outsource, 21 September 2011, 9:24 am
Writing a well researched article and at the same time ranking well for your keywords is quite an art in itself. Exquisite presentation.
north face outlet, 22 September 2011, 10:09 pm
North Face outlet is a great place to find cheap north face jackets at discounted prices. At the north face outlet store you will be able to find all sorts of gear from http://www.nfoutlet.com
North face jackets, 23 September 2011, 10:53 am
Retro classic casual style of bag, North face jackets , full of suede leather material seems quaint atmosphere, exquisite workmanship and it looks great texture.
outsourcing web developers, 24 September 2011, 8:46 am
Thanks a lot for sharing this amazing knowledge with us. This site is fantastic. I always find great knowledge from it.
True Blood Season 3 DVD, 24 September 2011, 10:56 am
Hppy Halloween!!1000s of dvd box sets discount!!!!
more for you at http://www.dvdboxroom.com
new tv show
outsourcing web developers, 26 September 2011, 1:31 pm
Through your excenlently article, we have gone out of just an amateur to a specialist in the area. It truly is a homage to your action. thank you!