Merb gets merged into Rails 3!

It’s christmas, baby, and do we have a present for you. We’re ending the bickering between Merb and Rails with a this bombshell: Merb is being merged into Rails 3!

We all realized that working together for a common good would be much more productive than duplicating things on each side of the fence. Merb and Rails already share so much in terms of design and sensibility that joining forces seemed like the obvious way to go. All we needed was to sit down for a chat and hash it out, so we did just that.

What this will mean in practice is that the Merb team is putting their efforts into bringing all of the key Merb ideas into Rails 3. Yehuda Katz will outright join the Rails core team, Matt Aimonetti will work on a new evangelism team, and Carl Lerche and Daniel Neighman (hassox) will be co-starring the effort to bring all this over. We’ve immortalized the merge with plaque page at rubyonrails.org/merb.

What’s being brought over?
Some of the key ideas that they’ll be taking with them from Merb into Rails 3 are:

  • Rails core: Yes, Rails is a full-stack framework and will remain so, but there’s no reason we shouldn’t also make it possible to run with less than the full monty. Rails 3 will make it easy to run just a bare minimum and then allow you to opt in just the stuff you want, if that’s necessary for your particular situation. Think “rails myapp —core” (and “rails myapp —flat”).
  • Performance optimizations: Merb has a lot of Rails pieces rewritten to be faster. We’ll be bringing all that good stuff over. We’ll also bend the architecture in the places where that’s necessary for a big yield. In short, Rails 3 will get all the performance attention that the Merb guys are known for.
  • Framework agnosticism: Rails will always have a default answer to every question within the stack. If you don’t care about testing frameworks, you’ll get test/unit. If you don’t care about which ORM, you’ll get Active Record. But some people do care and want something else. Some people want RSpec for testing, others want to use Sequel or Data Mapper for ORM, others again prefer Haml for templating, and some might prefer jQuery for Ajax. All these people should feel like Rails is welcoming them with open arms. Yes, we’ll have a default, but we shouldn’t have any form of discrimination against alternatives.
  • Rigorous API: Too many plugins break when Rails is updated because it’s not clear where they can safely hook into the internals and when they’re monkeypatching and should expect things to break. The Merb guys committed to a public API with tests to ensure that it wouldn’t break. They’ll bring over that line of thinking and give Rails 3 a tested and documented API for extensions that won’t break willy-nilly with upgrades.

This is not a big bang rewrite
It’s important to understand, however, that this is not a “big bang” rewrite of Rails. We’re far beyond the time when we could just throw out everything and start over. This is going to be a progressive improvement of Rails that’ll carefully judge new initiatives on their impact on backwards compatibility as well as their general utility.

I’m sure there’ll be some parts of Rails 3 that are incompatible, but we’ll try to keep them to a minimum and make it really easy to convert a Rails 2.x application to Rails 3. The Merb guys will also be working hard on giving existing Merb users a manageable upgrade path to Rails 3. We’re working with lots of ideas including allowing existing Merb controllers to be mounted alongside new Rails 3 ones. We’ll see how it all plays out, but play out it will.

Also, the Merb guys aren’t just abandoning the existing Merb user base and their applications. They’ll still be doing bug fixes, security fixes, and work on easing the upgrade path to Rails 3. This will all progress in a nice, orderly fashion.

The timeline
Rails 2.3 is just around the corner. We hope to wrap up and release in January. It’s a blockbuster release packed with goodies to the tilt. But as soon as that’s done, all eyes will be on Rails 3.

The probably-overly-optimistic goal is to have at least a beta version ready for RailsConf 2009 in Las Vegas. Who knows if we’ll make it, but we’ll certainly have made tons of progress on it by then.

So all of these changes are pretty much effective immediately. We’ve already started the collaboration and we’ll be rolling out a bunch of public initiatives announcing the concrete elements of the work under the Rails 3 milestone very shortly.

No hard feelings, just kumbaja
This is quite a dramatic turn of events. We went from testy relations to coming together in not very long at all. But I’ve been incredibly positively surprised at how well everyone on both sides have been gelling behind the scenes. The more we talk, the more we realize that we want the same things. And in the few cases were we do care about something different, it’s usually complimentary.

I really hope that everyone within both communities will deal with this news as gracefully as the key contributors from both camps. Let’s just wipe the slate clean on anything that has gone before and cherish that we can now move forward in unity instead of as fractions of the same ideas.

Rails 3 is going to kick ass.

Also read what Yehuda wrote about this and Carl Lerche and Ezra and Matt.

Performance of Rails Metal

Josh recently added Rails Metal, which has been getting a fair bit of publicity. Metal is a great piece of functionality for those rare cases where the speed of your framework actually matters.

However, people have been reporting 25x speed increase over a regular Rails action, and that just doesn’t seem right. So I decided to do some benchmarking of ‘Hello World’ Rails action v/s Metal. Here are my results :

Rails action

Time per request : 1.244 [ms]
Throughput : 800 request/second

Metal

Time per request : 0.386 [ms]
Throughput : 3000 request/second

You can find more details about benchmark command/code at http://gist.github.com/38080

Of course, these are not very scientific benchmarks and your results may vary a little from what you see here. You should also make sure you run your benchmarks in production mode.

Now, if you compare these results, 3000 r/s against 800 r/s, you may think you’re seeing a 3x performance increase. However, that’d a wrong perception and throughput isn’t the best metric here.

Difference in Time per request is what you should looking at. In my benchmarks, speed increase I get when using Metal is about 1 millisecond. And that’s a constant speed increase I’ll get over a regular Rails action. It’s very important to understand that it’s a constant speed increase. It’ll always be 1 ms for me.

For example, if my Rails action takes 12ms, when I reimplement it all in Metal, it will take about 11 ms and not 4 ms.

To conclude, I’ll just quote DHH :

But for those few, specialized cases where you just need as much raw speed as possible, Metal can be exactly what the doctor ordered. It allows you to have the best of both worlds in one package. The lowest possible overhead for a Rack application when that matters and the full-featured goodness of Action Controller when it doesn’t.

This Week in Edge Rails

December 13-December 19, 2008 Edition

The Rails team hasn’t been hibernating this week: 50 commits hit the edge branch, from a variety of developers. Here’s my pick of the most interesting and significant new core code for the week.

Rails Metal

If you’ve been keeping in touch with Rails at all, you’ve heard about Metal this week: superfast endpoints inside of your Rails applications that bypass routing and Action Controller to give you raw speed (at the cost of all the things in Action Controller, of course). This builds on all of the recent foundation work to make Rails a Rack application with an exposed middleware stack. Rather than explain Metal in more detail, here are places you can read more about it:

Rack-based Session Stores

A big change pushed the underpinnings of Action Controller session storage down to the Rack level. This involved a good deal of work in the code, though it should be completely transparent to your Rails applications (as a bonus, some icky patches around the old CGI session handler got removed). It’s still significant, though, for one simple reason: non-Rails Rack applications have access to the same session storage handlers (and therefore the same session) as your Rails applications. In addition, sessions are now lazy-loaded (in line with the loading improvements to the rest of the framework). commit

A related change provides persistent session identifiers for cookie sessions, with API compatibility with the server-side stores. commit

MIME Type Handling

There are a couple of changes to the code for handling MIME types in Rails. First, MIME::Type now implements the =~ operator, making things much cleaner when you need to check for the presence of a type that has synonyms:


if content_type && Mime::JS =~ content_type
  # do something cool
end

Mime::JS =~ "text/javascript"        => true
Mime::JS =~ "application/javascript" => true

The other change is that the framework now uses the Mime::JS when checking for javascript in various spots, making it handle those alternatives cleanly. commit commit

Active Record Cleanup

Even though Active Record has been a core part of Rails basically forever, people are still eking out performance and usability gains here and there. This week saw commits to stop generating some useless queries when working with belongs_to associations (commit), give better error messages on failed find_by_foo! calls (commit), fix some association preloading issues (commit and commit), and improve performance with the MySQL adapter (commit).

Odds and Ends

Remember the cleanup for noise in the log files that edge got a couple of weeks ago? Building on that, we’ve now got prettier printing for output from any Gem in the backtrace. commit

If you’re on edge, Rails now enforces the requirement for Mocha 0.9.3 or higher, so that you can run the tests. commit

ETag handling has been cleaned up a bit: Rails will now skip sending an ETag header when there’s no body to the response or when sending files with send_file. commit

If you want to track down who worked on Rails in the past, it’s gotten easier thanks to a huge data collection and patching effort by Xavier Noria: he went through all of the changelogs and normalized author names, so we don’t have the confusing mash of names, handles, and email addresses in there any more. commit

Introducing Rails Metal

Rails Edge adopted Rack a while back and we’ve been exploring ways to expose that better. The first thing we did was to make it really easy to hook up any piece of Rack middleware in front of a Rails request. In your config/environment.rb file, you can do:

config.middlewares.use(Rack::Cache, :verbose => true)

This will make your application use Ryan Tomayko’s excellent Rack::Cache middleware for better HTTP caching. There are lots of other great generic, reusable Rack middleware pieces. Do go shopping.

But there’s another use case where Rack also comes in handy. If you have a Rails application that has service end points that need to be really, really fast. So fast that the few milliseconds that a trip through the Rails router and Action Controller path is too much.

For this scenario, we’ve built a thin wrapper around the generic Rack middleware and given it a place in the hierarchy along with the name “Metal”. Rails Metal is meant for things like the Campfire poller service, which is doing hundreds of requests per second and needs to be really fast and is at the same time really simple (2 database calls and returning a string). This service is currently written in C, but we’ll look into converting it to Rails Metal instead.

Rails Metal is a subset of Rack middleware
Think of Rails Metal as a subset of middleware intended for application-specific end points that need the extra speed (“write to the metal”, hence the name). Thanks to recent adoption of Rack sessions, these pieces of metal even have access to the same session as Rails. You can either run them as part of the Rails process or run them as individual services through rackup.

We’ve even included a generator for Metal, which by default will create a stub ala:

class Poller def self.call(env) if env[“PATH_INFO”] =~ /^\/poller/ [200, {"Content-Type" => “text/html”}, “Hello, World!”] else [404, {"Content-Type" => “text/html”}, “Not Found”] end end end

As you can see, all this does is capture requests to /poller and return “Hello, World!” (and the request is not for /poller, it’ll let the Rails router handle it — that’s what the 404 return does). It will do that just about as fast as is technically possible in Rack (cue useless metric of how Rails now has the fastest, most meaningless “hello world” stat!!).

Note that this is a pretty specialized tool. While you could write an entire Rails application using just metal, it would be a terrible idea in most cases. The vast majority of applications out there will never do what they do so fast that the overhead of the already heavily optimized Rails router and Action Controller execution path will matter one bit (pick up Michael Koziarski’s talk on Rails Performance through torrent for more on this).

Also, if you’re writing a generic filter in the vein of Rack::Cache, Metal is not the right choice either. You should just be creating regular middleware which can be shared across all Rack applications.

But for those few, specialized cases where you just need as much raw speed as possible, Metal can be exactly what the doctor ordered. It allows you to have the best of both worlds in one package. The lowest possible overhead for a Rack application when that matters and the full-featured goodness of Action Controller when it doesn’t.

We hope you like this. Josh Peek has putting tireless hours into getting Rails on Rack and this is just the latest product of his efforts. Thanks a bunch, Josh!

Update: Jesse Newland wrote up a great introduction to Metal as well. It includes benchmarks and even more examples!

Update 2: Changed code example to match latest style.

Hosting Ruby on Rails with Passenger

Phusion Passenger aka mod_rails has been on a tour de force lately and rightfully so. It makes Rails deployment so much simpler and combined with REE faster and with less memory overhead. So I’m really happy to see that lots of the hosting companies in the Rails world are adopting it and making it available to their customers.

Rails Machine recently announced that Passenger is now part of their standard stack and that they’ve made it silly easy to switch from a Mongrel-based setup to Passenger. The latest Accelerator from Joyent has Passenger preconfigured as well. At the shared hosting end, Dreamhost has been supporting Passenger for a while (nice tutorial using Passenger at Dreamhost).

I’ve personally been setting up Passenger at Slicehost with Ubuntu and having great results with that. At 37signals, we’re already running Ta-da List (on EC2) and Backpack (at Rackspace) on Passenger and plan to move over the rest of the applications shortly. Our system administrators certainly appreciate not having to funk with Mongrels any more.

Lots of other hosters are in advanced testing with Passenger as well. Brightbox has been building Ubuntu packages for Passenger and is putting one together for REE. They should have complete Passenger support shortly. Most other Rails hosters I’ve talked to are at least looking into it as well.

But just because Passenger is a big step forward for Rails hosting, it doesn’t mean that other approaches are suddenly useless. There may still be situations where a traditional proxy/Mongrel setup would be relevant. For example, if you for some reason are unable to use Apache, that’s still the way to go. Rails will continue to support both FCGI, proxy/Mongrel, any Rack web server, and of course Passenger.

The change is that if you do not already have an investment in an alternative solution, or if you’re feeling pain with that solution, you should definitely consider Passenger to be the default choice for Rails.

Update: Phusion has posted a guide to how you control the Rails and Ruby environment variables under Passenger. Useful for tweaking the GC settings etc.

Videos from Paris on Rails '08

I love how so many conferences are recording their sessions. Paris on Rails is another of those conferences and they’ve recently published their recordings as both video and audio. Most of these sessions are in French, but Michael Koziarski did a great session on Rails Performance that’s in English and so was my iChat Q&A session.

This Week in Edge Rails

December 6-December 12, 2008 Edition

Here’s your weekly roundup of the action out on the edge, where the code is exciting and sometimes even stable. There were only 32 commits this week; perhaps we’re going into a bit of year-end hibernation.

Memory Improvements

A couple of memory leaks got squashed this week. There was one that occurs in Rails 2.2 when you set config.log_level = :warn (this fix has been applied to the 2-2-stable branch as well as to edge) and one that turns up when you call reset_session in an around filter (this fix is on edge, and will be ported to 2.2 soon). commit commit

Application Templates

There was a bit of cleanup work on the application template feature that was introduced last week. This included support for installing plugins as git submodules, a couple of small cleanup items, and a rake task to apply a template to an existing application . With this task, you can now run


rake rails:template LOCATION=~/template.rb

And the changes indicated in the template will be layered on top of whatever code the project already contains.

I18n Cleanup

A few more bits of the Rails infrastructure are now exposed to internationalization, including connectors in Array#to_sentence and separators in Active Record validation messages . Also of interest to the international Rails community is a patch to add an ActiveSupport::Multibyte::Chars#ord method to properly return the Unicode value of the first character in a string.

Odds and Ends

There’s a new version of the memcache-client (1.5.0.5) bundled with edge Rails. This fixes some bugs in the original 1.5 and adds better support for failover and timeouts. You can read more about it on the Lighthouse ticket commit

Active Record’s to_xml support gets just a little bit more flexible with the addition of a :camelize option, which converts all of the column names in the output to camel case. commit

If you test a lot of view code that uses partials, you’ll be happy to know that assert_template now takes a :partial option that can look for them in the output directly. commit

And finally, there’s a patch that has not been applied to Rails yet, but that you might want to go look at in Lighthouse – it adds support for HTTP status code 418 so that Rails can conform to RFC2324 . I’m sure the core team would welcome further discussion of this change.

RailsWay - German Rails Magazine

If you can read German and you develop with Rails you’ll be happy to know that it republik, the same company that also produces a Java, .NET, and PHP magazine, is about to release a Ruby on Rails Magazine called RailsWay.

The magazine is in print and will be out in Germany, Switzerland and Austria by December 17, 2008. For 2009 they plan to release six issues (in the German language), and I hear they’re planning on an English version, but no ETA on that. The content of the first issue includes:

  • Getting started with Ruby on Rails
  • AJAX and Rails
  • RAILS 2.2 – what’s new
  • IDE’s for RAILS
  • Behaviour Driven Development
  • Metaprogramming With RUBY
  • RESTFull Web Services
  • Featured Website: Qype.com
  • Tips, Tricks, Performance & Security

A DVD comes with the magazine which includes tutorials, videos, free rails books, and a bunch of IDEs. The Editor in Chief is Michael Johann, who has done a great deal to promote Rails in Germany.