On Twitter, Rails, and Community

Posted by kev Sun, 15 Apr 2007 20:13:00 GMT

An interview by a twitter developer has gotten a good deal of press lately. For many, it has reawakened the giant that is the “does Rails scale” debate, and others feel compelled to defend the framework.

The problem with Rails isn’t that Ruby is slow. It’s also not that Rails is slow, or big, or bloated, or doesn’t scale. The problem isn’t that Rails doesn’t support multiple databases or composite keys out of the box. The problem is that this is how the vocal minority in the community tends to respond to criticism.

So let’s all slow down and take a look at how we got here and what we can do to avoid it. That’s of course assuming we want to. Inflammatory posts certainly get lots of traffic, but I think Rails is in the spotlight enough we can actually talk like reasonable people and still get things done.

More after the jump.

Alex Payne inadvertently opened the flood gates when he gave a 5 question interview about working on Twitter, the highest traffic Rails app in deployment. When commenting on how Rails was scaling, he said:

The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time. The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases, neither of which are quick fixes to implement. So it’s not just cost, it’s time, and time is that much more precious when people can[’t] reach your site.

Just so we’re all on the right track here, keep in mind that he didn’t say “Rails just doesn’t scale, there’s nothing you can do about it, abandon hope all ye who enter here”. Alex says that scaling Rails is more than a matter of money, it’s a matter of time as well.

Those who skimmed the article and didn’t actually listen to what Alex had to say began to use the interview as tinder for Anti-Rails posts and flamewars, and soon DHH responded. Here’s where the real problems begin. David turns Alex’s explanation for Twitter’s scaling problems into a discussion on blame (emph added):

Rails makes the act of developing such a pleasant experience that when you need to follow the same scaling path as every other shared-nothing stack, the contrast can feel stark. And perhaps it’s a natural reaction to feel a need to blame something for that contrast, however natural it is.

[…]

Second, when you work with open source and you discover new requirements not met by the software, it’s your shining opportunity to give something back. Rather than just sit around idle waiting for some vendor to fix your problems, you get the unique chance of being a steward of your own destiny.

[…]

Once the stress of having to deal with that in the moment subsides, I’m convinced that the team will grow beyond the blame game, get their hands dirty as full participants in an open source community, and contribute back their advances to the framework.

The conversation shifts from where bottlenecks come from when scaling Rails to “you’re saying it’s Rails’ fault, but it’s not”. This is where we start to slide down the slippery slope. It’s easy to argue about blame. David pulls out his “I’m not a vendor, I don’t owe you shit” speech, and says that because of the beauty of open source, Twitter is in charge of their own destiny.

He’s absolutely right. He’s also a big jerk about being right. David seems to imply that Twitter has been sitting on their hands, doing nothing, refusing to solve their own problems, and not contributing back to the community. This is false. It’s also completely beside the point.

Early in the discussion we started to imagine blame where it didn’t exist. Alex never says that Rails should support doing 11k requests a second. He simply said it doesn’t, which is something no one will dispute.

So, we’ve moved into rough territory, but someone goes and does something nice (Woo hoo!). Dr Nic writes a plugin to use multiple databases with Rails. Way cool. Really way cool. Awesome job Nic. This is the part of open source that I love, when people help each other just because it’s an interesting problem and they’re a nice person. Hugs and kittens all around.

DHH tosses down an ”I told you so”. David, what are you doing man? Yes, Dr Nic is awesome, and yes, Ruby is wondrously hackable, and yes, Rails allows plugins easily, BUT why do you keep beating this dead horse? No one said anything about a critical flaw. No one said Rails is inherently flawed and can never be adapted for high traffic sites. Just the opposite is true! Alex and the other guys at Twitter are those who are pushing Rails further than it’s ever gone. These are the pioneers. They’ve gotten to the hard part, and they’re still going, despite the difficulty. Really, does anyone think that sites at that load are running out of the box copies of Rails? It’s just not the case. That’s ok. The fact that Rails can be modified is one of its strengths. There’s no need to be defensive here. That’s the big problem we all need to face.

Before we post (or hit the enter button in IRC, or send off that email, or …) we all need to take a deep breath and figure out if there’s really any attack taking place.

I’m as guilty of violating this as anyone, and I’m sorry for it, but this needs to be what happens in the majority of cases, or we’re all going to turn into assholes.

There are some things we’re just going to need to get used to hearing.

Ruby and Rails are not as fast as some alternatives. But in most cases, it probably doesn’t matter. In some cases it does.

With Rails there is no vendor and no obligation to fix things. But just because Rails doesn’t do something out of the box doesn’t mean it’s broken. In fact, many things probably shouldn’t be “fixed” in Rails itself. We’ve got a wonderful plugin architecture just to fit that need.

Just because someone doesn’t contribute to a specific problem doesn’t mean they haven’t contributed to the community. Blaine Cook, one of the Twitter developers, recently released the awesome Jabber::Simple library, is a regular contributor at SF Ruby, and is going to be giving a talk at SVRC on making Twitter push the traffic it does. He even notes on his blog that Twitter is planning to release some of the tools they’ve developed to solve their particular problems.

There will always be growing pains. It’s natural. We’ll get through it. Let’s not tear ourselves apart in the process. Rails is what it is because of the community. This will continue to be a great ride with an interesting community doing exciting new things… as long as we don’t kill each other in the process.

Posted in , ,  | 29 comments

Comments

  1. Avatar Wade Winningham said about 4 hours later:

    I agree with what your post completely. It’s great to see Dr. Nic’s reaction. I think you’d find more people willing to help out like he did if you simply asked the community to make it a challenge rather than a defensive measure.

  2. Avatar Damon Clinkscales said about 7 hours later:

    Amen.

  3. Avatar Tom said about 11 hours later:

    I hope your post gets as much publicity as the others, because in my view this one is the most important.

  4. Avatar Steve Tooke said about 13 hours later:

    Really well said.

  5. Avatar Pius said about 18 hours later:

    Kudos.

  6. Avatar ThatAnonymousGuy said about 18 hours later:

    I wish I could write like you! well said.

  7. Avatar harper said about 20 hours later:

    I agree 100%. I hope that people see your post and see that rails people are not all like DHH and his mass of fanbois.

  8. Avatar Jerrett said about 21 hours later:

    Great post, thanks for saying what needs to be said

  9. Avatar josh said about 21 hours later:

    Thanks for being the voice of reason. We don’t need anymore gasoline thrown on fires for a while.

    For me, the bottom line is this: if you don’t treat people with respect, they’ll stop listening to you. Acting like an asshole is the second fastest way to marginalize yourself. (Lying is the first.)

  10. Avatar nil said about 21 hours later:

    well put.

  11. Avatar Joe Grossberg said about 21 hours later:

    Bravo. People need to stop being so damn defensive.

  12. Avatar carlos said about 24 hours later:

    DHH often says stupid things. Are you guys really suprised? DHH even thinks he invented ruby mixins: http://weblog.jamisbuck.org/2007/1/17/concerns-in-activerecord comment #28. Just ignore him, end of story.

  13. Avatar Kevin Clark said 1 day later:

    carlos: No one says they’re surprised, but it does need to stop happening. I’m happy to have people involved in the discussion, but let’s work on productive discourse instead of bashing. I’m annoyed at him too, but I think things can change.

  14. Avatar Bob Aman said 1 day later:

    I really couldn’t have said it better myself.

  15. Avatar Koz said 1 day later:

    Nice piece kevin.

  16. Avatar AEM said 1 day later:

    Man, thanks for putting in words what I was feeling. Like many others I was following links today and could not believe DHH reaction. Sad, really. But it is great that there is people that actually read the full article and keep an open mind.

    Thanks again,

    AEM

  17. Avatar Aaron said 1 day later:

    Good post.

  18. Avatar izidor said 1 day later:

    First, we are here because DHH created Rails. A bit of perspective and humility would be in order before dispensing advices left and right.

    Second, the lesson is: be positive or be quiet.

    You whining people already cost me a project which will be done in Java because of “Ruby can’t scale” (app for 10 users :-)

    Remember that decision makers are just skimming and the interview message is “Twitter says Rails can’t scale and Ruby is slow”, so “Rails bad, Java good”. Don’t put out such stuff until even stupid people accept Ruby (and Rails) as alternative which is worthy of evaluation. Whenever we reach evaluation stage, we win on technical merits, but we need to get there! See exquisite article on O’Reilly by Steve Yegge.

    Any whining is completely unnecessary (and brings only bad results), because this is open-source and you can do anything you want. It’s not as DHH is holding you back or something. The only result is bad publicity – is that what you want? There are mailing lists for technical discussions, no need to shout on the web about “Ruby is slow”.

    And how sad that the irony is lost on you people – the guy who single-handedly created the stuff is proclaimed to be “vocal minority”. Just get rid of this vocal minority and all will be right in Rails community, I guess…

  19. Avatar izidor said 1 day later:

    Oh, and since there is no direct link to Steve Yegge’s fabulous addendum, which you need to read also, I had to add another comment entry.

  20. Avatar itchy said 1 day later:

    At this point, DHH has become like Alan Greenspan. Everything he says gets amplified.

    Since all credit and blame for every aspect of what Rails does and should do gets funneled to him, I’m betting this was the 10th “Rails can’t do X” he had heard that day.

    I cut him a lot of slack. Yes, he’s ‘opinionated,’ as he’s often said himself. I’ve never met him, but he seems pleasant enough in general. Nevertheless, I’m sure he has bad days.

    When someone says “Rails can’t do X,” no, they’re not necessarily blaming Rails, but sometimes they are. “Rails can’t do X” is frequently an abbreviation of: “Rails can’t do X, so Rails is useless.”

    So I can see where DHH got defensive: Here’s a company doing bleeding-edge stuff, choosing to use a framework for it, then saying the framework doesn’t have the stuff out-of-the-box to handle the requests.

    I agree, it looks like the Twitter guys didn’t mean it as a criticism, but I can see easily where DHH would see it as such.

    Maybe he should be more measured with his responses, but, hey, so should many people.

  21. Avatar Dave Hoover said 1 day later:

    Well said Kevin. Keep up the great work.

  22. Avatar Amy said 1 day later:

    Well put.

    I’d like to further add that this seems like a classic case of text communication problems.

    We all have to remember to read carefully and try not to add coloring and undertones that aren’t really there. When you don’t know a person, it’s especially easy to do.

    I’ve met both of the major parties here and watched them interact with others1 and I’m pretty positive that they could have had a friendly, productive discussion about this face-to-face.

    But through the power of the Internet, everything has been blown out of proportion. :)

    I say, when in doubt, write things friendlier than you have to. Things that sound perfectly reasonable in your head can easily be misinterpreted in the absence of body language, intonation, smiles, etc. Frankness and friendliness don’t have to be at odds when it comes to writing.

    [1] I am not a creepy stalker.

  23. Avatar chuck said 1 day later:

    I think maybe DHH, and a lot of the Rails community, are just stuck in defensive mode from trying to fend off all the flames from Java-land. But those are staring to die down. So relax. Don’t treat everything like an attack.

  24. Avatar Ryan Davis said 1 day later:

    itchy: the difference is, DHH hit the send/submit button on that blog post of his.

    Bad day: maybe. Bad action/decision: definitely.

  25. Avatar Jakob S said 2 days later:

    Right on, the first sensible blog post I’ve seen about this debacle.

    If there’s something technical to take away from this, I’d say it’s the fact that Rails indeed is able to scale and handle the 11.000+ requests per second Twitter is serving. Cheers to Twitter for getting up there and proving this.

  26. Avatar Alex Payne said 3 days later:

    Thanks much for contributing some sanity :)

  27. Avatar AkitaOnRails said 6 days later:

    Just to give you an example: this discussion landed even down here in Brazil. I’ve been following the flow of events and I am as overwhelmed as the next guy. I did understand the whole point since the first statement from Alex, and the minute I read the DHH post I was blown away: “What is he doing!?”

    Fortunately it didn’t go as far as it could. I will not judge DHH on this because I made similar mistakes in the past as well. It’s an inherent human quality to defend itself. Sometimes the instincts go over our rationale.

    As far as a community is made of human being, mistakes like this are prone to happen. The only thing we can do is learn from them and avoid future repetitions.

  28. Avatar Jim Greer said 12 days later:

    Needed saying, and well said.

  29. Avatar Gabe da Silveira said 19 days later:

    First of all, great post. This is exactly what I’ve been thinking about the whole matter.

    @izidor – when people talk about fanboyism this is it. Don’t try bury legitimate criticism. If your clients/boss don’t trust you then you need to find better people or work on your credibility.

    @itchy – I totally agree. DHH is human and flawed just like everyone else. He has every right to post whatever he wants. However he could stand for a little introspection about his defensiveness. I mean he’s one of the most influential programmers of the decade and he’s not even 30 yet. Rails is a smashing success and used by thousands and thousands of very smart people. I can see how the interview rubbed him the wrong way, but it’s not as if there were any actual lies in there that needed rectifying. For his own sake I wish he could let this stuff roll off his back.

Comments are disabled