If your Rails application’s primary users are on your company’s intranet, and you host your app using mongrel_cluster and Apache’s mod_proxy_balancer or something similar, you may have noticed a potentially unwanted side effect: users who cause exceptions in your application will get a stack trace instead of your configured error page, even when running in production mode. This is a good excuse to peek into the Rails internals and explain how Rails determines what page to display when it encounters an error.

First, if you have absolutely no interest in how this all works, and just want to know how to fix it, click here, and shame on your lack of curiosity.

Master of Puppets

The part of Rails that handles errors in your controller actions is located (unsurprisingly) in ActionController.

actionpack-2.0.2/lib/action_controller/rescue.rb:

    # Exception handler called when the performance of an
    # action raises an exception.
    def rescue_action(exception)
      log_error(exception) if logger
      erase_results if performed?
    
      # Let the exception alter the response if it wants.
      # For example, MethodNotAllowed sets the Allow header.
      if exception.respond_to?(:handle_response!)
        exception.handle_response!(response)
      end
    
      if consider_all_requests_local || local_request?
        rescue_action_locally(exception)
      else
        rescue_action_in_public(exception)
      end
    end

In that final if/then, rescue_action_locally is the method that renders the stack trace you might expect to see only when running in the development environment, while rescue_action_in_public will use your custom error documents if available. Now, the first condition, consider_all_requests_local, is set to true in config/environments/development.rb for your Rails app, which explains the stack trace always showing up in development mode.

What is local? (Baby, don’t hurt me no more.)

If you’re done groaning at the above header, let’s move on to the second condition, local_request?, which is clearly not what it seems.

actionpack-2.0.2/lib/action_controller/rescue.rb:

    # True if the request came from localhost, 127.0.0.1. Override this
    # method if you wish to redefine the meaning of a local request to
    # include remote IP addresses or other criteria.
    def local_request? #:doc:
      request.remote_addr == LOCALHOST and request.remote_ip == LOCALHOST
    end

After reading the comments above, you might wish to join me (and a certain wise old owl) in expressing your incredulity. It’s pretty clear the people on your company’s intranet aren’t making their requests from your server’s loopback IP, am I right? Right? Well… no.

Technically, if you’re running Apache with mod_proxy_balancer on the same server as your mongrel_cluster using a configuration similar to the one outlined here, then request.remote_addr will always be 127.0.0.1, because the request is really coming from your Apache server. Still, hundreds of sites run using this type of configuration, and they don’t all display a stack trace on exception in production mode. Rails tends to be pretty smart about things like this, so something else must be going on. This is where request.remote_ip comes in.

actionpack-2.0.2/lib/action_controller/request.rb:

    # Determine originating IP address.  REMOTE_ADDR is the standard
    # but will fail if the user is behind a proxy.  HTTP_CLIENT_IP and/or
    # HTTP_X_FORWARDED_FOR are set by proxies so check for these before
    # falling back to REMOTE_ADDR.  HTTP_X_FORWARDED_FOR may be a comma-
    # delimited list in the case of multiple chained proxies; the first is
    # the originating IP.
    #
    # Security note: do not use if IP spoofing is a concern for your
    # application. Since remote_ip checks HTTP headers for addresses forwarded
    # by proxies, the client may send any IP. remote_addr can't be spoofed but
    # also doesn't work behind a proxy, since it's always the proxy's IP.
    def remote_ip
      return @env['HTTP_CLIENT_IP'] if @env.include? 'HTTP_CLIENT_IP'
    
      if @env.include? 'HTTP_X_FORWARDED_FOR' then
        remote_ips = @env['HTTP_X_FORWARDED_FOR'].split(',').reject do |ip|
          ip.strip =~ /^unknown$|^(10|172\.(1[6-9]|2[0-9]|30|31)|192\.168)\./i
        end
    
        return remote_ips.first.strip unless remote_ips.empty?
      end
    
      @env['REMOTE_ADDR']
    end

See that huge regular expression? Rails is doing some pretty clever stuff on our behalf here. A lot of larger Rails installations are going to be running the load balancing proxy (or proxies) on completely different machines from the mongrel servers on the back end, but it’s highly likely that no matter how many proxies the request gets forwarded through before it reaches your Rails app, on your network or your client’s network, they’ll be somewhere in the RFC 1918-defined IP space for internal networks. So it strips those out, hopefully leaving you with the external IP that originated the request.

One problem: if your clients are all on your internal network, this will leave the array empty, which means we’re now falling back on @env[‘REMOTE_ADDR’], which is – you guessed it – 127.0.0.1. And this is why your Rails app continues to show all its naughty bits to your users, but only if your users are on your internal network. It thinks they’re connecting from the local machine.

That wasn’t all that interesting…

OK, so you just want to know how to fix it? Fine. You’re no fun! Throw this in your application.rb to prevent stack traces from being shown on exception unless the app is running in development mode.

app/controllers/application.rb:

    def local_request?
      false
    end

It’s as simple as that. Still, it’s helpful when overriding methods to know why you’re overriding them, I think.