I love Mustache for rendering. Its simplicity pretty much forces you to do The Right Thing™ in code that uses it, and it’s got an implementation in just about any language you might find yourself using. That being said, I did find myself wishing for one feature: multipass rendering. So, loving both Mustache and Ruby, I decided to add support for multipass rendering to the Ruby implementation of Mustache.

What’s multipass rendering?

Multipass rendering is a term often used in relation to 3D rendering. It refers to splitting the rendering work into multiple parts, or “passes”, for any of a number of reasons. One of these reasons, and the one which is most likely to apply in the area of template rendering, is the case when some rendering passes change while others stay the same.

For instance, consider the following template:

Hello, {{ name }}!
Multipass rendering is {{ adjective }}!

Now, if you want to render this template once, it’s easy enough:

Mustache.render(template, :name => 'Ernie', :adjective => 'awesome')

Hello, Ernie!
Multipass rendering is awesome!

But what if you want to render the same message hundreds or thousands of times, while only changing the name? Sure, you could render the entire message over and over again, but that gets increasingly expensive as the template grows. Besides, what if you wish to split the rendering up across different apps or machines, not all of which share access to the same contextual data?

Now you have a problem.

Problem? What problem?

“No biggie. I’m super-smart. I’ll just supply Mustache tags as values!”

Great, let’s try that:

Mustache.render(template, :name => '{{name}}', :adjective => 'awesome')

Hello, {{name}}!
Multipass rendering is awesome!

Success!

Well, not really. What if you want to wait for a second pass for an entire object and not just a single value? Given a template:

Hello, {{# person }}{{ first_name }} {{ last_name }}{{/ person }}!
Multipass rendering is {{ adjective }}, and you are {{ person.adjective }}!

How might you accomplish this one? You might try:

Mustache.render(
  template, :adjective => 'awesome',
  :person => {
    :first_name => '{{first_name}}', :last_name => '{{last_name}}',
    :adjective => '{{adjective}}'
  }
)

This would give you:

Hello, {{first_name}} {{last_name}}!
Multipass rendering is awesome, and you are {{adjective}}!

So, this isn’t what you want. You’ve lost the tags that “step in” to person, and you’ll need to make the values include a person prefix as a result. And then you find you want to allow triple-stache (unescaped) values for some attributes, and the whole thing falls apart again.

This isn’t working.

It’s just Ruby (in Ruby)!

It’s time to look at the Mustache source. Digging around, we find Mustache::Context#find, which looks like this:

def find(obj, key, default = nil)
  hash = obj.respond_to?(:has_key?)

  if hash && obj.has_key?(key)
    obj[key]
  elsif hash && obj.has_key?(key.to_s)
    obj[key.to_s]
  elsif !hash && obj.respond_to?(key)
    meth = obj.method(key) rescue proc { obj.send(key) }
    if meth.arity == 1
      meth.to_proc
    else
      meth[]
    end
  else
    default
  end
end

Sweet. So it turns out that a Mustache context figures out whether an object in its stack has a value by seeing if it looks “hashy”, and if so, treating it like a hash. Otherwise, it falls back to sending messages to Ruby objects. We can use this knowledge for good.

What if we had an object that had any key we could ask for, and returned the “right” value for that key, according to how it was being used in the Mustache template? Well, it wouldn’t be that easy, since the object doesn’t really know how it is being used. When you call Mustache#render, Mustache converts the object passed as the first parameter into a Mustache::Template, which is what actually does the rendering. If we take a look at Mustache::Template#render, we can see how this happens:

def render(context)
  # Compile our Mustache template into a Ruby string
  compiled = "def render(ctx) #{compile} end"

  # Here we rewrite ourself with the interpolated Ruby version of
  # our Mustache template so subsequent calls are very fast and
  # can skip the compilation stage.
  instance_eval(compiled, __FILE__, __LINE__ - 1)

  # Call the newly rewritten version of #render
  render(context)
end

Mustache “compiles” the template into a string consisting of Ruby code which accepts a context and returns the actual output. It then defines a singleton method named render on itself, which is the result of this compilation step, so the compilation step doesn’t need to be repeated for this template, and calls this newly-defined method.

With me so far? Good.

What’s compile look like, you ask? Simple:

def compile(src = @source)
  Generator.new.compile(tokens(src))
end
alias_method :to_s, :compile

Mustache::Generator creates the aforementioned string of Ruby source from a tokenized version of your template, stepping through the tokens and firing events as they are encountered, SAX-style. This happens in Mustache::Generator#compile! (the root expression is a :multi containing an array of other expressions):

def compile!(exp)
  case exp.first
  when :multi
    exp[1..-1].map { |e| compile!(e) }.join
  when :static
    str(exp[1])
  when :mustache
    send("on_#{exp[1]}", *exp[2..-1])
  else
    raise "Unhandled exp: #{exp.first}"
  end
end

The various parts of your Mustache template are split into chunks of static content and Mustache tags of various kinds:

• section - {{#person}}{{name}}{{/person}}
• inverted_section - {{^person}}Nobody here!{{/person}}
• etag - Escaped tag, like {{name}}
• utag - Unescaped tag, like {{{html_content}}}

So, now we have some idea of what we need to do. We’ll need:

• Objects that appear to have all keys, and return appropriate values.
• A modified version of Mustache, which converts template data to a…
• …modified version of Mustache::Template, which compiles using a…
• …modified version of Mustache::Generator, which handles our special objects in its generated code.

Thankfully, this is all just Ruby! We can subclass the Mustache classes and add our own behavior.

Let’s get to the code!

Great, so now we have a plan. What might our special object look like? It turns out that it’s super-simple. Let’s call it LaterContext:

class LaterContext

  def initialize(prefix)
    @prefix = prefix
  end

  def has_key?(key)
    true
  end

  def to_s
    @prefix.to_s
  end

  def [](key)
    LaterContext.new([@prefix, key].join('.'))
  end

end

What’s it do? Well, it expects an initial “prefix”, which serves as its string representation, and it returns a new LaterContext for any key, appending the key’s name to its existing prefix using Mustache’s dot separator.

You may also be familiar with Mustache syntax like:

<ul>
  {{#people}}
  <li>{{.}}</li>
  {{/people}}
</ul>

In Mustache templates, a bare dot like this refers to the object that’s been “stepped into” via a section. So given the following context:

{ :people => ['Bob', 'Jim', 'Tom'] }

The rendered template would be:

<ul>
  <li>Bob</li>
  <li>Jim</li>
  <li>Tom</li>
</ul>

In Ruby, the dot is essentially an alias to the object’s to_s method (or key), because the Mustache parser creates “fetch” expressions that split the value inside the key on the “.” character. When the generator encounters an empty array to fetch, it returns the Ruby code: ctx[:to_s] (where ctx is the current context in the stack).

So, we would really like to handle that case as well. We need an object that is a string (so it can return itself on to_s calls or keys) but can be differentiated from any other “.” by the generator. We’ll add it to the stack any time we step into a LaterContext via a section. Let’s call it DotContext.

class DotContext < String

  def initialize(val = '.')
    val = '.' # Ensure we're always a dot
    super
    freeze
  end

  def to_s
    self
  end
  alias :to_str :to_s

  def has_key?(key)
    true
  end

  def [](key)
    key == :to_s ? self : LaterContext.new(key)
  end

end

By returning a new LaterContext when keys are retrieved, this should allow us to generate non-prefixed tags inside sections.

With those classes out of the way, it’s time to do some Mustache subclassing!

M-M-M-MULTISTACHE!

Since we’re subclassing Mustache to do multipass rendering, what better name than Multistache? I’ll save you the time: there is no better name.

We’ll set up the class method that “templateifies” objects to do so using a to-be-created subclass of the normal Mustache::Template class, and we’ll add a factory method for creating a LaterContext while we’re at it.

class Multistache < Mustache

  def self.templateify(obj)
    if obj.is_a?(MultistacheTemplate)
      obj
    else
      MultistacheTemplate.new(obj.to_s)
    end
  end

  def self.later(prefix)
    LaterContext.new(prefix)
  end

end

Now, let’s create MultistacheTemplate, which will use our to-be-built custom generator:

class MultistacheTemplate < Mustache::Template

  def compile(src = @source)
    MultistacheGenerator.new.compile(tokens(src))
  end
  alias_method :to_s, :compile

end

All of those pieces don’t mean anything if we don’t have a generator that knows what to do with our new objects. This one is going to look a lot hairier than it really is. Let me explain what we need to do here.

So, here’s Mustache::Generator#on_etag, which handles generating code that returns an HTML-escaped string, such as {{name}}:

def on_etag(name)
  ev(<<-compiled)
    v = #{compile!(name)}
    if v.is_a?(Proc)
      v = Mustache::Template.new(v.call.to_s).render(ctx.dup)
    end
    ctx.escapeHTML(v.to_s)
  compiled
end

It’s pretty straightforward, though the parameter name is a little misleading. Here, name isn’t actually a name, but a parsed expression like:

[:mustache, :fetch, ["name"]]

Earlier, we looked at the compile! method. I didn’t mention the :fetch type of Mustache tag then, but there it is. So, as you might imagine, there is also an on_fetch method which handles this case, and takes care of returning code that will fetch the attribute from the context. In this case, it will generate…

ctx[:name]

…which does about what you’d expect, returning the result of reading that key from the Mustache context using a hash-like syntax. The context takes care of searching through its stack to return the proper value.

So, then, the generated code from the default on_etag method would look like this, in the case of {{name}}:

v = ctx[:name]
if v.is_a?(Proc)
  v = Mustache::Template.new(v.call.to_s).render(ctx.dup)
end
ctx.escapeHTML(v.to_s)

To do multi-pass rendering, we want to generate code that looks more like:

v = ctx[:name]
if v.is_a?(Proc)
  v = MultistacheTemplate.new(v.call.to_s).render(ctx.dup)
end
case v
when LaterContext, DotContext
  "{{#\{v.to_s}}}"
else
  ctx.escapeHTML(v.to_s)
end

In other words, if the value returned from the context is a Proc, we want its return value to be treated as our special kind of template, and not the default Mustache one. If the value ends up as a LaterContext or DotContext, we want to render a string that corresponds to the Mustache syntax for an “etag”, to be rendered in a different pass.

The other 3 tag types are handled similarly, with the interesting part being that, as mentioned, when we step into a section, we push a DotContext onto the stack, so that we don’t actually render any attributes inside the section. This is because we don’t have the necessary information about the object’s keys in order to know whether or not any given attribute would be fetchable from the deferred object or would fall further down the stack. An inverted section has no such requirement, since it doesn’t step into the object in question.

It’s unfortunate that in order to get this behavior, we have to duplicate so much of the existing Mustache code, but because these methods are generating code, and we need to modify behavior inside the middle of that generate code, it’s not just a matter of calling super. Bear in mind that while this code listing is kind of long, the only changes from stock Mustache are the kinds I outlined above.

Enough talk. Here’s the code:

class MultistacheGenerator < Mustache::Generator

  def on_section(name, content, raw, delims)
    code = compile(content)

    ev(<<-compiled)
    if v = #{compile!(name)}
      if v == true
        #{code}
      elsif v.is_a?(Proc)
        t = MultistacheTemplate.new(v.call(#{raw.inspect}).to_s)
        def t.tokens(src=@source)
          p = Parser.new
          p.otag, p.ctag = #{delims.inspect}
          p.compile(src)
        end
        t.render(ctx.dup)
      elsif v.is_a?(LaterContext)
        outside = v.to_s
        ctx.push(DotContext.new)
        inside = #{code}
        ctx.pop
        "{{##\{outside}}}#\{inside}{{/#\{outside}}}"
      else
        # Shortcut when passed non-array
        v = [v] unless v.is_a?(Array) || defined?(Enumerator) && v.is_a?(Enumerator)

        v.map { |h| ctx.push(h); r = #{code}; ctx.pop; r }.join
      end
    end
    compiled
  end

  def on_inverted_section(name, content, raw, _)
    code = compile(content)

    ev(<<-compiled)
    v = #{compile!(name)}
    if v.is_a?(LaterContext)
      outside = v.to_s
      inside = #{code}
      "{{^#\{outside}}}#\{inside}{{/#\{outside}}}"
    elsif v.nil? || v == false || v.respond_to?(:empty?) && v.empty?
      #{code}
    end
    compiled
  end

  def on_etag(name)
    ev(<<-compiled)
      v = #{compile!(name)}
      if v.is_a?(Proc)
        v = MultistacheTemplate.new(v.call.to_s).render(ctx.dup)
      end
      case v
      when LaterContext, DotContext
        "{{#\{v.to_s}}}"
      else
        ctx.escapeHTML(v.to_s)
      end
    compiled
  end

  def on_utag(name)
    ev(<<-compiled)
      v = #{compile!(name)}
      if v.is_a?(Proc)
        v = MultistacheTemplate.new(v.call.to_s).render(ctx.dup)
      end
      case v
      when LaterContext, DotContext
        "{{{#\{v.to_s}}}}"
      else
        v.to_s
      end
    compiled
  end

end

Nice. So how do I actually use it?

With everything in place, we can now revisit the template we were discussing earlier:

Hello, {{# person }}{{ first_name }} {{ last_name }}{{/ person }}!
Multipass rendering is {{ adjective }}, and you are {{ person.adjective }}!

Then, we can render it with:

first_pass = Multistache.render(
  template,
  :person => Multistache.later('person'),
  :adjective => 'awesome'
)

We’ll get a new, partially-rendered template in first_pass:

Hello, {{#person}}{{first_name}} {{last_name}}{{/person}}!
Multipass rendering is awesome, and you are {{person.adjective}}!

On one or more second passes at rendering, we can then supply different people to render!

people = [
  {:first_name => 'Ernie', :last_name => 'Miller', :adjective => 'nifty'},
  {:first_name => 'Joe', :last_name => 'Blow', :adjective => 'lame'},
]
rendered = people.map { |p| Multistache.render(first_pass, :person => p) }
puts rendered.join("\n")

Hello, Ernie Miller!
Multipass rendering is awesome, and you are nifty!

Hello, Joe Blow!
Multipass rendering is awesome, and you are lame!

And there you have it. But this post wouldn’t be complete without a few…

Caveats

This all works really great, but keep a few things in mind:

1. If you’re deferring rendering of sections, then any contextual references contained inside the section will be deferred until the second pass, for reasons explained earlier. This means if you refer to contextual information that won’t be available in the second pass, you’re gonna have a bad time. In practice, this isn’t generally an issue, but it’s important to keep in mind. If your second rendering pass has a context that is always a superset of the first pass, you can disregard this warning altogether.
2. Remember that an inverted section will have its content rendered on the first pass, and only the {{^key}} {{/key}} tags will be left unrendered. If you are referencing contextual data that isn’t available until the second pass, you’ll want to supply LaterContext values for them, to defer rendering.
3. You’ll note that the name of the context key is specified as the parameter to Multistache.later. Yes, this means you can actually rename a contextual reference on the first pass for use in the second. I have no idea why you’d want to, but it’s kind of cool.

Wrapping up

I hope you found this journey through the innards of Mustache interesting, and the code examples useful. By the way, all of the code is bundled up for you in this gist.

Thanks for reading!