Scroll - Webquills.net

Mason's caching improves performance, relieves database pressure

2008-09-23T00:41:58Z

HTML::Mason is a template system with power tools built in. Here's a case study in the usefulness of Mason's internal caching tools.

Mason reads from database

As part of a web site registration application, the user is to be presented with a list of email newsletters the site makes available. During registration, the user may select one or more of the newsletters and be automatically subscribed. So I coded up a Mason component to display this list.

<%once>
use ListDB;

<%init>
my @newsletters = ListDB::get_mailing_lists();

% for my $newsletter (@newsletters) {
    
    <% $newsletter %>

%}

Of course this example is simplified, the real code has more HTML and a bit more display logic. It's not pretty, but it gets the job done. In the once section I import the library that interfaces with our mailing list database (which is managed by a separate application). Then we simply get a list of available newsletters and display each with an input checkbox.

When database is busy, Apache gets tied up!

As the website and the newsletters became more popular, we began to see problems. Specifically, when the mailing list software was performing some database intensive task, as it frequently did, the simple query underlying the call to ListDB::get_mailing_lists() would take a very long time to return. As a result, the user had to wait a minute or more for the page to load, and an Apache process was occupied waiting for the database to return. If we ever had a time when the web site and the database were both busy at the same time, all our Apache processes could end up waiting for the database query, leaving none to serve pages. That's bad news for a web site!

The problem here, obviously, is that the code queries the database every time a user requests the registration form. This is not a problem that is unique to Mason, all dynamic web applications eventually run into it. Nor is the solution unique to Mason. But Mason makes the solution trivial to implement.

Mason cache smoothes database bumps

The solution, of course, is to cache the response from the database, so that the next web request does not need to hit the database in order to render the list. The list of newsletters doesn't change very often, so there is no harm in caching it. In fact, since the HTML rendering of the list won't change either, why not cache the HTML and save the rendering next time too?

If only Mason had a built-in cache system so I wouldn't have to go digging around for extra modules on CPAN and write another 20 lines of code. Oh wait, it does!

<%init>
return if $m->cache_self(expire_in => 3600); # THE SOLUTION! (ALMOST)
require ListDB;
my @newsletters = ListDB::get_mailing_lists();

% for my $newsletter (@newsletters) {
    
    <% $newsletter %>

%}

See, I told you Mason makes it easy! You can arrange for any component to cache its output with this simple incantation. If the component has previously been cached, $m->cache_self will return a true value, and the component will not execute again. Instead, the output will be fetched from the cache, without any further action on your part. Pretty neat, huh? But that's not all it does. If the component has not been previously cached, this routine will set a "hook" that will transparently cache the output of the current component execution once it has finished. Now that's just cool! I'll bet you can guess what the expire_in argument does. Yup, I'm telling it to cache the output for 3600 seconds (one hour), if the current cache happens to be empty.

Notice the other change I made. Instead of useing our ListDB library in the once section, I have changed to a require statement. Why? Because in the general case, the library will not be needed for as much as an hour, because the content will be served from the cache. To save memory, we don't import the library until after the call the $m->cache_self, when we know we'll need it.

After the cache expires, le deluge

Ah, but there's a flaw in your brilliant plan, I hear you say. What happens if the database happens to be slow just when the cache expires? You're right back where you started, with an empty cache and a whole bunch of Apache process waiting for data to fill it with.

Doggone it, why do you have to be so observant? I guess I'll have to break another tool out of Mason's bag of tricks. Watch this!

<%init>
return if $m->cache_self(expire_in => 3600, busy_lock => 60); # THE SOLUTION!
require ListDB;
my @newsletters = ListDB::get_mailing_lists();

% for my $newsletter (@newsletters) {
    
    <% $newsletter %>

%}

See what I did there? No? Look again at that first line of the init section. The new busy_lock argument is the key. Basically, that's telling the Mason cache, "Hey, if this object has expired, extend the life of the old cached value by 60 seconds so other processes will still get a cached value. Meanwhile, I'm gonna calculate a new value and cache that."

Now, when the cache "expires", only one Apache process will notice that fact. That one process will proceed to hit the database for the new value. Meanwhile, other processes will continue to find the old value in the cache for another 60 seconds. If the database does happen to be slow, only one user (per minute) will have a problem, and only one apache process will be hung waiting for data.

HTML::Mason references

The Mason home page
The Data Caching section of the Mason developer guide
Get "The Mason Book", Embedding Perl in HTML With Mason (also available online )
Get Mason from CPAN

Perl: Readability, Expressiveness, and Concision

2008-07-07T11:05:06Z

Writing readable code means expressing yourself as clearly and correctly as you can, not targeting the lowest common denominator of reader.

There are many people out there who are preaching the gospel of readable code. I myself am one of them. I am a strong proponent of expressive variable and subroutine names, for example. Unless you are a fire-and-forget contract programmer, you are going to be reading your code a lot more often than you write it. But honestly, if code readability is your primary concern, use Python, not Perl.

Reading good Python code is like reading a novel by Dan Brown or Michael Crichton. The language is clean and simple, allowing you to move forward easily, and quickly grasp the point of the story. The vocabulary is easily attainable and won't stretch your brain too far. I like reading Python code. It's an easy read — a beach read.

Reading good Perl code is like reading Shakespeare. Every line is deeply expressive, both in its content and its rhythm. You probably need a dictionary to look up some words that, while not perhaps in everyday usage, express exactly what the author intended. It's still English, but you have to work a little harder at reading it, because it's beautiful, powerful English.

When I write code, I am concerned about readability. But I am also concerned about expressiveness and concision. I have something I am trying to say. I don't always want two paragraphs of easy prose that approximate my point. Most of the time I want le mot juste, just the right word, a precise expression of exactly what I mean that requires no further elaboration.

Now, I don't intend to imply that my Perl is of a class with Shakespeare's verse (nor Flaubert's prose). What I am saying is that some stories cannot be told in terms of Dick and Jane and Spot. Complex problems often require complex language to describe their solutions. I love writing such solutions in Perl, because Perl, like English, offers me a broad and rich vocabulary of idiom I can use to construct complex works easily.

Those who speak pidgin-Perl can usually puzzle out what the code is doing, but they are going to get hung up at some points, and they are going to miss a lot of detail. That is not a flaw in the language, nor an error on the part of the writer. It is simply the nature of the form. But those who understand the depth of the language can truly appreciate the subtlety of a nice, tight hack.

The key to writing code that is both readable and expressive is to build a bridge with your comments that help the unfamiliar reader find his way through the complex and obscure portions. If you find yourself using a rare or arcane idiom because its the right thing to do, don't dumb it down into fifty lines of loops for novice programmers. Instead, write a full paragraph of explanation above your one-line hack. Include references to man pages, books, and web sites. Scholarly works always have footnotes. The reader will not only gain a better understanding of your code, but perhaps learn a new technique that improves his own.

Writing readable code does not mean writing for the lowest common denominator of reader. It means expressing your point as clearly and correctly as you can. Try to make your code readable, but don't sacrifice concision for readability. You should expect more from your reader, rather than expecting less from your code.

P.S. For another view on the subject, you can read about the Unobfuscated Perl Code Contest.

P.P.S. For more deep(?) thoughts about Perl programming from Webquills.net, read the feed or get new posts via email.

The DBI Imp strikes!

2008-06-26T17:17:04Z

Somewhere in the process of doing something or other in the last few weeks, I managed to break my installation of MySQL on my MacBook Pro. I can't remember now how I managed to break it, but I had shoved a note onto my "to do" list to fix it.

After downloading and installing the latest stable version of MySQL (5.0.51b), I discovered that Perl could no longer access the database. I figured I just needed to recompile DBD::mysql against the new MySQL libraries. But when I tried, I got complaints similar to this one:

Can't use dbi_imp_data of wrong size (127 not 124) at /System/Library/Perl/Extras/5.8.8/darwin-thread-multi-2level/DBI.pm line 1190.

It seems I did two things wrong. First, I downloaded the "Mac OS X 10.5 (x86_64)" version of the MySQL server. Apparently, although Leopard (OSX 10.5) is supposed to be 64-bit, the included Perl 5.8.8 is compiled as 32-bit. So for compatibility, I scrapped MySQL and re-downloaded the 32-bit version. Of course, I may be mistaken about this, because...

I still got the errors! Some more searching suggested that some bug fixes in DBI for FreeBSD might be related. I was using the pre-installed DBI 1.52, and that was error number two. So I upgraded to the latest DBI (1.605), make realclean, and tried again.

I got a warning when running Makefile.PL:

Multiple copies of Driver.xst found in: /Library/Perl/5.8.8/darwin-thread-multi-2level/auto/DBI/ /System/Library/Perl/Extras/5.8.8/darwin-thread-multi-2level/auto/DBI/ /Library/Perl/5.8.6/darwin-thread-multi-2level/auto/DBI/ at Makefile.PL line 759

It seems to be detecting both the original Apple version of this file, and the new one I just installed from CPAN. However, I was still able to build the software, so apparently this is a pickled herring or whatever you call those things that look scary but don't actually matter.

Whew! Now we're back up and running. I'm tempted to try the 64-bit version again to see if it will work now that I have updated DBI, but that's too much like work, and since I only need it for testing, it isn't very important.

The moral to the story? Erm ... Don't break stuff? I'm not sure, but anyway, if my experience helps you get your MySQL working faster, I'll be happy.

ORM - PITA?

2008-06-14T23:25:35Z

Recently I read a note called DBIx::Class Gotchas over at Perl Alchemy. I've been struggling myself for several months with a bundle of old Class::DBI code, and I can relate. Both packages fall into the category of Object-Relational Mapping (ORM), software that tries to map programming objects around entities stored in a relational database.

For a long time I was a huge proponent of ORM software, and I still am for certain types of projects. But the honeymoon is over for me, as I've started to hit some of those limitations they always warn you about. ORM buys you a big gain in productivity, because the software reduces database access patterns from a dozen lines of code to just one or two. If you need to get a small project up and running fast, ORM is a tool that can help a great deal.

The trouble with ORM is that your objects don't always map directly to the queries you want to make. To simplify the thorny problem of SQL generation, often the underlying implementation is doing things in Perl code that would be more efficiently done in the database, or executing multiple SQL queries when a single join would have done the trick. When your project gets very large, very complex, or very dependent on database performance, the ORM layer suddenly seems to hinder as much as it helps. I now spend as much of my time coding around the ORM layer as using it.

Still, without ORM I would spend a lot more time writing "select * from table" queries and the like, and I almost always have the need to optimize for development time rather than execution time. So I have this love/hate relationship.

What's your experience with ORM software? Love it? Hate it? Tolerate it?

Perl 5: Hash slices can replace loops

2008-05-26T16:00:57Z

How many times have you written a for loop to do something simple with a hash and thought, there must be a better way to do this? Using hash slices instead of simple loops can save you lines of code and execution time.

A hash slice is a syntax for accessing the values of multiple keys of a hash in a single statement. It is a succinct and efficient technique, but it is also one of those collections of punctuation that give Perl a reputation as a write-only language. Once you have learned it, however, you will feel much more clever! Here are a few examples of how I use hash slices to make my code shorter and faster. (Note that you can also slice arrays, but today we are just talking about hashes.)

Basic hash slice syntax

You perform a hash slice by using a list as a hash index, rather than a scalar value, and preceding with the @ sigil rather than the $ sigil you would use to get a scalar value.

my %number_for = (one => 1, two => 2, three => 3);
# Regular access to scalar key
print $number_for{one}; # 1
# Hash slice accesses multiple keys. Note the '@'
print @number_for{qw(one two three)}; # 123
# This also works
print @number_for{'one','two','three'}; # 123

A cautionary note: notice how the scalar index uses a bare word as the key. Perl gives you the quoting for free in this case. With a slice, Perl doesn't help, so you have to do the quoting yourself.

Merging two hashes

Since hash slices can be lvalues, they can be used to merge one hash into another. A common example is when you get configuration information from more than one source, but you want to consolidate it to look up in just one place.

my %your_numbers = (two => 2, four => 4, six => 6);
# I get all your numbers! 
# (And your number will override mine if they differ)
@number_for{keys %your_numbers} = values %your_numbers;
print sort values %number_for; # 12346

Accessing keys in a particular order

Here is a common thing you run into in web development. You have received input from a web form and validated it. (You have validated it, right?) The data lives in a hash, and you want to store it in a database. You have your SQL statement all prepared, but it requires that the values be bound in exact column order. Unfortunately, the values function cannot be relied upon to return the values in the order you want. (And besides, you don't want to store the value of the submit button!)

# get valid data from your validation code
my %validated = %number_for;
# Columns of your table, in order needed by your SQL
my @columns = qw(six one three);
# Get the bind values with a slice
my @bind = @validated{@columns}; # 6,1,3

Accessing values sorted by keys

Say you want to sort a hash by its keys, and then use the values in that sorted order. Using the above data, perhaps we want to print numbers in alphabetical order.

print @number_for{sort keys %number_for}; # 41632

Slicing a hash reference

Eventually you will find yourself with a reference to a hash, and you will discover that the above syntax does not work. You may try three or four different combinations of curlies and arrows that just generate errors. Don't give up! You can slice a hashref! First, let's review using a hashref to get at scalar values.

my $num_for = \%number_for;
# Common syntax for dereferencing and getting a scalar index
print $num_for->{one}; # 1
# Alternate syntax, the lazy way:
print $$num_for{two}; # 2
# Alternate syntax, the explicit way
print ${$num_for}{six}; # 6

The key to slicing a reference to a hash is to use the alternate syntax shown above, replacing the initial $ sigil with @.

# The lazy way:
print @$num_for{@columns}; # 613
# The explicit way:
print @{$num_for}{@columns}; # 613

Note the distinct absence of the "arrow" syntax. The arrow implies a scalar, and we want a list.

Powerful syntax

The hash slice is an advanced syntax demonstrating Perl's concision and expressiveness. Now you should be able to recognize it when you see it, and hopefully apply it to your own projects to save time and space. (But remember, use your Perl superpowers only for Good, never for Evil!)

For a semi-regular diet of great Perl programming tips, subscribe to the Webquills.net feed or get Webquills.net via email.

Best Error Message Ever

2008-03-15T22:12:33Z

Oh, the irony is so heavy, it's breaking me!

Perl Tool Tip: module_info

2008-02-19T02:45:00Z

When developing and deploying Perl code that relies heavily on CPAN modules (and if yours doesn't, it's either really simple, or you are not using Perl to its full potential), I find myself asking the same set of questions over and over again.

Do I have ${ModuleX} installed on this machine?
What version of ${ModuleX} is installed?
There might be more than one copy of ${ModuleX}. Which one is my code using?
${ModuleX} is not behaving. I want to have a look at its source.

A handy (and free) tool to help answer these questions is available from CPAN in the Module::Info distribution. In addition to the library, this distribution also includes the module_info command line script. Run it to find out the version and location of the module in question. For example:

vince@Vince-Laptop:~$ module_info Module::Info

Name:        Module::Info
Version:     0.310
Directory:   /Library/Perl/5.8.6
File:        /Library/Perl/5.8.6/Module/Info.pm
Core module: no

You can quickly find out what version of the module is installed, where it lives, and even whether it is a part of the Perl core distribution. Sadly, Module::Info itself is not in the Perl core, so you'll have to install it and its dependencies from CPAN.

If you can't install the CPAN module for some reason, I hacked together a little script I call qmod that I could carry around on a thumb drive with my dot-files and such. Being just a quick hack, it's not as functional as module_info, but it gets the job done. Here it is in its entirety (less documentation).

#!/usr/bin/perl
use strict;
my $max = 0;
my @list;
for (@ARGV) {
    $max = length($_) > $max ? length($_) : $max;
    eval  "require $_;" ;
    if ( $@ ) {
        push @list, [$_, 'Not Found', ''];
    } else {
        my ($version,$file,$which);
        ($file = "$_.pm") =~ s{::}{/}g; 
        my $varname = $_ . "::VERSION";
        eval "\$version = \$$varname;";
        push @list, [$_, $version, $INC{$file}];
    }#END if
}#END for
printf "%-${max}s  %6s  %s \n", @{$_} for @list;

If you just need to know the location of a module file, here's a nice tip from brian d foy's Mastering Perl: perldoc -l Module::Info.

Inside-out Templates in Perl

2008-02-09T12:57:44Z

Pop quiz, hotshot. Your team has undertaken a project to completely redesign your web site. Fancy-pants designers are hard at work generating not one but three new designs. The designs will be put through a battery of usability tests, after which the best parts will be combined into the final design. They need you to create server-side logic that will populate content areas, navigation, and various design elements according to how the user manipulates the site. But the HTML is nowhere near finished, and the designers will be tweaking it right up to the day of the test. What do you do?

Your normal workflow is shot. Typically you receive finished HTML from the designers, break it down into reusable components, add template tags to represent program objects and logic, and prepare the whole thing to be run through a template processor to generate output. That's out. Any attempt to mess with their HTML will be counter-productive. The priority here is to perfect the design for usability testing, and that means rapid iteration in visual design tools. Template tags will get in the way.

The time crunch is serious. They need a functional prototype to test with, but you can't embed the functionality into the template. They're holding your templates hostage.

My solution: Shoot the hostage.

]]> Binding logic to document structure

The key to solving a seemingly impossible problem is not to come up with a seemingly impossible solution. It is to reframe the problem into something that's easier to solve.

In this situation, the problems seems to be that I have to embed my template logic into these HTML files at the last minute, with no hope of having time for testing. Since this is nearly impossible, let's break down the problem. What is the template logic going to actually do?

Extract content from a database, keyed by the requested URL, and wrap it in one of the provided designs.
Rewrite the navigation section according to the URL of the current page.
Perform several variable substitutions based on the site being tested.

None of these things are really dependent on the particular HTML design they are going into. They are just inserting or changing some content at a particular position in the document structure. I can write this logic without having the HTML. The trouble is, the logic does not do anything until it is bound to the document structure in some way. Template systems perform this binding by embedding the logic into the structure itself. Since I can't put template markers into the HTML, what I need is a way to bind the logic to the document structure from the outside.

Being a Perl hacker, my first thought was, can I rewrite the content using a regular expression string substitution? Having a look at the HTML in progress, I quickly decided this was an unreliable method. If they changed the order of tags or attributes, which they were sure to do, everything would break, and I would be up all night debugging regular expressions. Not my idea of a party.

If only this were a client-side application, I found myself thinking. Then I could use jQuery or another javascript toolkit to rewrite the DOM at the client. In old-school javascript, this wouldn't be much better, but modern techniques like Hijax allow you to separate the code from the structure to manipulate the DOM from the outside.

Hang on a minute. At the server, I have access to everything the client has and then some. Can't I build a DOM at the server and manipulate it in Perl?

An inside-out template using HTML::TreeBuilder

Off to CPAN Search, and in minutes I was writing my first experimental script with HTML::TreeBuilder. I started with something like this:

#!/usr/bin/perl
use HTML::TreeBuilder;

my $tree = HTML::TreeBuilder->new();
$tree->parse_file('wrapper.html');

my @links = $tree->look_down( _tag => 'a');

printf 'Found %s links', scalar(@links);
print "\n";
for (@links) {
        print $_->attr('href'),"\n";
}

Sure enough, I was able to print out all the linked URLs just like that!

My next step was not code, but social engineering. I talked to our designers and established that all the HTML designs would use the same HTML id attributes for the same document elements, and that these id attributes would not change. I needed this structural stability for my code to work. Fortunately, our clever designers were already on the ball and didn't have to change a thing.

I quickly implemented some procedural code that would perform the replacements I was after. Most of the code was spent loading the strings and structures that would serve as replacement text, so it isn't really useful to display here. But here are a few tips that might help you out if you decide to use HTML::TreeBuilder in your application.

TreeBulder Tip 1: Comments are stripped by default

The designers needed to see their "trace" comments in the output of the processed template. HTML::TreeBuilder throws comments away by default. To keep them:

my $tree = HTML::TreeBuilder->new();
$tree->store_comments(1); # Before parsing!

This makes comments appear as separate nodes in the tree, with a tag of ~comment. You can also preserve PHP code and other processing instructions using $tree->store_pis(1), and preserve the DOCTYPE declaration with $tree->store_declarations(1), although I didn't need to.

TreeBuilder Tip 2: Enable parsing XML-style empty element tags

My designers were using tools that produced (for the most part) valid XHTML. In XHTML, the empty tag syntax is used for tags like link and img.

TreeBuilder's default parse mode (inherited from HTML::Parser) treats these trailing slashes as character data. This didn't cause any actual problems for me, but better safe than sorry. I explicitly enabled support for empty tags this way:

$tree->empty_element_tags(1); # Before parsing!

TreeBuilder Tip 3: Be explicit with `as_HTML()`

To turn the in-memory model back into HTML text for output, you use the as_HTML() method. This method can be called with no arguments, in which case it implements some default behaviors which are almost certainly not what your designers want.

The first argument describes what character entities should be HTML encoded in PCDATA (i.e. text nodes). By default, it encodes everything that HTML::Entities knows how to encode. This is probably overkill. The docs say that you can be compatible with previous versions by passing '<>&'.

The second argument is the "indent string". By default this is undefined, which is nice for production since it eliminates unnecessary white space in the output. But when you are debugging the output, it's nice to have it "pretty printed". So I specified a four-space indent. With this setting, TreeBuilder neatly aligns open and close tags, making debugging much easier.

The third and most important argument defines elements whose end-tags are "optional", and by optional it means omitted. If you want your HTML to validate, and more importantly not be broken, you need to pass an empty hash reference for this argument. Missing

tags can seriously screw up a CSS based layout in certain circumstances, as I discovered the hard way.

So the correct way to call as_HTML() is like this:

$w_tree->as_HTML('<>&','    ',{});

TreeBuilder Tip 4: Don't use `as_XML()` unless you really mean it

If you are producing XHTML code, you might be tempted to call as_XML() instead of as_HTML() to serialize your tree. And you would be correct. But remember that XML is very strict about how things must be structured, and TreeBuilder complies with this strictness.

One of the sneaky things that will bite you is that in XHTML, script tags have a content model of PCDATA. This means that a correct XML generator must encode entities contained by the tag. And if this happens (as it will if you call as_XML()), your Javascript just got completely mangled and will no longer run. (Unless, of course, you used the magic incantation to wrap your script contents in CDATA sections.) You can read more about the issues on Moz Dev. However, I found the safest route was to output HTML 4.01 instead. Using an HTML 4 DOCTYPE still triggers standards mode in web browsers, but doesn't have all the pitfalls of serving XHTML.

TreeBuilder Tip 5: HTML::Element::Library

I discovered this too late to use it myself, but HTML::Element::Library extends HTML::Element with some finger-saving shortcuts. Several times I ended up writing a code stanza that looked like this:

my $e = $tree->look_down(id => 'replaceMe');
$e->delete_content();
$e->push_content($replacement);

With HTML::Element::Library, you can do this instead:

 $tree->look_down(id => 'replaceMe')->replace_content($replacement);

TreeBuilder Pros and Cons

I think TreeBuilder was the the right tool for the job I had to do. It allowed me to manipulate HTML documents at the server without embedding code-specific markers in the document itself. This allowed my team to generate both HTML and server-side Perl code in parallel, and we met our targets with room to spare.

That said, I don't think I would want to use TreeBuilder as my regular template tool. Binding logic to the HTML structure using procedural Perl takes a lot more code than just embedding it into the template. More code means more typing, more testing, and more bugs. HTML::Seamstress tries to help, and certainly should be considered if you are using more than one or two templates with this method, but Seamstress doesn't eliminate code so much as generate it for you. Better, but still not what I want.

What I really want is a concise, declarative syntax for binding logic to the document structure. No, not XSLT. I said "concise". I can definitely write Perl code faster than XSLT, and the resulting code is shorter and easier to read.

What would be nice is something very like a CSS stylesheet, but with code blocks instead of style attribute assignments inside the curly braces. Or maybe just a Perl port of jQuery. That would be way cool. Anybody want to build that for me? (UPDATE: see pQuery)

Pop culture note

For those too young to remember, "Pop quiz, hotshot", "What do you do?", and "Shoot the hostage" are references to the 1994 action movie Speed, a classic of my generation.

HTML::Mason 1.39 - Now with Memcached

2008-02-04T03:25:14Z

Speaking of templates, I was just catching up on my mailing list deluge, and found this announcement for the latest release of HTML::Mason:

1.39 Jan 30, 2008

[ ENHANCEMENTS ]

CHI may now be used as the backend for $m->cache as an updated alternative to Cache::Cache. Among other things, this facilitates easy use of Cache::FastMmap and memcached for data caching. Cache::Cache is still the default for now, and is still listed as a prereq for Mason.

-dave

Memcached support puts a smile on my face! I need to test that out soon! Read the docs.

Choose the right template system for your team

2008-01-30T03:53:18Z

What is the best template system? Ask five people and you'll probably get six different answers. Perl has more than its fair share of template tools, from the Swiss Army Chainsaw of Template Toolkit, through HTML::Mason and Text::Template down to the ever-tempting "variables interpolated in a here-doc" method.

As always with this type of question, there is no "correct" answer. The best template system is the one that works best for you, in your circumstances. If you are the only programmer in a room full of professional designers, your optimal template choice is likely to be different than if you are just trying to stick a loop into a text file for your own cron job.

In my experience, the key factors in choosing a template system are usually the composition of your development team, and the development workflow. Who has to work with the templates, and what are their skill sets? Is the code being designed around the interface, or is the interface being pasted on top of the code?

Read on for a comparison of the major template systems in Perl, and my recommendations of which systems fit which circumstances.

]]> Solo Perl Hacker: Text::Template

If you are a Perl programmer and you are producing your own templates, whether they be an HTML front-end or a plain text email, you can hardly go wrong by starting with Text::Template. This is the shortest path from Perl code to output, short of variable interpolation in a double-quoted string. The syntax is just plain old Perl, and everything you know about Perl translates directly to effective templates.

The down-side, of course, is that you do not enforce a clear separation of concerns. As a programmer intimately familiar with the internals of your application, you will be tempted to embed all sorts of application logic into the templates themselves. This makes your application harder to test and harder to maintain.

There is nothing in Text::Template that forces you to mix business logic into your templates or to write unmaintainable code, but it doesn't go out of its way to encourage modularity or cleanliness either. For larger projects, meaning either lots of templates or several programmers, you may consider a more complex solution that enforces some rules. (Template Toolkit might be a good step up, read below.) But for quick and dirty (or quick and clean) templates, Text::Template is awesome. Also, if your application is not a web application, this is a good place to start.

Perl Team, Web App: HTML::Mason

HTML::Mason is an excellent solution for a small team of Perl programmers writing their own HTML. Although it is not limited to HTML (it works fine with any kind of text), Mason excels at web development. It ships with a light-weight web framework of its own, although it can also be plugged into web frameworks like Catalyst.

Like Text::Template, Mason works by embedding Perl into your templates, so Perl programmers can pick it up fairly quickly. Much of Mason's added syntax is geared around triggering code at different points in the request cycle, and creating time-saving features like automatic wrapping of content with layout components that are "inherited" based on the directory hierarchy.

Mason allows you to think about HTML design like a programmer, decomposing a web site into reusable components that are effectively Perl objects, each with its own methods and variable scope. This makes it highly effective for code-centric applications, where the HTML is built after the primary program logic.

Mason is not necessarily the best system for HTML designers who are not Perl hackers. Because it encourages decomposition into components, you rarely have a template that represents a complete HTML document. Mason's syntax uses pseudo-tags to demarcate embedded logic, which tends to upset HTML-centric design tools.

I find Mason to be especially useful for applications where I need to create a web interface to some existing data store, such as a SQL database created by another application. Combined with a clean database access layer like DBIx::Class, you can get from legacy database to structured web front-end extremely quickly.

I'm a big Mason fan, and I encourage you to visit Mason HQ to give it a try.

Dedicated HTML Coder: HTML::Template or Template Toolkit

If your team has designers who are good with mark-up but not experienced programmers, then you probably want a template system that will insulate them from program logic while giving them the flexibility to construct simple loops and conditionals.

HTML::Template is a good choice for the mark-up coder who is less programmatically inclined. Its syntax is close to HTML, which flattens the learning curve, and it doesn't expose too many internal moving parts that would be hard to learn and easy to break. On the other hand, the syntax is so similar to HTML that HTML editing tools might see it as bad mark-up and complain about it. As with Mason, your templates will almost certainly not validate on their own. [UPDATE: As jerje pointed out in the comments, HTML::Template allows an alternate syntax that hides the template instructions in HTML comments, which allows them to validate.] As a programmer, I find the functional limitations frustrating.

If your HTML expert is also comfortable writing Javascript code (or some other non-Perl language like Java, Python, or Ruby), you might shift toward Template Toolkit. TT allows you to use program objects in the template using a Javascript-like dot syntax, and gives a bit more flexibility in logical constructs. TT also has the advantage that it can scale to large and complex template sets, and its numerous plugins give you a lot of usage options. Template instructions will not be confused with HTML mark-up, either by humans or editing tools, so if you're careful you can get your unprocessed templates to validate.

Fancy-pants Designers vs. Engineers: Template Toolkit, Petal, Template::TAL

If your web application is heavily design-oriented, what your designers want from you is probably what they might describe as "server-side Javascript". That is, you are probably working from a design first, and building program logic to hang functionality onto the design.

Template Toolkit can fit pretty well into this kind of process. It makes sense to designers familiar with Javascript or some other programming language, and the more programmatically inclined designers enjoy exercising program logic in their templates. But not all designers are programmatically inclined. I refer to these types (lovingly) as "fancy-pants" designers.

Typically, fancy-pants designers will start with a "wireframe" sketch and move up to fully elaborated Photoshop mocks of their web designs. Only after working in this form for several iterations will they start to generate an HTML model of the design. Their first goal is to create an HTML model that exactly mimics their Photoshop images, including sample text and images in the content areas. This helps them to visually prove that their designs will display correctly in a web browser (vitally important for a design-intensive web application).

Using Template Toolkit or any of the above template solutions adds a step to the design workflow where the HTML model must be dismantled, and bits of pretty HTML filler replaced with unattractive template tags and symbols. Most designers don't care for this, because it makes it more difficult to visualize the product as it was meant to be viewed. Design changes that come after this transformation can be painful, because the designers usually have to go back to their mocked HTML model to visualize the changes, and then they (or you) must repeat the dismantling and replacement process.

Petal and Template::TAL are potential solutions to this problem. (Both are Perl ports of a Python tool, Zope Template Attribute Language.) Their technique solves the problem by hiding template instructions in the HTML as extra attributes on proper HTML tags. Designers can feel free to fill the template with sample text and images that will make the template display cleanly in any web browser or HTML editing tool. Instead of taking this filler away, they (or you) add template instructions that dictate how the filler should be replaced by the output of program objects when the template is processed. Using namespaces, this can result in templates that are valid XHTML documents.

And the winner is...

What have we learned? When choosing a template system, you must consider the needs of the team as a whole, not just the software developers. Any of these solutions might be perfect for your team and your workflow.

But since you insist: My personal favorite is HTML::Mason. I've been using it to build web applications for years. The community is very supportive, I'm comfortable with it's quirks, and most importantly, I'm extremely productive with it. However, if you currently have no experience with any of these template systems, and you want to learn only one system, I recommend you pick up Template Toolkit. It has a vibrant community around it, which I consider a must for any tool I use. You can get started quickly. And you can reasonably use it in small, simple applications as well as large complex ones, so it leaves you plenty of room to grow.

If I forgot to mention your favorite template system, I'm sorry. I can only write about what I know, and these are the template systems that I am familiar with. Feel free to plug your own favorite solution in the comments!

Working widgets in MTOS

2007-12-28T16:17:21Z

Yesterday I upgraded Webquills.net to the "Boxing Day" build of MTOS. I was pleased to find that the widgetized sidebar is now activated for the default theme. I had tried (admittedly not very hard) to get the widgetized sidebar working in a previous build, but all I got was empty space.

Actually, with the Boxing Day release, you still get empty space by default, because although the widgetized sidebar is enabled, the sidebar components aren't actually populated with any widgets.

Creating your own widgets works pretty much like you would expect. I created a Google AdSense widget just by pasting the AdSense code into a new widget and wrapping it in a couple of nested divs classed with "widget" and "widget-content" respectively, code that I copied from the pre-built widgets. I was also able to modify some existing widgets without any trouble.

But I see that a new nightly build is out, so this is probably old news. Still, I'm gratified to see steady progress being made. The nightly builds are firming up. Some people have reported intermittently about upgrade issues, so you'll probably want to wait for the stable release to upgrade an older Movable Type installation. If you're starting up a new blog, the nightly builds should prove good enough at this point.

Perl 5: The grass is plenty green

2007-12-22T15:33:19Z

A few months ago I wrote a missive lamenting the coolness going on in other languages in terms of web frameworks and the dearth of magical leaps forward in Perl. Well, I take it all back. Serves me right for not recognizing hype for what it is.

Each of the frameworks from other languages that I examined promised great leaps in productivity for programmers of that language. But what I came to realize was that these were improvements that I had already experienced in Perl. The streamlining did not come from the language at all. One scripting language, it seems, is pretty much as good as the next when it comes to productivity for the skilled programmer. No, the great leap forward came from applying a well organized web application development framework and its associated tools to the problem of web development, where before there had been no organized solution and only a rudimentary tool set.

But I have been using Perl web application frameworks for years. (I use the plural of framework because TIMTOWTDI in Perl.) Apache+mod_perl is an amazingly rich web application framework on its own. Perl was (I believe) the first scripting language to be embedded in the web server itself, and is still the best option for manipulating the rich, complex internals of Apache. HTML::Mason's ApacheHandler is a simple layer on top of mod_perl that is a highly effective solution to certain classes of problems. Richer, more modern frameworks like Catalyst and Jifty do more for you, but also require you to do more learning to get productive. In Perl, we're spoiled for choice, which is a problem all its own. But the point is, the other scripting languages haven't beat Perl in the web framework race. They are just now catching up!

This is certainly not a call for Rails developers to switch to Perl. Why should they? The fact of the matter is that web frameworks in whatever language are substantially similar to one another, because they are each a solution to the exact same problem, built under similar constraints.

Programmers should use whatever language fits their mind best, because that is where they will find themselves most productive. For some people, that's Java. (Some people are weird.) For some its Python or Ruby. More power to you. But if you are productive in Perl, don't be lured away to some other language by the greener-looking grass on their side of the fence.The grass is plenty green right here.

Don't believe it? Subscribe to Webquills.net via RSS or Email and learn all about it.

Wordpress to MTOS: An easy migration (so far)

2007-12-18T21:07:51Z

With the recent and long-awaited release of Movable Type Open Source, I undertook to port my blog at Webquills.net to the new MTOS.

Why MTOS?

I've never felt comfortable publishing my content in proprietary software. Even if I could obtain it gratis, I was always paranoid that the license would change out from under me and I would suddenly find myself in a panic trying to port my content to some other system. So I started blogging on the GPL-licensed Wordpress, and I liked it well enough.

I am a Perl programmer by trade, and though my PHP skills are reasonably good, I would feel much more comfortable hacking on Perl code. MTOS is written in Perl so it gets points. That's my first reason for switching.

My second and more pressing reason to switch to MTOS is its static publishing model. I run my blog on a web server that's light on memory, and what memory there is, I like to use for my own web development. Using a heavy Apache/PHP process to serve a page whose content has not changed in weeks is a luxury that I'd rather not have to pay for. I'm willing to trade slightly delayed publishing for lightweight static file serving.

Installing MTOS

Installing MTOS was pretty simple. Not quite as simple as installing Wordpress, but simple enough that anyone comfortable with a Unix shell (or well-skilled with their FTP client) can get it done without breaking a sweat.

Be warned, however, that as of this writing no "stable" builds of MTOS have been released. I'm using one of the nightly builds, and it works fine for me, but you may run into some issues. If that bothers you, stick with the commercial version until the first MTOS stable release comes out.

I pretty much followed the installation instructions for Linux. I created a MySQL database for MTOS to use, and a new user with full permissions on the database. Then I grabbed the latest nightly build from the MovableType.org archive. Simply untarring it into my chosen destination was enough to get started. Then I walked through the "install wizard" filling in the blanks as needed. None of the questions were particularly tough to answer. :)

After completing the installation, I got some warnings that MTOS did not have permission to write to the /mt-static/support/ directory. Apparently this is where MTOS stores uploaded files and its own reports. I did a chown www-data mt-static/support so the web server could write there, and the complaints went away. I also had to do this to the output directory where I told MTOS to write my blog.

Migrating the content

The step that surprised me the most was migrating my Wordpress content to MTOS. It was shockingly easy. I entered the Wordpress admin area and selected Manage -> Export from the menus. Pressed the button, and I have a Wordpress eXtended RSS feed (WXR file) containing all my posts, comments, and categories (and probably other stuff too).

Then I went to MTOS and clicked to System Overview -> Import. I set the type to WXR and uploaded my Wordpress data. Snap! Just like that MTOS had my posts, comments, and categories all sorted out and ready to use. I punched the publish button and had a web site ten seconds later.

Final cleanups

I picked the most boring MTOS style I could find (because a blog should resemble its owner :P), and made just one customization. I wanted to point users to my FeedBurner feed instead of the local one. I picked Design -> Templates from the menu, then picked Template Modules and edited the Header module. I also had to change the "subscribe" link in the Sidebar modules. (Well, it wasn't as easy as it sounds. It took ten minutes of clicking around to find the right bit to edit. Still, it wasn't exactly hard, either.)

That done, I republished, and was ready to go.

The trickiest part of the migration had little to do with either Wordpress or Movable Type. It was simply that the URLs for my blog entries were going to change under the new system. But cool URIs don't change, so I needed to create a translator for the old ones. I handled this using a simple Apache directive with some Regex Fu.

 RedirectMatch permanent ^/(\d\d\d\d/\d\d/[^/]*)/?$ \
http://www.webquills.net/scroll/$1.html

More to come?

I'll probably publish some more comments about MTOS as I get used to using and customizing it. In the interim, I can say that the migration process from Wordpress to MTOS is surprisingly easy and should not be considered an excuse for not switching!

Forking Perl

2007-11-16T12:11:51Z

One of my favorite sites, Perl Buzz, just pointed me to Kurila, an experimental Perl 5 fork. I'm not sure how I feel about that. On the one hand, I do love to see innovation in the Perl space. Perl 6 has been promised by Christmas, they just won't say what year. I figure 2014. But Perl 6 isn't Perl anyway, it's a new language in (tattered) Perl clothing. Perl 5 marches on in 5.10, though, which keeps me from complaining too much.

On the other hand, I would hate to see Perl stagnate so much that a fork like this actually became a viable alternative for the average Perl developer. Not because the fork is bad code or has bad language implications (on the contrary, I like some of Karila's ideas), but because of what that would mean about the old Perl. Namely, that it started to suck worse than the alternatives.

Movable Type Open Source - again

2007-06-20T03:08:25Z

[UPDATE: I stand corrected. Informed commenters point out that though the source was available, MT was never technically "open source". To quote Richard Blain, "I was misinformed" -V]

It's ironic, to me at least. When I first got interested in blog software, I heard that Movable Type was the one to beat. It was written in Perl, which was my language of choice; ~~it was open source (so you could make your own modifications)~~; and it was by many accounts the most advanced blogging software out there (but not without argument, of course, this is the Internet).

Five minutes after I discovered it, the company announced that their 3.0 version would carry a new, commercial license. Suddenly being Perl was useless, because I could no longer modify, fork, or crib from the code. I moved on, and finally settled on Wordpress.

Now, Movable Type is ~~once again~~ opening its source code. Thanks to Ben and Mena for contributing this code back to the community. Ultimately, open source software is not about the software, it's about the community of users, and we users appreciate it when the companies we support share with us. Keep up the good work!