Pareto Perl

In this case, the stream is our server logs, each record consists of a cookie variety, quantity, and customer email address, and we're aggregating the number of orders per cookie variety.

Let's start simple and iterate. We'll create a hash where each key is a cookie variety and the value is the number of cookies of this variety.

The regex is the same as before. && does short-circuit boolean evaluation just like in Java or C, so the effect is to increment a per-customer counter on each match.

Notice that we never declare our hash variable, x, nor do we explicitly initialize its values to 0. This works because Perl automagically initializes and assigns a hash to x the first time it is used (search perldoc perlref for 'autovivification' for further reading). Then when we try to perform addition on an undefined scalar, Perl helpfully treats that value as 0.

Perl's autovivification and context-sensitive coercion of variables initially struck me as bizarre. The perldocs are full of WTF-inducing gems like this one, from perldoc data.

To find out whether a given string is a valid non-zero number, it's
sometimes enough to test it against both numeric 0 and also lexical "0"

Sometimes, but hey, maybe not! You never know.

Taken in the abstract, this design choice sounds arbitrary and convoluted. But it's a perfect example of the pragmatism that makes Perl so convenient.

Compare our one-liner in Perl…

…to the equivalent Python:

There's nothing wrong with this Python implementation. It's certainly easier to follow than our dense, Perl, one-liner. But it's also 13 lines long and requires a dedicated file. If all you need is a quick answer to a one-off question, a couple of lines of throwaway Perl is almost always the faster route.

Back to our Perl implementation.

The only other interesting part of the code is the END block. Everything in this block is excluded from the implicit loop created by the -n flag and is executed exactly once at the end of the loop.

Now armed with cookie counts, our chef gets busy baking. But now our delivery department comes knocking. They need to prepare to package these orders. Problem is, cookies that contain nuts need to be packaged separately from nut-free cookies. So for each customer we need two numbers: the number of cookies they've ordered that contain nuts, and the number of cookies that don't.

This painfully contrived scenario calls for the second recipe in our Perl cookbook: aggregating over a hash of hashes.

Let's break it down.

Our regex hasn't changed, but the do block bears some explanation. For each order, we check if the cookie variety contains nuts and assign it to one of two categories: cookies with nuts are keyed off the string 'nutty', others are 'non_nutty'. We map each of these strings to a counter in each customer's hash, and increment the appropriate counter for each order.

The END block makes use of Perl's final data type: the array. For each customer in our hash, we initialize an array inline and print its comma-separated contents. The elements of the array are the customer's email address, their count of cookies with nuts, and their count of nut-free cookies.

This works, but it's getting unwieldly. If you ever need to hand this script off to a coworker, there will likely be a strong negative correlation between the number of $'s per line and the peer feedback rating on your annual performance review.

The typical lifecycle of my Perl scripts usually looks like:

1. Craft a quick one-liner, edited and executed directly from the shell. Iterate until I start having to squint.
2. Use fc to edit the inline script in vi, and throw in some newlines for readability.
3. Accept that this is no longer a one-liner and save the command to a file in /tmp. Open this file in a split tmux pane for quick iteration.

At this point, we've reached the final phase. We've had some great times with our one-liner and we'll always cherish the memories we create together. But if we need to extend this any further, it's time to admit that we've outgrown each other and move on from the shell.

If your first language is Python or Ruby then I wouldn't blame you for falling back to what you know best at this stage, given that we're forfeiting Perl's killer advantage: it's inline scripting capability.

But just for fun, let's go through the exercise of turning this jibberish into a respectable script.

First let's add some whitespace to make this a little more readable.

Notice that we've dropped the -n flag in favor of an explicit while (<>) loop. This does exactly the same thing and allows us to drop the END block. We could do better, though.

That eq or eq test is triggering my obsessive compulsive urge to refactor. It also affords an opportunity to introduce the idiomatic way of representing sets in Perl: a hash where each member of the set maps to the value 1.

The logic above is exactly the same, except that we've replaced that gross "if-or-or-or…" with a call to the exists function, which tests for membership in the set of nutty cookies.

Also, now that we're no longer constrained by the width of our terminal, we should start using sensible variable names.

Wait, 0 orders? That's obviously wrong. Ready for a neat debugging trick? We can use the Data::Dumper module to pretty-print data. Let's dump the contents of %orders to see where our refactoring went wrong.

Now the problem is apparent: we're using the wrong key in the customer hash. It should be "nutty" or "nut_free", not the cookie variety.

There, fixed. That debug output is just clutter now, though. Let's hide that behind a debug flag. Time to introduce a new Perl concept: subroutines. You declare a subroutine with the sub keyword. The arguments are available in the array @_. We can access the first argument by calling shift (as in, "shift left and pop the first element"), which operates on @_ if no argument is specified.

But wait, $is_debug is set - where's our debug output?

Perl tolerates reckless behavior such as multiplying strings and tossing references to uninitialized variables around willy-nilly. But once you've eaten your pig slop and come crawling back, begging for a little discipline in order to save you from yourself, Perl will graciously oblige. All you have to do is use strict and use warnings.

Yikes! That's a lot of warnings. Fortunately most of them are just telling us that we need to declare all of our variables using the my keyword before we reference them. Let's do that.

Sans clutter, the problem is easier to spot. The interpreter is giving us a hint here:

Global symbol "$debug" requires explicit package name (did you forget to declare "my $debug"?) at /tmp/extract-orders line 8.

In our debug subroutine we reference a scalar named $debug, which we never declared. That's because we actually meant $is_debug. Thanks, interpreter!

Fixed. But it's annoying to have to edit the code every time we want to toggle debugging. Let's accept a flag from the command line to enable debugging. The arguments to our program are available in @ARGV, and we can get the length of an array by resolving it in scalar context.

Much better! At this point, you could probably pass this script off to a coworker without fear of them throwing something at you.

And with that, we've covered just enough Perl to be dangerous. Let's do a quick review of the concepts that we've touched on.

-lne covers 95% of your use cases.

-l

automatically strip the record separator (newline, by default) off each input record and append it to each output record

-n

wrap your code in an implicit loop that iterates over each input record

-e

Execute the following string as a Perl program

See perldoc perlrun for more information.

Perl has three fundamental data types: scalars, arrays, and hashes. See perldoc perldata for further reading.

Construct a regex with / /. The syntax should look familiar if you've worked with regexes before. Captured groups are placed in $1, $2, $3, but you can also use destructuring assignment to put the captured groups into variables. Perl supports advanced constructs like positive and negative look-ahead and look-behind assertions, should you need them.

For more information, see perldoc perlre.

BEGIN and END blocks let you do things exactly once before and after an implicit while loop created by the -n flag.

Subroutines, or functions, are declared with the sub keyword. Arguments are passed to the function via the array @_. Functions can be invoked with or without parenthesis around the arguments.

See perldoc perlsub for more information.