Wildbit Bookshelf: Clojure Applied

"Clojure Applied: From Practice to Practitioner" by Ben Vandgrift and Alex Miller is a recent addition to our office bookshelf. After seeing all the enthusiastic feedback on Twitter, and hearing the authors talk about their book on Cognicast, I decided it would make for a good Quiet Friday read. Here at Wildbit, Quiet Fridays are one of those little things that form our culture. Any team member can take a Friday off work to read, learn, or dedicate some time to a labs project they’re passionate about. To paraphrase one of our core values: most of the things are not urgent — take some quiet time; think, then do.

Clojure is a functional programming language for Java Virtual Machine designed with focus on simplicity and practicality. We’ve been using Clojure in production to power business critical parts of Wildbit products for over 3 years now. It’s been a smooth run, but, as with everything else, there is always more to learn. With Clojure being relatively young, both the tools and recommended practices evolve over time, as the Clojure’s never-resting community figures out better ways of handling things. With that in mind, I spent last Friday in the office hammock reading "Clojure Applied".

The book assumes some familiarity with Clojure, so it’s more likely to be one’s second book on this programming language. This isn’t a hard requirement though. In my opinion, if you have any experience with Lisp and you can read some basic Clojure code then you should be able to find your way through “Clojure Applied” without much problem.

“Clojure Applied” guides the reader through the essential steps of building a production app. While the focus remains on Clojure, some ideas are easily applicable to any other programming language. The book encourages the reader to keep the program focused on their problem domain and provides advice on how to model its entities and their relationships using Clojure’s core data structures: records, maps, sets, vectors, and lists. It then teaches how to express the app’s business logic in a form of operations on sequences and channels, how to separate concerns in components, and, finally, how to compose everything in one functioning piece. For when it’s done, it describes various deployment options from popular cloud platforms to your own servers.

I was surprised to see how much attention the authors dedicated to sequence processing and Clojure’s concurrency primitives. Having read several comprehensive books on Clojure in the past, I found these chapters re-iterating a lot of material covered in detail by other books, like "Clojure Programming" by Chas Emerick, Brian Carper, and Christophe Grand. This information was of some use for me though, as "Processing Sequential Data" chapter had notes on using transducers, which we haven’t adopted in our codebase yet. Still, I’d have personally preferred a higher level overview of the practical applications of these tools.

While reading "Clojure Applied", I tried to remember what was the most difficult for me when I was building my first production Clojure program (DMARC digests for Postmark). A list below is what I put together in my notebook with some short commentary combining the authors’ advice with my personal experience.

1. What libraries do I use for X and Y? There is usually more than one library that can help you with a task. Picking the right one, or deciding to roll your own solution is one of the choices that you have to make early, but which can have far-reaching implications.
2. Do I always prefer a Clojure library to a Java alternative? Another difficult one. Aside from “pure” Clojure libraries (possibly using some Java under the hood), there is a whole family of wrappers on top of popular Java packages. In my experience, if the Clojure version feels even a little bit like using the underlying Java library directly, it’s not worth having this thin layer on top of it in the first place. Java interop in Clojure is pleasant enough, while the price of having this additional abstraction layer can be too high.
3. When do I convert types returned by Java interop to Clojure data structures, and when I just pass Java objects around instead? Passing (potentially mutable) Java objects around may feel “dirty” and “barbarian” in the Clojure’s world of precisely controlled runtime state. Still, I’ve found the benefit being worth the risk most of the time.
4. How do I structure my program in namespaces? I feel like this is the first one that had a complete and comprehensive answer in “Clojure Applied”. Reading it early would’ve certainly helped me avoid all those cyclic namespace dependency errors.
5. How do I handle runtime state, such as network connections? Another great piece of advice by the authors. They recommend representing these as explicit stateful components. While I was aware of this approach and knew about Stuart Sierra’s component library, none of our Clojure projects are currently employing it. I do regret it though. While it seemed like an overkill in the beginning, this approach would have provided us numerous benefits in a long run.
6. How do I go from one machine to many? The book doesn’t touch this topic, and I’m afraid there is no Clojure-specific answer to it. You can scale Clojure apps horizontally as you would any other app, but the language itself neither helps nor gets in your way when going in this direction. Clojure gives you exceptional tools for handling concurrency and parallelism on a single machine, but it doesn’t have much to offer for when you need to go beyond. Serialization, a topic covered in the book in detail, definitely plays a part, but there is much more to it (distributed shared state, messaging, synchronization, hierarchy, consensus) that you currently have to implement yourself.
7. How do I avoid runtime errors? Another big one. For a book on a dynamic programming language, "Clojure Applied" introduces schemas surprisingly early (in the first chapter), and also has a dedicated chapter on testing your apps (including in-REPL, unit, and property-based testing). Still, because of Clojure’s dynamic nature, tracking down all possible thrown exceptions and reasoning about collection types can be a tedious job. Because the language compiler doesn’t help you with this, you’ll almost inevitably run into uncaught runtime exceptions at some point.
8. How do I deploy to my own server? The book provides a high level overview of possible deployment destinations including your own servers. Unfortunately it doesn’t delve into operational details such as JVM health monitoring, live debugging, or built-in nREPL server. Our present deployment workflow for Clojure apps is a result of a years-long trial-and-error process. We use DeployBot’s atomic deployments with some in-house startup scripts that set desired settings for JVM. On top of that, all of our production services are continuously monitored with monitd and New Relic.

While “Clojure Applied” didn’t answer all of the questions I had as a beginner, I’d still recommend it to anyone who has learned a bit of Clojure here and there, but doesn’t know how to apply it. Even if it saves one from doing half of mistakes that I did, it’s worth it. Clojure is a one of those programming languages that will challenge your understanding of programming in general. From my experience at Wildbit, it works best for data processing and aggregation and leveraging the power of existing open-source Java libraries. With Clojure’s help we’ve built various systems ranging from data mining to VCS bindings to web APIs, and we all agree that it would be much harder to pull it off using our company’s primary programming language Ruby.

A software developers’ job includes a tremendous amount of learning. For most of us it means going with the flow and learning the tools as we face the problems. But if I have to pick one important thing Clojure has taught me, it would be going beyond the tools and trying to understand the nature of problems you face. The beauty of Clojure is in how it allows you to switch your thinking from writing code to the problem domain. It might not be the best programming language, but it is arguably the best programming experience. Give it a try, and "Clojure Applied" will help you get started!