michaelrbernste.in

The impossible fragility of Dockerfiles

Thu, 02 Feb 2017 00:00:00 +0000

The impossible fragility of Dockerfiles

Docker has done some amazing things for me, and for businesses that I’ve worked with, even in the very short time that it has existed. Certain large, technically complex problems that I’ve been a part of solving could not have been solved elegantly if it weren’t for Docker. I’ve seen people’s careers made by it, an entire secondary industry even, and yet, I still find their basic unit, the Dockerfile, to be a complete nightmare.

When I first started using Docker, I thought “this is a tough problem now, but it will probably get better.” Now, several years later, I’m not so sure. To me, the fundamental problem remains this:

Dockerfiles attempt to abstract over infrastructure by allowing units to be composed and extended. The fundamental issue is that that’s just not how infrastructure really works.

As an example, I recently spent time, as a very experienced Docker user, attempting to create a Dockerfile for an application which uses the following technology:

Ruby
Rails
MySQL (as in the client, not running a DB server inside this container)

Perhaps the three most vanilla, obvious choices for a language, a web framework, and a database to back them.[1] Instead of being straightforward for me to put this together, it was anything but. There is no off the shelf, easy way to “Dockerize” this type of application, and any solution that I found was broken in one way or another, made one or more wrong assumptions, or just straight up didn’t work.

While it’s true that once something has been successfully pushed into a Dockerfile it works nicely, it remains, as of the beginning of 2017, to be way, way too difficult and time consuming to make even slight changes to existing Dockerfiles to suit general purpose needs.

Again, I’d like to say something positive sounding like “I’m sure this will be solved eventually, and that this problem will go away,” but I’m really not sure that this is possible. What are some possible solutions that can make the Dockerfile problem tractable? How can infrastructure, so fragmented and granular in its natural state, ever become reliably composable? Someone help me out.

For now, I’ll be over here banging my head on a desk over and over again.

[1] If your takeaway here is “it’s Ruby that’s broken, not Docker!” – that’s kind of proving my point. How can Docker be successful when the things it purports to abstract remain so desparately broken?

A Framework for SaaS Pricing

Tue, 29 Nov 2016 00:00:00 +0000

After more than three years helping to build a business around a developer facing SaaS product, I've left to start my own consulting company called Reify. We do sales and Marketing software and process consulting for developer-facing, business to business SaaS companies. Get in touch if you make a product for developers and need help!

A Framework for SaaS Pricing

For many, pricing a product remains a dark art. Without a framework to work within, many people who create products simply copy competition, or just straight-up guess at what good prices might be. Talking to founders and startup executives who have struggled to get a grasp on how they should approach pricing has led me to search for relevant research on the subject, in order to be able to help them better. During this search I came across “Cloud Services Pricing Models” by Laatikainen, Ojala,and Mazhelis, which extends an existing framework for product pricing to be more specific and appropriate for cloud/SaaS software.

The existing SBIFT framework is named for the dimensions in the chart:

The idea is that a pricing model can be described by choosing a set of positions along a set of dimensions. Scope refers to the size of the product, base is the perspective that is most heavily used to inform pricing, influence describes various parties’ ability to control pricing, formula is another way of describing packaging, and temporal rights refers to the length of time a user can use the product.

Starting with this model, the authors extend it to be more specific to pricing and selling cloud-based software services:

Scope changes to move away from language using “packaging” and “attribute” toward language that uses “bundling” and “unbundling,” which are more common terms in the SaaS domain. “Pure bundling” refers to software which is sold in a way where features are only purchasable in pre-defined bundles, and “Unbundling” refers to the opposite, where the scope of features can be dynamically determined. Performance-based pricing, or pricing that is based on a certain SLA, is added to the base dimension.

The formula dimension gets the addition of a tiered-pricing item, which is a widely used packaging mechanism is SaaS products. Subscription is added as an item in the temporal rights dimension, and other items are removed, vastly simplifying this dimension, which is also appropriate given common sense understanding of the way SaaS works. Pay per use is also added in this dimension.

The degree of discrimination dimension is added in this cloud-informed version of the SBIFT framework. The authors describe price discrimination this way:

“Price discrimination is used when the same product/service is offered for different buyers for different price. This strategy is extremely important for providers of digital goods, since the low marginal costs allow them to sell the offering also for customers with low willingness to pay.”

Essentially, discrimination describes a wide range of techniques (discounts, etc.) that companies can use to specialize their pricing based on the customer, the circumstance, or the offering.

Finally, the dynamic pricing strategy dimension attempts to capture the strategies that companies can use in the medium and long term to describe how their pricing might change. I recommend reading this great post by Tom Tunguz on pricing strategies for more context and information on some of the terminology in this dimension.

Once the model has been constructed based on research, the authors characterize the pricing implementations for 54 companies selling cloud services. These are divided into three groups, SaaS, IaaS, and PaaS, which serve to illuminate a few segmentation differences in the outcomes of the research.

Beyond the interesting takes on how to classify elements of a pricing framework, the paper’s real power comes when survey results are overlaid on top of this newly defined model:

Tracing through all of the elements aside from Base and Dynamic pricing strategy, which did not have enough data, you can see the choices that most companies make, segmented by sector: SaaS, IaaS, or PaaS. Popular choices in each dimension are highlighted with boxes, and otherwise segmented percentage breakdowns are offered.

There aren’t a ton of huge surprises in the breakdowns, but it is definitely instructive to see how the companies they researched actually are distributed amongst these different dimensions and items. An interesting exercise would be to pose as a question to someone what they thought these breakdowns might be, and then compare them to the research presented.

More than anything, the usefulness of this research is summed up best by this quote from the paper:

“In this research, we attempted to find a systematic way to describe the pricing models in order to help decision makers plan, develop and speak about pricing alternatives.”

For folks who are looking to demystify pricing in order to be able to speak intelligently about their decisions and those of their competitors, this is priceless. In working with companies who are started by technical people and market software to developers, it’s one of my missions to help technical people see marketing and sales as a science backed with frameworks that can help them understand their problems, much as they do with software. I think this model is a great step in that direction.

A quick note on starting a business with someone else

Thu, 24 Nov 2016 00:00:00 +0000

A quick note on starting a business with someone else

I’m more inclined to work with someone who wants to build a company of the right size than with someone who wants to build the right thing.

Five qualities of great marketing

Tue, 08 Nov 2016 00:00:00 +0000

Five qualities of great marketing

Here are my current, crystallized thoughts about what I’m trying to convey to Computer Modern clients.

Great (B2B SaaS content) marketing…
   …has a single, specific purpose.
   …addresses a specific audience.
   …actually helps that audience.
   …exhibits domain expertise.
   …has diverse touchpoints.

The hidden secret of the CAC ratio

Wed, 02 Nov 2016 00:00:00 +0000

The hidden secret of the CAC ratio

The 2016 Pacific Crest Private SaaS Company Survey, which this year aggregated and analyzed data from 336 SaaS companies, was recently released. Finding data about private companies without paying for it isn’t easy, and the Pacific Crest survey is one of the best, most well-respected free sources of data available on the subject.

There are plenty of good roundups of the report that surface each year, so instead of a full recap, I’ll spend a few posts pointing out slides in the presentation which were particularly interesting to me, and to the companies I’m talking to at Computer Modern.

Today’s is about the cost of acquiring customers, a question which was posed to participating companies in this way:

“How much do you spend on a fully-loaded sales & marketing cost basis to acquire $1 of new ACV from a new customer?”

So your CAC ratio, according to this report, is the amount you spend per dollar you receive from a new customer. Here’s the graph:

The sidebar explanation here is interesting:

“Respondents (excluding the smallest companies) spent a median of $1.13 to acquire each dollar of new ACV from a new customer. The result drops to $1.00 if we include companies with <$2.5MM in revenues.”

That shows us the distribution of the CAC ratios, but it would be interesting to see how that changes with the size of the company. Let’s take a look at one more graph:

This one’s got a nice little summary on the sidebar too:

“Larger companies tended to report increasing CAC ratios for new ACV from new customers.”

There are some lessons to be learned here, especially if you’re running a company that makes less than $2.5MM annually, or you’re thinking about turning your idea into a product.

The median cost of $1.13 for $1.00 of new ACV shows that in SaaS, the money is in longterm happiness and success of customers.
Larger companies paying more money to acquire new customers shows that they have proven mechanisms to expand revenue during the lifetime of the customer.

To turn this into advice for SaaS companies who are trying to figure out their positioning and go-to-market strategy:

How much money you should spend to acquire a customer should be a crucial part of how you think about your company, from the beginning.
The lifetime value of your customers should be a crucial part of how you think about your company, from the beginning.
How you can provide your customers with more value over time, and thus charge them more money over their lifetime, should be a crucial part of how you think about your company, from the beginning.

Don’t just think about how to get people to pay you initially, think about how to keep them onboard, to keep them satisfied, and to find ways to provide them more value over time. That’s the hidden secret of the CAC ratio.

Yes, your team should be blogging

Wed, 02 Nov 2016 00:00:00 +0000

Yes, your team should be blogging

I tweet a lot. Enough that when one particular tweet gets more attention than others, it makes me wonder why. Today, that happened with this tweet:

An engineering blog with actual things that actual engineers have shipped is the best recruiting tool there is /c @patio11
— mrb (@mrb_bk) November 2, 2016

I’m hoping that a good number of people saw this tweet and agreed, and that’s why they told me they “loved” it, or Retweeted it. I started to think about the reasons why your developer-facing company should be blogging, and realized that recruiting is only one of them. So, I decided to take my own advice and blog.

Here are 10 more reasons why technical blogging should be just as important to your team as feature development or fixing bugs:

It helps with quality assurance - if you’re speaking out about it publicly, you’ll go the extra mile to make sure its battle tested
It helps create a rhythm around shipping
It signals to the outside world that the company behind the product is highly-functioning, and that people are excited about and invested in their work
It shines a spotlight on individuals or teams within a company who don’t typically get enough recognition
It helps diversify the voice of a company and its audience as a company grows
The best marketing communication is helpful and authentic, and being a developer writing to an audience of developers makes great writing that much easier to pull off
It helps promote technical ideas and Open Source projects that might otherwise have a hard time gaining momentum
It explicitly involves technical individuals in the revenue side of the business, something that is sorely needed in B2B startups
It helps outsiders understand products that they love to use
It lets you brag about how amazing your team is

…and I’m sure there are many more.

In addition to blogging, your team should be giving talks, and documenting, and contributing to evergreen resources, and a lot of other things, but start with blogging. In my opinion, there are few things with a better ROI for developer marketing and community growth.

The most important question to ask about your SaaS business

Thu, 20 Oct 2016 00:00:00 +0000

The most important question to ask about your SaaS business

After doing an initial announcement about my work with Computer Modern, I was very happy to hear from a number of companies (around a dozen) who were interested in the type of go to market strategy help I’m trying to provide. As I made my way down the list of founders, marketing executives, and managers who got in touch to chat, I had a great time refining my pitch, trying to get at the heart of the problems I’m trying to solve.

At first, I made the rookie mistake of saying way too much about the kinds of things I’m working on without doing one of the most important things you can do in sales: shut up and listen. After I got a few of these under my belt, I learned my lesson, and began to ask questions first before I tried to pitch my services. Out of all of the questions I asked, one seemed to get people thinking about the kinds of concerns I wanted to discuss better than any other.

Here it is:

How many customers do you want?

At first this might seem like a silly question. Who doesn’t want a lot of customers, right? Wrong! I didn’t ask “how much revenue do you want” (though it’s also wrong to assume that every company is interested in making billions of dollars), I asked “How many customers do you want?” What’s great about this question is that to answer it, you have to have a lot of other questions answered first. Here are just a few:

Will you sell to startups or enterprises?
Will your software be available for self-service purchase?
How much money on average will your customers pay you?
How will you reach potential customers?
Who is your target buyer within your target company?
How much money will your customer pay you initially?
How much money will your customer pay you over the lifetime of their engagement with you?

And many, many more. My position here is based on the belief, espoused by folks like Hardie and Fader, that Customer Lifetime Value is the key to understanding your business, especially in a contractual sales setting. Everything flows from how much your customers will pay you.

If you’re trying to figure out go to market, or how your idea or product can become a company, ask yourself this question, and force yourself to answer the other questions it poses.

How to Sell SaaS

Tue, 18 Oct 2016 00:00:00 +0000

How to Sell SaaS

When I decided to start trying to sell my services as someone with unique insight into marketing and selling software to developers, a thought occurred to me: I should probably back my experiences up with some research. I had read some papers relating to understanding customer behavior and revenue patterns, and delved a bit into the statistics and theory behind the KPI measurements that startups tend to rely on, but I hadn’t done much searching for research about sales and marketing. I wondered what was out there.

After some searching, I came across a nice paper titled “How to Sell SaaS: A Model for Main Factors of Marketing and Selling Software-as-a-Service” by Pasi Tyrväinen and Joona Selin. In it, the authors develop and refine a model for understanding the important components of marketing and sales in a Software-as-a-Service (SaaS) business model. This paper interests me because helping companies develop go to market strategies is on the top of my list of projects I’m conquering with Computer Modern.

After some reflection, I realized that the reason this paper appealed to me is because it helps to do one of my favorite things in an elegant way: it breaks down a large, messy subject into constituent parts, backed by research to bolster its claims. Papers which provide overviews of existing disciplines are always interesting to me, and I look forward to reading more deeply in this area.

The authors are out to achieve a few basic goals in this paper:

“Our purpose is to initiate academic research on marketing and selling SaaS by presenting a model of the key factors for marketing and selling SaaS and related key performance indicators (KPI).”

Now that’s what I’m talking about. It’s great to see work aimed at taking situations where most people go on gut, and attempting to transform that into a more scientific, data-driven approach. All too often in my career I have made the mistake of assuming too much without bothering to check details, and I’ve seen other leadership do the same. This paper is definitely a step in the right direction.

In this post I’ll pull out some of the more pertinent aspects of the paper to the work I’m doing with Computer Modern. Since I can’t cover it all, I hope you’ll read the paper in full and share your thoughts with me.

Definition of SaaS

One of the more useful components of academic work is its ability to provide definitions for complex ideas. Though the definition of SaaS as provided by the authors may not be complete or accurate for every case, it’s a great starting point, and a useful one if you’re explaining the concept to someone who is not familiar with it. The authors define the “criteria for compliance with the SaaS model” as:

Software is used with a Web browser or other thin client making use of standard internet protocol.
A standardized software product is provided with no customization.
There is no need to install software to the customer site.
Deployment requires no major integration or installation.
Customers pay for use of the software rather than licenses.
The same multitenant installation is provided for several customers.

It’s interesting to see how this definition contrasts with one you might put together - this can definitely be an enlightening exercise. The result of a business which exhibits the above criteria is typically that a customer can easily adopt and buy the software, turning the standard enterprise sales model on its head.

It’s worth noting that the cloud represents a major shift in how SaaS can be delivered - “on premise,” single-tenant products (often known as “enterprise versions”) are often sold alongside the multitenant ones. Offering dual product lines is so pervasive that companies like Replicated have sprung up to serve SaaS providers looking to provide single-tenant versions.

An initial model

The heart of this paper is the development of a model representing “The main factors of marketing and selling SaaS collected from the literature.” The authors surveyed a number of papers and online sources in order to determine a set of variables that determine strategies for marketing and selling SaaS products.

Each dimension has a scale, and the model is meant to be understood this way: “the categories in the middle of the diagram are likely to co-occur and the categories in the outer rings are likely to co-occur.” For example, under the “Service and implementation model,” you would expect Self-Service products to co-occur with small customers paying a small amount of money (from the Customer Size and Entry Transaction Size categories), and Consultation based services to be sold for more money to larger customers.

It was rewarding for me to see this diagram because it reinforces something that I bring up a lot when discussing go to market strategies with founders, CMOs, etc. - no decision in the realm of the “business side of business” can be made in isolation. Customer Lifetime Value, initial deal size, customer persona, sales channels, how to market, customer size, etc. all influence each other, and how you approach one of these aspects will influence your ability to successfully approach others.

An evidence-based approach

Once the authors have described an initial model based on their synthesis of existing research, they document their multi-case study, where they survey a number of Finnish startups in order to see how the companies actually function with respect to their initial model. The resulting model is a very interesting set of tweaks on the first:

The basic refinements here are based on recognizing that of the original eight dimensions, there are groups which are so tightly coupled as to necessitate being collapsed into one. This has the benefit of turning something from being eight-dimensional to four-dimensional, which is considerably easier to reason about.

Additionally, these four categories are augmented with KPIs for short and long term success in each given area. This is one of the most useful aspects of the refined model for me - seeing KPIs paired with the dimension of the model they correlate with makes it easy to have conversations around the numbers, not just the ideas. Churn, Customer Lifetime Value, Customer Acquisition Cost, and Entry Transaction Size are all crucial numbers that are useful for measuring performance, but they’re also incredibly important when discussing strategies and tactics for growth. Determining how the decisions you’re making will impact these numbers isn’t always possible to predict, but being in the mindset of keeping these numbers in your head is inarguably effective.

The revised “clover” model that the authors present is a fabulous resource. Starting from an idea and trying to turn it into a business, with a developed go to market strategy, is extremely difficult. Structured ways of thinking about how your idea will actually turn into a product in the market are crucial to measurable, sustainable, repeatable growth, and can make these exercises considerably more efficient and less prone to biased thinking.

The big picture

So how does all of this add up? What kinds of things should startups looking to develop a go-to-market strategy and grow be paying attention to? There are a few quotes in the conclusion of the paper which speak to these questions - I’ll pull out a few them here.

“The dominant factor to determine the target group of customers was the number of potential users, which is directly related to the headcount of the customer organization.”

Whenever I have the opportunity to discuss business strategies with technical founders, I always ask them this question first: “How many customers will you need to be successful?” This is a good way to put a real face on what is kind of an abstract question. The idea is that if you need $1MM in ARR to reach your first milestone, how many companies will that consist of? 10? 100? 1,000? Do you really think you’ll want to support 10,000 customers? If so, how will you grow their revenue, etc.?

See? A good starting point. I love that question because of all of the other questions it forces you to answer.

“The main sales channel was direct personal sales supported with Internet-based marketing communication. Internet as such was not much used as a sales channel, and advertising was not used in marketing communication.”

Developers who make products to sell to other developers often get trapped by attempting to apply to many tools too early. Online ads are an easy thing to pay for, but are they the right thing for your company right now? I’m not sure if the conclusions the authors make here regarding the relative dearth of online advertising applies to everyone’s strategies, but it’s definitely something worth being critical about. The next quote implies why this is the case.

“The most important performance indicator for marketing and sales was customer acquisition cost.”

This is a big one. What I think startups should do is invest earlier on in making sure they’re nailing their go to market strategies. What I DON’T think they should do is overspend on frivolous audience building activities or even hire someone full-time unless they’re fully capable of understanding the domain of the product until they’re clear about their goals with respect to the four “leaves” of the clover. Messing up and overspending to acquire customers who will cancel quickly and not provide obvious value is a recipe for the dead zone. Avoid it at all costs!

“Finding a marketing and sales approach … will be critical to success of any SaaS firm in the near future.”

This quote really spoke to me, because it’s the whole reason I’ve been ranting about this stuff lately: how you package our product matters. Go to market is the thing that will separate your popular Open Source project from a sustainable business that can keep you going for a long time. Don’t wait until its too late to get invested and excited about the positioning, packaging, and pricing of your product.

Wrapping up

Overall, the authors do an admirable job of collecting data and impressions from a wide variety of sources in an attempt to codify something very complex: how your SaaS business should approach Sales and Marketing. Whether you agree or disagree with their categorizations or choice KPIs, the “clover” is a great conversation starter, and I’ll definitely be breaking it out when chatting with folks through Computer Modern.

Works Cited

Tyrväinen and Selin, “How to Sell SaaS: A Model for Main Factors of Marketing and Selling Software-as-a-Service” [PDF Link]

Five failure modes for technical founders

Sun, 09 Oct 2016 00:00:00 +0000

Five failure modes for technical founders

Developers who also have a knack for products and want try their hand at being an entrepreneur are in an amazingly advantageous business position: they can build and sell whatever they want. Usually, though, they’re much clearer on the build part than the sell part. I’ve been thinking a lot about how to make and sell products to developers over the course of more than three years at Code Climate.

Witnessing things for myself and talking to lots of technical founders, I’ve found five common ways that technical founders can get it wrong. Correct these issues early on and stay on top of them, and your chances of succeeding will be greatly improved.

If you’re a founder and are struggling to build a business, find product market fit, and just plain sell, you may be doing one or more of these things:

Not making something you and your colleagues would use
Not charging enough money
Not knowing your buyer
Not automating enough
Not embracing sales and marketing

Let’s briefly take a look at each of these issues and why they’re important:

Not making something you and your colleagues would use

You got into this business because you’re a developer and you know developers, right? If your product doesn’t solve a problem you or your friends and colleagues have had, it might not be the right product at all. It’s very hard to sell something you don’t believe in, and believing in a product starts with knowing that it’s going to be useful, and that its value proposition is clear.

Not charging enough money

As a developer, one area where you shouldn’t listen to your friends and colleagues, for the most part, is when it comes to pricing. Developers can be very price sensitive, sometimes unrealistically so, and they often just don’t have the insight necessary to know what it takes to make a business meet your business goals. If you want one free piece of advice, before I even take a look at your pricing page for you, it’s this: DOUBLE YOUR PRICES. You’re welcome.

Not knowing your buyer

Thinking about, obsessing over, and modeling the persona of your ideal buyer at software companies is one of the most challenging but crucial aspects of selling SaaS products successfully. Your marketing, pricing, and functionality all hinge on who it is you’re trying to sell to.

Not automating enough

Sales and marketing can be really hard, but thankfully we have software to help us! Use your skills here and automate. Get crafty about sales and marketing the same way that you get crafty about development and operations. Thinking like an engineer here is a gift - there’s lots of ways that being lazy can help you sell better.

Not embracing sales and marketing

It’s never too early to start investing in sales and marketing. As the founder of a company you’re the best one to sell its products, but that doesn’t mean that you shouldn’t be getting help from experts.

What do you think? Have you experienced these problems? Think I’m full of shit? Get in touch and let me know!

The Power of the Material, or: Make something and keep it on your desk

Fri, 29 Jan 2016 00:00:00 +0000

The Power of the Material

or: Make something and keep it on your desk

One of the first pieces of pottery I ever made ended up getting a much more royal treatment than a beginner’s piece typically receives or deserves. After being thrown on a wheel and then fired once in an electric kiln at my studio, it was finished in a wood-burning kiln at Baltimore Clayworks, a cultural institution dedicated to ceramic arts that, like me, dates back to 1980.

Wood-burning kilns were the traditional way to high-fire pottery in certain parts of the world for certain periods of time, but in modern times they are increasingly rare. It takes nearly a full week from start to finish to fire pottery in the kiln at Clayworks: several hours to load the kiln (an art in and of itself), 24 hours to fire the kiln (during which shifts of people feed the voracious fire until the kiln temperature reaches nearly 2,500 degrees farenheit), 3 days to cool it down, and another several hours to unload, clean, and get ready to start all over again the next week. Contrasting this with an electric or gas kiln which takes the time down to 1-3 days, and you can see why most people don’t do this kind of thing anymore.

I was lucky enough to participate in a wood-firing about 5 months into my time as a potter, and it was amazing. I learned the folklore of the kiln, about how different historical glazes were influenced by the chemicals in the wood from trees indigenous to their regions, how ash and smoke flow through the giant oven, and more. It was awesome. Here’s my favorite piece from the firing:

It’s a small, lopsided, not very useful “bottle” form that I sometimes put little branches or weeds into. It’s not particularly skilled, it’s heavier than it should be, the lip is uneven, the glaze should go further down the body, there’s not great ash from the kiln, etc.

Even though all of these critiques are true, I’ve kept this bottle on my desk every day since the kiln firing. I have turned it over and over in my hand thousands of times while on video calls, while thinking about problems, while trying to waste time. I’ve considered it from so many different angles, felt it move slowly in my palms with my eyes closed, thrown it up and down in the air to see how it’s weighted.

I know exactly how I would make this form again if I had the chance, how I would glaze it differently, where I would try to get it placed in the kiln. I have some ideas about ways to decorate it, thoughts on the types of flowers that would look good in it.

I’ve thought very deeply about this object, and then I spent a bunch of time thinking about why I’ve thought so deeply about it. My conclusion turned out to be pretty simple: it’s been so easy to consider this piece so deeply because it’s right in front of my face all the time. I can pick it up. It’s physically there. It’s physical presence helps me understand it.

And then I started to think about the things that we create which aren’t physical, and the problems that we tend to face with maintaining, knowing, and understanding those things. Teams of very smart people who spend hours and hours fighting against software systems, for instance, that everyone knows some part of but no one could possibly know in its entirety.

While I don’t have a solution or a good answer about how to make software more physical, or how to consider the immaterial things I create as deeply as the material ones, I can definitely say one thing: making something physical and then keeping it on your desk can teach you a hell of a lot.

It can teach you about the consequences of small, seemingly meaningless decisions on final outcomes. It can teach you about how much you learn about something once it’s been used, and that you shouldn’t stop thinking about something or consider it’s “done” just because the initial phase of making it is over with. And lots more.

I’m trying to schedule time for another wood-firing this summer - it will probably take me 4 or 5 months to prepare this time. Now that I’ve gone through the process once, and spent enough time thinking about it, I know that it will be much easier for me to achieve my desired results. Just another lesson that having a lumpy bud vase on my desk for six months has taught me.

I'd like to introduce you to a few of my closest friends - CUSEC 2016 Keynote

Fri, 15 Jan 2016 00:00:00 +0000

I’d like to introduce you to a few of my closest friends

This is the transcript and slides of a keynote talk I gave at CUSEC 2016 in Montreal.

I’d like to introduce you to a few of my closest friends

Michael R. Bernstein

CUSEC, Montreal, QC, 15 Jan 2016

Hello, and thank you for having me - I’m honored to deliver a keynote speech to this group of wonderful people for a few specific reasons.

Firstly, I used to be a teacher, and I miss being around students. Your energy and intelligence never cease to amaze me. Thanks for all of the great conversations I’ve had with organizers, attendees and fellow speakers so far this weekend - I’m looking forward to a lot more.

Secondly, I’m used to speaking on very specific topics to a room full of industry professionals or seasoned Academics, some of whom have heard it all before, know everything, and yet still continue to attend conferences! It boggles the mind. In this case, I get to address a group of students about a topic entirely of my own choosing! Though you probably still think you know everything if college-aged me is any indication.

David and the rest of the CUSEC committee are very brave for inviting me to do this, and after I got the invitation I felt like I had to give the kind of talk that you wouldn’t typically hear in this scenario.

I really didn’t want to let you all down, so I started to think long and hard about what I can personally bring to the table to talk about that would be unique to me.

I’m not a well-renowned programmer, nor have I ever innovated or created any kind of popular world-changing programming language or web framework, so talking about a specific technology was out.

I’m not a researcher presenting original thought about Computer Science, so unfortunately today I won’t be giving you a sneak peek at my current work, nor giving a retrospective of my famous work that you’re already familiar with. Mostly because I’m not working on anything original and I don’t have any famous work that you’re already familiar with.

I wasn’t employee #3 at a famous social media startup that makes a product that everyone in the world uses. Your parents and that weird cousin of yours who lives in the middle of nowhere have never heard of anything I’ve worked on.

I’ve never started a company, or made a lot of money from a Venture Capital based exit, or written fiery thinkpieces that have changed the course of … wait, do fiery thinkpieces ever change the course of anything?

All of these poor credentials of mine pushed to the side, I began to get a little bit scared. What was I going to talk about? There must be something that is unique to me that I can write an hour’s worth of words about.

So here I am, blessed with the singular opportunity to address a room full of promising, wonderful individuals, with not much to talk about. At this point the conference is a mere few weeks away, and I still haven’t cemented what I wanted to communicate to you.

So I took a deep breath, and did what I usually do in this kind of situation. I looked to one of my friends for inspiration. In this case, I cracked open a book of poems by one of my favorite poets, Wallace Stevens.

I turned to a well worn page in my Stevens anthology, the one where my favorite poem of all time lives. It’s called “Thirteen Ways of Looking at a Blackbird,” and I’m going to read it to you now.

“Thirteen Ways of Looking at a Blackbird” by Wallace Stevens

I

Among twenty snowy mountains,
The only moving thing
Was the eye of the blackbird.

II

I was of three minds,
Like a tree
In which there are three blackbirds.

III

The blackbird whirled in the autumn winds.
It was a small part of the pantomime.

IV

A man and a woman
Are one.
A man and a woman and a blackbird
Are one.

V

I do not know which to prefer,
The beauty of inflections
Or the beauty of innuendoes,
The blackbird whistling
Or just after.

VI

Icicles filled the long window
With barbaric glass.
The shadow of the blackbird
Crossed it, to and fro.
The mood
Traced in the shadow
An indecipherable cause.

VII

O thin men of Haddam,
Why do you imagine golden birds?
Do you not see how the blackbird
Walks around the feet
Of the women about you?

VIII

I know noble accents
And lucid, inescapable rhythms;
But I know, too,
That the blackbird is involved
In what I know.

IX

When the blackbird flew out of sight,
It marked the edge
Of one of many circles.

X

At the sight of blackbirds
Flying in a green light,
Even the bawds of euphony
Would cry out sharply.

XI

He rode over Connecticut
In a glass coach.
Once, a fear pierced him,
In that he mistook
The shadow of his equipage
For blackbirds.

XII

The river is moving.
The blackbird must be flying.

XIII

It was evening all afternoon.
It was snowing
And it was going to snow.
The blackbird sat
In the cedar-limbs.

Dedicated to Wallace Stevens, and all XIII Blackbirds

That poem, published almost 100 years ago in 1917, was one of the first poems I ever read multiple times. I really don’t even remember how I found it, and I have an impossible time explaining why I like it so much. But for some reason, I look to it often for answers, and sometimes, mercifully, it provides them.

This talk is the product of reading this poem many, many times, so I de facto dedicate this talk to Wallace Stevens, and one of his earliest poems, about which he would probably be terribly embarrassed if he were still alive.

So what was the realization that I came to while reading this poem for the fifty-eleventh time?

That most software engineers don’t count poems among their greatest influences, or at least if they do, they don’t spend enough time talking about it.

It dawned on me somewhere between the 8th and 9th birds in the poem that the best thing I could do to encourage a room full of eager, open-minded students, was simply to be myself. I should use the things I know the best, the things that are closest to my heart. And be vulnerable.

So that’s what this talk is all about. I’m going to introduce you to 10 (well, 11 if you count Stevens) of the individuals who have impacted who I am and how I think, from the beginning of my intellectual explorations as a child until today.

I call these individuals my friends because in my mind that’s what they are - companions, there for me when I need them most, unwavering in their support, often helping me understand my surroundings but just as often asking new questions that I could never hope to have the answers to.

I spend a lot of time doing things that have nothing to do with computers. The longer that I’ve worked in technology, the more those outside experiences have shaped my job choices and day to day activities.

If you’re confused or uncomfortable right now, or if you keep looking around to see if you’re really at a conference about Software Engineering being addressed about poetry by a bedraggled and bearded weirdo, then I’m doing my job. Stay with me.

Since this talk is about me and what I’m calling my friends but are actually my influences, let me just tell you a little bit about myself first.

I’m Obsessed

I used to begin my talks with this slide. I would put this slide on the screen, like it is now and say “Hello, my name is Michael R. Bernstein, and I’m obsessed. Not with anything in particular, just, you know, in general.”

While it was good for a laugh, I realized over time that that’s probably not the best way to introduce myself, for various reasons.

First of all, even if I was “obsessed in general,” that doesn’t sound like something I should be boasting about, and it certainly doesn’t lead to very healthy behavior.

Second of all, it doesn’t at all convey what I thought it would, and instead tended to give people the impression that I feel that being obsessed is the only way you can learn things. This is far from the truth.

So, I decided to make a change.

I’m doing just fine thanks

Hi, my name is Michael R. Bernstein, and I’m doing just fine - thanks.

So enough about me. Today I’m going to introduce you to some of my friends, roughly in the order that I met them. I’ll distill the lessons that I’ve learned from these individuals into one or two words, which is by nature super reductive but I think serves the purpose of illustrating how all of these disparate pieces fit together inside my head.

So without further ado, let’s get started.

Thelonious Monk - Improvise

I first met Thelonious as a teenager. I was young and didn’t know anything about music. Vanilla Ice was popular and I was cool with that - this is the kind of state of mind I was in back then.

One day my friend Mark, who was learning how to play the trumpet at the time, came over with a CD of Miles Davis’ seminal album “Kind of Blue.” We listened to it on my crappy JVC boombox, and I was super confused.

Where were the words? What kind of weird songs are these? No electric guitar? Were these people even stopping, collaborating, and listening? Turns out they were.

I didn’t have access to “Kind of Blue,” but I knew that my dad used to play the trumpet and was at least a little bit into this kind of music, which I discovered was called “Jazz.”

I went upstairs to the attic and rifled through his record collection, disappointed to find that he didn’t have “Kind of Blue” - I still need to ask him why the hell he didn’t have a copy of that record, by the way - but I did find this record: “Monk’s Dream” by Thelonious Monk.

I took that record from his old dusty box and brought it downstairs, where I had set up his old turntable and connected it (without a pre-amp, I might add - ask me later why this is an important detail if you don’t know what a pre-amp is), and played it over and over again. It dawned on me that music didn’t always need words.

I started to make all kinds of musical connections because I was fortunate enough to have access to this album. I realized that music conveys emotion on its own, that vocals and lyrics can be secondary to the music, that song structure is something that can be played with, bent, molded, and shaped.

After listening to this album a bunch, I asked Mark about it, and about “Kind of Blue.” What was going on here? Did they always play the songs the same way twice? Some of the tracks were long, and it seemed impossible that they could recreate this whenever they wanted.

Mark told me that no, they don’t play the same thing every time and that Jazz musicians are known for something called “improvising.”

“Improvising? Sounds cool, what’s that?” I asked.

“Well, the musicians work within a pre-defined structure, but they improvise within it. You don’t know much about music but there are basic rules, and improvising is all about expressing yourself, while staying as close to these rules as possible, but also trying to push things, you know, make them sound interesting.” Mark replied.

Wow. I was floored. First of all I loved music A LOT up until this point and though I had tried to take some piano lessons that never really stuck, I had never really thought about what music was made of, how it was produced, how it worked, how we understood it. But I was completely fascinated by the idea, and it really stuck with me, to say the least.

Here’s a picture of me about 10 years after I first learned about improvised music. I’m playing a homemade synthesizer in a quartet of improvised musicians and wearing tie-dye. So really, not much has changed in the 10 years that have passed since this picture was taken.

So Thelonious (and tangentially, Mark, wherever the hell you are): thank you for teaching me about improvisation. I rely on this concept all of the time. I try to give myself permission to improvise.

Improvisation is about starting down a well-known path and ending up somewhere you could never imagine, and is a crucial skill for human beings in general and software engineers in particular.

In your career as programmers, project managers, entrepreneurs, psychedelic synthesizer warriors, etc, you will often be expected to find your own solutions by starting somewhere known. Your ability to improvise might not help you get there any quicker than anyone else, but I can guarantee that you will have a more interesting journey.

If you’d like to know more about Thelonious and his amazing life, I highly recommend this authoritative biography by Robin Kelly. It will change the way you think about the history of music, and if you’re reading closely enough, it will change the way you think about the world in general.

Gilles Deleuze - Question Everything

Speaking of changing the way you think about the world, a few years after I got deep into spending a lot of time with Thelonious and Miles, I started to spend time with some interesting (if not kind of dour) people named Franz, Friedrich, and Gilles.

I was a classic pretentious teenager with pretentious friends who collected difficult books and tried as hard as possible to sound intelligent, therefore sounding very unintelligent indeed.

Regardless, something about the work in Gilles Deleuze’s “A Thousand Plateaus” stuck with me and seriously continues to impact my thinking in ways that I don’t think any other person’s thoughts have done for me.

“A Thousand Plateaus,” which is subtitled “Capitalism and Schizophrenia,” is a book that is very difficult to describe. It is the second volume of a collaborative effort between Deleuze, a philosopher, and Felix Guattari, a psychoanalyst.

If you were to ask me what this book was about, I would hesitate to say “everything,” but would probably end up saying “everything” anyway. As Deleuze and Guattari romp through subject matter as diverse as economics, biology, psychology, and more, they make one thing very clear: everything is to be questioned.

What we are taught, on the surface, about how to think, how to read, how to exist in a society that is permeated with the need to be more productive, and happier, and better, and richer, should all be questioned. “A Thousand Plateaus” is not to be read in a linear order, it is to be absorbed, osmosis-like, in any order you see fit.

“A Thousand Plateaus” teaches us that societal and economic control are not the big, scary monsters that you think - they are the combinations of millions and millions of small, supple actions that in aggregate present to us, on the surface, the society we are a part of.

After reading and re-reading and trying to read and failing and reading again, this line of questioning seemed very deep to me - is the binary really binary? Is God a lobster after all?

What does an obscure book that is ostensibly about Postmodern theory have to do with software engineering? Part of the point of this talk is just that - books, art, music, etc. don’t have to have “anything to do with” software engineering to help you become a better software engineer.

Without blowing the rest of the talk for you by spoiling the ending, you’re going to hear echoes of this idea several more times: to read and digest art, music, philosophy, poetry, and human creativity is to become a better person. If you ever hope to be an amazing software engineer, someone who changes the world with your code and your ideas, it is imperative that you focus on becoming the best person you can be at the same time.

Well-roundedness is not something that you use to pad your resume for work. Well-roundedness is something that you pursue because it is the very damn reason you were put on this earth to begin with - to learn and share.

So thank you, Deleuze, and thank you Guattari, for this amazing beautiful impossible book which taught me to take nothing for granted, to assume nothing, to experience everything.

Baudelaire - Redefine Beauty

From philosophy to poetry, I did seriously attempt to rack up as many points as possible in the column of wannabe smart teenager around this time in my life. Sadly, I probably modeled my idea of what constituted “smart” more from movies and magazines than anything else, but what can you do?

Though I postured in many other ways at this time during my life, claiming allegiances that I didn’t really have, pretending I was something I was not more often than not, I really did see something in Baudelaire. I was fascinated by the anger and disgust depicted in the book, the tortured changing definition of beauty, and these things have never really left me. My deep and enduring love of poetry is something that I guard very closely and am very proud of.

Baudelaire’s most famous work is (pardon my horrendous French accent) “Les Fleurs du Mal,” or “The Flowers of Evil.” A long, meandering, disturbing book, it nevertheless manages to do what I believe Baudelaire hoped it would - it expands your notion and understanding of what is beautiful, of what words can describe, of what a book can make you feel.

In an homage to peeking out of windows, Baudelaire looks upon the world, reports on it, and provokes the reader who asks him whether his observations of the world are in fact “the right story.” Baudelaire replies:

“What does it matter what reality is outside myself, so long as it has helped me to live, to feel that I am, and what I am?”

The preface to the book contains this “malediction,” another provocation, a defensive answer to the readers who he knows will misinterpret his work in some way:

“It’s Ennui!—his eye brimming with spontaneous tear / He dreams of the gallows in the haze of his hookah. / You know him, reader, this delicate monster, / Hypocritical reader, my likeness, my brother!”

There are many poets who have since taught me lessons that have helped me redefine my aesthetics in subtle ways. Baudelaire was simply one of the first to bring a hammer to my naive, narrow notions of the capabilities of words.

So thank you, Baudelaire. Every time I find myself stopping on the sidewalk to stare at some peeling paint, every time I decide to not limit myself in order to experience as much as possible, every time I manage to find beauty in what others pass by or reject, I think of you.

I’m not going to try and connect Baudelaire to software engineering because I think that would be even a bit profane for Baudelaire’s tastes, but I will simply restate - reading poetry makes you a better human, and being a better human makes you better at whatever it is you’re trying to do.

Brian Eno - Hone Your Aesthetics

From “Music for Airports” to his famous pop albums of the 70s, Brian Eno is one of the deepest and most successful sound thinkers of the 20th century. I learned about Eno late in my High School career. Again I don’t really know how I first came across him.

Eno was a maverick who created an entire world of music out of nothing. He was one of the founders of the group Roxy Music, who infused a hell of lot of weirdness into the 1970s UK Rock Scene. Roxy Music, like the Velvet Underground, is one of those bands about whom you can say that “Not many people liked Roxy Music, but all of them went on to form their own bands.”

Eno’s solo work is defined by an amazing, precise attention to detail. “Music for Airports” is ambient music, to be played in the background. It is composed in a way that if you try to pay too much attention to it, it just squirms out of your reach, defying definition and stasis. If you do what Eno suggests, and let this music define your environment and impact your mood, that’s how you can really get the full picture of what Eno is trying to achieve.

The amazing and brilliant thing about Eno’s music is that he was not a trained musician, did not really know “how to play instruments,” and was still capable of defining a fabulous aesthetic arc that continues to this day. Listening to his work with the recently deceased David Bowie, for example, you can hear what sounds like thousands and thousands of layers of sound rushing together to create a warm pool of sound that you could spend your whole life trying not to drown in.

Though Eno is often touted as “experimental” and his music “obscure,” he is fact one of the most fastidious and purposeful creators of sound in the history of modern music. Though he embraced chance and John Cage-like systems that allowed him to create dense, shifting landscapes of sound, he never released anything that wasn’t, at its heart, beautiful.

Eno has lots of somewhat obvious lessons to you as a group of individuals who are in various stages of your journey into the real world. I’ll share two that come to mind:

First, regardless of how good your idea is, implementation is what counts. Take the time to make your work beautiful. Take pride in your craft, make it count.

Second, it’s often people who “shouldn’t” that end up changing the world. By all accounts, an art school drop out whose only skills were fixing reel to reel tape machines and synthesizers “shouldn’t” have made some of the most beautiful music this world has ever seen.

In your travels, you’ll meet people like this - people who don’t have the same training or background as you, but remain stubbornly amazing at what they do. Don’t shun them, or make them feel bad. Embrace them, and learn from them, and watch as they amaze you in ways you never thought possible.

So thank you, Mr. Eno, for never giving up, and for proving that what counts is what you leave behind, not the so-called legitimacy of your history.

Virginia Woolf - Use Your Real Voice

The first time I ever picked up a novel to be completely blown away by the voice and perspective that it was written in was Viriginia Woolf’s “To The Lighthouse.” This was around my Senior Year of High School or my first year of college, in a literature survey of some kind, and I was completely not prepared for what was contained in the book.

A somewhat mundane story ostensibly about a family and their travels to a summer home in Scotland, “To The Lighthouse” is really about thinking and voice more than anything else. In it, Woolf extensively employs a stream-of-consciousness style of writing where the thoughts of any number of characters are laid bare for the reader to interpret.

Lacking the omniscient narrator that both authors and readers tend to rely on as an anchor in a novel, Woolf instead relies on the constantly moving thoughts and changing perspectives of the characters, not so much to “tell a story” as to get you to think.

This brilliant technique really worked for me. I recall an assignment related to “To The Lighthouse” where we were instructed to write something in a stream-of-consciousness style ourselves. I absolutely loved it. I tried to write about what I was thinking. I ended up writing about thinking and thinking about writing. And thinking about voice, and what it meant to “have a voice.” What it meant to change perspectives. How many different ways there were to tell the same story.

One of my favorite quotes from the book either sums this whole section up perfectly or completely muddles what I’m trying to say, rendering this section useless. That’s for you to decide:

“What is the meaning of life? That was all—a simple question; one that tended to close in on one with years. The great revelation had never come. The great revelation perhaps never did come. Instead there were little daily miracles, illuminations, matches struck unexpectedly in the dark; here was one.”

So thank you, Virginia Woolf, for showing me the malleability of perspective, for encouraging me to use and develop my own voice, by showing me that you could use and develop your own.

And that’s the lesson I want you to take away from Woolf and “To The Lighthouse” - that your ideas, unique and wild as they may sound to you in your head, are worth exploring. That it’s possible to change BIG THINGS, like the way stories are told. On top of that, it never hurts to be inside someone else’s head for a while - it often helps you once you end up back in your own.

Harry Smith - Obsess

Though I no longer wish to introduce myself in association with obsession, I still feel that we have a lot to learn from people who are or have been truly obsessed. My favorite obsessive individual is by far Harry Everett Smith, so I’d like to introduce you to him.

Harry Smith was a truly American icon of the 20th century, someone who most people don’t know but was a big inspiration for people much more famous than him, including poets like Alan Ginsberg and authors like William Burroughs. Smith grew up in the Pacific Northwest of the United States, where he was fascinated with his surroundings and began earnest documentation projects in his early teens, notably recording the ritual and ceremonial music of the Kiowa Tribe who lived near to his home.

Over the course of his life, it Seemed that what Smith collected was less important than the fact that he was collecting, and this was for a very logical reason: his primary interest was in the relationship between things, not with the things themselves. Smith had a way of seeing every day life in ways that revealed subtle, intricate systems and connections that inspired him to continue his life long mission of collection, curation, and creation.

Here’s the story of how Harry Smith ended up moving to New York City, where he completed his most famous projects - an epic abstract film and a collection of folk music for Folkways records which led directly to the proliferation of American rural and spiritual music to the emerging folk scene that birthed Bob Dylan, Joan Baez, and more:

“In 1950 Smith received a Guggenheim grant to complete an abstract film, which enabled him first to visit and later move to New York City.[10] He arranged for his collections, including his records, to be shipped to the East Coast. He said that “one reason he moved to New York was to study the Cabala. And, ‘I wanted to hear Thelonious Monk play’.”[11]”

Here’s a list of some of the things that Harry Smith collected throughout his lifetime:

Songs
Films
Knots
Paper Airplanes
Books
Pop-up books
Beaded costumes
Things shaped like other things, including spoons shaped like ducks, banks shaped like apples, and anything shaped like a hamburger.

And lots lots lots more.

So Harry Smith: thank you for teaching me how to obsess, how to look just below the surface of the world in front of me. Thanks for teaching me to consider and reconsider facts and their contexts in ways that might allow me to see things that other people might not.

Grant Achatz - Persevere

An obsession that I have been indulging for the last decade or so in earnest has been food and cooking. Because I have a tendency to go “all in” when trying to understand someone’s work, I will often read whatever I can about a Chef or restaurant when I find it interesting.

When I first started hearing about Alinea in Chicago, a restaurant that, by the way, I’ve never been to, I was immediately intrigued. I had experienced some of the type of food that the chef, Grant Achatz, was meant to be serving there: progressive or experimental cuisine, molecular gastronomy, whatever you want to call it.

Achatz was experimenting with ingredients and presentations in a way that seemed to elevate the gimmickry which a lot of this type of food is prone to. Alinea was a serious restaurant, with a serious chef, who just so happened to have a signature desert preparation, shown here, that involves preparing the desert course at the end of a tasting menu directly on the diner’s table. Achatz then implores diners to scrape every last bit directly off the table with provided spoons.

Two years into the life of Alinea, after the restaurant and the chef had garnered multiple awards, including the best restaurant in the country in more than one publication, Achatz announced that he had been diagnosed with stage 4 cancer of the mouth. He discovered that he had this cancer because of a pain in his tongue.

The intense agony and irony of a world-famous, finally renowned chef getting cancer of the tongue, threatening his sense of taste itself, is an almost unbelievably scary story. Achatz underwent aggressive treatment and was able to recover completely from the cancer, but not before running Alinea as normal, but without the ability to taste at all.

This story is captured in a book called “Life, On The Line” which Achatz wrote with his collaborator and business partner, and is a highly recommended, short non-fiction depiction of a team of individuals thrown into crisis and emerging a success.

He relays an amazing anecdote in the book, one that I think is applicable to almost anyone who spends time working with other people. After learning that he wouldn’t be able to taste properly, at least for some period of time, Achatz realized that he didn’t have as much to worry about as he feared. He was surrounded by an extremely talented and dedicated team who knew all of the flavors and textures that were supposed to be present in the food they were serving. He said that he knew he could relax and “let them be his tongue” as he underwent this ordeal.

Because he had collaborated and trained his colleagues so well, he knew that he could trust them.

After this nearly disastrous bout with a deadly form of cancer, Achatz went on to earn three Michelin stars two years in a row at Alinea, which is remarkable for a restaurant in Chicago, not typically known for being a dining destination previously.

So thank you, Grant Achatz, for reminding me that perseverence is critical to success, that my bad days are probably not so bad, and that if I work hard and trust those around me, I can achieve way more than I could ever hope to on my own.

Nancy Lynch - Formalize

The last three friends I have to introduce you to are all people that I’ve met relatively recently. I’ll start out with Nancy Lynch, someone whose work I got to know very well a few years back when I was preparing a research-heavy talk about distributed systems.

To give you a sense of the depth and breadth of Lynch’s work appropriately would be a difficult thing to do in the time allotted, but I thought this image, which is a screen shot from Lynch’s homepage, would sum up at least the magnitude of her contributions quite nicely.

You can see here that Lynch has been consistently publishing work since the early 70s. What you can’t see here is how many different areas of interest she has had. Lynch is very well known for her work with Fischer and Patterson in the “Impossibility of distributed consensus with one faulty process” (colloquially known as the ‘FLP Result’), and her work with Seth Gilbert, where they turned Eric Brewer’s CAP conjecture into the CAP theorem by successfully formalizing Brewer’s notions regarding communication in distributed systems.

My favorite Nancy Lynch paper, though (not that I have read them all by a long shot) is 1989’s “A Hundred Impossibility Proofs For Distributed Computing.” In this paper she describes the phenomenon of published “impossibility results,” which are essentially proofs where instead of proving that something can be done, you prove that something cannot be done. Roughly speaking. Very roughly speaking. Very, very roughly speaking.

The paper is very entertaining, ranging from head scratchingly dense and challenging to very funny and light hearted. I love, too, how the paper (which is a talk transcript) ends:

“I’ve tried in this talk to give you a good picture of the history, status and flavor of research in impossibility proofs for distributed computing. I hope you’re convinced that it is an interesting and fruitful area for research. Now with some luck, skill and inspiration, we can continue to make great strides, proving more and more things to be impossible!”

When I met Nancy Lynch I admired from afar her ability to put down on paper and formalize what seemed, forgive me, impossible to formalize. Her work is a reminder of the necessary rigor that underlies much of our day to day work. Many of the decisions that we make as software engineers are attributable to the work that Lynch and her colleagues have been hammering away at for decades.

It’s very common for people who studied software engineering in university to do the most formal work they ever do while they are still students. Once you get a job in the industry, it seems, keeping up with the latest trends replaces your ability to continue with the formal, research oriented aspect of your work. The reason I wanted to introduce Nancy Lynch to those of you who don’t know her is for this reason - I think that this is a mistake.

While your muscles for dense, deep exploration still exist, please flex them. Where they are currently weak, please build them up. The world of software, that is to say, the entire world itself, needs more people who know what it means to provide rigorous proof that something is or is not.

Beyond that, I think that these skills are only going to become more important as there are more opportunities to apply formal techniques in every day software engineering practices.

So thank you, Dr. Lynch, for your amazing rigor, your hard work, and your continued explorations of the invisible worlds of digital communication.

bell hooks - Empathize

bell hooks is one friend that I have to say that I wish I had met earlier. It was only 2 years ago that I first read her work - after it had been recommended to me a number of times by a number of people I admired, I decided to give it a shot.

After ordering it online, the small book arrived and I read the entire thing in a few hours. It impressed me so much when I read it that I immediately got in touch with the publisher, purchased 10 extra copies, and then mailed them to random strangers on the internet who requested a copy from me via Twitter.

What the book is about is right there in the title - “Feminism is For Everybody.” hooks, a feminist scholar, author, and activist, collects some short essays she’s written that are meant to describe and lay out what Feminist politics means for her. In the introduction, she powerfully describes the typical interactions she has when people discover that she is a cultural critic, a radical feminist.

The word “feminism,” hooks describes, has gotten so lost and tangled in the capitalist landscape that defines modern American society in particular, that people don’t even know what it’s about anymore. They think that feminism is about “angry women who want to be men,” instead of what it’s really about. “It’s about rights,” hooks says,” equal rights for women.”

hooks delves into a number of topics in the book, including the damage that the patriarchy imposes on people who identify as men. This nuance was new for me - I hadn’t in the past spent much time thinking about the damage that macho posing and predatory teaching does to young boys who grow up into men. hooks has a way of stating in plain terms how her experiences have impacted her, how a patriarchal system really hurts us all, holds us all back, denies us the love that we are due.

So thank you, bell hooks, for teaching me to empathize more deeply with girls and women in their struggle for equality. Thank you for holding no punches, for telling it like it is, and for giving me the courage to stand here and relay these lessons to this group of students.

We all have a lot to learn about the roles that we play in the structures of power that help to shape our lives. As software engineers entering into a field that is rife with prejudice and discrimination, it is your job to ensure that you do your part to dismantle as many of these structures of power as possible.

hooks can teach you a lot about not standing idly by while the world continues to churn in a seemingly endless downward spiral. hooks can teach you a lot about yourself, and make you a better person. So please, read her work, and share it.

Michele Alexander - See through the system

Six months or so ago I was having a conversation with a friend who is well versed in the literature about race, economics, and American History. After hearing some of the texts I was looking into along these lines, he recommended that I read a book that had a lot of buzz around it at the time - “The New Jim Crow” by Michele Alexander.

The central thesis of “The New Jim Crow” is that the incarceration state and its current stronghold in the United States mirrors the post slavery laws, known as “Jim Crow Laws,” which effectively dismantled the rights of freed slaves to the point where it seemed no real progress had been made at all.

In a very precise, systematic, and deeply sad way, Alexander outlines how the combination of discretion in policing, judging, and incarcerating Americans has created deeply troubling patterns for young people of color all over the United States, from coast to coast.

In a world which is supposedly “post-racial,” Alexander shows that the effect of whitewashing hateful rhetoric (which is making an unfortunate resurgence thanks to certain Republican presidential hopefuls with terrible hair whose name I won’t mention) is not what people think. Just because the rhetoric has become less overtly racist, Alexander argues, means nothing when it comes to the lives and times of young people of color, particularly those in major cities.

Alexander reminds us all to not fall into the trap of forgetting that many, many people in the United States suffer from being born into a situation where so many cards are stacked against them that it not only seems impossible for them to rise out of it, it often times is impossible.

The lack of social mobility in America’s poorest cities, the concentration of police, the movement of jails outside of cities to rural areas where local counties can profit off of them, the terrible schools, the non-existant or underpaying jobs. All of this, every bit of it, is part and parcel with America in the 21st century.

So thank you, Ms. Alexander, for reminding me to not become complacent. For reminding me that many, many people have almost no opportunities to thrive, and yet somehow some manage to, against all odds.

I want you all to look into Alexander’s work and contemplate how you might impact your society with the work you do. And before you go down that road - no, I don’t think that an iPhone app is going to help you solve systematic racism.

Only a lifetime of dedication to justice can bring real change. Who of you will pick up that mantle?

Conclusion

So there you have it. 10 lessons from 11 friends.

Improvise
Question everything
Redefine beauty
Hone your aesthetics
Use your real voice
Obsess
Persevere
Formalize
Empathize
See through the system

And two final provocations:

First:

“The best thing you can do to be an amazing software engineer is spend some time doing something besides software engineering.”

Second:

“The opportunity is yours to help people live the lives that they deserve to live.”

Thank you for hanging out with me for an hour.

I hope some of my friends will soon become mutual friends to all of us.

What I Am About

Sun, 13 Sep 2015 00:00:00 +0000

# #

What I Am About

You don’t know me. I mean, you might actually know me, but chances are, you don’t. Which is great. That’s kinda what this whole thing is about. If you want to know me, you should know what I am about. What makes me tick. What motivates me.

Today’s a Sunday. I’m traveling in the opposite direction of my wife and kids to be with some other family and then to spend a few days apart from any family doing some work. When I’m moving in this direction, I tend to get reflective. I’ve wanted to write this post for a while, but then 3 things happened in two days that reminded me again, so here we are. Here’s what I am about:

Stopped at a traffic light yesterday, I noticed a telephone pole with 20-30 layers of the same advertisement stapled to it. One after another, the same ad. Torn, replaced, stapled, torn, replaced, stapled. Over and over again. You could see the worn wood of the telephone pole through the center of the sinkhole of yellowing paper ads. I chuckled to myself, thinking how beautiful it looked. I looked over at my wife to tell her about it, and she was starting at it too. She noticed it at the same time as me, and also found this totally strange and mundane thing beautiful.
My daughter had her first slumber party at our house with a friend who used to be her classmate. She’s five years old. While she and her friend were playing, I walked into my room and noticed that my three year old son was sitting on my couch, looking out the window. I laid on the couch next to him, he snuggled into my arms, and we stared at the gentle breeze moving the trees for ten minutes. We told each other what we liked about it.
Recently, my wife and I were fortunate enough to buy our first home. It’s a big and strange house, over 100 years old, mysterious and graceful in its way. It has multiple screened-in porches. The front yard gets a lot of sun. As we’ve settled in, my wife and I have been planning how to landscape around the house - it could use some work. Talking to a friend in the neighborhood who had recently done something similar, they were frustrated to find out that growing a decent hedge takes about five years. That amount of time sounded just about perfect to me.

I think you know me a little bit better now, or at least you know what I think I’m about, at 35 years old, on September 13, 2015.

Thanks for reading.

Propositions As Types: Papers We Love 2015

Thu, 03 Sep 2015 00:00:00 +0000

# #

Papers We Love NYC: Propositions As Types

I was honored to return to the NYC chapter of Papers We Love to discuss Philip Wadler’s paper “Propositions as Types.” See the PWL crew’s round up page about the event here and see below for slides and video.

Live Every Time

Thu, 03 Sep 2015 00:00:00 +0000

# # Note: this was originally published in May, 2007 on the Volcanic Tongue website. Volcanic Tongue was a record distributor and label run by my friends Heather Leigh and David Keenan out of Glasgow, Scotland. Volcanic Tongue is over and I wanted to document some of my ‘other’ writing on my blog, so I’m re-posting this here. Below is the original introduction written by David, followed by the article itself. Photo by Chris Gray, originally published with the article as well.

“Live Every Time” By Mike Bernstein

Mike Bernstein is a central cog in the Northeastern USA art/noise scene. As a member of Double Leopards he brought a sophisticated feel for primitively re-wired electronics and hi-jacked music software to the table, lending their already fathoms-deep sound a couple of miles more boom. As a member of Religious Knives alongside his partner Maya Miller and Nate Nelson from Mouthus he has convincingly formulated new approaches to rock song that would incorporate the most advanced noise syntax. As co-owner of the Heavy Tapes imprint he has been at the forefront of the analog renaissance and has set new standards for homemade art documentation, an area of interest that he has further expanded upon via his curatorship of the current Leaderless: Underground Cassette Culture Now exhibition at Printed Matter in New York City. We’re pleased to welcome him as a Volcanic Tongue columnist with a very personal piece on Bootleg Culture.

It started at a head-shop/record store next town over called “Prime Cuts” that my Safta (Grandma to you) misheard as “Chaim Putz” when we asked her to take us there. She waited outside in the car while my brother and I went in to spend some of that beautiful grease that we always got floated when Safta was around.

I wandered around and checked out CDs in long boxes, some dusty records, and some truly righteous t-shirts while my brother looked through a binder, filled out forms, and waited around for…a cassette? Turns out he was picking live Grateful Dead sets out of a list of chronologically arranged recorded shows that were available for something like $2.00 each and copied onto Maxell Chrome c90 tapes.

He would pick the date, they would copy it for him in the back, use a pricing gun to mark the date on the tape (it doesn’t cost $1,210.69, it’s a show from 12/10/69) and then he would copy the set list out of the book onto a J-Card of your choice (Calvin & Hobbes, Dancing Bears, pictures of the boys, etc.) and he would walk out with a whole new show he’d never heard before.

I thought the band was weird for supporting the tapers, got generally mystified by the whole experience, and then started to buy my own pieces. It was an easy bike ride from my house. I’d pick based on the song titles, even if I didn’t know what they sounded like or were about. Simple things intrigued me – “Beat it on down the line” was always abbreviated as “BIODTL” and I was very intrigued on what a “China Cat” and a “Sunflower” had to do with each other and how a –> could really connect them. What kind of connection was that?

I learned how to listen to a band live by listening to these tapes. They’re always live; live every time. Your walkman is a time machine and so is your turntable. I loved the applause, the stage banter, and of course, the songs. Some of the tapes were recorded so well that the transparency was almost frightening. Others are noisier, rife with conversation, exploding with applause, shock, and amusement.

I wasn’t deep enough to make it through a Dark Star –> Drums –> Space 45 minute side yet, but the depths of the tunes, the great ones, really got to me. 15 years later I’m still obsessed with collecting live shows. I’ve never been to any of these gigs, but I’ve read about them, studied pictures of them, and pretended I was there.

Some bands and the rare solo performer just transcend “the gig,” but it’s a mixture of complex emotions that draws fans to pursue these recordings. It’s more than just collecting. It’s the closest thing we have to musical transubstantiation. The bootlegs for some bands, like the Dead, are popular because the band played so often, and recorded so little in the studio. Others are so guarded with their lives or ideas that people can’t help but be curious.

What do they reveal when you can get a real sense of the environment when they’re playing? In my opinion this is why Dylan’s “Great White Wonder” was the first popular rock bootleg – because the dude was so inscrutable! You know that if people were camping out outside his house that they were collecting his live shows. Dylan might have considered both of those things trash at the time, but his live work still continues to astound. His shiniest moments, including the epic tour with The Hawks in 1965-66, the Basement Tapes work (kind of a bootleg in this sense, but still private and mouldy as anything in the basement of Safta’s house in Queens), and the unbelievably rich Rolling Thunder tour from 1975-76 are rife with timeless classic moments of pathos, joy, and musical discovery.

Live! As many times as you can handle it.

Neil Young is another great example of an oft-bootlegged performer who was free and easy in so many contexts that his fans needed (and still do need) some crib sheets to get ready for the next time around, even if they ended up being useless half of the time. Neil is famous for being justifiably but perhaps excessively controlling about his recorded output, and the examples of his live performances on unauthorized LPs are rare but very powerful treasures – He makes jokes! The audience cracks up! He’s up there alone, with a piano and a harp around his neck, wearing white jeans, a white t-shirt, and suspenders. His hair is short, his eyes are red. He’s alone.

Part of collecting bootleg LPs is not only getting closer to the band, but getting closer to the fans. Deadheads, for instance, made awesome looking “fan club” and “giveaway” LPs of live shows which heavily feature crudely drawn skulls, scrawled wasted aphorisms, and no dearth of grinning bears. Neil Young fanatics crudely and sometimes disgustingly hid their obsessive releases with garish covers of tropical getaway scenes and red-lipped vaudevillians. Don’t ask.

The shows that they cared about end up being the shows that you care about. Some call it piracy, theft, greed, whatever. What is it that draws someone to release a live show on LP? I don’t care, but the results are the real deal – re-contextualized pictures of your heroes. Cheap.

The tops for me, live, are the obvious ones…I’m easy. The Grateful Dead, Velvet Underground, Neil Young, Bob Dylan, Quicksilver Messenger Service, and the Rolling Stones. Not to mention Throbbing Gristle, The Fall, Joy Division and Sonic Youth, Black Flag, Bad Brains, and also Jefferson Airplane, Trad Gras Och Stenar, and man, Coltrane, Miles Davis, and recently Wooden Wand, Hair Police, Aaron Dilloway, and Vibracathedral Orchestra, amongst others.

But the best live, on wax, are the Dead, the Velvets, and Neil. No doubt about it in my mind and I have stacks of horribly packaged, poorly pressed and unevenly recorded documents amongst the few princely examples of high quality recordings (packaged “simply” but “effectively” – that triple LP set from Germany, for instance, or the official looking Bottom Line joint) – to prove it.

There’s certain kinds of record shops that you can just tell will have good bootlegs as soon as you walk in. There was that one in Brussels where the ROCK section was so swelled with live shows that I ran out of time halfway through the H’s. In NYC there’s Subterranean, which has Television and Velvets posters all over the walls and has as many live albums as studio albums for most of the bands listed above.

The dusty ones will almost always set you up – by now I’ve trained my senses and my spine tingles when I cross the threshold, usually onto a wooden plank floor and often to the chagrin of the shopkeeper who is all but passed out in a rocking chair. Mooncurser, we’ll miss you.

I’m not shy about wishing I was there. It’s a well documented fact, personally bolstered by my own (admittedly wimpy in comparison) experiences “on the road” and “in the studio,” that you take it to a whole new place when you have to play it live every night. When you’re trying to make it happen like that, on that scale, it usually does.

I like sections of my record collection to have some real depth. The breadth of any collection should be wide within its given niche, but it’s the depth, for me, that has proven most evocative. Having the studio albums, the extra tracks, the singles, and live shows all next to each other and available allows you to see the cubist view of the artist. It’s an almost profane desire that we have to understand as much as we can about the artist whose words and sounds impress us so deeply.

Lately though, in the past few years I would say, I have been obsessed on a new level – beyond imagining myself in Wembley Empire Pool in 1972 with gloves on because it was so cold, watching the Dead’s first and second shows in their Europe ‘72 tour – one that frightens me but compels me equally.

I started to download the shows from the above performers, which are posted all over the internet for free in uncompressed formats which sound amazing. All of the people who used to trade tapes now trade CD-Rs and download on both public and private networks. Everything is digital which means everything can be shared. And shared it is!

Some of the resonance is missing, and I still prefer vinyl when I can get it, but this kind of trading has made the disparate pieces fit together that much more easily. I have found clarity in a lot of these recordings which are passed around this way – tapes or LPs that I have of shows from the 60s and 70s have been submitted to the virtual meat-grinder and emerge spotless!

While LPs can often be made from whichever recordings the label could get their hands on, only the highest quality recordings are traded in these networks. This doesn’t mean you can’t hear the applause and laughter, it just means that no songs are skipped, nothing is edited to fit the side of an LP, and you can generally hear all of the instruments and a good strong vocal. The Dead took their live mixes very seriously. They knew it was how the real fans heard them, in a concert hall.

Anyone can download this stuff and I’m posting some links to where it can be gotten from, along with a list of favourite bootleg LPs, easier to get CDs, and some reading material. Be sure to check mp3 blogs as well for this material – they have flourished as a fantastic resource to anyone obsessed enough to care.

Bootleg LPs

Quicksilver Messenger Service “Maiden of the Cancer Moon” (contact me if you’re selling this)
Grateful Dead “Rampens Revenge,” “More Cosmik Messages,” “Mountains of the Moon”
Bob Dylan “Great White Wonder,” “The Complete Basement Tapes”
Velvet Underground “and so on…,” “Live 1966,” “Sweet Sister Ray”
Neil Young “Live at the Bottom of the Hill,” “Live in San Bernadino”

Meeting J-Bob: Notes on 'The Little Prover' Chapters 1-5

Wed, 12 Aug 2015 00:00:00 +0000

# #

Meeting J-Bob: Notes on “The Little Prover” Chapters 1-5

Daniel P. Friedman has authored or co-authored an alarming number of Computer Science books that Programming Language enthusiasts would consider classics. Among them are a few members of the series of “Little” books, which started in the 70s with “The Little LISPer,” and continues 40 years later, with 2015’s “The Little Prover,” co-authored with Carl Eastlund.

All of the “Little” books follow a similar structure and embrace a similar pedagogy: concepts are taught through a series of questions and answers side by side, presented in “frames” separated by lines. I have studied some of the other books before, particularly “The Reasoned Schemer” (which doesn’t have the word “Little” in it but is still a part of the ‘series’), but have never really clicked with how the information is presented.

Probably because I had a good reason to feel that I needed to understand what they were presenting, the technique finally worked for me with “The Little Prover.” Reading the pages over and over again I got into a rhythm and started to absorb the information. Another thing that was blocking me before was a lack of intuitive understanding of Scheme (pairs and lists everywhere confused me, cons and car and cdr irked me). For some reason, that vanished for me too when I started with this book, so yay! It only took me around 8 years to be able to digest Scheme properly. Not too shabby, mrb, not too shabby.

Why was I so compelled to grok this stuff? I committed myself to presenting about Phillip Wadler’s “Propositions as Types” at the New York City chapter of Papers We Love, and I wanted to have a concrete, interactive experience to go along with the theory and mechanics I was trying to grasp from the paper. “The Little Prover” coming out around the same time as I was preparing seemed serendipitous - it describes a theorem prover written in Scheme and shows how theorem proving and writing Scheme code can actually be the same thing.

To put a finer point it, I love this book and highly recommend it. It has lots of lessons to teach, which aren’t limited to theorem provers. What the authors can do with a ‘simple’ system written in Scheme is phenomenal and very deep, and importantly, challenging concepts are made fun and easy to access given enough time and patience. I found pieces of some of my favorite aspects of philosophy, math, and Computer Science in the book, and think you can make connections to:

Compilers
Static Analysis
Logic
Language and truth (what is like, real…man?)
Kernel languages
Small things
Breakfast

And I’m pretty sure if you’re reading this, you like at least some of those things.

I decided to write down some notes and musings from each of the 10 chapters in the book in order to bolster my understanding, provide an artifact of my learning in case anything becomes fuzzy to me later (which it certainly will, given historical evidence), and to prove to myself that I can explain this stuff in the context of Wadler’s paper. This post covers the first 5 chapters. For each one, I’ve written down what the “Big ideas” were for me - what resonated, what I think the authors were after, and what I could take away. Here we go!

Note: There is some Scheme code below that is illustrative, and some that is meant to be executed by the J-Bob proof assistant, the design and implementation of which is one of the central focuses of the book. Understanding which is which is a little bit confusing, but check out The J-Bob Source Code on GitHub for some clues.

Chapter 1 - Old Games, New Rules

Big ideas:

We can know cool things about a program without having to run it
Rewriting programs according to rules is powerful and pervasive
You can prove theorems by writing code
Holy shit I finally understand Scheme

The Scheme code (car (cons 'ham '(eggs))) is equal to to 'ham.

Each of the following lines of Scheme code are also equal to 'ham:

(car (cons 'ham '(cheese)))
(car (cdr (cons 'eggs '(ham))))
(car (cons (car '(ham')) '(eggs')))

The Scheme code (car (cons 'ham '(eggs))) is equal to the value 'ham. It is not equal to any other value.

What value is the Scheme code (cons a b) equal to? We don’t know what a and b are, so we can’t know the value of (cons a b).

An atom in Scheme is defined as anything that is not a pair, and a pair is something that is produced with cons. This should give you the idea:

(atom? 1) => 't
(atom? 'nil) => 't
(atom? '()) => 't
(atom? '(1)) => 'nil
(atom? '(1 2)) => 'nil

Play around in your friendly Scheme REPL if you feel like testing more expressions. I’ve certainly spent long enough doing so, I wouldn’t want to begrudge you the fun!

So, back to the question “What is the value of (atom? (cons a b))?”

Even though we don’t know what a and b are, we know the value of (atom? (cons a b)) - why?

Because we know the definition of atom?, which returns 'nil for anything that is the result of a cons operation.

So the value of (atom? (cons a b)) is 'nil, regardless of the values of a and b. This is cool, and reminds me of static analysis.

Does the expression (equal? 'flapjack (atom? (cons a b))) have a value?

If we know that (atom? (cons a b)) is 'nil, does that help us? It does! We can substitute 'nil for any instance of (atom? (cons a b)) that we come across. That seems useful.

So (equal? 'flapjack (atom? (cons a b))) becomes (equal? 'flapjack 'nil) which is 'nil.

The big idea here is that:

We can know interesting things about a program without executing it.

The authors are also introducing the idea of a focus within a piece of code that is being transformed/evaluated in this strange new way.

Our attention is drawn to the fact that this transformation happens in discrete steps and rewriting is applied to specific, addressable subexpressions of the term at hand. To follow these steps:

Starting from the code (equal? 'flapjack (atom? (cons a b))), the focus is (atom? (cons a b)), which we can rewrite to 'nil.
After the substitution occurs, the focus is on the whole expression, which is (equal? 'flapjack 'nil). We know this is equal to 'nil.

So two steps and two focuses later, we have reduced the original expression to a value without having to know everything about that program, e.g. the values of a and b.

We relied on certain known facts in order to execute the above transformation. These facts are called axioms.

“An axiom is a basic assumption that is presumed to be true.”

Here are some axioms:

(atom? (cons x y)) is always equal to 'nil.
(car (cons x y)) is always equal to x.
(cdr (cons x y)) is always equal to y.

Stated in code, using the dethm construct:

(dethm atom/cons (x y)
  (equal? (atom (cons x y)) 'nil))

When axioms are built up to prove things, that’s called a theorem:

“A theorem is an expression that is always true.”

And to put a finer point on it:

“Axioms are theorems that are assumed to be true, whereas other theorems must be shown to be true.”

Here are the steps of rewriting the following expression until it becomes 't:

(car
  (cons (equal (cons x y) (cons x y))
    '(and crumpets)))

We notice that (equal (cons x y) (cons x y)) will always be 't, regardless of what x and y are. So, we can replace it with 't:

(car
  (cons 't
    '(and crumpets)))

Likewise, the static values 't and '(and crumpets) can be collapsed using cons. Finally:

(car '('t and crumpets))

The car of '('t and crumpets) will always be 't. So we arrive at:

't

Without executing the code, we know that it will always evaluate to 't.

The Law of Dethm (initial):

“For any theorem (dethm name (x1...xn) body-x), the variables x1...xn in body-x can be replaced with any corresponding expressions e1...en. The result, body-e, can be used to rewrite a focus p to become q provided body-e is either (equal? p q) or (equal? q p).”

(atom? (car (cons (car a) (cdr b))))

e1 -> (car a)
e2 -> (cdr b)

body-e -> (equal (car (cons (car a) (cdr b))) (car a))

p -> (car (cons (car a) (cdr b)))
q -> (car a)

(atom? (car a))

That’s what it looks like to apply the Law of Dethm to some functions. We’re also told that J-Bob, an assistant in the form of a computer program, can help us automatically do some of these transformations.

Chapter 2 - Even Older Games

Big ideas:

Theorems in the system are written according to the Law of Dethm
Using an abstract example to understand a deep principle is helpful - see The Jabberwocky
We will be able to prove conjectures using a library of axioms

What is (car (cons a b)) in this code equal to?

(if (car (cons a b))
  c
  c)

a, since the car of (cons a b) is always a. This is the result:

(if a
  c
  c)

We know we can do that because of the axiom of car/cons. What is the expression above equal to?

The result is c, regardless of a. Chapter 2 introduces axions around if to help us rewrite if code the same way that we rewrote with car/cons above.

The axioms of If: if-true, if-false, if-same:

(dethm if-true (x y)
  (equal (if 't x y) x))
  
(dethm if-false (x y)
  (equal (if 'nil x y) y))
  
(dethm if-same (x y)
  (equal (if x y y) y))

These axioms each represent very simple ideas that turn out to have very powerful implications.

Every if expression has three parts: (if Q A E): The Question, the Answer, and the Else.

The axioms of Equal: equal-same, equal-swap, equal-if.

The Law of Dethm (final)

For any theorem (dethm name (x1...xn) body-x):
The variables x1...xn in body-x can be replaced with any corresponding expressions e1...en.
The result, body-e, can be used to rewrite a focus as follows:
1. body-e must contain the conclusion
2. The conclusion must not be found:
3. in the Q of any if
4. in the argument of any function application
5. if the conclusion can be found in an if A or E, then the focus must be found in an if A or E with the same Q.

Explaining the “Law of Dethm” with the Jabberwocky example:

“Sometimes our intuitions get in the way.”

The if-nest axioms are amazing:

(dethm if-nest-A (x y z)
  (if x (equal (if x y z) y) 't))
  
(dethm if-nest-E (x y z)
  (if x (equal (if x y z) z) 't))

Reminds me of the first time I realized that statically typing terms of certain dynamic languages was possible. Consider the following snippet:

function fooTester (foo){
  if (foo > 3){
    return "cool";
  } else {
      return "uncool";
  }
}

Think about it long enough and you can see that there are both assumptions that can be made about the types in this function, and also some things you know. An assumption is that foo is a number. We test it against 3 using the > operator, which we know operates on numbers.

Something we know for sure is that this function returns a String. Regardless of the value of foo, this if expression returns a String.

Start to stack these assumptions up, dream of an interactive tool that can tell you the types of these expressions without having to know them, and throw in a type checker that can assert what you annotate, and you can get a sense of a pathway from dynamic to statically typed programming.

In this chapter we also get The Axioms of Cons, atom/cons, car/cons, cdr/cons:

(dethm atom/cons (x y)
  (equal (atom (cons x y)) 'nil))

(dethm car/cons (x y)
  (equal (car (cons x y)) x))

(dethm cdr/cons (x y)
  (equal (cdr (cons x y)) y)

More intuitive but not intuitive facts about list construction in Scheme than you knew were useful. This chapter really nails the fluid, bizarre methods of applying rules and rewriting code that goes into creating mechanized proofs on a computer using a programming language.

Through applying some sensical and some nonsensical techniques, the authors illustrate that like programming, you can gain an intuition for the idioms of theorem proving through repetition and deep understanding. Write and read enough programs and you’ll learn how to write programs properly. Write and read enough proofs, it seems, and the same will happen. An interesting perspective on the correspondences that make the ideas in this book possible.

Chapter 3 - What’s in a Name?

Big ideas:

A claim is an unproven theorem
J-Bob can help you prove that your claims are theorems
J-Bob has a language for describing how to rewrite code

The value of the Scheme code (pair 'sharp 'cheddar) is '(sharp cheddar).

The value of (first-of (pair 'sharp 'cheddar)) is 'sharp.

The value of (second-of (pair 'sharp 'cheddar)) is 'cheddar.

The function pair is written like this:

(define (pair x y)
  (cons x (cons y '())))

And here’s first-of and second-of:

(define (first-of x)
  (car x))
  
(define (second-of y)
  (car (cdr x)))

Here’s a claim about first-of:

(dethm first-of-pair (a b)
  (equal (first-of (pair a b) a))))

“A claim is an as-yet unproven theorem.”

We can make a claim into a theorem! By proving it!

“A proof is a sequence of rewriting steps that ends in 't. If we can rewrite a claim, step by step, to 't, then that claim is a theorem.”

(pair x y) -> (cons x (cons y '()))

(pair a b) -> (cons a (cons b '()))

Defun: Given the non-recursive function (defun name (x1...xn) body), (name e1...en) = body where x1 is e1,...xn is en.

In other words, for a non-recursive function, you can replace the function call with the function definition, substituting the arguments from the definition with those of the function you’re proving.

(equal (first-of (pair a b) a))
(equal (first-of (cons a (cons b '()))) a)
(equal (car (cons a (cons b '()))) a)
(equal a a)
('t)

So first-of-pair is a theorem, by Law of Defun, car/cons, and equal-same.

Is second-of-pair a theorem?

(dethm second-of-pair (a b)
  (equal (second-of (pair a b)) b)

Yes, by car/cons, cdr/cons, equal-same, second-of, and pair.

For the first three chapters, the authors cleverly use colors in the printed text to show the “focus” of an expression as it is being rewritten.

When you are using the J-Bob prover that the book introduces and shows you how to use and implement, you describe these focuses using a query language of sorts describing the “path” to the part of the expression that needs to be rewritten.

This works because all code is a list. “Code as data” in the Lisp tradition has a lot of awesome applications and the authors of this book, one of whom is easily one of the deepest functional programming thinkers of all time, leverage this property to the fullest in this text.

“If we can find a proof one way, we can always find it in another. If the second way goes wrong, we can ‘back up’ to where we started and do it the first way again. But some approaches will find a proof faster than others.”

Let’s look at in-pair?. What does it do?

(defun in-pair? (xs)
  (if (equal (first-of xs) '?)
    't
    (equal (second-of xs) '?)))

It determines whether the two-element list xs contains '?. Let’s try to prove this claim!

(dethm in-first-of-pair (b)
  (equal (in-pair? (pair '? b)) 't))

Is in-first-of-pair a theorem? Yes! By car/cons, equal-same, if-true, pair, in-pair? and first-of.

“Skip irrelevant examples: Rewriting a claim to ‘t does not have to go in any particular order.”

This chapter shows us the mechanics of using code to prove theorems, which are expressed as code. These proofs happen by rewriting code using pre-defined rules to begin with a conjecture and then end with 't, which completes the proof.

Chapter 4 - Part of This Total Breakfast

Big ideas:

J-Bob can be used to prove the totality of functions, recursive or not
Proving totality of functions with natural recursion uses J-Bob’s size function, and is a neat way to prove totality
Partial functions are problematic because they can be used to introduce inconsistencies into a system

We meet a new function - list0?. Here’s how it works:

(list0? 'oatmeal) => 'nil
(list0? '()) => 't
(list0? '(toast)) => 'nil

What does list0? do? It looks like it tells you if something is an empty list. Here is the definition:

(defun list0? (x)
  (equal x '()))

Is list0? total? What does total mean?

“list0? is total means that no matter what value v is passed to list0?, the expression (list0? v) has a value.”

In other words a function f composed of functions x1...xn and if statements if1...ifn can be total if all of the outcomes of if1...ifn produce a value and x1...xn are all total.

We learn that, somewhat surprisingly, all built in operators are total. When we’re trying to determine if a function is total, we know that all functions used in a function must be total for a function to be total. For example:

(defun list1? (x)
  (if (atom x)
  'nil
  (list0? (cdr x))))

Is total because atom, list0?, and cdr are all total, and the if statements A and E produce a value. Now that we know what totality is, we can learn about the final Law of Defun:

“Given the total function (defun name (x1…xn) body), (name e1…en) = body where x1 is e1, …, xn is en.”

In other words, we can replace a function’s use with it’s body (once variable substitutions are performed) if the function is total. Why does it matter if it is total? If it isn’t, a function can be used to prove a contradiction. Let’s take a look.

We go through the process of trying to prove list0?, list1?, list2?, list3?, etc. as total. There must be an easier way, and it sounds like a job for recursion. Can we use recursion like that? What would that look like? How about like this:

(defun list? (x)
  (if (atom x)
    (equal x '())
    (list? (cdr x)))))

The list? function produces 't if its argument is a list, otherwise it produces 'nil. So, is list? total?

It looks like in order to know if list? is total, we need to know if list? is total, because the function is recursive. So what can we know about how list? is called from within list??

We see that list? is always called with (cdr x) as its argument. That means that by definition, calls to list? from within list? always have fewer conses than the function started with.

The way that we use this fact to prove that list? is total is with a mechanism known as the measure of a function.

“The measure is an expression that is included with a function definition. It may only refer to previously defined, total functions and to the function definition’s formal arguments. The measure must produce a natural number that decreases for every recursive call to the function.”

So if we definte list? with a measure, J-Bob will know how to measure whether or not recursive calls to a function will actually cause the function to terminate and produce a value.

(defun list? (x)
  (if (atom x)
    (equal x '())
    (list? (cdr x)))))

measure: (size x)

The size function measures the size of a function’s measure. Counting cons cells, for example:

(size '((10 (A 20)) B)) => '6
(size '(10 (A 20))) => '4
(size '10) => '0

The way that we make the totality claim for list? using a measure is by expressing “If x is not an atom, than (size x (cdr x)) must be smaller than (size x).”

(if (natp (size x))
  (if (atom x)
    't
    (< (size (cdr x)) (size x)))
  'nil)

We are introduced to some Axioms of Size which give us certain truths upon which we can rest the burden of our totality claims! Handy.

(dethm natp/size (x)
  (equal (natp (size x)) 't))
  
(dethm size/car (x)
  (if (atom x)
      't
        (equal (< (size (car x)) (size x)) 't)))

(dethm size/cdr (x)
  (if (atom x)
        't
        (equal (< (size (cdr x)) (size x)) 't)))

So using these rules and the rules we’ve seen before, we can indeed reduce the totality claim above to 't. Here’s how it looks using J-Bob:

(defun defun.list? ()
  (J-Bob/define (defun.list2?)
    '(((defun list? (x)
         (if (atom x) (equal x '()) (list? (cdr x))))
       (size x)
       ((Q) (natp/size x))
       (() (if-true (if (atom x) 't (< (size (cdr x)) (size x))) 'nil))
       ((E) (size/cdr x))
       (() (if-same (atom x) 't))))))

The totality claim for partial is as follows:

(if (the (size x))
  (if (< (size x) (size x))
  't
  'nil)
  'nil)

Which can be reduced to:

(if (< (size x) (size x))
  't
  'nil)

Which can’t go any further - certainly (< (size x) (size x)) could never be true or reconciled within a consistent system. That’s why partial is partial, and why totality matters - because partial functions can introduce inconsistencies into our system.

Chapter 5 - Think it Over and Over and Over

Big ideas:

There are techniques you can apply to help produce efficient, straightforward proofs
Wow proofs really are a lot like programs, aren’t they? Wait, which are we writing again?
This really reminds me a lot of static analysis

We start out looking at memb?, a function which looks for '? in a list, and remb, which removes the first instance of '? from a list. Are they total? Let’s look at memb? with a measure:

(defun memb? (xs)
  (if (atom xs)
    'nil
    (if (equal (car xs) '?)
      't
      (memb? (cdr cs)))))

measure: (size xs)

You can determine if memb? is total by proving its totality claim:

(if (natp (size xs))
  (if (atom xs)
    't
    (if (equal (car xs) '?)
      't
      (< (size (cdr xs)) (size xs))))
  'nil)

It is total, and so is remb, so we should be able to build theorems out of these functions. Which is exactly what we do next, with memb?/remb0, which states that remb removes '? from a 0-element list. Here’s the dethm:

(dethm memb?/remb0 ()
  (equal (memb? (remb '())) 'nil))

We can try to prove this using the axioms we’ve learned so far. The bit to focus on is:

(equal (memb?
          (remb '()))
        'nil)

The first thing we should do is use the definition of remb, which we know to be total. We’ll replace the xs in the definition of remb with '().

(equal (memb?
          (if (atom '())
              '()
              (if (equal (car '()) '?)
                (remb (cdr '()))
                (cons (car '())
                  (remb (cdr '()))))))
          'nil)

What can be simplified from this? Quite a bit, it turns out. This expression becomes:

(equal (memb?
          '())
        'nil)

By atom and if-true, because '() is an atom. Now, let’s use memb?, again replacing xs with '():

(equal (if (atom '())
           'nil
           (if (equal (car '()) '?)
               't
               (memb? (cdr '())))
           'nil)

Again we see the expression atom '(), which we know is true. So the above becomes:

(equal 'nil
       'nil)

And we know that memb?/remb0 is indeed a theoreom, by atom, if-true, and equal-same. Here’s how the above explanation looks when it’s written in J-Bob:

(defun dethm.memb?/remb0 ()
  (J-Bob/define (defun.remb)
    '(((dethm memb?/remb0 ()
         (equal (memb? (remb '())) 'nil))
       nil
       ((1 1) (remb '()))
       ((1 1 Q) (atom '()))
       ((1 1)
        (if-true '()
          (if (equal (car '()) '?) (remb (cdr '())) (cons (car '()) (remb (cdr '()))))))
       ((1) (memb? '()))
       ((1 Q) (atom '()))
       ((1) (if-true 'nil (if (equal (car '()) '?) 't (memb? (cdr '())))))
       (() (equal-same 'nil))))))

This is an executable program that will prove that this function is a theorem - cool!

The authors take us through a similar procedure to prove successive functions on top of memb?/remb0 - memb?/remb1, memb?/remb2, etc. They lead up to the question that would occur to anyone paying attention to the book up until this point - couldn’t this be generalized? What techniques can we use to perform recursion like proofs? The answer is induction, and it is the big idea for the second half of the book.

While we’re shown how proving the successive functions to memb?/remb0 isn’t exactly as straightforward as you’d think, we’re given some nice insights into how to effectively write proofs with our system:

Rewrite from the Inside Out

Start with the statements “inside” the code, the things that are nested and squirreled away. Simplify them and then the entire proof will become simpler

If Lifting

We’re shown this technique which really resonated with me as a programmer, because again it reminds me of the kinds of insights you form about programs in a certainly language after writing with it for a decent amount of time. Most experienced programmers will be able to recognize certain simplifications that can be made to code that will preserve its meaning but make its intent clearer, improve performance, etc. Similarly, we’re given techniques that simplify proofs without changing their meaning. Pretty sweet.

Pull Ifs Outward

This is a piece of wisdom based on the mechanic provided above - that is, use if lifting! It can simplify what you’re trying to prove radically.

Keep Theorems in Mind

Another piece of advice from the authors which I can’t help but draw a parallel to software engineering about. Keeping theorems in mind is about recognizing the patterns in your code that can be simplified using axioms, but also about recognizing when you’re close to a pattern. When you’re close to a pattern, make some minor changes around your statements, and you might reveal that pattern. This reminds me a lot of the ways that software developers apply patterns, and look for “smells” or clues that they might be doing something wrong, or every once in a while, something right.

Two Programmers

Wed, 13 May 2015 00:00:00 +0000

# #

Two Programmers

With Apologies to Arnold Lobel

Two programmers sat at a computer with an opened text editor, writing in the only programming language they knew. It was nice. Their programs worked there. “This programming language is very nice,” said the first programmer. “But I wonder what the other programmers are using?”

“We do not know. We never will.” Said the second programmer.

One day a senior developer walked by. “Suzie, could you tell us what kind of programming languages the other programmers are using?” Asked the two programmers. Suzie thought about it for a second and said, “They are using paradigms you’ve never seen, languages of unbridled efficiency, clarity, and performance.”

The first programmer said, “All of those languages sound amazing. All of the other programmers are using them.”

“How sad.” Said the second programmer. “We cannot use them, we never will.”

The two programmers sat at their terminal. They felt sad for 100 days.

One day a junior developer walked by. “Joey, could you tell us what kind of programming languages the other programmers are using?” Asked the programmers.

Joey said quickly, without thinking, “They’re using the same languages as you!”

The first programmer said, “Suzie told us a lie. The other programmers are using the same exact languages as we are.”

“Oh good!” Said the second programmer. “We feel happy now. We always will.”

The end.

Don't Bet Against Yourself

Wed, 15 Apr 2015 00:00:00 +0000

# #

Don’t Bet Against Yourself

In startup land these days, there’s a lot of discussion around the “right tool for the job.” You should “never invent here,” you should “use boring tools,” you should “use whatever the hell you want.” I’m currently involved in my third consecutive startup, and have the dubious distinction of having a lot of friends who are in similar positions - they’ve seen a lot, made a lot of mistakes, and have tried to learn what impact that the tools we choose have on our ability to do the three things that matter when starting a business:

Making something that works
Making customers happy
Making money

While I hardly know everything, I am decent at observing patterns. In these discussions (commiserations), one theme is predominant:

“Don’t bet against yourself.”

The idea here is that today’s throwaway prototype is tomorrow’s successful prototype (“rails new with rockets strapped to it”, to paraphrase the ever-wise Coda Hale) and that each decision you make to please yourself, to try something new, or to just fuck around will have consequences later.

So when you’re choosing the tools for the job, it’s a bit of a quandary. It can feel like choosing “too conservatively” will have the negative impact of not being exciting enough for people to work on. Examining this idea quickly makes it fall apart, though. If you’re only interested in working with the kinds of developers that are only attracted by using new, unproven technology, what does this say about them? Or you? To me it seems like an easy way to perpetuate the state we’re in is to keep doing the same old thing, which in this case, ironically, is to stop doing the same old thing and try something new. Confusing, right? The status quo is to choose something new. How did we end up in this situation?

On the other hand, choosing “too liberally” will leave you with a whole lot of uncertainty. You don’t know if the tools you’re playing around with will even exist in N years. This might be worth it to you, but what’s important is considering the tradeoffs up front. Perhaps your choice is so revolutionary, so bold, and so right that you can’t help but make it. In that case I say to you - congratulations, but with a word of caution:

“Being right doesn’t pay the bills.”

Further Listening

Aaron Quint and I discussed some of these ideas on our podcast, Beats, Rye & Types. Check out Episode 9: Magic.

The False Economy Of Metaprogramming

Fri, 06 Mar 2015 00:00:00 +0000

# #

The False Economy Of Metaprogramming

Modern web frameworks often attempt to provide “convenience” and “expressiveness” to their users by generating code using a technique commonly called either “template programming” or “metaprogramming.” It often starts with code like this:

class Foo < SomeVeryLargeORMClass
  attribute :bar, Boolean         
  attribute :baz, String
end

Here we’re declaring a class Foo which inherits from SomeVeryLargeORMClass. We’re using the class method attribute, which comes from SomeVeryLargeORMClass to declare a field named bar, which is a Boolean, and a field named baz, which is a String.

There is already a whole lot of functionality being included in our class simply because we inherit from SomeVeryLargeORMClass, and much of it is useful. We don’t have to connect to the database directly from this class to query it, and it’s nice and neat to be able to declare our database using a nice DSL like attribute. Unfortunately, that’s not where it tends to stop.

In addition to getting setter methods like Foo.new.bar= and getter methods like Foo.new.bar “for free,” we also get a lot of other things. Things that might seem nice on the surface, but cross an important axis: they generate more incidental complexity than the convenience they supply is worth.

An example of this is a method like Foo.new.bar?. For Boolean fields in ORMs, it is common to see “question mark” methods generated alongside other getters. The ORM uses sophisticated reflection and metaprogramming techniques to determine the type of the field and then insert other methods accordingly. This leads to situations where a developer might write the following code inside the Foo method:

def a_method
  if bar?
    # do something
  else
    # do something else
  end
end

The method bar? is generated for us because bar is a Boolean. The method is not something you can find by searching through the code base. It is only something you can know if you know how all of the underlying machinery of SomeVeryLargeORMClass works. After a while, this stops being convenient, and starts being annoying and expensive.

According to everyone’s favorite primary source of infallible and always perfect truths, Wikipedia, a false economy is defined as:

“… an action that saves money at the beginning but which, over a longer period of time, results in more money being spent or wasted than being saved.”

If the “money we spend” at the beginning is everything that we get for “free” by inheriting from SomeVeryLargeORMClass, then the money “being spent or wasted” is our attention, our ability to know what is going on, to reason about our code easily without having to boot into SomeVeryLargeORMClass every time.

So called “question mark” methods are a prime example of code that does nothing more than “look nice” without giving us any other benefit: in fact I feel they cost us more than they provide us with. For code to be “expressive” does not mean that it should “look” or “feel” like “natural language.” Code can’t look like natural language and anything that goes this far is a costly half-measure that we will regret in the long run.

Yeah, but what are you gonna make?

Tue, 03 Mar 2015 00:00:00 +0000

# #

Yeah, but what are you gonna make?

I love to tell people about the things I’m interested in and learning. Often, when I’m talking to someone about what I’m currently obsessed with, they’ll ask me:

“Yeah, but what are you gonna make?”

Usually, I’m not gonna make anything.

And I’m okay with that.

And you should be okay with it too.

Since you never know how what you learn now will impact what you do in the future, only learning for the sake of making doesn’t work for everyone. If you learn by doing, then do! But don’t be afraid to make and publish your toys and experiments.

Actually, don’t be afraid at all. Fear sucks.

Programs That Eat Programs: Software GR 2/2015

Thu, 26 Feb 2015 00:00:00 +0000

# #

Programs That Eat Programs

The awesome folks at Software GR in Grand Rapids, Michigan invited me to come and give a talk about … anything I wanted! It was quite an honor to be asked and I delivered a talk I’ve been wanting to give for a long time. I decided not to include the speaker notes this time because they’re still a work in progress. Consider this a first step towards a more well developed talk that I will give at the end of March. You can find the bibliography/works cited here.

A Metaphor For Abstract Interpretation

Sat, 31 Jan 2015 00:00:00 +0000

# #

A Metaphor For Abstract Interpretation

Recently, I’ve been studying a Computer Science concept known as “Abstract Interpretation.” Abstract Interpretation is a big idea that subsumes many smaller and more familiar concepts to programmers, one clear example being Static Analysis. But what is Abstract Interpretation exactly? How big is it? How do I understand it? Explain it? By reading, naturally!

There are a variety of well known resources about the subject, none of which are squarely aimed at basic understanding for the working programmer. One of the best explanations of the basic idea that I’ve seen so far comes from a tutorial paper titled “Using an Abstracted Interpreter to Understand Abstract Interpretation” by Daniel P. Friedman and Anurag Mendhekar. Since I’m a big fan of using metaphors as a pedagogical tool, I thoroughly enjoyed the first paragraphs from the paper, so I thought I’d share them here:

“The goal of abstract interpretation is to allow the user to do program analysis with a set of values that abstract another set of values. We present an example of abstract interpretation from real world experience. Suppose that we wish to travel from Nice to Paris. We are not sure how long the trip will take. Naturally, there is a very simple way to find out. Check the time just before we start our trip and check it again when we arrive and calculate the time based on these two numbers. That works, but we probably wanted to know how long it would be without actually taking the trip. What we needed was an abstract characterization of the road from Nice to Paris. Such things exist. They are called road maps. So, had we looked at such a map and used its scale, we could calculate the distance and by guessing our average speed, we likely would determine a reasonable answer to our question without actually taking the trip.

We can think about the actual taking of the trip as real computing and the analysis of the road map as abstract computing. What we learn when we do abstract computing can be incorporated into our knowledge when we do real computing. For example, knowing approximately how long the trip will take, will help us plan where we should stop to eat enroute to Paris.”

I could go on and on about the intricacies of the concepts involved and the metaphor at hand, but for once, I’m just going to shut up and try to let it sink in. More soon – stay tuned.

Learning Math: A Blank Is A Blank With A Blank

Thu, 29 Jan 2015 00:00:00 +0000

# #

Learning Math: A Blank Is A Blank With A Blank

Recently, I found myself in the curious position of trying to learn something again. Funny how that keeps happening. I’m trying to understand something called a “Galois connection,” and a kindly stranger on the Internet recommended a text called “The Galois Connection between Syntax and Semantics” by Peter Smith.

I found it online, started reading, and found this definition of a “partially ordered set” or “poset,” which is one of those annoying prerequisites for understanding what a “Galois connection” is. Here’s the explanation:

A partially ordered set - henceforth, a poset - is a set P, carrying an
ordering relation which is reflexive, anti-symmetric and transitive.

Okay, this seems simple enough but there are definitely some more prerequisites here - “ordering relation,” “reflexive,” “anti-symmetric,” and “transitive.” Where to go from here?

This post isn’t at all about what a Galois connection or a Poset is. Instead it’s about the shape of the definition above, a shape that is common enough that you see it everywhere, from math texts to computer science texts, programming language manuals, etc. I like to call it:

A Blank Is A Blank With A Blank

This kind of question comes up a lot with programmers who are trying to learn some more theory:

What is a Monad?
What is a Monoid?
What is a Functor?
What is a Lattice?
What is a …

They’re all Blanks, which are also Blanks, and have some kind of Blank that does Blank.

Math concepts, and the advanced programming concepts that are often based on them, are often tricky to learn because they can be very abstract. They’re not always distinct ideas themselves as they are circles around ideas that already exist. Groupings of ideas. For example, a set of numbers with an operation that defines some kind of order allows you to be sure that the set will hold certain properties.

Before we can learn and teach the actual big concepts themselves, like Monads and Lattices and Posets, we need to learn and teach the shapes of the abstractions that they rest on top of. Until a year or two ago this idea was pretty alien to me. I assumed that these big ideas must be big themselves. They’re often not. The results they produce can be big, and the insights they provide can resonate deeply, but often the ideas themselves are deceptively simple.

The simplicity of the idea of “A Blank is a Blank with a Blank” is meant to remind us that often what we’re doing is plugging concepts in to a larger framework that we’re building. Sometimes the cool thing about a sculpture isn’t the form that it holds, but the shadow it casts.

The Man Who Lived 300 Years

Fri, 02 Jan 2015 00:00:00 +0000

# #

The Man Who Lived 300 Years

On December 26th, 2014, in the warmth of the night in Mexico, a great person was taken from this earth way too soon. James Lennon Engels Golick was a dreamer, a doer, an inspiration to all he met. Though I’ve struggled to put in words what he’s meant to me, James always encouraged my words, so I needed to give it a shot.

I only knew him well for a few earth years, but I know him for many, many years in James Time. James Time is easy to calculate - you take earth time, multiply it by 10, and round up liberally.

During the 30 or so James Years that we had together, James taught me quite a lot: about computers, about life, art, music, coffee, booze. About esoteric things, grand things, general things. He loved things I didn’t understand and he didn’t understand everything that I loved, but his energy always made me feel welcomed, loved, wanted.

In the days after his passing, friends and family gathered in Toronto to remember him, and the recurring theme was an amazing one: James had what appeared to be a few dozen best friends. At first this was kind of off-putting - we all thought we were the only ones he talked to every day, complained to, bragged to, said hi to, etc. somewhat obsessively. Everyone from his mother to people he only met one or two times in the flesh reported the same experience - they knew his every move. They had shared their fears and regrets, their triumphs and achievements, and everything in between.

At first I struggled to understand how he had so much energy, and then I remembered: his day beat most people’s average week and his year would make most people’s decades seem modest.

So while James lived around 300 James Years, the tragedy remains: he should have probably been around for closer to 1000.

To everyone who feels the pain of his absence, we need to do what he would have wanted: to celebrate him, to remember him, to do well in his honor.

James and my son Asa, October 2014.

'Power' Mix: December 2014

Mon, 15 Dec 2014 00:00:00 +0000

# #

‘Power’ Mix: December 2014

I’ve been wanting to make this mix forever, and booking a gig at a bar in a couple weeks gave me the motivation I needed. I recorded a bunch of tracks from vinyl through an old Zoom H4 recorder from my #noiselife days, and then mastered everything in Audacity. I imported the tracks into Ableton Live, and using the controller I have for that, executed the mix below. It’s diverse and not shy about record noise - enjoy!

Tracklist:

Willie Colon - Angustia Maternal (1977)
Blops - Locomotora (1974)
Funkadelic - Funky Dollar Bill (1970)
Donald Byrd - Sister Love (1973)
Lonnie Liston Smith - Exotic Mysteries (1978)
Grover Washington - Mister Magic (1975)
Roy Ayers Ubiquity - Everybody Loves The Sunshine (1976)
Yellow Magic Orchestra - Citizens of Science (1980)
Eddie Harris - Zambezi Dance (1972)
Haruna Ishola and His Apala Group - Haruna Ishola Lo Ilu Oyinbo (1973)
McCoy Tyner - His Blessings (1970)
Hailu Mergia - Woghenei (1977)
Victor Uwaifo - Ekassa 24 (1973)
Celestine Ukwu & His Philosophers National - Onwu Ama Eze (1974)
Mulatu Astatke - Tezeta (Traditional)
Peacocks Guitar Band - Jesus Christ (1973)
Frank Croffie of Ramblers Fame - Odo E! (1980)
David Axelrod - The Signs (1970)
Caetano Veloso - Algeria, Algeria (1968)
Milton Nascimento - Clube de Esquina No. 2 (1972)
Ismael Miranda - Borinquen Tiene Montuno (1974)
Ray Baretto - Power! (1970)

You Are Learning Haskell Right Now (Or Anything You Want Really)

Thu, 11 Dec 2014 00:00:00 +0000

# #

You Are Learning Haskell Right Now (Or Anything You Want Really)

It wasn’t that long ago (about 6 months or so) that I wrote in the description for a talk I was giving in New York City that though I kept trying, I “still can’t write Haskell.” I wasn’t joking at the time, or being coy - I couldn’t put even a simple program together. Since then, I’ve turned a bit of a corner and have made modest progress working on an “Intermediate” level Haskell program involving nasty State Monad bits, Networking and disk persistence, not to mention a lot of brow scratching and internal trauma and anxiety.

People have asked me a few times “how I learned” or have asked me for recommendations for how they should learn the language. Since I took what I think is a very conventional but not often discussed path to “competency” with the language, I figured I’d write it down so people can learn from it.

Also, though this post is not a conference talk, it was partially inspired by the following tweet, in that I hope it exposes to you just how long it has taken me to do even basic things and how that is okay:

at future conferences I'd love to see some *men* talk about their vulnerability and impostor syndrome
— literally santa (@jennschiffer) December 11, 2014

I put together a rough time line of how I “learned Haskell.” It covered about 3.5 years worth of time during which I purchased ~4 books on the subject, asked for countless hours of advice and help from people on the internet and in person, and repeatedly try to make progress on even the most basic non-trivial programs I could think of. I thought I would publish that time line and fill it with things like key conversations I had with people or little insights I had, but then I realized that that was pointless. Let me explain.

The first documentation I could find of me messing around with Haskell at all was a little more than 3 years ago:

Also, Haskell is very intriguing.
— mrb (@mrb_bk) November 3, 2011

The point I wanted to get across with the incidental nature of the time line that I threw away is that the only thing that helped was passing time. It’s not true that “every time I tried to learn” it got easier, because how I was trying wasn’t consistent. The reason why learning a language like Haskell is hard is because there is a hell of a lot to learn.

Another reason to not publish a time line about how I learned a specific language is because in reality, it’s not at all that interesting. What the world needs is more posts encouraging people to forge their own paths and try, try, try to achieve the very challenging goals that they’re setting out for themselves. It doesn’t matter if it’s learning an “academic” language or something more “practical” - the lesson remains the same.

When I first started learning, I tried to mess with things I knew well, like parsing binary formats, or things I needed, like a fast little git application, or things I loved, like visual art or music processing. I tried networking, distributed programming. I tried to build from the bottom up. I tried to start from the top.

I tried everything I could think of and was frustrated over and over again. The tool chain was complicated, error messages made no sense. I didn’t know how to put basic things together and I struggled again and again. I knew what I wanted to create but was unable to. I lacked the knowledge and the ability.

I repeatedly felt like I had no idea what I was doing, and I wasn’t getting a whole lot of positive feedback in my attempts in general. The only thing that kept me going was wanting to be able to do this thing that I didn’t know how to do. Though I couldn’t even carry on a basic conversation about the language in person, I sought people to chat about it with. Though I couldn’t understand the type errors, I tried to type check my simple programs.

I tried over and over again to turn my self doubt into a pure functional program, and eventually, it clicked.

So I have come to the conclusion that the only thing that works is time and repetition, and that may seem daunting, but guess what? Time is passing. You’re learning Haskell right now. Or anything else you want to, for that matter.

P.S. I’m anticipating that some clever people will come to the conclusion that if I had only tried to “learn the language properly,” I would have had an easier time. My response is: if that’s your conclusion, you’re missing the point.

Two Things Types Have Taught Me

Thu, 30 Oct 2014 00:00:00 +0000

# #

Two Things Types Have Taught Me

I’ve been researching and playing around with programming languages with very interesting type systems these days, namely Idris and Haskell. Since what little code I write professionally these days is entirely in a dynamically checked language with none of the typically touted benefits of an advanced type system, it’s been a lot of fun.

After a few years of repeatedly banging my head on being able to express even the simplest programs using Haskell, I’ve recently had some breakthroughs and am able to construct some modest programs. I’ve made no bones about the fact that I think types are super cool and a very interesting area of research in Computer Science in general and Programming Languages in particular, but I haven’t said that much about what I’ve learned from using these type systems directly. This post is a start down that road.

Thing Number One: A type signature is not enough

Though type systems can allow for wonderfully expressive type signatures and programs which are shockingly declarative and almost literal, it is still my opinion that a type signature is not enough. Descriptive names and documentation are still requirements of excellent software and at least in terms of what is currently available to the day to day programmer, the whole package is necessary.

It’s (more or less) true in my experience that once you learn the meaning of things like \\, $, ., >>=, etc. in Haskell that they eventually become transparent, but two things make this challenging for people learning the language:

There are too many instances of these kinds of opaque functions, concepts, etc.
The language encourages people to create their own

It takes a lot of time to become familiar with all of the idioms of Haskell (I’m not even close in that respect). Excellent documentation in certain places has been the main thing that has made it easier (that, and talking to people) - so in my experience, we’ll always need more than types. We need comprehensive, awesome documentation, and people to support the software.

Thing Number Two: Type systems are excellent for globally enforcing invariants

One thing that gets me excited about type systems are their ability to enforce invariants in a way that is challenging or cumbersome to replicate in other contexts. One example that I have been playing around with is using a type system like the one found in Idris to enforce certain properties of functions statically that have previously only been enforceable at runtime. Because Idris uses dependent types, it is possible to design types which enforce function properties such as idempotence.

This would have awesome impact in the lives of professional programmers because there is increasing evidence that designing data transformations to conform to certain algebraic properties makes reasoning about data considerably easier in the long term. A type system that could tell you statically that your function is not fit in a job class which requires idempotent jobs would be amazing. Something like this is possible in other contexts (certain forms of testing, etc.) but to me it feels challenging or clumsy to generalize.

Conclusion

I’ve been thinking a lot about how fancy type systems can impact software development, and these two short thoughts are my first take at writing some of them down. I’d love to hear your thoughts - drop me an email or hit me up on twitter @mrb_bk.

Should I Read Papers?

Tue, 21 Oct 2014 00:00:00 +0000

# #

Should I read papers?

TL;DR: Yes.[1]

Papers, papers, papers. Papers have gotten a lot of attention these days. There are Twitter accounts, meetups all over the world, GitHub repositories, and lots and lots of chatter about reading papers. With all this talk about papers, people are bound to ask:

“Should I read papers?”

I think you should. I’m on the record saying so quite a bit and I continue to assert that for a variety of reasons, it’s a great idea for people in any field.

But this still leaves a lot of questions unanswered, which I’d like to address now. In the spirit of unpacking dense language, let’s look at each of the four words in the question:

“Should I read papers?”

Of the four words, the “should” is probably the most challenging. What do I mean you “should” read papers? Who am I to say? Do I really want to dictate your behavior?

No, I really don’t. If you don’t want to read papers, don’t. That’s fine. But I wanted it to be on record why I think you should.

I mean should in the same sense that you should eat healthful food, or you should get out in the fresh air as often as possible. I mean it in the sense that if you believe that you should bring purpose to your work, then you should be doing whatever you can to improve and to understand the universe you work in a little bit better each day.

“Should I read papers?”

Who is the “I” that I’m addressing here? My current milieu is that of people involved with computers in some way, but primarily I’m addressing software developers.

I was fortunate enough to study with some amazing people who taught me that if you make things with computers, it behooves you to understand how they work. To not accept the status quo in artistic or creative tools that computers provide, but to look under the hood, emulate, and make your own tools.

So if you’re one of those people, who sits down in front of a computer every day to make things either for yourselves or others, you’re included in that I.

“Should I read papers?”

Because of the amazing elasticity of human language, even the word “read” in this context is very loaded. Doesn’t read just have one meaning? If it did, most people would say something like:

“You have read something if you’ve consumed the entire thing from beginning to end.”

But that is a fantastically terrible definition for read in the sense of “Should I read papers?” To read a paper is to interact with in even the most surfacey of ways:

Skim the works cited
Google the author’s names and see what they’re all about
Get a grasp of the domain involved from the abstract

This is because to read a paper means to have a relationship with it. Over a long period of time. You may sit down and “give it a good read,” but really, if it’s worth reading, you’ll come back to it again and again.

In other words: you can’t be “too dumb” to read a paper. Anyone can read any paper at any time. Don’t be afraid, be encouraged. You will learn something.

Keep in mind that “reading a paper is not the same as reading a blogpost or a novel” and also don’t forget: you are not alone.

Even though the question at hand is “Should I read papers?” you’re barely ever alone. There’s almost always someone else going down a similar pathway as you. And now more than ever those people are reachable, either in person or virtually.

“Should I read papers?”

Finally (maybe this one should have been first?), what is a “paper?” A paper is a crystallized idea presented in written form. It could be tiny. It could be significantly longer than that.

It could be an idea that someone captured, or the result of a life’s work, or anything in between.

As fluid as the definition of read is above, that’s how you should think of a paper. The person or people who wrote it aren’t necessarily “smarter” than you. They almost certainly have very different life experiences. Who knows where they come from, or what they were trying to do?

I’m drawing your attention away from the paper and toward the people on purpose, because they’re one and the same.

In Conclusion

My answer to the question “Should I read papers?” is a resounding yes, but really what I mean is: you should stretch.

Give yourself a challenge.

There’s more beneath the surface than you could ever imagine.

Thanks to Zeeshan Lakhani, Tom Santero, James Golick, Alex Kahn, and Ken Keiter for their help with this post.

Notes

[1] This is the one and only “TL;DR” that I will ever publish.

Where Does Scope Come From?

Wed, 10 Sep 2014 00:00:00 +0000

# #

Where Does Scope Come From?

After several false starts, I finally sat down and watched the first of Frank Pfenning’s 2012“Proof theory foundations” talks from the University of Oregon Programming Languages Summer School (OPLSS). I am very glad that I did.

Pfenning starts the talk out by pointing out that he will be covering the “philosophy” branch of the “holy trinity” of Philosophy, Computer Science and Mathematics. If you want to “construct a logic,” or understand how various logics work, I can’t recommend this video enough. Pfenning demonstrates the mechanics of many notions that programmers are familiar with, including “connectives” (conjunction, disjunction, negation, etc.) and scope.

The connectives are demonstrated by showing the connection between intuitionistic or constructivist logic and the “introduction” and “elimination” rules, and how the “harmony” of the qualities of “soundness” and “completeness” fit together in a way that makes our logical systems consistent when applied to real world ideas.

Scope is demonstrated during this process as well. It turns out that in logic, as in programming, the difference between a sensible concept of scope and a tricky one can often mean the difference between a proof that makes no sense, and one that you can rest other proofs on. I am very interested in this kind of fundamental kernel - how the smallest and simplest ideas are absolutely necessary for a sound foundation in any kind of logical system. Scope is one of the first intuitions that new programmers build - can we exploit this fact to make the connections between logic, math, and programming clearer to beginners?

I’m working on a longer writeup of the idea, which I hope to publish here with some transcriptions of what Pfenning writes on the board in the video, but for those who are on the edge about watching these lectures, allow this post to be the push you need. Some audio quality issues during Q&A aside, this is pure gold - dive in!

Programming, Math, and Hot Dogs

Mon, 11 Aug 2014 00:00:00 +0000

# #

Programming, Math, and Hot Dogs

Programming is like eating a hot dog: you don’t have to know what it’s made of to enjoy it, and in fact it may even detract from your experience. [1][2]

Notes:

[1] Math. Programming is made of math. That’s the joke.
[2] On the other hand, some people take pleasure in knowing where their food comes from, understanding that even scraps have a home somewhere.

Know Your Types: GoRuCo 2014

Mon, 23 Jun 2014 00:00:00 +0000

Know Your Types: GoRuCo 2014

This is the video and unedited text of my talk at GoRuCo on bringing static types to dynamic languages. Enjoy!

I'll never stop calling it 'The City'

Tue, 17 Jun 2014 00:00:00 +0000

# #

I’ll never stop calling it ‘The City’

I don’t have a habit of posting anything personal on this blog, but there’s room here for anything.

I also don’t have really have the words to express how this makes me feel to say, but here it goes: On August 1st, 2014, my family and I are moving out of New York City to pursue a new life in the Washington, D.C. area.

There are many factors that make this the right choice for us right now, and equally as many that make it a gut-wrenching, impossible goodbye.

Really, I’m not going to try and write down what it’s like to leave before I leave.

For my local friends, we still have plenty of time together, and I’ll be back in “the city” often.

Also, I’ll never stop calling it “The City.”

See you all soon.

Types Are The Truth

Tue, 10 Jun 2014 00:00:00 +0000

This post is one of a series on Types and Programming Languages by Benjamin C. Pierce.

Types Are The Truth

In Chapter 9 of Types and Programming Languages, Pierce presents his exposition of the Simply typed Lambda-Calculus. It is a truly awesome chapter - each subsection in succession constructs a rich and very well detailed system of computation and Pierce is a master of precision, guiding the reader through the maze of theory very carefully. Exploring the mechanics of the various calculi Pierce presents reveal the hidden connections between this core kernel of an idea to many, many practical applications - type systems, compilers, program optimization, and more draw on these ideas.

On the other hand, studying Chapter 9 of TAPL will have you well on your way to hallucinating these mechanics everywhere. You may see their shapes, in passing, and then lose them. Sounds kind of scary, but we could all use some more hallucinations in our lives, right? Among the various sub-explorations in the chapter, “Section 9.5: Erasure and Typability,” grabbed my attention in particular. It contained a new bit of information for me, confirmed some interesting intuitions, and put a name on an idea I was curious about: type erasure.

What is Type Erasure?

Pierce begins the section by pointing out that the evaluation relation as defined on simply typed terms has the property that it carries along the type information in the terms, but does not actually use them during run-time. This is in contrast to how things work in “full-scale” programming languages:

“Most compilers for full-scale programming languages actually avoid carrying annotations at run time: they are used during typechecking (and during code generation, in more sophisticated compilers), but do not appear in the compiled form of the program.”

That’s a really cool idea: The safety that is guaranteed by type checks at compile time can be removed at run-time with no change in the results. This process of removal is called type erasure, a new concept to me.

I was previously aware that programs can be compiled down to intermediate representations which may not carry type information, and I knew that in the end the instructions carried out while executing the code have very little to do with my type annotations, but the impact of seeing the mechanics of it and how it actually works was very impactful to me. I think this is because the relationship between fully annotated programs and their “erased” counterparts is echoed in other important place in Computer Science theory. I’ll try to elaborate later. First let’s see the mechanics.

How Does Type Erasure Work?

Pierce explains that in most situations, “programs are converted back to an untyped form before they are evaluated,” but that doesn’t really speak to how it actually works. [1] A bunch of questions come to mind:

How can you be sure that the semantics won’t change?
Is this universally true?
Are there exceptions?
Is there a theorem?
Yes. It’s TAPL. Of course there’s a theorem.

With respect to type erased typed terms, Pierce notes that:

“This style of semantics can be formalized using an erasure function mapping simply typed terms into the corresponding untyped terms.”

For a term t, Pierce gives the following definition of erasure, naming the function erase. Rules are given for each of the three types of terms - variables, abstractions, and applications.

erase(x)         = x                   -- 1 - Variable
erase(λx:T1. t2) = λx. erase(t2)       -- 2 - Abstraction
erase(t1 t2)     = erase(t1) erase(t2) -- 3 - Application

For each of these rules in order, here is how I would write it out in english:

If you apply the erase function to a variable x, you get x.
If you apply the erase function to an abstraction which is annotated (in this case, a function that accepts a term of type T1), it is equivalent to an untyped abstraction with an erased input.
If you apply the erase function to the application of a term to another, it is equivalent to erasing them first and then applying them.

In other words:

“…it doesn’t really matter whether we evaluate a typed term directly, or whether we erase it and evaluate the underlying untyped term.”

Whoah, that’s awesome, and we just saw exactly how it works. While we’re at it, here’s one more pithy Piercism on erasure:

“evaluation commutes with erasure”

Let it sink in. That means that the commutative property holds between evaluation and erasure - the order you apply them does not matter. The proof is constructed by induction on the following derivations:

If t -> t’ under the typed evaluation relation, then erase(t) -> erase(t’).
If erase(t) -> m’ under the typed evaluation relation, then there is a simply typed term t’ such that t -> t’ and erase(t’) = m’.

It’s useful to write down some sample terms that follow this form to get a feel for the mechanics of it, but it’s not necessary to understand the proof component at this point to grok the bigger ideas at play.

The Implications of Erasure

So what are those echoes that I referred to above? Isn’t erasure kind of an obvious idea once you think about it for a moment? While not being the most complex concept with the most tendrils to unravel, I feel that its structure is quite elegant, and I believe that Pierce does too:

“For more interesting languages and more interesting compilers, it becomes a quite important property: it tells us that a “high-level” semantics, expressed directly in terms of the language that the programmer writes, coincides with an alternative, lower-level evaluation strategy actually used by an implementation of the language.”

That makes me wonder what other kinds of transformations you can make to terms in programs that won’t change their meaning. The fact that you can assert properties of programs with types, erase them, and have the same guarantees of semantics is fascinating to me. That’s the idea I was hinting at with this tweet:

Types are the truth: they're there whether you want them to be or not
— mrb (@mrb_bk) April 14, 2014

Whether your programming language “embraces” types or has a modern type system or not, these concepts are central to how programs execute, because they are a fundamental part of how computation works. Adding types to the lambda calculus is what transformed the primordial ooze of computation into the complex ecosystem of programming, and its mysteries and intricacies are manifold.

Works Cited

All quotes unless otherwise cited from Pierce.

[1] Benjamin C. Pierce. 2002. Types and Programming Languages. MIT Press, Cambridge, MA, USA.

The Art of Obsession: !!Con 2014

Sun, 01 Jun 2014 00:00:00 +0000

# #

The Art of Obsession: !!Con 2014

This is the video and unedited text of my keynote talk at !!Con 2014 in NYC. The slides can be found here. Transcription provided by the awesome Mirabai Knight - hire them for your next conference!

Transcript

All right. Allow me to… Uh… Bet on this working. All right. No? Oh! There we go. Okay. All right. Let’s see. This works. I’m pretty loud, so… What’s that? For the screen? That’s exciting for me. Is it accurate? Cool. I have bad hearing, and so… That’s really cool to see that. All right. Anyway, I’ll get started. So… This is a photograph of a really special person who I’d like to introduce you to today. How many people know who this person is? Okay. A few of you? Cool. So this is Harry Smith. I first saw this photograph when I was a teenager. About 14 or 15 years old. And I was at my friend’s house. And the thing about this photograph is that it was in a book of photographs by Allen Ginsberg, who was my favorite poet at the time. I fancied myself a bit of a poet when I was a teenager, as many of you did and I’m sure still do. So it was a really interesting photograph. I had no idea who this individual was. And the caption is really funny. It says Harry Smith, painter, archivist, anthropologist, filmmaker, hermetic alchemist, transforming milk into milk. So I had to know – who was he? I did a bit more digging. Found a photograph of him when he was younger. He had some interesting things going on earlier in his life as well. You can tell from that smile he has a lot of knowledge in his head. So it took me a long time to figure out information about him, because at that time, it was just a little bit more challenging to figure out not only basic facts about an individual, but to get your hands on the media was particularly challenging. So it turns out that he was all those things that Ginsberg said he was and a lot more. He was an anthropologist by trade, a linguist, a translator, a painter, and perhaps most of all, a collector, a very obsessive collector. I’m going to read you a quote about him. This quote and a lot of the images in this presentation are in part due to the good work of the Getty Research Institute, part of the Getty Museum in Los Angeles. So they preserved a lot of photographs about him and did research about him and read about him. You’re all reading the quote on the screen. Harry Smith liked to look for keys to the universal patterns that shape our cultures and the hidden realms of the human unconscious. He compared patterns in Native American music with the eccentric rhythms of jazz, the patterns in Seminole patchwork with Ukrainian Easter eggs, the (on screen)… Sounds pretty cool, right? He was a collector. So I’m going to show you… I’m not going to tell you so much anymore. I’m going to show you the things he collected. Tell you a little bit about them, but show you too. And perhaps most famously, he’s a Grammy award winning ethnomusicologist. He collected sounds and also recorded sound himself. Some of the recorded sounds he collected he presented in this 1952 epic release three double albums with booklets, called the anthology of American folk music, from the mid20s to the early 30 s, and released them – songs, social music, and ballads. He released in 1952 after collecting it for a long time, and you can imagine that in the ’40s and ’50s, collecting records wasn’t really a thing. These were not artifacts that people cared about. No one thought you would want to keep them around or had any idea that this music would be lost, along with the media. So the fact that he decided to collect and document them in that anthropological way was important, because he ended up influencing quite a lot of people by releasing this music that would otherwise have been lost to history. Including Joan Baez, a well known folk singer in these parts named Bob Dylan, who was not only influenced by this music. He covered some of these songs and used a lot of them in this music. And it influenced a personal favorite musician of mine, less well known, but equally brilliant. Named John Fahey. An American guitar player who took Smith’s direction a little bit even further. And here’s another interesting story for a musical project he worked on. In 1954, Smith was held for a week in the Anadarko, Oklahoma jail, on suspicion of stealing guns, and there he met several Kiowa Indians who introduced him (on screen) he returned to record them, because he heard the music and liked it, and he decided to set up his equipment in singers’ homes or his hotel room, rather than at the actual ceremony. Because he wanted to get commentary. And he was worried about mixing taking peyote and faithfully recording these rituals, which you can imagine might be challenging. Here’s the picture of that. It’s called the Kiowa peyote meeting and it shows a range of his interests. He published it on the same record label 20 years after he published the anthology of American folk music, and it shows the range of his interests and the rigor of how he was capable of applying his means of collecting to a variety of different genres of music. But he also collected a lot of other things, like eggshells on toilet paper rolls. You can’t really see that. There’s not a lot of light. When you see the slides later – it’s literally a box full of eggshells on toilet paper rolls. He collected gourds. These are the boxes from the Getty Museum collection of his collections. Tarot decks, he was into the occult and esoteric knowledge, and one of his other collections was paper airplanes. He collected so many paper airplanes that he donated them to the Smithsonian Air and Space Museum, and they had them there for a long time. Now they’re at the Getty. But the thing about the way he collected them was fascinating. He collected for years and years found paper airplanes. He lived in New York and he would walk all over the place and ended up finding a lot of paper airplanes. You never know what you’re going to find if you’re not looking for it. So I’m going to show you some of them. They hold fascinating cultural meaning, because as you see, the airplanes that he finds reflect components of the culture that they were found in. Like this one from 1967… That’s on a flyer entitled many smokes and spring seasonings, published by the organization to end the war in Vietnam. Or this from the cafe Agogo. Published by the American Folkways in 1967. Demonstrate against the war, the Vietnam peace parade committee. Here’s a paper airplane made from that flyer, a flyer from the Empire State Building. People used to like to fold them up on the top of the Empire State Building and throw them off and he would find them somewhere. A 1978 menu from Max’s Kansas City, inscribed with the word Excelsior over and over again, and here’s a page from a child’s connect the dots book. Whatever you want to call it. So besides paper airplanes, he loved string figures. He was really interested in string figures. Here’s an awesome picture of him with some string figures. It’s really… You know, very photogenic individual. So here’s him, recreating a string figure of some kind. Here’s a sketch that he made. Of a string figure from an unpublished manuscript on string figures that was left behind when he died. So he collected string figures, which is an interesting thing to collect. How do you collect them? Right? They’re an ephemeral thing, but they’re a reproducible form, because you can compose them. You can start with a string of a known dimension and perform certain operations on them and then end up with that string. Right? And so he did that. And so here are some photographs of the ones that he placed, and he would see someone making it, write it down, reproduce it, and photograph it.

And they look really cool. So much so that a gallery in 2012 in Brooklyn did a recreation of some of the facsimiles of his books. So they’re really cool. So besides all the other things he collected – images. He was a really famous painter, but also a really well known experimental filmmaker. He wound up in the Bay Area in the 50s, becoming a very well known avant garde filmmaker. Made perhaps one of the most beautiful and difficult abstract films of all time, called film number 18, also called mahogany. A very challenging film to project, a very challenging film to watch, very long, very abstract, but shows that he was a collector and a composer. So he used a lot of collected, found imagery. He would find film reels, he would record everyday life, and he would compose it into these grand collages, that were meant to invoke, you know, this idea of universal consciousness that he was so desperately seeking.

Here’s a couple more stills. This one is more indicative of his abstract work. So he also collected… A lot of other things. Books. Pop-up books, beaded costumes, things shaped like other things. Including spoons shaped like ducts, banks shaped like apples, and anything shaped like a hamburger. So I think we can all agree that Harry Smith was a master of obsession. Which is why I jumped right into telling you this story. This talk is called the art of obsession. My name is Michael Bernstein. This is !!Con. I’m really excited to be here. Thanks for having me. It’s an honor. This is the requisite title slide. This is my first keynote talk, if you can’t tell, and that’s enough about me. Let’s get back to the subject. So why did I tell you his story? Because I’m interested in the following questions and I wanted an excuse to ask them to you. Because you’re such a room full of beautiful and intelligent people. So what can we learn from Smith’s techniques? That’s one thing that I wanted to ask you. He found satisfaction in knowledge, in the intersection between his interests. Right? So it wasn’t that he liked music, and he liked film and he liked paper airplanes and he liked string figures. Right? He liked connections between things. He liked knowledge. And he really… He liked life. Life wasn’t always so kind to him, but he really liked it, and he wanted to experience it to the fullest. So paper airplanes were interesting, because of the stories they told as a whole. Each of them is an interesting artifact. Right? They’re cool, but if I said he collected a paper airplane, that would hardly be as fascinating as the fact that he collected so many over such a long period of time, in so many different places. And he actually sat and thought about what that meant. Why did people make them? What did finding them mean? What can you… What could you read from a map of reconstructed locations of found paper airplanes over space and time?

So paintings, songs, things that look like other things. They all communicate ideas. They’re more than the substance that composes them. So that’s one question. Another one is: how can we master this art of obsession? And why do we want to? It’s summed up to me by this quote that we saw from the Getty. I love the writing that they did about his work. So I’m going to read the same quote twice. Harry Smith liked to look for keys to the universal patterns that shape our cultures and the hidden realms of the human unconscious. Keys to universal patterns. This is the – I really like this phrase the best. From the work about him. So we can master the art of obsession by looking for these keys. That’s my thesis. And the universal patterns are – since universal patterns are composed of connections, I have a connection that I’d like to share. Which is that since string figures and knots were some of Smith’s favorite things, that’s a really interesting thing to me. He believed that they held some secrets to universal understanding. Knot tying can be found in many cultures, but not others, and there was a connection in Smith’s mind between – you know, how cultures and societies communicated with each other through written and spoken language and what kind of knot tying culture they had. He believed that they worked similarly to how the brain understood written numbers and letters. So Smith was really into knots. And also computer scientists and mathematicians are also really into knots. This was a figure from a book about ternary writing, and this picture that’s shown here has parallels to parallel execution and term rewriting. So you can look at this knot untying as a complex term that can be normalized to a specific value. And this connection is really deep and interesting to me. And it made me think that…knots are programs. And they signal – what is interesting about computation. Because computation is more than how computers work. Computation – our understanding of computation influences how we understand nature the world around us, influences how we live and communicate, and it’s one of the deepest human achievements, intellectually, that we’ve ever achieved. The achievements. You achieve them. That’s how you get them. That’s how they become achievements. So since knots are programs, that leads me to believe that Harry Smith would have loved computer science. It has universal application. It gets to the heart of so many human problems. Even though we don’t always talk about it in the most humane terms. It has its own codes, its own languages, it’s very esoteric in its own way, and it can be applied in a variety of contexts that are not immediately obvious by looking at it on the surface. And so if Harry Smith had some advice for us, he would have told us to get obsessed. If we want to learn more about computer science and understand it better, we should probably get really into it, and get obsessed in the way he did, by looking at the connections. Looking for connections between things. Because obsession is crucial to interdisciplinary thinking. If you want to understand one thing deeply, and you have your eyes open, and are available to – making connections between the thing that you’re studying and the world around you, then you can get a lot further, and Smith demonstrated this, in my mind. And the reason we want to think interdisciplinarily is because interdisciplinary thinking is crucial to advancement. The biggest, most radical achievements within the world of computer science, in my opinion, are those that have made connections to other disciplines and have leveraged other lessons from those other disciplines, and people that are aware of the world around them can make big, radical changes, in the way that we think about things, even as deep as computation. So there’s so many possibilities, when it comes to computers and computer science. There’s a lot of starting points for exploration. Things that you can get obsessed with. It’s very fractal in nature, as it’s often said. You keep studying and studying and studying, and you look at what you’re studying and you realize… Wow, this looks a lot like what I think I was studying three months ago, and I’m studying the same thing, but it’s somehow different. How am I solving the same problem over and over again? You’re in a knot or something like that. If I was a skilled speaker, I would have a knot analogy now too, but I don’t. So what kind of things with computers should we get obsessed with? I really like compilers. They’re my favorite type of program lately. I think that compilers are… Compilers are the mahogany, film number 18, of computer science. You know? They’re so amazing. They do everything. We… People spend a lot of time thinking about them, talking about them. If you want to teach me about how to write a compiler, you can… I’d love to chat with you after this talk. I like type systems a lot lately too. They get to the heart of how we express programs. There are these running programs that tell us information about our programs. They tell us what we can and what we cannot express. And they’re there, whether we want them to be or not. It’s sort of this funny thing, that you can program in one so-called paradigm and be influenced by this idea that you can choose to acknowledge it or not. And it’s still sort of, to a certain extent – influences what you’re capable of expressing, in your language of choice. I’m absolutely fascinated by them, and I think they’re one of the deepest… They’re clearly one of the deepest connections that we have between computer science and mathematics and logic, for one thing. So I think that Harry Smith probably would have gotten a really big kick out of the idea that programs are really connected to this 1920s logic technology. He probably would have said told you so, or something like that. Static analysis is cool. It’s also my day job. But programs that analyze other programs are really interesting to think about. Right? Talk about the connections between things. So I want to write my… I’m going to write a program in one language that tells you information about the program that you wrote in another language, because in the language that I’m writing in, it’s good for telling you things about your programs, but I wouldn’t want to write the kinds of programs that you write in the language that you write them in. Or something. So that’s a lot. That’s a big bag, that I would be careful before I open that one up, if I were you. Logic programming is really cool. You can solve problems that you didn’t know you have.

And in understanding and learning how to use logic programming, you can shatter your narrow paradigm world views. I have a lot of exclamation points in my notes. Just my… Title was lacking one, so I felt like I needed to make up for them. I’m not speaking in a very exclamatory way right now, but they’re there. I spoke a lot and thought a lot about distributed systems and they’re also cool and they’re also one of those fractal problems. The same problems that you have quote-unquote locally appear in a distributed way in a really different and interesting – the properties slightly change, in other words. Things look different when you see them from afar, and then you stare at them for a while, and they kind of look the same, but not quite. That’s what abstraction is, I guess. So you should study them. The world is already distributed. It’s time to catch up. Exclamation point. Programming language theory. It encompasses a lot of other things. Programming language theorists and type theorists have this cool ability to somehow talk about philosophy. But now you’re in the context of computer science. Right? So when you’re thinking about how we express the… How we interact with… What is our interface to computation, and to computers? It’s through these languages. Right? So they’re clearly very powerful. And they… And it’s well known that the language that you use influences what you can express. Right? That’s sort of a given. But it also influences how you think about computation in general. So if you speak enough in a certain language and you think you can communicate certain things, you write enough programs in a certain programming language, and it influences how you think for a long time. You know, I learned very… I’ve learned very uninteresting things when I was learning how computer programming worked. I wish that I had learned other paradigms first, because then I would not have such a broken brain right now. But anyway… Yeah. How you learn really influences what you think. And programming language theory encompasses a lot of that. So how do you learn all of those things?

How do you get obsessed, and stay obsessed? How do you understand these subjects in a way that will lead you to find the types of connections that Smith made and was so good at making? So here’s a little bit of advice on my take on that. This is my impractical guide to the art of obsession. So the first piece of advice is to follow the links. So find a resource. Look for the influences and connections between these resources and disciplines. Here’s a metaphor for that. Your favorite band, growing up, liked music that you thought was terrible. Right? So I have a really funny story. How many people like the band Black Flag? You guys like punk rock music? Hardcore music? You know who they are, at least? If you don’t, that’s okay. They’re a really famous band, and they were really tough. They were from California. And they thought that they were really cool. They were. They were really cool. The lead guitarist had a dirty secret, though. He loved the Grateful Dead. They were his favorite band. He wore Grateful Dead T-shirts and people thought that he was kidding. So these punks would beat up hippies and would be wearing Black Flag T-shirts while they were doing so. So the idea there is that… It’s not enough to learn what your influencers teach you. Right? You need to look back at what influenced them. In order to understand how they arrived at that conclusion. That’s a fundamental component to understanding how the world works. How computation works. And in order to make those connections, you have to be willing to step further than what’s obvious. And out of your comfort zone.

Because I love the Grateful Dead and Black Flag a lot. They’re both really great. Yeah. Cool. I’m glad I got to tell that story. So open your eyes. Be open to connections to your interests and unlikely places. This kind of goes with that. Like, not only… You know, when you’re doing something completely different, be open to the idea that you might learn something about what you’re not expecting to learn about. I’ve gotten a lot out of that, and I think Smith did too. He heard things in – you know, he saw patterns in Ukrainian Easter eggs that were also present in… You know, whatever other beaded things that he was studying. So… I missed it from the quote, but you get my idea, I hope. Pursue discomfort.

Regularly consult texts and concepts that are too hard for you. You know, pick it up. Don’t be afraid. You know, if it’s a heavy book, the worst that will happen is that you’ll drop it on your foot. And it won’t get damaged, though, because it’s a really big book. Your foot will also heal. Pick up that book and just look at what the shape of those things are that are too hard for you, and, like, stare at the page that’s impossible for you to understand. And remember that feeling. Because then when you study, and you get past that, and you squash it, right, you know, you squash that discomfort, and you learn, then you get to feel really good. Right? You’re like… Ha-ha! I remember when this looked like absolute gibberish to me, and now I’m talking about it on Twitter like I’m an authority.

So… Over time, watch them become understandable as you work hard and learn. Trust yourself, because you’re awesome. Exclamation point. You know, if you don’t trust yourself, and you’re lucky enough to have other people that trust you, you’ll probably still be okay, but if you really want to get as far as you can, you need to be the one to convince yourself that you’re doing the right thing, and it’s okay to do the wrong thing. You’ll do many more wrong things than right things. If you follow Harry Smith’s example, especially.

Contact your heroes. This is a good one. Because they want to teach and learn also. You know, they have things to learn from you. Whatever you’ve done in your life is definitely not the same as whatever your hero has done. Right? You think about individuals that accomplish things as somehow projections of those things that they’ve accomplished, but actually, they’re just bags of meat. Like you are. And they want to be friends with you and talk to you, most likely. And if they don’t, then that’s their loss, not yours. You have as much to teach as you have to learn. And that’s a really important thing to remember. People want to know what you think about the work that they’ve done. That’s why they do it.

Whether they want to admit it or not. I’ve had a lot of awesome experiences, particularly in the world of academic computer science, where, as an outsider, I just write something, and then guess what? The author finds it. Because they Google themselves. You know? And they’re like… I’m going to Google this paper that I wrote. And see who’s… See what the pundits have to say about it. Pundit. And it feels really good. They’re like – hey, thanks for reading my paper. I’m like yo, thanks for writing that paper. That was cool. What other papers have you written? What other books should I read? That kind of thing. And forget what is practical. It’s important to pursue concepts past the point of simple applicability. Don’t just learn the thing and then use it and walk away from it. Once you’ve learned it or at least a part of it, you’re in a pretty good position to – I don’t know, understand how it was made, why it was made that way, make your own decisions about how that thing should have been made, make it yourself, never use it again. Talk about it on Twitter. Whatever.

And then remember what is practical. Because it’s good to ship software too, and to do things. As opposed to talking about them all the time. You can… Yeah. I’m going to stand by that. You should also do things. Don’t just talk about them. It’s really good. Doing things feels great. I’ve done it a couple times. So… Yeah. And then in the end, I guess the advice is to be at least a little bit more like Harry. Don’t be too much like him. He disregarded his life and his physical form more than I would want for any of you. So… Embrace the world around you, and learn. Get obsessed, and that’s it. Thank you.

(applause)

Lisp is Abstract Syntax

Wed, 16 Apr 2014 00:00:00 +0000

# #

Lisp is Abstract Syntax

I’m currently working through a bit of Andrew W. Appel’s Modern Compiler Implementation in ML both to help me get more solidly grounded in ML and to help me more deeply understand compilers and interpreters. At the end of the chapter on abstract syntax, in the further reading section, I came across this gem of a quote about Lisp and the origins of abstract syntax. I’m not sure if its a spurious tale or not[1], but it’s too good of a quote not to share:

“The notion of abstract syntax is due to McCarthy [1963], who designed the abstract syntax for Lisp [McCarthy et al. 1962]. The abstract syntax was intended to be used writing programs until designers could get around to creating a concrete syntax with human-readable punctuation (instead of Lots of Irritating Silly Parentheses), but programmers soon got used to programming directly in abstract syntax.”

If the syntax for Lisp was meant to be a stepping stone, but programmers adopting it prevented advancement, perhaps this was the first time that software engineering and computer science’s fundamental differences manifested themselves in a way that impacted future directions of both fields.[2]

Thanks to Tom Santero and Greg Pfeil for corrections.

Update: Corrected the wording a bit to reflect the reality in the footnote below.

Notes

[1] It’s not! I had forgotten about M-Expressions. Practical Common Lisp by Peter Seibel has a good bit about this.

[2] While it’s true that the adoption of S-Expressions did lead to macros and many other interesting things, Appel’s account remains valuable (or at least entertaining).

Read Code Every Day

Fri, 11 Apr 2014 00:00:00 +0000

# #

Read Code Every Day

My friend and Open Source hero John Resig recently wrote an excellent blog post called “Write Code Every Day.” I love its message, its optimism, and probably above all, that it is pragmatic, applicable advice.

In an attempt to further an agenda that I’ve been harping on for a while, I’d like to add another perspective:

“Read code every day.”

Much along the same lines as John, I have had issues prioritizing my time between work and home life, wanting to advance my learning without wanting to interrupt my ability to do good work during the day, and wanting to spend enough time with my family, because missing them is the absolute worst.

I was having a hard time focusing on topics I wanted to learn more deeply or skills that I wanted to acquire. I found that I was building, building, building these things up in my head when what I should have been doing is getting in the habit of reading, thinking, and writing.

It reminds me of the years I spent working on music as a hobby. I found that the only time that I could really make any progress was when my gear was set up - when I had a practice space or enough room in my house where I could just sit down and get to work, without the distractions involved in setting up.

Building a habit of reading code, (or math, or prose about code or math) will have the same impact - over time you will make progress, grow stronger, and get more done - without the guilt and bullshit.

Modeling and Pestering

Mon, 07 Apr 2014 00:00:00 +0000

# #

Modeling and Pestering

People often ask me where my obsession with reading started. I can say without hesitation that it comes from my parents, and that they had two primary techniques: modeling and pestering.

The modeling, which my mom was almost entirely responsible for, went something like this: she was always reading. When she had a spare moment, when we were at the beach, when we were on an airplane, on vacation, she read. All the time. I knew in the back of my head that my father had to have read at some point, since he had two Master’s degrees and I was pretty sure you couldn’t even get out of High School without reading, but I didn’t see him read much. He had a different technique.

My father relentlessly pestered me to read. This took the form of intervening in any situation where I wasn’t reading, and, you know…suggesting otherwise. As a thank you to him, and to illustrate my point, I thought I’d share one particular form of his pestering that stuck with me for this long.

While many parents shared the common love of a sports team with their children, hanging the posters of the baseball or football team in their rooms, my father, one day, did his own version of this. It was 1992, and I was 12 years old. I was resting during the day in my room when my dad knocked on the door. He had in his hands a freshly printed, accordion folded pile of dot matrix paper. I groaned, recognizing it as something lame he had printed in PrintShop that he wanted to hang on my wall. It was just that. He unfurled the poster, which read, simply:

“Just Do It”

Not as in, achieve your sports goal or become president of the United States. In this case, Just Do It meant specifically, every time you don’t want to keep reading, look up at this sign, and keep going. In other words:

“Just Do It”

A pretty simple message, but hey - it worked.

I don’t know if I would have continued on with my dedication to reading if I didn’t have both the modeling and the pestering, but if you’re looking for inspiration, I thought one more data point couldn’t hurt.

The Well-Rounded Technologist

Wed, 02 Apr 2014 00:00:00 +0000

# #

The Well-Rounded Technologist

My friend Robert Pitts tweeted this last night:

Remember, just because you work in or with tech, it doesn't have to define you.
— Boring Dracula (@rbxbx) April 2, 2014

and I wanted to post it here because it is very fucking good advice. Beyond the obvious message that contained, that we should not be defined by our jobs, I also contend that in order to be both successful and happy, you absolutely must have something else.

I’ve been having a lot of fun and success the last year or two writing and speaking about technology, and I firmly believe that the only thing that makes me an entertaining and interesting speaker is that I’ve seen other things. In terms of work, I’ve been a video store employee, run a concession stand at a public beach, taught Middle School and high school, and been a gallery assistant. But that’s not very exciting, is it?

In terms of my personal life, I have a wife and two children, I’ve dedicated years to learning and playing music and have been lucky enough to play for audiences all over the US and Europe. I read Science Fiction, study subcultures of all varieties, and generally just try to keep my head above the water, to make sure that all of my perceptions aren’t colored by the lenses of being just another person in technology.

Our culture, as people in technology, has been plagued by problems with diversity, economic inequality, and stagnant innovation. Perhaps if we show more of ourselves, remember that we are more than employable, more than productive, excited about more than the latest tech, than we can start to steer this ship in a better direction.

P.S. In the spirit of this post, I shouldn’t only link to Rob’s technical blog. He’s also a musician and his music can be found on Soundcloud.

On Programming and Cooking

Tue, 25 Mar 2014 00:00:00 +0000

# #

On Programming and Cooking

We should be reading fewer recipes and more food science.

Notes

[1] For an actual book on food science, check out Harold McGee’s On Food and Cooking. It is a fantastically effective book on how to cook because it teaches you the science and history of food – you don’t only get the how, you get the why.

Sales For Engineers, 1-50

Mon, 10 Mar 2014 00:00:00 +0000

# #

Sales For Engineers, 1-50

In my capacity as a software engineer for the last 7 years or so, I’ve done all kinds of interesting things. Recently, I’ve dove into sales and sales engineering at my new gig. I found that acquiring customers and understanding their behavior is very challenging, and is also a lot of fun. The typical attitudes held by software engineers about sales are shitty at best, uninformed at worst. I’m here to tell a different side of the story and hopefully offer some advice that will get people excited. If you like this list, I’ll make another one.

Note that this list comes from my experiences of selling a Business-to-Business Software as a Service product. It may not pertain to everyone, but it should be entertaining and informative nonetheless.

Sales matters. Embrace it now, you’ll be better off in the long run because you’ll understand how shit works.
If you won’t put time and effort in to understand sales, how can you expect people to do so with technology?
Don’t take being a “cost center” personally. Actually just kidding, that shit is offensive.
Learn some statistics for the love of Edsger.
You already know everything about computers, programming, software engineering, etc, so why not something new?
Measure everything. I’m going to say this a couple times.
Understand your customer because that is who you build your software for.
Everything you build is in service of the customer, no matter how invisible to you they are.
A qualified lead is a customer who is already interested.
Following from #9, a qualified lead isn’t going to be annoyed when you reach out to try and help them.
Following from #10, you should help them by figuring out if they’re a good fit for your product.
The value in your product is in the eye of the customer.
If you can’t confidently sell your product at the price you sell it for, either don’t sell it, or sell it for a different price.
Sales and the interaction between human beings is the ultimate distributed system.
You have an advantage in sales because you understand the product in a specific way.
You have an advantage in sales because you breathe numbers and functions.
You have an advantage in sales because you can see through the data.
You have a disadvantage in sales because you understand the product in a specific way.
You have a disadvantage in sales because you breathe numbers and functions.
You have a disadvantage in sales because you can see through the data.
You can make the data come to life.
Use tools to determine who to speak to.
Speak to them a lot.
Use the telephone.
Seriously, use the telephone.
Get excited about knowing your potential customers.
Put yourself in the position of the customer.
Measure everything about everything.
Make graphs, not too many, mostly 95th percentiles.
How many trial customers do you have right now? You should know.
How many trial customers are on the last day of their trial right now? You should know.
Get my point? You should know a lot about your trial customers. As much as possible.
How long is your trial? Why is it that long?
Pricing is really hard, you will fuck it up at least once.
Listen to your customers about pricing, but be confident in the value you provide for them.
Cheap products seem cheap.
Cheaper plans churn more quickly.
Net churn vs. Gross churn - learn about them.
There’s a lot of science in sales - check this out, for example
Bruce Hardie is amazing. Learn about him.
How do your competitors sell their product? Sell it better.
Sign up for new trials all the time, see how other companies do it.
Next time you get the opportunity to take a sales call, do it. See how it works from the other side.
Assume that your customer is interested, and help them figure out if the product is right for them.
Measure everything - had to say it again.
Annual pricing is awesome, provide it when you can.
How much does it cost you to acquire a new customer?
How much does a new customer bring you in revenue?
What is the typical amount of recurring revenue that a customer brings you?
Sales is fun. Enjoy it.

Thanks to JD Maturen and Bryan Helmkamp for inspiration and James Golick for corrections.

The Promise of Relational Programming

Sun, 02 Mar 2014 00:00:00 +0000

# #

The Promise of Relational Programming

Will Byrd’s work in programming languages is fascinating and is only getting deeper.[2] At the end of 2013, Will attended the Code Mesh conference to speak alongside another awesome researcher, Nada Amin, and was interviewed about his research on the side.[3] I have often recommended that people check out MiniKanren (the main language of Will’s focus) and other areas of relational programming or logic programming research, but I haven’t done a good job explaining what I thought was interesting about it.

I think the interview that was recently published at InfoQ is now my go-to resource for a good explanation of why people should care about Will’s work and relational programming in general. I recommend watching the video or reading the transcript, and I wanted to highlight a specific quote.

Will starts out this particular section of the inteview discussing program synthesis and ends up describing the big idea behind pursuing relational programming, relating his work to other paradigms in a way that I really enjoyed:

“It is a similar situation to Scheme or Lisp or ML, these functional languages which have been around for a long time where in theory, you are supposed to do everything purely and you are supposed to write everything in this nice style where you are not doing any mutation, but those languages have mutation that you can change variable state and stuff like that all the time, whenever you get stuck. When Haskell came along and made a commitment to purity because of the laziness of the language required it and because of that, there was a lot of innovation in using monads to represent mutable state in a more pure way. I see a similarity here with what is happening with miniKanren relational programming versus, say, Prolog and logic programming.”[1]

I love the connection that Will draws here, and I think that once you witness the magic of relational programs, you can start to see that widespread application of this kind of program facility would be a powerful thing indeed. The ability to apply relational, rule-based logic inside languages of any given paradigm is worth pursuing because that is often where both imperative and functional languages feel clumsy.

I think this is the beginning of an important dialogue from which we can all benefit greatly.

Works Cited

[1]“William Byrd on Logic and Relational Programming, miniKanren” Interview, William Byrd by Werner Schuster. Available here

[2]William E. Byrd’s homepage. http://webyrd.net/

[2]Nada Amin’s homepage. http://people.csail.mit.edu/namin/

Translating a Typechecker from 'Math' to OCaml

Fri, 28 Feb 2014 00:00:00 +0000

This post is one of a series on Types and Programming Languages by Benjamin C. Pierce.

Translating a Typechecker from ‘Math’ to OCaml

Chapters 9 and 10 of Benjamin C. Pierce’s Types and Programming Languages are dedicated to the presentation and implementation of a Simply Typed Lambda Calculus (STLC). Computation with the Lambda Calculus is extended with a syntactic and semantic type system that allows for the annotation and verification of a term’s type, while preserving the progress and preservation theorems that we discussed last time.

After coming across this idea many times – a Lambda Calculus with types – and not really having any idea what that means or why it is of any importance, it is very satisfying to read a thorough and well-executed explanation. My take-away is that the formal basis of the semantics of the Lambda Calculus can be extended to elucidate the fundamental role that types seem to play in how we reason about programs and what they mean. Applying type theory to the Lambda Calculus has provided some of the deepest insights regarding our intuition about questions like “What is computation?”

Most impressive and resonant to me amongst the material, however, is one bit that Pierce mentions towards the end of Chapter 10. After describing the calculus and discussing its implementation, Pierce notes that:

“The typechecking function typeof can be viewed as a direct translation of the typing rules for the Simply Typed Lambda-Calculus, or, more accurately, as a transcription of the inversion lemma.”

The idea that moving between the mathematical/symbolic expression and code could be so straightforward appeals to me greatly - I’ve read positions advocating this way of thinking about programs, researched the idea, and even written and spoken about some of the implications of taking a more foundationally formal approach to program construction and algorithm verification. I wanted to unpack this statement to see what it really meant.

What does it mean to translate a proof into code, though, specifically? Why is a transcription of the inversion lemma alone enough to implement the typechecker?

“…because it is the inversion lemma that tells us, for every syntactic form, exactly what conditions must hold in order for a term of this form to be well typed.”

Well, that’s pretty clear. The lemma is a statement of each of the possible forms in the system which defines how to create expressions that are well typed. Pierce provides the code version of the typing rules, dedicating a chapter to describing it. It is very instructive to see it after the system has been thoroughly and formally described, and in the spirit of the book, I’m going to walk through the lemma show the equivalent OCaml code, to see how well the translation idea works out, from words to code. The lambda calculus has variables, abstraction, application, boolean, and conditional forms to be typed, so we will see how each of these work - how they look in their formal description, and what the corresponding code looks like.

If you’d like to follow along, I suggest at least opening up the full OCaml definition of the typechecking function here, or better yet, download and check out the full source code for the interpreter, as provided by Pierce.

The New Typing Relation

There are a few new ideas that are worth noting before walking through the translation, which stem from the question of how to actually apply types to forms like abstraction and application. A new typing relation is introduced, which needs to be able to type forms more complex than t:T where t is a term and T is its type. The new relation appears in terms like this - Γ ⊢ t1:T1. I’ll describe the symbols now and their meaning through the description of each type of form in the STLC which we will be translating.

Γ is known as the typing context, which consists of “a sequence of variables and their types.” For explicitly typed languages, it is enough to assume that the declared type should hold in order for safety to be preserved. This is essentially the “environment” for type-relations which holds the known facts about types.

The ⊢ “Turnstile” operator reads “under the set of assumptions”

The : operator assigns a type to a term, like t1:T1 for “term t1 has type T1”

So that Γ ⊢ t1:T1 reads “Under the set of assumptions in Γ, the type of t1 is T1”

The typeof function

The function which we are exploring has the job of ensuring that declared programs are well-typed. That means that it upholds the formal rules that are encoded into the type system, and that it is provable that programs that are verified with this typechecker are well-typed. Having an OCaml implementation of this is very interesting because it allows us to see the size and shape of a function with such deeply encoded and knowable semantics. Each of the sections below walks through one of the potential forms that the typechecker will encounter as a program is typechecked. Put all together, it can typecheck any well-formed program in the STLC.

Typing variables

The first step toward creating a basic type system for the Lambda-Calculus is to provide a mechanism for the typing of variables. As Pierce explains, there are type systems with different approaches to determining the type safety of their terms. Explicitly typed languages, like the Simply Typed Lambda-Calculus, rely on annotations to determine if terms are type safe. Implicitly typed languages use analysis and type reconstruction to determine types. In order to keep things basic, we begin with an annotated, or explicit language where variables are declared along with their types.

The first step of the inversion lemma on the typing relation reads as follows:

If Γ ⊢ x:R, then x:R ∈ Γ.

First, let’s translate this from the propositional form to a less symbolic sentence.

“If variable x is declared with type R, then the typing context contains the mapping from x to R.”

And now, the line of code in the typeof function which deals with variables:

TmVar(fi,i,_) -> getTypeFromContext fi ctx i

OCaml’s pattern matching facilities come in handy to declutter the function a bit. Each step in the inversion lemma will be accompanied by a similarly shaped form:

MatchedPattern(values...) -> ... (* execute when MatchedPattern is matched *)

When the interpreter happens upon a variable (TmVar(fi,i,_)), it can derive its type by looking up the type by index in the typing context. The function which accomplishes this, getTypeFromContext, does just this. It’s a simple intuition that is matched quite nicely by the code.

Typing abstractions

Typing abstractions (or lambda expressions, or functions as they are known in programming languages) is probably the most interesting aspect of constructing a Simply-Typed Lambda Calculus. If you’ve seen type systems like those in Haskell or OCaml, you may be familiar with function type syntax like the following:

# let square x = x * x;; (* entered by the user *)
-> val square : int -> int = <fun> (* returned by the type inferencer *)

Above, the user enters the definition of the square function in the REPL, and the type inferencer, based on the types for the * multiplication function, guesses that it is a function that takes an int and returns an int, which is written int -> int. The formalism for the Simply-Typed Lambda Calculus uses similar notation. The -> operator is used to denote that a function accepts a value with the type on the left hand side of the arrow, and returns a value with the type on the right hand side of the arrow.

In the initial presentation of these ideas in the book, typing functions actually comes first, and the other definitions follow. Since up until Chapter 9, we have only seen simply typed expressions such as variables with declared types, the typing relation we have no longer holds, as we saw above. As Pierce states:

“This changes the typing relation from a two-place relation, t:T, to a three-place relation Γ ⊢ t:T, where Γ is a set of assumptions about the types of the free variables in t.”

The second step of the inversion lemma deals with these, and reads as follows:

If Γ ⊢ λx:T1.t2:R, then R = T1 -> R2 for some R2 with Γ, x:T1 ⊢ t2:R2.

“If a function is declared which expects a term of type T1 and returns a term t2, then the function has type T1->R2 where t2:R2.”

It’s nice to see the formal construction of the -> operator, so now let’s see the code:

TmAbs(fi,x,tyT1,t2) ->
  let ctx' = addbinding ctx x (VarBind(tyT1)) in
    let tyT2 = typeof ctx' t2 in
      TyArr(tyT1, tyT2)

When the matched pattern is an abstraction, declare an intermediate representation of the typing context, adding the binding declared on the input variable in the abstraction. Calculate the value of the type for t2 by looking it up in the context.

Typing application

Application (also known function application) shows how we can compose terms to perform computation. t1 t2 could be, for example λx:Boolean.x true, which would apply the identity function to the value true. Here’s how it looks in the lemma:

If Γ ⊢ t1 t2 :R, then there is some type T11 such that Γ ⊢ t1:T11 -> R 
and Γ ⊢ t2:T11.

This can be read as:

“If the application of t1 to t2 has type R, then there is some type T11 where T11 -> R. Additionally, t2 has type T11.”

It has taken me quite a bit of time to be able to work through the symbols, but I believe that this captures the intuition. The OCaml implementation can be seen here:

TmApp(fi,t1,t2) ->
  let tyT1 = typeof ctx t1 in
  let tyT2 = typeof ctx t2 in
  (match tyT1 with
      TyArr(tyT11,tyT12) -> (* looking for the Arrow Type for functions *)
        if (=) tyT2 tyT11 then tyT12
        else error fi "parameter type mismatch"
    | _ -> error fi "arrow type expected")

Here we see that typing function application boils down to checking the types of the two terms in question and looking for appropriate supporting types in the typing context. The types of t1 and t2 are stored and then the left hand side and right hand side of an arrow type for the function t1 are composed and searched for in the typing context. You can follow through by applying arguments to this function mentally and seeing what you arrive at - the errors are useful for guidance.

Typing Boolean values

Our Simply Typed Lambda-Calculus has been extended to include boolean values, which is a good thing, because otherwise it would be pretty boring. Typing boolean values is straightforward because they are built into the language. I’ll repeat the definitions for true and false for the sake of completeness:

If Γ ⊢ true : r, then R = Bool.

“If term r is true, then its type is Bool.”

TmTrue(fi) -> 
  TyBool

If the interpreter encounters the value true, we know it is of type Boolean.

If Γ ⊢ false : r, then R = Bool.

“If term r is false, then its type is Bool.”

TmFalse(fi) -> 
  TyBool

If the interpreter encounters the value false, we know it is of type Boolean.

Typing conditional statements

After we have typed variables, functions and their application, and booleans, we can type conditional statements. The Simply Typed Lambda-Calculus has the facility for an if...then...else statement, and here is its formal statement, the final step in the inversion lemma:

If Γ ⊢ if t1 then t2 else t3 : R, then Γ ⊢ t1 : Bool and Γ ⊢ t2, t3 : R.

As often as we have used if statements as programmers, we don’t always see them looking like this. Here’s an interpretation:

“If t1, t2, and t3 are the three parts of a well typed conditional, then t1 is a boolean and t2 and t3 are the same type.”

And now the code:

TmIf(fi,t1,t2,t3) ->
  if (=) (typeof ctx t1) TyBool then
    let tyT2 = typeof ctx t2 in
    if (=) tyT2 (typeof ctx t3) then tyT2
    else error fi "arms of conditional have different types"
  else error fi "guard of conditional not a boolean"

We check the type of the first term to make sure it is a boolean and then compare the types of t2 and t3. Again, seeing the error messages helps us see the form of the conditional in the code.

We now have conditionals! Nice. And with that, we can compute quite a bit.

Conclusion

Type systems in the wild are often large, scary, unapproachable things. Working through some relatively simple logic in “Math” and seeing how it manifests as code has helped me see that at their core, they are really not very different at all. Understanding the boundaries between theory and application is important for practitioners as well as theorists, and seeing how these systems are implemented is very instructive in this regard.

In my estimation, Pierce is correct and the idea of a translation does work. When the inversion lemma is written as OCaml code, it bears the realities of the system it runs on: it has to handle errors with words, use specific names for functions, etc. To me, that’s a beautiful thing.

Works Cited

All quotes unless otherwise cited from Pierce, all code from provided implementations.

[1] Benjamin C. Pierce. 2002. Types and Programming Languages. MIT Press, Cambridge, MA, USA.

Papers We Love: A Unified Theory of Garbage Collection

Mon, 24 Feb 2014 00:00:00 +0000

Papers We Love: A Unified Theory of Garbage Collection

I was invited to speak at the first meeting of the NYC Papers We Love meetup, on the awesome “A Unified Theory of Garbage Collection” paper by Bacon, Cheng, and Rajan. Video is here:

And here are the slides:

What is a Type System for?

Mon, 17 Feb 2014 00:00:00 +0000

This post is one of a series on Types and Programming Languages by Benjamin C. Pierce.

What is a Type System for?

Coming from an imperative and mostly dynamic language background, learning my way through the worlds of functional and statically typed programming languages has been a fascinating and intuition destroying exercise. Likewise, going knee-deep into the philosophical wonderland of Van Roy and Haridi’s Concepts, Techniques, and Models and then jumping into Pierce’s notoriously intense Types and Programming Languages (hereafter TAPL) has done way more to make my brain hurt than it has to actually improve how I write or at this point even understand code.

On the other hand, the hazy picture I have had about certain aspects of computation are becoming somewhat sharper, and studying TAPL has reminded me what is at the core of what I’m pursuing in studying Computer Science as a professional programmer. I want to understand more in order to accurately express programs in a way that will preserve my intended meaning without being too intrusive to construct.

Whether you prefer to use a programming language with an expressive type system or not, Pierce shows that types provide a way to understand deep constructs in computation. The systems constructed to formalize types as shown in this text have provided some of the deepest insights for me in my Computer Science self-pedagogy and I’m eager to share some of these with you. Before that, however, I thought it had to be asked:

“What is a type system for?”

This question is sort of central to a lot of other questions that you hear asked more often:

Why do some programming languages have advanced type systems while others do not?
What is the central purpose of a type system?
Why do I need types?

I’ll try to touch on these questions, but first, some symbols.

Some Symbols

In this post and the others on Pierce’s book, I’ll be using the following conventions for referring to terms, Types, and evaluation:

t is a term - something that we want to be evaluated in whichever system we’re working in.

T is a type - a dimension of a term that describes its possible values.

: is the typing relation, and t : T reads t is of type T

-> stands for term reduction and t -> t’ - reads t reduces to t’

A value is a primitive, or atomic element in a system - numbers and booleans are classic examples.

Safety, Progress, and Preservation

And now, on to that question - what is a type system for? Pierce states it nicely, in the context of the first simple type system discussed in TAPL:

“The most basic property of this type system or any other is safety (also called soundness): well-typed terms do not ‘go wrong.’”

That sounds great! I certainly don’t like it when my programs go wrong, and believe me, it happens quite often. But now that I think about it, what exactly does ‘go wrong’ mean in this context?

“…it means reaching a ‘stuck state’ that is not designated as a final value but where the evaluation rules do not tell us what to do next.”

Earlier in the book, Pierce discusses stuck states in a bit more detail, and equates them with runtime errors. This was a very deep connection for me. Stuck states are terms that are, according to the operational semantics of a given system, meaningless. Powerful stuff. Pierce continues:

“What we want to know, then, is that well-typed terms do not get stuck. We show this in two steps, commonly known as the progress and preservation theorems.”

The proof-heaviness of TAPL scared me off for a long time, and while stepping through the proofs is instructive, it isn’t necessary in this case - you can still form a nice intuition for the purpose of a type system without driving the details through your brain a million times.

The purpose of a type system is to provide safety, and safety consists of progress and preservation. So to get to the bottom of our question, we have to look at two theorems:

Progress: If t is a well-typed term (that is, t : T for some T), then either t is a value or else there is some t’ with t -> t’.

Preservation: If t : T and t -> t’ then t’ : T

In other words, if the program can continue executing, and each execution preserves types properly, then the program is safe. Or as Pierce says it:

“Safety = Progress + Preservation”

Not the most complex equation on the surface, but its tendrils are myriad. I encourage you to stare at the moon and think about that one for a while.

Falling for Types

While many professional programmers have opinions about dynamic vs. static programming languages or the need for expressive type systems, I don’t often see the central point of types being discussed. The idea that the purpose of a type system is to prevent undesirable application states eluded me for some time, and I believe this is the case for many other people as well.

As with any other aspect of the discipline, choosing to use a type system or not shouldn’t preclude you from learning more about them and understanding the inherent tradeoffs that are involved in their application. It is clear to me that expressive type systems like those found in OCaml and Haskell are rich and well-considered; they definitely have light to shed on problems encountered in almost any modern developers day-to-day work, but whether or not they can be unobtrusive, for me, remains to be seen.

Works Cited

All quotes unless otherwise cited from Pierce.

[1] Benjamin C. Pierce. 2002. Types and Programming Languages. MIT Press, Cambridge, MA, USA.

A Taste of Logic Programming In Ruby

Wed, 01 Jan 2014 00:00:00 +0000

# #

A Taste of Logic Programming In Ruby

In getting recently re-obsessed with Logic Programming and its relationship to the future of both programming and computation (with a healthy dose of obsession for its history, of course), I recalled coming across a few relevant Ruby libraries last year that I wanted to write up a bit after playing with them here and there. I was interested in seeing some small implementations of unification algorithms and logic programming systems, and bingo, I came across that and more in one person’s unassuming profile.

GitHub user jimwise who apparently is in the general part of the world as me but I have no idea who he is (the Internet is awesome) has a trio of libraries that implement unification a la Ehud & Shapiro, a small nondeterministic programming library that utilizes Ruby’s callcc functionality, and a small Prolog system also influenced by The Art of Prolog. Since most systems you see that tackle these problems are written in Scheme, Pascal, Haskell, etc, it was nice for me to see them in Ruby, which for better or worse is still very much my code lingua franca.

This post is a quick overview of what each of these libraries can do, and I encourage you to check them out if you’re curious about what bringing features of other languages to Ruby can look like, or even if you’re just interested in language implementation details in general.

unific - A Unification Engine

Unification is a concept that is at the heart of understanding how logic programming systems work. Toward that end, unific provides an interface to Logic Variables, a flexible Environment, and the ability to chain unifications. Here’s a small sample:

irb(main):008:0> a = Unific::Var.new("a")
=> #<Unific::Var:0x007f8c5a833ab8 @name="a">
irb(main):009:0> b = Unific::Var.new("b")
=> #<Unific::Var:0x007f8c5a828fa0 @name="b">
irb(main):010:0> e = Unific::unify([a, 1, 2], [0, 1, 2]).unify([a, b, 5], [0, 3, 5])
=> #<Unific::Env:0x007f8c5a0dec40...>
irb(main):014:0> e[a]
=> 0
irb(main):015:0> e[b]
=> 3

Does this smell like pattern matching to you? Unification is often described as “double-sided pattern matching,” so you’re on the right track if so. Also, what is a Var? If you’ve never experienced logic variables, stare at unific for a while, they’re an awesome way to think about how data can flow through a program.

You can also trace unification calls in unific to see how the engine is working, making the algorithm that much simpler to understand.

ambit - Nondeterministic Programming

The ambit library provides a platform for nondeterministic programming in Ruby. This programming paradigm is quite complex and a little bit abstract to be sure, but the the ambit README has a good introduction. I came across this particular library because it used in the rulog library that I talk about below. Also interesting is that ambit relies on MRI’s callcc functionality to implement nondeterministic choice. The code is interesting, so I’ve included it here. If you’re uninitiated, callcc is worthy of your study:

  # Given an enumerator, begin a generate-and-test process.
  #
  # Returns with the first member of the enumerator.  A later call to #fail!
  # on the same generator will backtrack and try the next value in the
  # enumerator, continuing from the point of this #choose as if that value
  # had been chosen originally.
  #
  # Multiple calls to #choose will nest, so that backtracking forms
  # a tree-like execution path
  #
  # calling #choose with no argument or an empty iterator 
  # is equivalent to calling #fail!
  def choose choices = []
    ch = choices.clone          # clone it in case it's modified by the caller
    ch.each do |choice|
      callcc do |cc|
        STDERR.print "choosing from " + choices.inspect + ": " if @trace > 0
        @paths.unshift cc
        STDERR.puts choice.inspect if @trace > 0
        return choice
      end
    end
    self.fail!                  # if we get here, we've exhausted the choices
  end

  # Indicate that the current combination of choices has 
  # failed, and roll execution back to the last #choose, 
  # continuing with the next choice.
  def fail!
    raise ChoicesExhausted.new if @paths.empty?
    cc = @paths.shift
    # if it quacks (or can be called) like a duck, call it -- it's either a Proc
    # from #mark or a Continuation from #choose
    cc.call
  end

If you think you’re seeing things, you’re probably starting to see how that code works. (Note: It’s not as scary as it looks, it’s scarier.)

rulog - A Prolog in Ruby

rulog is a Prolog engine in Ruby that relies on unific and ambit. I found rulog when searching for an implementation of a logic programming engine, because I needed to see the implementation to understand how it worked, or so I thought. It turns out that I actually needed to learn what the hell I was trying to learn - logic programming is such a foreign paradigm that when I first encountered it I had no idea what I was seeing.

Through exposure to books like Concepts, Techniques, and Models of Computer Programming, The Art of Prolog, and The Architecture of Symbolic Computers, I have started to see real value in exploring declarative programming and how it can intersect with modern, practical approaches to computation. While rulog is a simple and somewhat rudimentary Prolog, it is great to be able to see it up close, and the author has made some thoughtful design decisions.

Here is how you can define the rules for Peano’s Axioms - the number, equal, and plus relations:

rs1 = Rulog::rules(
   Rulog::declare{  num(0)  },
   Rulog::declare{  num(s(v(:x))) [ num(v(:x)) ]     },

   Rulog::declare{  equal(0, 0)  },
   Rulog::declare{  equal(s(v(:x)), s(v(:y))) [ equal(v(:x), v(:y)) ] },

   Rulog::declare{  plus(v(:x), 0, v(:x)) },
   Rulog::declare{  plus(v(:x), s(v(:y)), v(:z)) [
                         plus(s(v(:x)), v(:y), v(:z)) ] }
   )

Here, we turn rulog’s tracing to a verbose setting, and ask the engine to solve the equation x + 2 = 4:

irb(main):030:0> 3.times{ rs1.trace }
irb(main):030:0> rs1.solve(Rulog::declare{  plus(v(:x), s(s(0)), s(s(s(s(0))))) })

# marking!
# choosing!
# chose num(0).
# chose num(s(?x)) :- num(?x).
# chose equal(0, 0).
# chose equal(s(?x), s(?y)) :- equal(?x, ?y).
# chose plus(?x, 0, ?x).
# chose plus(?x, s(?y), ?z) :- plus(s(?x), ?y, ?z).
# marking!
# choosing!
# chose num(0).
# chose num(s(?x)) :- num(?x).
# chose equal(0, 0).
# chose equal(s(?x), s(?y)) :- equal(?x, ?y).
# chose plus(?x, 0, ?x).
# chose plus(?x, s(?y), ?z) :- plus(s(?x), ?y, ?z).
# marking!
# choosing!
# chose num(0).
# chose num(s(?x)) :- num(?x).
# chose equal(0, 0).
# chose equal(s(?x), s(?y)) :- equal(?x, ?y).
# chose plus(?x, 0, ?x).
# finished rule
# finished rule
# finished rule
# 
# goal:
#   plus(?x, s(s(0)), s(s(s(s(0)))))
# answer:
#   plus(s(s(0)), s(s(0)), s(s(s(s(0)))))

The author’s decision to put tracing in is absolutely awesome! We can see how the whole thing works.

In Conclusion

It is one of my goals in 2014 to get more people excited about logic programming and how we can leverage it in a multi-paradigm context. Embedding pattern matching, non-deterministic choice, and logic programming into an Object-Oriented language like Ruby has many interesting possibilities. We’ve seen some traction with these ideas in the Clojure community, and I hope this small tour of some easy to digest libraries by one awesome author helps get you excited about their potential.

Works Cited

Special thanks to Jim Wise, all code from his GitHub repos.

[1] Sterling, Leon and Ehud Shapiro, The Art of Prolog, MIT Press, 1994

A Certain Kind of Book and a Thank You

Tue, 31 Dec 2013 00:00:00 +0000

# #

A Certain Kind of Book and a Thank You

Computer Science is a deep discipline. The range of areas of specialization is dizzying and encompasses enough fascinating applications to keep anyone busy for a lifetime. Arguably, all of these areas rely on the basic understanding of the foundation of our field: computation. What is computation? Why is understanding it important, and how can we learn it from a book? It turns out you just need to find that certain kind of book.

That certain kind of book is one that has a point of view, that is obviously the product of a lot of thought, that has obviously made the rounds and kicked around for long enough that it’s here to stay. It has a system built in, answers as many questions as it can, and compromises as little as possible.

In other words, books work and are worth it. I’ve spent a decent amount of 2013 trying to understand what computers do without actually sitting in front of them, and I’m pretty sure it’s working.

As 2014 draws near, why not consider indulging in a year long project of reading one or more of the following books, which for me are that certain kind of book.

Structure and Interpretation of Computer Programs by Gerald Jay Sussman and Hal Abelson

Learn from the masters, through the lens of Scheme. Available online for free and a great place to start.

Concepts, Techniques, and Models of Computer Programming by Peter Van Roy and Seif Haridi

Take a small kernel language and evolve it beyond your imagination, implode paradigms, and break all rules. I recommend the hardcover edition, a highlighter, and several free months.

Essentials of Programming Languages by Daniel P. Friedman and Mitchell Wand

Learn to compute by learning to understand interpreters in Scheme. Dense and rich, I’m planning to work through this in 2014.

The Architecture of Symbolic Computers by Peter M. Kogge

Cast off the shackles of von Neumann and study the implementations of Functional and Logic programming languages. Worth the trouble to find it for a reasonable price. Another 2014 project.

Thank You

Finally, I’d like to say thank you all for your support and for reading my writing in 2013. Your comments and encouragement make it worthwhile.

Why We Need Explicit State

Tue, 10 Dec 2013 00:00:00 +0000

This post is part of a series exploring Concepts, Techniques, and Models of Computer Programming by Van Roy and Haridi. Check the blog index for more.

Why We Need Explicit State

The fact that the authors of Concepts, Techniques, and Models of Computer Programming wait until page 405 (about halfway through the book) to introduce the concept and implementation of explicit state is a joke that tells itself. Explicit, mutable state is the cause of enough confusion and complexity that entire languages and paradigms are based around the idea of avoiding it, and yet many beginning professional programmers and computer science students learn with it up front. Unfortunately, in the long run, the joke is on these beginners, who often struggle in a sea of complexity.

“Explicit State” is the title of Chapter 6 of Concepts, Techniques, and Models of Computer Programming. Far from a diatribe, the treatment of the subject is subtle in the hands of Van Roy and Haridi who use introducing explicit state as an opportunity to consider the pure world of declarative programming in the harsh realities of the real world.

CTM is a book that is distinguished amongst the Computer Science textbooks that preceded it in a variety of ways, from its early introduction to dataflow variables to its multi-paradigm approach. Certain aspects of this radical departure from previous literature at times seems like a direct reaction to certain books in particular, one of them being Abelson and Sussman’s epic The Structure and Interpretation of Computer Programs (SICP). Interestingly, in holding off on the discussion of state and trying to balance a description of its importance with warnings about its cost, the two books take very similar approaches. I’ll cover a bit of SICP’s approach as a way of reinforcing some of Van Roy and Haridi’s ideas in this post.

During the course of this tour of explicit state, the stateful model of computation is introduced, which bears a strong resemblance to the model used in Chapter 5’s message-passing model. A few different applications of the stateful model are discussed, including the implementation of Abstract Data Types, which see some very nice improvements over their declarative counterparts. Because complexity comes along with the introduction of new ideas to the computational model, ideas for handling it are discussed, and finally the fundamental challenge of the sequential nature of state is addressed.

What is State?

Before explaining the new techniques we have at our disposal, the authors want us to ask ourselves: What is State? Or maybe more specifically, What is a state?

“A state is a sequence of values in time that contains the intermediate results of a desired computation.”

The authors explain that there are two types of state - implicit or declarative state, and explicit state. We have seen implicit state before in the book, such as recursive functions that encapsulate values over time. However, the authors distinguish this type of state as implicit because the consumer of the function in that case is not concerned with the implementation detail as being stateful, it simply produces the desired computation.

Explicit state, on the other hand, is defined as:

“An explicit state in a procedure is a state whose lifetime extends over more than one procedure call without being present in the procedure’s arguments”

This kind of state wasn’t possible in other models that we’ve seen, and we have been warned repeatedly about the potential for complexity that allowing this semantics can introduce. Still, this type of programming with explicit state is ubiquitous and necessary, so understanding how it works is important.

The Stateful Model

The stateful model is a tweak on the declarative model of computation that adds a mutable store alongside the immutable store and the semantic stack that we see in the most basic kernel language implementation:

The Stateful Model of Computation[1]

Stateful programs in this model are always sequential. The “Semantic Stack” above represents the ordered operations that comprise a program written in the kernel language. The “Immutable Store” is present in all of the kernel languages we have seen so far, and is the main “environment” which preserves the stated relationships between immutable variables and values. New here is the “Mutable store” which allow a new syntactic construct enabling a powerful semantic addition to the language: mutable state. This requires the addition of a minimal number of operations to the kernel language:

The Stateful Model Kernel Language[1]

Two convenience methods for manipulating cells can make the language even more expressive, but amount to syntactic sugar for the Exchange operation:

The Stateful Model Kernel Language: Cell Operations[1]

With this new sematic device (the mutable store), we can create, assign value to, read value from, and swap the values of the contents of mutable cells. Here is a simple program that uses these new constructs:

local
  C ={NewCell 0}
in
  fun {SumList Xs S}
    C:=@C+1
    case Xs
    of nil then S
    [] X|Xr then {SumList Xr X+S}
    end
  end
  fun {SumCount} @C end
end

Here we can see a new Cell being created and used within the scope of a function. Repeated calls to SumList in the program will encapsulate the state of the Cell, allowing the consumer of the function to worry only about the value, and not the implementation details of the function.

This is the essence of the power available to us, and the reason that almost every practical programming languages is willing to pay the cost of introducing a technique that requires a radical rethinking of the implementation of the mechanics of computation.

One way to think of it is that as we “don’t care” about the implementation details of the SumList above, we also don’t have access to change it in a fundamental way. This tradeoff is something that is very well covered in Abelson and Sussman’s treatment of the same topic, so I’m going to take an opportunity to indulge a tangent in that direction.

The SICP Perspective

SICP is a book which is primarily concerned with teaching functional programming through the use of the Scheme programming language. Similarly to CTM, it spends a lot of time explicating the mechanics of computation, but through the lens of one paradigm, and primarily from a mathematical perspective. To best teach those that understood engineering and mathematical principles and get them to understand computation, the approach is taken to begin with functions that are pure, and to work from there. This probably sounds familiar if you’ve read the other posts on this blog about CTM.

To highlight the elegance of a bit of well-placed explicit state, the authors of SICP ask us to consider a random number generator and the use of that kind of function in a larger program, like a Monte Carlo simulation. Without explicit state, the following random number generator, written in Scheme, would not be able to provide the nice interface of just calling rand:

(define rand
  (let ((x random-init))
    (lambda ()
      (set! x (rand-update x))
      x)))

Instead, you would have to pass the current value of the random number generator each time, which quickly becomes a pain in even a trivial program like the one the authors demonstrate.

To get a sense for the fundamental change that takes place when SICP introduces the set! operation, it is important to realize that up until the point in the book where state is introduced, one mechanic (the substitution model) had been used to compute programs. This model has the same declarative, referentially transparent properties as the early models in CTM.

The authors of SICP are quick to praise the transparency and simplicity of the substitution model, and once state is introduced, they wax poetic about its loss. The following quote is from a lecture series the authors gave using SICP as a textbook. After demonstrating how encapsulated explicit state can behave as the function above does, Sussman relates that:

“And what we see is the same expression leads to two different answers, depending upon time. So demo is not a function, does not compute a mathematical function. In fact, you could also see why now, of course, this is the first place where the substitution model isn’t going to work. This kills the substitution model dead.”[3]

Anyone who has struggled with understanding why they find imperative programs so complex, especially at medium or large scale, should review the transitions in these two books to help them put a finer point on their understanding. Seeing the authors of two prominent texts articulate this idea so clearly, that complexity comes along with state and that state is a function of time was a revelation.

One aspect of the use of set! in Scheme that Sussman highlights can be seen in the following implementation of a factorial function that uses explicit state internally:

(define (factorial n)
  (let ((product 1)
        (counter 1))
    (define (iter)
      (if (> counter n)
          product
          (begin (set! product (* counter product)) ;; Assignment 1
                 (set! counter (+ counter 1))       ;; Assignment 2
                 (iter))))
    (iter)))

What Sussman highlights is that when writing this program, the programmer is forced to make the decision for what order to assign the variables in, as shown in the lines labeled Assignment 1 and Assignment 2 above. If you switch the orders of these two statements, which would preserve the semantic meaning of the program (what you want to do is set the product and increment the counter, the order is meaningless).you will introduce a bug. And as Sussman states:

“It’s a bug that wasn’t available until this moment, until we introduced something that had time in it.”[3]

I thought this was a fascinating way of stating what is at the heart of the complexity of large imperative programs. After Abelson and Sussman introduce this bug, in the footnotes of the textbook they write:

“In view of this, it is ironic that introductory programming is most often taught in a highly imperative style. This may be a vestige of a belief, common throughout the 1960s and 1970s, that programs that call procedures must inherently be less efficient than programs that perform assignments. (Steele 1977[4] debunks this argument.)”[5]

Tracing the preference for educational choice to decisions made based on uninformed understanding of computer architecture is probably too good to be true, but it is an interesting question, and one worth thinking about more - what does the tendency towards imperative programming languages in educational contexts buy us?

Abstract Data Types

Abelson and Sussman’s idea, that we pay for the costs of explicit state because it makes certain other aspects of practical programming that much more elegant and potentially modular, is echoed in CTM’s coverage of Data Types (ADTs), or “a set of values together with a set of operations on these values.” The authors had covered ADTs earlier in the declarative model, and demonstrated that there are various properties, such as openness and security, that define how a programmer interacts with instances through their interfaces. With the stateful model, however, things have changed:

“Now that we have added explicit state to the model, we can present a more complete set of techniques for doing data abstraction.”

There are eight ways to organize a data abstraction according to the book’s formulation, and many of them are demonstrated. A simple stack data structure is presented in the declarative style and then shown in other variations until the stateful version is shown. In all cases the stateful version looks like the modern practical programmer’s idea of an ADT, with a proper interface and knowable performance characteristics.

Polymorphism as it comes to operating on ADTs and parameter passing in the stateful model are covered, wherein the ideas of the costs and benefits of state are thoroughly examined. After all of these topics are under our belts, we get introduced to some nice implementations of “Stateful collections,” which are of great interest to modern programmers, who tend to prefer maps to lists.

Tuples, records, arrays, and dictionaries are designed to meet a common interface for access and setting content and position where appropriate. It is once we can encapsulate values within data type objects and make them conform to common interfaces that we really see the promise of explicit state. When well contained, it is indispensable, and though it would be nice to never need it, because everything would be declarative, we know that that just isn’t realistic.

Reasoning With State

After discussing the potential pitfalls of the stateful model, Section 6.6 discusses some ideas for tools that can be used to reason about stateful programs. The reason this is necessary is summed up nicely here:

“Programs that use state in a haphazard way are very difficult to understand. For example, if the state is visible throughout the whole program, then it can be assigned anywhere. The only way to reason is to consider the whole program at once. Practically speaking, this is impossible for big programs.”

Invariant assertions are introduced as a way to get back some of the transparency that comes with reasoning about declarative programs. The idea is that each operation associated with a data structure (as in any of the ADTs discussed above) would have pre- and post-conditions specified. These specifications would serve to bolster the design of the system and how a programmer can come to understand what its goals are. The fact that they are separate from the implementation is touted as an advantage that decouples the interface from its code. Assertions like this have been seen in the Eiffel programming language amongst others, and the possibilities of extending popular dynamic languages with these capabilities is worth exploring.

To offer a higher level of abstraction for reasoning about programs, proof rules for the stateful model (without aliasing) are shown, which is an intense read that I am still trying to form an intuition for. In one of the admittedly most gentle intersections of mathematical proof notation and computer science theory I’ve come across, the authors show the proof rules for binding, assignment, procedures (with and without external references) and loops. Doing this, they have built a sound foundation for formal reasoning with respect to the operational semantics of the kernel language. New concepts can be invented, proof rules created, and so on, and the authors state that this can be a means of reasoning about stateful programs. The interface for this is unclear to me, and I’ll need to do some more reading to really grasp at what level of abstraction it is supposed to be applied. Nonetheless it is worthwhile for how rules for one statement can be applied to others to form an intuitive system of related operations.

A Reason For State

The authors end the chapter with a pair of warning statements:

“The real world is parallel”

“The real world is distributed”

Both of these present challenges to stateful programming because of its sequential nature. If bringing this up under the heading of “A Reason For State” seems odd, my point is that these notions help us put a finer point on when we need to use state and how we should use it.

Programming languages that embrace distribution, like Erlang, naturally have a better handle on how to encapsulate state than languages that don’t. The process abstraction in Erlang, like the message passing model discussed in an earlier chapter of CTM, is an example of how we can isolate state in a way where reasoning about it becomes simpler, so you do not in fact have to “hold the whole program in your head.” Of course in practice this is difficult because programs in any language can be written in complex, order sensitive ways that defy logic, and so we must press on and continue to develop tools for reasoning about programs.

The practice of examining what state is and why we need it is a fundamental component of understanding practical computation. We have seen bits of state here and there, encapsulated in lazy computation, in message-passing concurrency, etc. The idea of understanding all of the complexities that state introduces is not to search for a way where it isn’t necessary, but to search for a system where dealing with it is tractable. One goal for modern programming languages is to create a system where applying explicit state where it is not necessary would not be an attractive or “easy” way of solving a problem.

State is a fundamental component of how we reason about computation and thus the tradeoffs we accept are worth it in the long run. How we introduce it into our programs can be positively impacted by a deeper intuition for how it works and how it got there in the first place.

Works Cited

All quotes unless otherwise cited from Van Roy and Haridi. All Scheme code above from [5]

[1] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

[2] Abelson, Harold and Sussman, Gerald Jay. Structure and Interpretation of Computer Programs Lecture Series 5A: Assignment, State, and Side-effects Available online here

[3] Eric Grimson, Peter Szolovits, and Trevor Darrell, 6.001 Structure and Interpretation of Computer Programs, Spring 2005. (Massachusetts Institute of Technology: MIT OpenCourseWare). http://ocw.mit.edu (accessed 12 02, 2013. License: Creative Commons Attribution-Noncommercial-Share Alike. PDF Of Lecture Transcript

[4] Steele, Guy. Debunking the ‘Expensive Procedure Call’ Myth, or, Procedure Call Implementations Considered Harmful, or, Lambda: The Ultimate GOTO MIT AI Memo #443 PDF available here

[5] Abelson, Harold and Sussman, Gerald Jay. Structure and Interpretation of Computer Programs, second edition. MIT Press, 1996. Available online here.

Knots and a Mathematical Notion of Equivalence

Sun, 24 Nov 2013 00:00:00 +0000

# #

Knots and a Mathematical Notion of Equivalence

A knot, like everything else in mathematics, has a very specific definition. Let’s look at a photo:

Left: An unknot, Right: A trefoil

Both of these are the kinds of knots that mathematicians interested in knots study: they are joined at the end. The unknot above is a trivial knot, the simplest kind of knot imaginable. The knot on the right is a trefoil, a simple sort of knot that is not equivalent to an unknot.[1][2]

Practical and theoretical computer science are both concerned with notions of equivalence. Simple values are easy to equate in almost any system, while the ability of a programming language to allow the programmer to reason about equality in simple but expressive ways is highly valued.

What do we mean when we consider equality of abstract types?

If you take the time to create the two knots above, you can get a physical sense for how they are not equivalent in the knot theory sense. This level of intuitive connection to foundational concepts of our theoretical frameworks is what we should strive for in our models as practical and theoretical computer scientists.

Works Cited

[1] http://en.wikipedia.org/wiki/Knot_theory

[2] “Terese Lite,” Excerpts from the book TERM REWRITING SYSTEMS by Terese. From the Course Term Rewriting Systems at the Vrije Universiteit, fall 2006.

Distributed Systems Archaeology: Ricon West, 2013.10.30

Fri, 22 Nov 2013 00:00:00 +0000

# #

Distributed Systems Archaeology: Ricon West, 2013.10.30

This post collects the available resources from the talk I gave on the history and future of distributed systems theory and practice at Basho’s Ricon West Conference in October 2013.

Video has been published and is available on YouTube:

And the rest of the resources can be found here:

Part one : The Mind
Part two: The Proof
Part three: The Market and the Future
Works Cited
Slides can be found here.

Distributed Systems Archaeology, Part Three

Thu, 21 Nov 2013 00:00:00 +0000

# #

Distributed Systems Archaeology, Part Three

This post presents the final part of a narrative form of the talk I gave at Ricon West 2013. Part one can be found here and part two can be found here. The works cited can be found here, and the slides can be found here. Video is forthcoming.

The Market

The impact of commerce on distributed systems research

So far we’ve discussed the philosophical and humanity-based origins of distributed systems research in the work of Licklider, Minsky and Hewitt, and the formal origins in the work of Dijkstra and Lynch.

One of the motivating factors in my decision to study distributed systems more deeply was that I needed to understand how they worked better in order to be professionally competent. From this I can extrapolate (not that I have to, people are complaining about this constantly) that many other developers are in similar situations, suddenly finding themselves interacting with distributed systems as their projects grow, or simply as they have the realization that they have in fact been distributed systems programmers all along.

I think the existence of this conference alone proves that there is great commercial interest in distributed systems in 2013, so an interesting question came to me - has this always been the case?

There were few explicit commercial applications in Minsky and Hewitt’s work, and though it could be argued that Licklider’s money from DARPA that helped to create the lab at MIT did come from somewhere, by all accounts, researchers were free to explore at a time when things seemed new, fresh, and possible. Dijkstra and Lynch also published at the fringe of commercialism for a long time, focusing on the formal and mathematical underpinnings of the field.

I was very happy to find, however, that someone I was very intent on studying, Leslie Lamport, did have very interesting interactions with industrial work on distributed systems early on. As I mentioned before, author’s web pages with lists of published works are goldmines for researchers, and it is worth nothing that Lamport’s deserves to be in the world wide web hall of fame for his. It is amazing.

Amongst the many gems on Lamport’s page is a paper from 1978 called SIFT: Design and Analysis of a Fault-Tolerant Computer for Aircraft Control. Lamport annotates this entry with the following quote:

“When it became clear that computers were going to be flying commercial aircraft, NASA began funding research to figure out how to make them reliable enough for the task.”

Part of that funding was the SIFT project, which was charged with the task of designing the hardware and software for the aircraft and formally verifying its correctness. Lamport explains that the technology described in this paper is notable because the distributed system that composed the airplanes hardware and software could tolerate malicious, also known as Byzantine faults.

In 2003, Lamport published High-Level Speciﬁcations: Lessons from Industry, a collaboration with an intel engineer he worked with to formally verify multiprocessor memory designs. Lamport has applied techniques of formal verification to a variety of industrial applications, and this is why he straddles the section on the Market and the section on Formalism. In this paper, Lamport claims that high Level Specifications, such as the tools provided by his TLA+ language are essential to verifying industrial systems, concurrent algorithms, and more.

TLA+ allows you to provide specifications which get “compiled” into proofs. In many ways, I feel that in the long run his work on TLA+, which makes proving systems more accessible, will be of great importance. I’ve seen mention of it being used at Amazon, for example, and in other industrial modern applications. It shows that Lamport has made the connection between the theory of distributed systems and one form of its practice, although a form of practice that is very different from what most of us do.

Another luminary in the field of distributed systems, Ken Birman, has had quite a bit to say over the years about the mixture of commercialism and research and its impact on practitioners. Birman is noted for his work on Virtual Synchrony and the Isis toolkit, which is very well covered by his own bit of archaeological work in 2010’s A History of the Virtual Synchrony Replication Model.

Virtual synchrony is a system for considering work in distributed systems and has had various formulations over the years. Birman’s flirtations with industrial application of distributed systems are storied - his work was used by the New York Stock Exchange, amongst other important clients, for many years. Additionally he is an outspoken, reflective writer who has participated in workshops and produced papers about the history and impact of distributed systems research.

To get a better sense of Birman’s involvement in industry, a famous exchange in the form of academic papers from 1993 between Birman and two other authors in the field, Cheriton and Skeen, can and should be consumed by fellow archaeologists. Cheriton and Skeen came out with Understanding the Limitations of Causally and Totally Ordered Communication, which Birman claimed was a thinly veiled attack against his work on Isis in A Response to Cheriton and Skeen’s Criticism of Causal and Totally Ordered Communication.

The interesting aspect of this exchange is that Birman indicts Cheriton and Skeen for having financial skin in the game, and for over-simplifying his work in order to prove a relatively lame point. Fascinating reading, and important in that it reminds us that researchers are living, breathing human beings who have to survive and want to advance their ideas.

Fast forward to the mid 2000’s and two more documents that have Birman’s name on them, 2006’s How the Hidden Hand Shapes the Market for Software Reliability, and 2008’s Towards a Cloud-Computing Research Agenda both contain critical looks at practitioners, researchers, and the market in general. In these works Birman urges his fellow researchers to pursue practical and thus humane solutions to the problems that actual people face. He has many interesting things to say, from the impact of the “impossibility” idea I discussed above to the blow that the applicability of transactions and database theory had on the field of software reliability.

Overall, however, the take away from his work is that we need to be aware of the impact that the market has on our work, and thus our lives.

To end the section on the market, I wanted to briefly touch on a phenomenon that has had a prolific impact on the theory and practice of distributed computing that is a direct result of commercialism: modern industrial research, such as the work produced at Google and Microsoft.

Google’s papers in particular have been crucial to the field and many practitioners who I spoke to in preparing for this talk point directly to these papers as the initial sources of interest and access for them. Here you have companies at a scale that most people will never see actually publishing the techniques they use to do the seemingly impossible.

This feedback loop between large companies and academia is seen by some as a mixed blessing, however. If the deep academic work pursued by some is considered inapplicable, how could the average practitioner ever hope to leverage the lessons of a company who counts synchronizing atomic clocks between international data centers among its problem areas?

The answer is that sometimes they are applicable, and sometimes they aren’t, and as usual it is up to the practitioner, who often has no training, to make this determination.

Okay, that was a lot. Now that I’ve covered each of the three threads, and exposed a few obvious sources of tension for the modern practitioner, I have two recommendations in the form of directions for the community to pursue: Language, and Humanity.

Programming Languages

In pursuing my archaeological project, I came across many many “languages for distributed computation,” and I also know of some interesting work going on right now in this field. However the idea that a “language for distributed computing” that isn’t Erlang could possibly exist is not known to many developers, and I think it is high time to destroy this myth.

Two books that I have been very fond of lately that are directly applicable to why I feel that it is important for researchers and practitioners to pursue the advancement of languages for distributed computation are Van Roy and Haridi’s Concepts, Techniques, and Models of Computer Programming and Carlos Varela’s Programming Distributed Computing Systems: A Foundational Approach.

Concepts, Techniques and Models, also known as CTM, and its accompanying whitepaper Teaching Programming With The Kernel Language Approach is a revolutionary Computer Science textbook that completely changed my brain and finally got me to understand the connection between comptuer programming and computer science, no easy task to be sure - just ask Dijkstra, or anyone unfortunate enough to work with me.

In the paper, Van Roy and Haridi state that Teaching programming in terms of a single paradigm or language has a detrimental effect on programmer competence and thus on program quality …and that is indeed how many practitioners are taught. They are taught to bend the will of the languages that are commercially popular to the needs of distributed computing at the same time that they are expected to learn the foundations of the problems themselves. This is catastrophic.

CTM is an important book for many reasons, chief amongst them being that it makes the reader realize that small, simple, understandable languages and formal models that can be evolved into more complex ones are very powerful for forming intuitions of problems in computer science. In the book you are exposed to a basic language with a simple underlying formal model that is made more and less advanced over time as various subjects are treated - state is added here and taken away, distribution is included when it is needed, etc.

Carlos Varela is an author who is clearly inspired by Van Roy and Haridi’s work, and his excellent book on distributed computation takes the position that understanding concurrent computation is essential to understanding distributed computation, and proceeds to elucidate various modern formal process calculi that he argues should be the basis for future languages.

Varela describes the terms distribution and mobility as essential properties for distributed models. Distribution is the idea that computation can occur in different locations and mobility is the idea that computation can move between these locations. The combination of distribution and mobility is what most modern developers are actually dealing with, but they simply do not have these tools.

In other words, from both Van Roy and Haridi and Varela’s work we can take the lessons that languages devoted to distribution are necessary to teach the concepts of distribution, and that there is great potential in formal models that encode the ideas of distribution and mobility that have not yet been directly applied in the operational semantics of a programming language.

Humanity

The most fruitful work that we have achieved in the field of Computer Science is a direct result of the application of resources towards the ends of furthering and better understanding humanity. It is a simple fact that the longer we ignore this reality, the more it is to our peril.

Two papers will end this talk. The first is On Proof and Progress in Mathematics by the mathematician William Thurston. I came across this paper when Thurston died and it was recommended to me by many smart people on the internet, which is the way that I discover most of the interesting things that I read. I wasn’t sure what to expect but I certainly wasn’t prepared. This paper is an absolute brain-breaking work of painful beauty, and I won’t say much about it besides the fact that everyone here should read it, and that it contains keys to the questions I’m trying to bring to your attention in this talk. As a short summary, however, Thurston deals with the idea that “progress” in mathematics is often measured by proof, and attempts to understand the impact that it has. Section 4 “What is a proof,” in particular also has direct applicability to computer science researchers and practitioners because of Thurston’s fondness for technology - he often integrated computers into his proofs.

Lastly, a paper that I learned about very recently, Papadamitriou’s Database metatheory: asking the big queries hits on many of the notes that I’ve brought up here in a much deeper and more intelligent way. The author discusses the impact of the definition of the field by negative proof and compare’s Kuhn’s theory of revolutions in natural sciences to Computer Science - definitely a worthwhile read. Papadamitriou’s attempts to understand and contextualize the work done by researchers at various points in innovation cycles is a poignant reminder that our place in time impacts what we do and how effectively we do it.

In Conclusion

The Distributed Systems "community triangle"

Given all of the lessons above, my hopes for the future can be summed up as follows:

Practitioners: To the best of your ability, recognize the formalisms your work is based on, understand the details of the papers you’re reading, and be careful with how you communicate these ideas to your peers.
Academics: Guide the community and strike a balance between alleviating current pain and making the future path clear.
Industrial Researchers: Provide complete implementation details in papers, be generous with your Open Source contributions, try to give advice directly to practitioners.

In conclusion, distributed systems is an incredibly deep and rich field. Studying it has been absolutely thrilling and in addition to a fascinating body of artifacts that are ripe for more archaeological work, the community is generous, motivated, and forward-thinking.

I hope this talk inspires you to be reflective about the challenges of programming and understanding distributed systems regardless of your position in the “triangle” above, and remember, together we can do some amazing things.

Distributed Systems Archaeology, Part Two

Wed, 20 Nov 2013 00:00:00 +0000

# #

Distributed Systems Archaeology, Part Two

This post presents the second part of a narrative form of the talk I gave at Ricon West 2013. Part one can be found here. The works cited can be found here, and the slides can be found here. Video is forthcoming.

The Proof: Formalism

The role of the proof in the history of distributed systems research

In parallel with the work on artificial intelligence at prominent universities around the world, computer science was growing as a field, and the lines between mathematics and its application to technology were being teased out.

As a backdrop, it’s important to state that at the time I started researching this work, I had a hard time grasping how you could formalize something as complex as a distributed system. It seemed like too many moving parts, like it was too abstract. I didn’t understand the techniques used to apply something like a mathematical proof, wherein you have to “reinvent the entire universe” to something like distributed systems. The work of three individuals helped me understand these techniques.

Edsger Dijkstra
Leslie Lamport
Nancy Lynch

I’ll discuss Dijkstra and Lynch in this section, and move on to Leslie Lamport in a little while.

Edsger Dijkstra’s place as the most prominent name in the field of distributed systems is both fitting and telling. In 1965 he published a one page paper titled Solution of a Problem in Concurrent Programming Control where he describes an algorithm for mutual exclusion (known as a mutex), a technique that had not been well known up until that point.

It is with Dijkstra’s work that we see the beginnings of the thread of the prominence of formalism in the field of distributed systems. Dijkstra authored many important papers and was rigorous and thoughtful about proving the work that he produced, and over the course of his career was very vocal about the importance of formalism to computer science.

Dijkstra published another seminal paper Self-stabilizing Systems in Spite of Distributed Control in 1974 which was also ahead of its time, rigorous, and elegant. Self-stabilization became its own area of study in distributed systems research, and Dijkstra set the stage for the next generation of computer scientists, beginning a thread that was picked up by many.

Perhaps chief amongst those are that followed in Dijkstra’s footsteps are Nancy Lynch and Leslie Lamport, both of whom have provided a continuous stream of impressive theoretical work in the field. I want to highlight them specifically because of how closely we still feel the impact of their research, and because the specific results that they produced were incredibly groundbreaking and influential. Unfortunately, they are also often misunderstood.

Nancy Lynch was one of the authors of 1982’s Impossibility of distributed consensus with one faulty process which set a major direction in distributed systems research by proving that in an asynchronous network, distributed consensus was “impossible” (for a very specific definition of impossible). The impact of this paper is manifold, and much other work is based on its ideas. It is interesting to consider, and I will bring this up again a bit later, that an impossibility proof, also known as a “negative result” should be so important to defining the work in our field.

Towards the end of the paper, Dr. Lynch states that:

“These results do not show that such problems cannot be ‘solved’ in practice; rather, they point up the need for more refined models of distributed computing.”

It would seem to me that not many people make it this far, because this is hardly the subtle point that is made when this paper is discussed.

In 1989 Lynch published A Hundred Impossibility Proofs for Distributed Computing which is an awesome paper that I stumbled upon while digging through the list of publications on Dr. Lynch’s website. As an aside, the publications list on researcher’s pages are some of the best sources of information for archaeological digs. What I found so great about “A hundred impossibility proofs” is how Lynch playfully collects, distills and reports on the work in the field up until that point, a field of which she was a pioneer. Amongst the One Hundred proofs that Lynch surveys, she finds that they all have but one thing in common. They are all concerned with,

“The limitation imposed by local knowledge in a distributed system”

In other words, asynchronous networks and the potential for failure in other nodes makes certain assumptions impossible. By 1989, over 100 papers had been published based on these ideas. What kind of influence did that have on those who followed?

In 2002, Lynch published Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services, which contained a proof Brewer’s CAP conjecture - it was now known as the CAP theorem. The impact of this paper is still present and in fact many people at this conference will likely be discussing this result, 11 years later.

By the time I looked into this paper, I was already searching for those assumptions - what are those frozen FACTS about the system that allow the result to be produced? The proof in the paper uses the terms Consistency, Availability, and Partition-tolerance, three terms which are very carefully defined as part of a formal model. In other words, the C, A, and P in CAP are to be understood for very specific definitions of those terms and are not to be applied generally. Unfortunately, it turns out that it is very easy to apply these ideas generally and get confused about the meanings of them. You can end up seeing C, A, and P everywhere you look.

The point I am trying to make here is that while the work of Dijkstra and Lynch have been a huge part of establishing an indispensable foundation of theoretical work in the field, it is also very easy to get confused about what these results really mean. It is perhaps an irony that those who actually produce rigorous proofs are those that are the targets for the most misunderstanding as a result. When something is “proven,” people tend to treat it as gospel, divorcing the result from the proof and the work that went into it.

Even worse, perhaps, is that the elegance and simplicity of the formal models is completely lost to those who hear the information, telephone-style, from bad implementations, misunderstood blog posts, and other misguided studies.

I admit that as I read this work for the first time, the point didn’t really hit home. It was only later, upon reading some more modern work, that I started to form an intuition for the relationship between the underlying formal model assumed in academic work and the applicability of its results to practitioners.

Dijkstra and Lynch, in addition to Leslie Lamport, whose work I am going to cover in the next section, have produced incredible bodies of work that have helped to define the field of distributed systems and hoped to imbue a sense of responsibility on the part of academics to provide a formal model for their work.

Next, we will look at ‘The Market: The impact of commerce on distributed systems research, and I’ll wrap things up. Stay tuned.

Distributed Systems Archaeology, Part One

Tue, 19 Nov 2013 00:00:00 +0000

# #

Distributed Systems Archaeology, Part One

This post presents the first part of a narrative form of the talk I gave at Ricon West 2013. The works cited can be found here, and the slides can be found here. Video is forthcoming.

Introduction

My name is Michael Bernstein, and I’m Obsessed. Not with anything in particular, just in general.

This talk is called “Distributed Systems Archaeology,” a phrase I designed to be a catchy talk title before actually thinking through what it meant. The more I thought about it, however, the more I convinced myself that it had meaning, and so here we are.

If Archaeology is “the study of human history through the artifacts,” then “Distributed Systems Archaeology” is an attempt to apply an artifact-driven approach to understanding the history of the field of Distributed Systems.

Why would I want to do that?

In order to understand that you’d have to know more than you want to know about me, so I won’t say more than this: during the course of my career as a programmer at startups, I have found it impossible to avoid interacting with and producing distributed systems. They’re everywhere.

In addition to their overwhelming ubiquity, I also found reasoning about distributed systems very difficult. I didn’t understand how the tools worked or even what tools existed, and even worse, no one around me really had any more experience than I did in the matter.

To top it off, the resources available to people in my position, which are practitioners who are trying to help build small companies and turn them into medium sized and hopefully giant companies, are scant.

There were some merciful exceptions, like Jeff Hodges’ Notes on Distributed Systems For Young bloods, which begins with this quote:

“I’ve been thinking about the lessons that distributed systems engineers learn on the job.”

I read that and thought, “well shit, I’ve been thinking a lot about that too!” In this essay Hodges states that while developers will come across literature, they will not come across applicable lessons. This was my experience, and this delta, between the literature and the lessons is what I am trying to understand.

So I saturated myself in the literature in order to get some clarity around what the field was all about, and why it is so challenging to interact with distributed systems precisely at a time when we need them to be easy.

To help answer this question, I started out by searching for lists of classic papers and books, reaching out to experts, and absorbing as much as I could. As I started to get a sense for the field, I started to recognize some common threads as I traced the influences and trends.

I’m going to use these threads to tell a story about three influences on the field of distributed systems research and practice:

The Mind
The Proof
The Market

Each of these categories represents a common thread that I have traced through the literature, and each of these threads contained information and wisdom that helped me understand where we started, and how we got to where we are today.

The Mind: Artificial Intelligence

The roots of distributed systems research in artificial intelligence

First, the mind. Around the time I pitched this talk, I was heavily influenced by trawling through the archives of the AI Group at MIT, particularly the insanely rich and amazing group of documents known as the “AI Memos.” Amongst these memos there are papers by many of my intellectual heroes, dealing with a very wide range of topics. Since the origins of the research funding for computer research was heavily based in Artificial Intelligence, it makes sense that some of the most forward thinking research and thought at the time was coming out of this laboratory.

AI researchers had access to hardware and also the freedom to dream about what the future of hardware and software, and how they interact, would look like. AI had a heavy influence on the origins of distributed systems and provided the breeding ground for some of its brightest thinkers.

My dream history of distributed systems starts a very long time ago (I’m going to publish an extended bibliography), but for the sake of time, I’ve chosen to focus on three sources: Licklider, Minsky, and Hewitt. Their works is fundamental to the field of artificial intelligence, and in my estimation, to distributed systems.

Licklider and Minsky each contributed massive amounts of energy and numerous novel ideas to the field of artificial intelligence and computer science in general, helping to provide the foundation for the very definition of hackers, software, networks, the internet, and more.

One of the best anecdotes about Licklider I’ve heard came from an interview I watched with Tracy Licklider, JCR Licklider’s son. According to Tracy, Licklider was an eccentric. He did eccentric things like drink Coke for breakfast. He did eccentric things like dreaming about connecting the entire human race and abolishing the barriers to knowledge inherent in an unbalanced society. He also did eccentric things like calling that project the Intergalactic Computer Network because he knew the scientists he was pitching to would think it was funny.

Licklider spoke of Man-Computer Symbiosis and truly believed in the distributed future.

His The Computer as a Communication Device was a futuristic take on technology. Licklider spread money around to big thinkers, started programs, and believed, above all, in humanity.

Here is an image from “The computer as a communication device” where Licklider is trying to work out how information could travel through networks. We see terms like “nodes,” “ports,” “message processor.” He is using biology as an inspiration for modeling the flow of information:

Licklider imagines computational biology

Among the programs that Licklider helped to fund was project MAC at MIT, where Minsky was a teacher and Carl Hewitt was a student. Minsky’s interests were oriented toward understand humanity as well - his obsession was the brain and the mind. Minsky pushed the Artificial Intelligence agenda and inspired the use of computers to help understand and extend the human experience.

In 1970 he published the 1968-1969 Progress Report which covers the depth of the work going on at the MIT lab at the time. Work began to transition from purely theoretical, to implementation based, and back. The primary concern was understanding how computers and humans could interact. This brief summary does Minsky no justice - please read about him to discover more.

Carl Hewitt’s work is an inflection point for me in the history of distributed systems and his ideas are familiar to many of the people at this conference even if they don’t know it.

In 1973 he published A Universal Modular ACTOR Formalism for Artificial Intelligence which defined the Actor model of Computation and discussed his language PLANNER. The actor model describes computation as being performed by a single kind of object, known as an actor.

Actors have simple rules which allow them to perform general purpose computations, and because of the loosely coupled nature their definition and implementation, they can be inherently distributed.

In 1976 Hewitt published a paper, Viewing Control Structures as Patterns of Passing Messages (A.I. memo #410) which discussed the work he had done up to that point with the actor model and message passing. Although there were several technical achievements being pursued and accomplished, the work Hewitt was doing was still primarily framed in human terms - “Modelling an intelligent person” and “Modeling a society of experts” are both stated goals early on in the paper.

The actor model turned out to be one of the most enduring ideas produced from the work done on AI at MIT. Modern programming languages like Erlang have a semantics that is based on the actor model, which has been formalized several times. Hewitt’s work is an inflection point for me in the history of distributed systems because of how implementation based it is - there is formalism, there is code, there is hardware.

To recap, Licklider, Minsky, and Hewitt all were inherently concerned with the distributed nature of knowledge and how technology could be used to solve a wide variety of problems. The origins of the field of distributed systems can be found here, at the very intersection of technology and humanity.

Next, we will look at ‘Formalism: The role of the proof the history of distributed systems research. Stay tuned.

The Only Sure Thing in Computer Science

Wed, 13 Nov 2013 00:00:00 +0000

The Only Sure Thing in Computer Science

Everything is a tradeoff.[1]

Works Cited

[1] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

Distributed Systems Archaeology: Works Cited

Wed, 06 Nov 2013 00:00:00 +0000

# #

Distributed Systems Archaeology: Works Cited

I have received many requests for a list of papers and books from my Ricon talk with links for downloading where applicable. In all but two cases, these works are freely available. The two which aren’t are newer, commercially available books which I encourage you to pursue and purchase if possible.

Please note that for brevity, these works are listed by their primary authors but in many cases there were other authors involved. Additionally, many of the links below are direct downloads, please be prepared for that fact.

Where possible I have linked to the author’s publication pages.

Hodges, J “Notes on Distributed Systems for Young Bloods”
Licklider, JCR “Memorandum For Members and Affiliates of the Intergalactic Computer Network”
Licklider, JCR “Man-Computer Symbiosis”
Licklider, JCR “The Computer as a Communication Device”
Minsky, M “1968-1969 Progress Report”
Hewitt, C “A Universal Modular ACTOR Formalism for Artificial Intelligence”
Hewitt, C “Viewing Control Structures as Patterns of Passing Messages”
Djikstra, E “Solution of a Problem in Concurrent Programming Control”
Dijkstra, E “Self-stabilizing Systems in Spite of Distributed Control”
Lynch, N “Impossibility of distributed consensus with one faulty process”
Lynch, N “A Hundred Impossibility Proofs for Distributed Computing”
Lynch, N “Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services”
Lamport, L “SIFT: Design and Analysis of a Fault-Tolerant Computer for Aircraft Control “
Lamport, L “High-Level Specifications: Lessons from Industry”
Birman, K “A History of the Virtual Synchrony Replication Model”
Cheriton & Skeen “Understanding the Limitations of Causally and Totally Ordered Communication”
Birman, K “A Response to Cheriton and Skeen’s Criticism of Causal and Totally Ordered Communication”
Birman, K “How the Hidden Hand Shapes the Market for Software Reliability”
Birkman, K “Towards a Cloud Computing Research Agenda”
Van Roy & Haridi “Concepts, Techniques, and Models of Computer Programming”
Varela, C “Programming Distributed Computing Systems: A Foundational Approach”
Thurston, W “On Proof and Progress in Mathematics”
Papadamitriou, C “Database Metatheory: Asking the Big Queries”

The Unexpected Outcomes Of Code Review

Wed, 16 Oct 2013 00:00:00 +0000

This post originally appeared on the Code Climate blog, follow this link to join the discussion there.

#
#

The Unexpected Outcomes Of Code Review

Modern software developers have many tools at their disposal to help them feel confident that the code that they produce is high-quality and defect free. Amongst the most valuable of these tools are those that aid in inspecting source code written by another developer, a practice known as code review.

While these tools are becoming more pervasive, their value is difficult to quantify and the motivations of those who use them are not well known. Alberto Bacchelli and Christian Bird of Microsoft Research recently published a paper, “Expectations, Outcomes, and Challenges of Modern Code Review” which concluded that while most developers feel that code review is meant to find bugs in code, the process reveals that other outcomes are more prevalent, as this chart from the paper illustrates:

The 'Outcomes' chart: ranked categories extracted
from a sampling of code review data.

In this blog post I will briefly introduce the authors’ concept of “modern code review,” discuss the techniques and findings of this fascinating study, and discuss the unseen benefits of this process.

What is “modern code review?”

Code review has its origins in the formal practice known as a code inspection, introduced by Fagan in 1976 and described by Bachhelli and Bird as “cumbersome, time-consuming, and synchronous.” It involved line-by-line break-downs of code, lengthy explanations, people in suits, and other undesirable traits.

In contrast, the authors define modern code review as a form of code inspection which has the qualities of being informal, tool-based, and frequent. The most well-known engineering organizations today typically participate in a whole lot of modern code review: Microsoft, Google, and Facebook not only advocate this practice but have all authored their own tools for the purpose.

How the study was conducted

The study sets out to answer two primary questions:

What motivates developers to participate in (or managers to advocate for) code review?
How does code review actually benefit those who participate in or advocate for it?

Bachhelli and Bird relied on access to employees, previous work, and data from Microsoft to conduct their study. They dealt with the use of one specific tool: Microsoft CodeFlow, an internal interactive tool that creates a central location where the discussion of code and the subsequent fixes can be discussed either in real-time or asynchronously. CodeFlow also indexes data for searchability, a fact the authors relied on heavily in determining their findings.

In addition to using data from a previous study, the authors observed and interviewed “17 developers with various degrees of experience and seniority across 16 separate product teams with distinct reviewing cultures and policies.” Additionally they “manually inspected and classified the content of 570 comments” from CodeFlow, and surveyed 165 managers and 873 programmers.

Code improvements vs. finding defects

What the authors found was that while a large portion of developers, managers, and testers are motivated to perform code review because they hope to find bugs, the results of the study show otherwise. After categorizing the discussions in actual code reviews, finding defects ranks fourth, as shown in the chart above. To contrast the outcomes with the motivations, here is a graph that shows the motivations of the developer segment of those interviewed:

The 'Motivations' chart: the ranked motivation categories
from the developer segment of the interviews.

The top answer for the largest number of developers was “Finding defects,” and it ranked second overall amongst developers. The most prevalent topic discussed in code reviews ostensibly pertains to “code improvements:” comments or changes about code in terms of readability, commenting, consistency, dead code removal, etc.

This mismatch naturally led the researchers to wonder why this delta was so severe. The paper points to the idea that the challenges in finding bugs in other’s code essentially boils down to one idea: understanding.

Being capable of internalizing and understanding code is essential to being able to review code properly and find bugs, yet today’s large codebases and fragmented engineering organization make this difficult. As a result, an impedance mismatch is created that prevents decent overarching defect detection from occurring.

Conclusion

While there is a significant difference across the Microsoft organization between the expectations and outcomes of code review, this is not an entirely negative thing. Although we don’t seem to be getting the exact value from code reviews that we expect, we may be getting quite a bit more.

The paper states that in addition to a modicum of somewhat superficial bug fixes, teams get benefits such as “knowledge transfer, team awareness, and improved solutions to problems” from participating in modern code reviews. Additionally, the idea that code review is good at “educating new developers about code writing” is a compelling point.

So without knowing it, developers, managers, and testers rely on code review for a wide variety of purposes including communicating the design goals of implementations, enforcing style, and more. If we tweak our expectations and prepare for the inevitable outcomes, over time we can improve our code and use our tools to maximum effect.

Works Cited

All quotes and figures from:

[1] Alberto Bacchelli and Christian Bird “Expectations, Outcomes, and Challenges of Modern Code Review” May 2013, Proceedings of the International Conference on Software Engineering.

Available online here.

Formalizing Concurrency, Distribution, and Mobility

Mon, 23 Sep 2013 00:00:00 +0000

# #

Formalizing Concurrency, Distribution, and Mobility

Carlos A. Varela’s Programming Distributed Computing Systems: A Foundational Approach is a slim but impressive volume which takes a methodical approach to exploring the landscape of distributed computation, with very rich results.[1] A major premise of the book, that an awareness and understanding of the various models of concurrent computation is necessary for reasoning about various methods of distributed computation inspired this post, which will cover the major models used.

A computational model is a formal mathematical model that is used as a representation of computation, the most prominent example being the the λ (lambda) calculus, developed by Church and Kleene in the 1930s. Varela’s coverage of the models of computation is carried out with precision, and is based on the idea that, at an even higher level than the discussion of distributed computing:

“Theoretical models of computation are critical to be able to reason about properties of software, including the programming languages used to develop the software. Examples of software properties include expressive power, correctness, performance and reliability.”

Nothing revolutionary perhaps, but no less meaningful. Considering that each of the five models introduced could alone occupy several years if not a lifetime of research, this post aims to give an overview, and to provide some insight into the relationship that these models have to each other. Many of the most practical lessons one can take away from theoretical computer science result from comparing and contrasting computational models, and that practice is at the heart of this book.

Covered in the text are the λ (lambda) calculus, the π (pi) calculus, the actor model, the join calculus, and the ambient calculus. Most distributed computing work uses ideas taken from the π calculus or the actor model, though others are becoming more prominent. For me the impressive aspect of Varela’s work is how through rigorous comparisons and painstaking research, he shows the direct relationships between the computational models and the semantics of the programming languages which implement them. The direct encoding of the concept of concurrency into the post λ calculus models is an abstract but powerful concept that Varela communicates quite effectively.

The λ Calculus

The earliest and best known model is notable for its ability to concisely express sequential computation. While the λ calculus does not explicitly encode concurrency, it is the basis for all other models (even appearing “inside” the actor model) and understanding sequential computation is crucial for understanding concurrent computation.

The λ calculus (like all of the other models presented) is Turing-complete, that is “any computable function can be expressed and evaluated using the calculus.” The usefulness of a formal model like the λ calculus is that its simplicity and high level of abstraction make it very easy to reason about. This quality makes it well suited for studying and formalizing even complex programming language concepts.

A direct connection with programming languages is made in covering the various evaluation orders that have been explored for the λ calculus. Evaluation order refers to the rules for reducing or “solving” λ calculus expressions. This concept maps cleanly onto the semantics of modern functional programming languages. Normal-order is the simplest in many ways and is the “best-behaved” with respect to termination guarantees, but most programming languages use applicative order, for efficiency reasons. The Haskell programming language is well known for implementing a by-need or lazy evaluation order where partially computed functions are passed around to represent the results of not-yet-needed work. The following trivial example shows how values are passed in to a function and evaluated:

(λx.x*2 2)
2*2
4

The value of 2 is captured by the free variable x in a substitution. The function x*2 becomes 2*2 which becomes 4. This should start to give you a feel for the semantics of the calculus, which are simple but very powerful, but by no means represents what is possible. If the λ calculus is new to you, seek its beauty elsewhere.

In addition to evaluation order, other familiar concepts to functional programmers are covered included Currying, higher-order programming, sequencing, and recursion. Finally some details on how numbers and booleans can be represented in the calculus are discussed, which is a fascinating bit of detail that many professional programmers might not be familiar with. Combinators used to express numbers and true and false as a curried function is not day-to-day programming, but can provide some deep insights into the foundations of the systems we become intimately familiar with through use.

While it is amazing how deeply the model tracks with a host of mathematical concepts, as the author proceeds to introduce other models, the depth of the fundamental hook of explicitly representing multiple processes, which is missing from this model, becomes clear.

The π Calculus

The π Calculus is a formal model for specification, analysis and verification of systems composed of communicating concurrent processes, developed by Robin Milner et al in 1992, as a continuation of Milner’s earlier work in specifying a process calculus known as the calculus of communicating systems (CCS). In the π calculus, concurrent computation is modeled as processes communicating over shared channels. The operational semantics of the π calculus is presented, which explains how a well-defined set of inference rules determine the evaluation order and scoping rules of a π calculus expression.

The π calculus syntax includes operations for reading and writing values on channels, in addition to operations for declaring, composing, comparing, and executing processes. Take the following basic example of a π calculus interaction:

a(x).P | āb.Q.
P{b/x} | Q.

The | operator denotes concurrent composition. Process P and process Q are communicating over channel a. The first line represents the initial state, and the second line represents the state post-transition. The Q process will write the value b over the channel a, written here as ā because it is being written to. The value b has been communicated from Q to P over a. Being able to model concurrent processes in practice make expressing complex systems easier and formal reasoning about them possible.

The model relies on the concept of strong bisimilarity to provide a notion of process equivalence. In using modern programming languages, we sometimes take for granted our ability to compare two abstract entities and ask the computation at hand if they are in some way equivalent. This idea drills all the way down to the model level, where those decisions have to be made, based on the rules of reduction, possible inputs, etc. In other words, it is mathematically possible to prove, even in the full π calculus, that two processes are functionally equivalent.

We also see various examples modeled in the π calculus that very viscerally demonstrate how concepts common to a practitioner (a “get server” and a “set server” are modeled, along with a mutex and an implementation of the dining philosophers problem) relate to the expressiveness of the underlying computational model. Varela says “The semantics of a concurrency model explains precisely the behavior of its elements.” We see written “in math” what is normally written in a scripting language to make similar points. Varela fearlessly tackles this challenge, providing these examples not only in each computational model discussed, but also in programming languages which rely on these models later in the book.

The key difference between the π calculus and its predecessors in CCS and C.A.R. Hoare’s communicating sequential processes (CSP) is its explicit ability to send channel names between processes, allowing dynamic topologies, a property known as mobility. The remaining models in the book also have this property, which was inspired by the actor model. Since the π calculus can explicitly model changing topologies, it takes a very strong step in the right direction of unifying the practice with the underlying computational model.

The Actor Model

Of the models covered in this book, the actor model is probably the best known besides the λ calculus. Embodied in programming libraries and languages like Erlang, the actor model defines actors as the primary components of computation, where shared-nothing memory and asynchronous communication are assumed. Varela’s snappy definition says that:

“Actors are first-class history-sensitive entities with an explicit identity used for communication.”

In practice, actors are somewhat complex when combined in large numbers. The rules on a micro level, however, are simple. Actors can perform one or more of the following actions upon receiving an asynchronous message:

send a message to an acquaintance, an actor whose identity is known to the actor a,
create a new actor a’, with a given behavior b, or
become ready to receive a new message with new behavior b

This behavior allows the actor model to be an open system, so called because new components can be added by creating new actors dynamically, and because an actor can change behaviors dynamically. This means that a few actors can create a large graph of actors, and that the messages between them can include the names of actors, making very complex systems possible.

The actor model assumes fairness in a way that the other models do not - message delivery is guaranteed, and thus message processing is guaranteed under the correct conditions. It is worth noting that while assuming fairness makes reasoning simple, it potentially makes implementation challenging and expensive.

The formalized version of the actor model presented in this book is a more modern semantics that is sensitive to the realities of the implementations of actors in programming languages. The computational expense of creating new actors impacted the decision for which primitives to include, favoring a more efficient approach. Varela has this to say on the influence of the actor model on programming language theory:

“The actor model has influenced the development of several programming languages. Sussman and Steele developed Scheme in 1975 in an attempt to understand the actor model first conceived by Carl Hewitt (Sussman and Steele, 1998). Early actor languages developed at MIT include PLASMA (Hewitt, 1975) and Act1 (Lieberman, 1981). More recent actor languages include Acor, Rosette, ABCL, Erlang, E, and SALSA.”

Actors stand outside the other process calculi presented in many ways - seemingly spontaneously conceived and radically different in formation then the others, actors seem more tangible, can be explained simply, and model real world problems well. But because they are “higher level” and guarantee fairness, but do not explicitly encode notions of location, for instance, research continues toward designing models that approach being isomorphic with production distributed systems.

The Join Calculus

At this point in the book it becomes clear that we are searching for a model which allows for explicit reasoning about distribution and mobility. As straightforward as the translation from the actor model to a programming language is, given my background, the join calculus was equally abstract and bizarre to encounter. Varela has a concise definition, which is nicely fleshed out throughout the chapter that covers it:

“Concurrent computation is modeled in the join calculus as a reflective chemical abstract machine, where heating and cooling of molecules and atoms occurs according to reaction rules that can evolve over time.”

Molecules are processes running in parallel, which can combine and evolve into other processes over time according to the reaction rules. This sounds similar to how in the actor model, actors can change behaviors. In the join calculus, distribution and mobility are represented with a combination of molecules and reaction sites that model localities and correspond to rules with statically scoped names. Communication is constrained to reaction sites which are lexically scoped definitions. The join calculus, like the π calculus, but unlike the actor model, does not address fairness.

The join calculus is the first model I encountered that represents systems using the idea that conditional actions in a distributed system can be modeled as conditional reactions occurring at explicitly visited sites. Isolating the logic of a system to the sites where they occur is definitely a step in the right direction. To show a very simple example, as shown below: reactions are denoted D or J ▸ P where J denotes a join pattern that molecules must match for a reaction to take place, and P denotes the resulting molecules from the reaction.

D = ready<printer> | job<file> ▸ printer<file>
D ⊢ ready<laser> | job<f1>
D ⊢ laser<f1>.

The initial solution shows that if the pattern ready<printer> | job<file> is matched, the reaction printer<file> will take place, ostensibly sending the data from file to the printer. The second line shows ready<laser> and job<f1> visiting the reaction site, thus activating the condition above. The final line shows laser<f1>, the data from f1 is sent to the printer called laser. This kind of program can become arbitrarily elaborate, but this simple program shows a novel approach. The challenge remains of how to represent the rules in a distributed system in as concise, declarative, and central manner. There is a later chapter in Varela that is devoted to programming in JOCaml, an OCaml variant that uses the join calculus model for concurrency.

Processes in the join calculus can be compared for equivalence based on the idea that “Two programs are considered to be equivalent if they behave the same when placed in any observing contexts.” The join calculus is extremely complex and the semantics are hard to grasp. While I still struggle with them, I have taken away the idea that the join calculus provides at least the power of the π calculus with some apparent gains in expressiveness. Additionally since the join calculus explicitly reasons about distribution and mobility, the challenges of addressing locality, for example, in distributed systems becomes possible where it is now very challenging.

The Ambient Calculus

Besides having an awesome name, the ambient calculus is a fascinating formulation of a very difficult type of system to model - one where mobility and hierarchies of process are explicit. From a paper by the authors of the model:

“The ambient calculus is a process calculus whose basic abstraction, the ambient, represents mobile, nested, computational environments, with local communications. Ambients can represent the standard components of distributed systems, such as nodes, channels, messages, and mobile code. They can also represent situations where entire active computational environments are moved, as happens with mobile computing devices, and with multi-threaded mobile agents.” [2]

Varela’s treatment of the ambient calculus conveys the novelty of the expressive power of the model, illustrating how ambients, which contain calculations, are reduced from complex expressions to calculated outcomes. Notions of scope, memory sharing and fairness are covered, as well as how the semantics of the model relates to the others presented. The following example is given to introduce the expressive powers and aims of the model:

m[p[out m.in n.<M>]] | n[open p.(x).Q].
m[] | n[open p.(x).Q] | p[in n.<M>].
m[] | n[open p.(x).Q | p[<M>]].
m[] | n(x).Q | <M>].
m[] | n[Q{M/x}].

These five lines represent an initial state, three transitions, and a final state for a simple interaction where a network packet p moves between two machines on a network represented by two processes m and n. The keywords out, open, and in should stand out, as they are used to represent the notion of ambients representing mobile data in a system. <M> is data ready to be written and (x).Q is data to be read. To step through:

Process m is an ambient which contains one other ambient, p. Ambient n contains some instructions pertaining to ambient p.
Ambient p moves outside of ambient m.
Ambient p moves inside ambient n.
The open expression inside ambient n is now activated: an ambient named p is on the same level and a message is ready to be sent.
p has been opened and destroyed, and the processes can now interact, the data is read, and instances of x are replaced with M.

While a bit confusing out of context, the above should illustrate how far we have come in being capable of modeling in a formal sense what we have had to deal with on a library level for so long. Concurrency, distribution, and mobility being baked into our formal models can have potentially huge payoff. As Varela says:

“The key difference between the π calculus and the ambient calculus is the latter’s ability to model hierarchical process boundaries, movement of active ambients across those boundaries, and the restricted interprocess communication to inside an ambient boundary.”

Unique to the ambient model are the concepts of entry, exit, and open, which are the operations the model uses to specify the movement of ambients. The notation is illustrated and although even basic examples take time to get through, it is worth it to see how a correct program concerning the communication of distributed processes might look, however tiny.

In fact the idea that the calculus is capable of explicitly modeling mobility and hierarchies makes me believe that some day we will have verified distributed programs written in higher level languages whose semantics are modeled after advanced models of computation such as the ambient or join calculi.

Ontological Commitments, Expressive Power, and Semantics

One of the highlights of the book is the chapter that acts as a bridge between the description of the various models and the example implementations. After laying out the models, we see how to compare them, which prepares us well for how they will be implemented in a programming language and used in various examples.

The ontological commitments of each model consist of two primary decisions - synchronous vs. asynchronous communication and shared vs. distributed memory. The actor model stands out amongst all of the others: it alone assumes an asynchronous approach to communication and a distributed approach to modeling memory. If you’ve wondered about the depth of Hewitt’s work on the actor model but haven’t been convinced, this chapter will likely give you pause - it is truly a radical, simple, powerful abstraction.

The section about the expressive power of each model is dense but fascinating. Each model is shown to be Turing complete, and each is expressed in terms of either the λ or the π calculus, which is mind-bending and something that I surely do not fully grasp the depth of. You might, in which case I recommend this book highly.

Finally the semantics of each model is discussed, with the following properties being highlighted:

Can it model unbounded concurrency? According to Varela:

“One of the key differences between earlier models of concurrency – such as Petri nets, calculus of communicating systems (CCS), or communicating sequential processes (CSP) – and more recent ones is the latter’s ability to model unbounded concurrency.”

In this context, unbounded concurrency relates to the expressive power of the model.

Composability refers to the ability to combine simpler systems into more complex ones. In order to be composable, a model must be modular. Modularity relates to how reasoning that applies to the parts of a system hold when the system becomes more complex through composition. Each of the modern models of concurrency supports composition through an interface that allows the model to consider processes interacting with each other.
Fairness is “a semantic property that ensures that valid (infinite) computation paths do not consistently ignore one of the possible ways in which a computation may evolve.” The only model that explicitly supports fairness is the actor model. The others are designed in a way which hopefully mitigates this necessity.

The discussion of the final two semantic properties, distribution and mobility, drive home some of the central points this book is trying to make.

“Distributed computing is inherently concurrent. However, distributed aspects go far beyond concurrency. Of particular importance from a modeling perspective is the capability to reason about the location (and potential co-location) of concurrent computations, the heterogeneous cost of interaction, the security aspects of interaction across multiple locations, and the potential for partial failures.”

Each of the models presented allow us to reason about distribution to varying degrees. The π calculus has no explicit representation of locations, although it can implement a variety of (potentially expensive) abstractions to represent mobility. The actor model similarly does not provide a semantic distinction between processes on the same machine and processes distributed across a network.

The formal aspects of this chapter were very helpful in my attempts to understand the subtleties and value of each model, and the categories above are valuable when evaluating other models.

Conclusion

The five computational models presented as the basis for understanding concurrent computation in Varela’s Programming Distributed Computing Systems present an array of approaches for representing computation in a distributed system. What is made explicit in this text is that computational models exist which explicitly model qualities of distributed computation. Programming languages based on these computational models will inherently be more powerful and potentially more expressive (if they are implemented properly) because they do not have to consider or implement qualities which their models do not account for.

The last part of the book covers programming languages which implement these various models. While far from perfect, these languages and their summation provides evidence that there is purposeful work to be done if we are to achieve a natural system of designing programs destined to run on a distributed platform. Layers and layers of software currently represent what could be shed if assumptions were baked into the design of systems from the outset. Distributed systems practitioners could benefit from researching the various process calculi and their predecessors, to appreciate the deep connection between their work and the theoretical underpinnings which sometimes seem so distant.

Works Cited

All quotes unless otherwise noted from Varela

[1] Varela, Carlos A. Programming Distributed Computing Systems: A Foundational Approach MIT Press, hardcover, ISBN 978-0-262-01898-2, 2013

[2] Luca Cardelli, Giorgio Ghelli, and Andrew D. Gordon. 2002. Types for the ambient calculus. Inf. Comput. 177, 2 (September 2002), 160-194.

Message-Passing and Concurrent Program Design

Wed, 11 Sep 2013 00:00:00 +0000

This post is part of a series exploring Concepts, Techniques, and Models of Computer Programming by Van Roy and Haridi. Check the blog index for more.

Message-Passing and Concurrent Program Design

After a thorough discussion of the semantics and application of declarative concurrency in chapter 4, Van Roy and Haridi turn their attention to an alternative design solution for the same challenging problem space. Chapter 5, “Message-Passing Concurrency” covers the necessary changes to the kernel language to make this kind of programming possible, discusses the design of programs using this paradigm, and takes a brief aside to look at the Erlang language and its OTP framework.

Beyond covering the technical aspects of concurrency, the authors also emphasize the application of these concurrency primitives as design solutions for particular kinds of problems. Concurrent program design, reasoning, and proofs are given a fair amount of treatment, and attention is paid to the realism of applying the ideas as they are presented in the book. The authors advise programmers to keep programs simple, to isolate non-declarative components, and to model all possible states in advance of writing code.

Message Passing and Observable Nondeterminism

The concept of studying computation as the passing of messages between independent agents was first explored by Carl Hewitt in his work on the Actor Model. Primarily concerned with modeling computation for the purposes of exploring Artificial Intelligence, Hewitt realized early on that concurrency was an inevitable area of interest for Computer Science, and the Actor model and his work on Planner were revolutionary in their explorations towards these ends. [2] Van Roy and Haridi introduce message-passing concurrency in part to address limitations in the semantics of the declarative concurrency model presented in chapter 4, which enables concurrent stream programming but does not allow for any observable nondeterminism. Consumers of stream data must know where the next messages will come from, making many-to-one or many-to-many communication impossible in any practical sense.

In the message-passing paradigm, however, programming with streams loses its declarative properties because it allows observable nondeterminism. In practical terms this means that since the name of a stream can escape the local scope and be accessed by other parts of the computation concurrently, we can no longer rely on the soundness of our computation with respect to knowing its outcomes. Declarativeness is the assurance that computation meaningful to the environment is localized to the extent that you simply cannot be surprised by the outcome. Ports are asynchronous and give streams a name that can be embedded in a data structure, passed around between scopes, and written to or read from freely. This is contrary to the kinds of programs we have seen up until this point, which have ardently adhered to the strictness (and resultant sacrifices in clarity and expressiveness) of declarative programming. It is both a welcome relief and a terrifying prospect to gaze into the abyss of state and nondeterminism.

Extending The Kernel Language with Ports

The transition from declarative concurrency to message-passing concurrency is small but powerful. To keep the magnitude of these changes in perspective, I find it instructive to take a look at the language specification, which as usual fits in a neat little box:

The Message-Passing Concurrency Kernel Language[1]

As succinct as that language looks, we know that Van Roy and Haridi are always good for a pithy quote. Here is how they sum up the changes:

“The message-passing concurrent model extends the declarative concurrent model by adding just one new concept, an asynchronous communication channel.”

The asynchronous communication channel is implemented as an abstract data type which combines a FIFO (first-in, first-out) queue called a Port and two operations: NewPort and Send. A fundamental implementation detail of ports is that they rely on streams to exchange information, giving them a name and allowing other parts of the computation to write to and read from them. Here are more complete definitions of the two new operations:

{NewPort ?S ?P} creates a new port with entry point P and stream S.
{Send P X} appends X to the stream corresponding to the entry point P.

These operations are integrated into the execution state by extending the abstract machine definition inherited from the previous chapters with a Mutable Store μ. That makes the execution state (MST, σ, μ) where MST is the multiset of semantic stacks, σ is the single assignment store, and μ is the mutable store (this earlier post in the series contains an overview of computing with the abstract machine). To illustrate the layout of the concurrent model, the authors provide an excellent illustration:

The Message-Passing Concurrent Model, Diagrammatically[1]

This diagram is useful for understanding what role the mutable store plays in the execution state, and what happens when a NewPort operation is executed. The two bubbles at the top of the diagram contain program fragments that represent threads of execution. This code is interpreted and the data is combined with the two semantically distinct data stores to create an environment. The immutable store σ, well covered in other posts (and also known as the “single-assignment store”), contains most of the data in the environment, while the newly introduced mutable store μ contains pairs of the form x:y to represent Ports, where x and y are variables of the single assignment store. The visual separation between the two stores in the diagram is telling with respect to the isolation of the nondeclarative components of the message-passing concurrency model. The combination of the immutable and mutable stores provides a neat, contained mechanism for defining ports.

A line of code containing a {NewPort ?S p1} call can be reduced to the semantic statement ({NewPort <x> <y>}, E), which does the following:

Creates a fresh port with identifier or “entry point” p1
Binds E(<y>) and p1 in the immutable store σ
If binding succeeds, adds E(<y>):E(<x>) to the mutable store μ
Raises an error condition if binding fails

The noteworthy aspect of this computational model is how the mutable store isolates the mutability, thus allowing for further isolation of the nondeclarative properties of message-passing. The mutable store binds the name or “entry point” p1 to the unbound tail element of the stream ?S, so that other threads of execution may safely read from the tail of the stream in a read-only manner. In order to add elements to the stream, the Send operation is used.

A send operation {Send p1 1} can be semantically reduced to ({Send <x> <y>}, E), which does the following:

If the activation condition is true (E(<x>) is determined, or has a binding):
- If (E(<x>) is not bound to the name of a port, raise an error condition
- If the mutable store contains E(<x>):z, then do the following:
  - Create a new variable z' in the immutable store σ
  - Update the mutable store μ to be E(<x>):z'
  - Create a new list pair E(<y>)|z' and bind z with it in the store.
If the activation condition is false, suspend execution

Note that changes have to be made to the memory manager to redefine reachability in light of changes to the abstract machine. These changes are simple and are covered in the chapter.

Between NewPort and Send, programmers using the message-passing concurrent model have the ability to model a wide range of applications. As is the case with most primitives introduced to the kernel language, there is some nice syntactic sugar we can apply to make the creation and manipulation of ports simpler.

From Ports to Port Objects

The authors introduce a new data type called a port object, which combines one or more ports and a stream object. Port objects are superior to stream objects in a variety of ways, but the focus in this section is on the means by which they allow for more complex communication and encapsulate state transitions. Streams are limited to communicating with the thread that extended them; semantically this means that streams must always “know where their next message is coming from.” Port objects as they are known in the kernel language are known by other names elsewhere - Erlang’s processes are similar, though they are extended and allow for more sophisticated handling of messages. The authors build up to a sophisticated abstraction with an example program, starting with the existing abstractions, adding a trivial example with the new abstractions, and then showing everything in action.

Here is what a port object built from the existing abstractions might look like:

declare P in
local S in
	{NewPort S P}
	thread for M in S do {Browse M} end end
end

The port object P can called in the form {Send P hi}, which will eventually display hi in the Browse window. With the NewPortObject abstraction, however, it can be easier and more flexible to program with port objects.

Two port object functions are introduced in this chapter: NewPortObject and NewPortObject2. NewPortObject is used to encapsulate state transitions, and NewPortObject2 simply executes a given proc when a message is received.

NewPortObject is called in the form {NewPortObject Init Fun}. Init is an initial state and Fun is the state transition function. The state transition function is of the type <fun {$ TS TM} : TS> where TS is the state type and TM is the message type. We are going to focus on port objects that have no internal state, however, because they are considerably simpler.

NewPortObject2 is a function that allows the user to specify a Proc to be executed each time a message is received, thus maintaining no internal state:

fun {NewPortObject2 Proc}
Sin in
	thread for Msg in Sin do {Proc Msg} end end
	{NewPort Sin}
end

Here is some simple client-server code that the authors use to illustrate Remote Method Invocation using NewPortObject2. First, the server, which will accept messages from remote clients, perform a polynomial computation (the proc passed in to NewPortObject2) based on the message received, and then bind the result to a dataflow variable passed in by the client.

Note that the of calc(X Y) then and of work(Y) then syntax below is the notation for matching incoming messages based on a pattern.

proc {ServerProc Msg}
	case Msg
	of calc(X Y) then
		Y=X*X+2.0+X*2.0
	end
end
Server={NewPortObject2 ServerProc}

Server is now available for other parts of the program to access, and the nice syntactic sugar added makes for some pretty neat separation of server and client code.

proc {ClientProc Msg}
	case Msg
	of work(Y) then Y1 Y2 in
		{Send Server calc(10.0 Y1)}
		{Wait Y1}
		{Send Server calc(20.0 Y2)}
		{Wait Y2}
		Y=Y1+Y2
	end
end
Client={NewPortObject2 ClientProc}
local X in {Send Client work(X)} {Browse X} end

This allows for us to guarantee that calls are executed sequentially by the server, thus simplifying a lot of the reasoning typically necessary in a complex data interchange situation.

While powerful, the stateless port objects are somewhat limited, and the stateful port objects discussed in Chapter 5 allow for very flexible expressions of complex concurrent programs where NewPortObject is used to define distributed state machines. The NewPortObject examples are complex and challenging to cover in a blog post, so I recommend seeking out the chapter to see these in action.

Using Message-Passing Primitives Directly

Port objects are one way to program concurrently using the message-passing model in the kernel language, but it is also possible to combine threads and ports to create useful abstractions like this concurrent queue:

fun {NewQueue}
	Given GivePort={NewPort Given}
	Taken TakePort={NewPort Taken}
	proc {Match Xs Ys}
		case Xs | Ys
		of (X|Xr) | (Y|Yr) then
			X=T {Match Xr Yr}
		[] nil | nil then skip
		end
	end
in
	thread {Match Given Taken} end
	queue(put:proc {$ X} {Send GivePort X} end
		    get:proc {$ X} {Send TakePort X} end)
end

The put and get operations on this Queue allow for concurrent access, and even further, can do cool things because they can take advantage of dataflow variables:

declare Q X in
thread Q={NewQueue} end
{Q.put 1}
{Q.get X}
{Wait X}
end

The above code instantiates a Queue, puts 1 on it, and then runs a get with an unbound variable. Because there is a value to consume on this queue, the ‘Wait’ will proceed and the call will end. You can similarly run a put onto the Queue before the value is bound, allowing your code to synchronize access in a variety of ways.

Concurrent Program Design

In addition to providing an exhausting quantity of abstractions and ideas for concurrent programming with message-passing, the authors provide guidelines and wisdom about concurrent program design. Because CTM is a textbook that is aimed at communicating a deep philosophy, the authors know that presenting a well-rounded and practical view to their readers will be persuasive, and in this case they do an admirable job.

“Designing a concurrent program is more difficult than designing a sequential program, because there are usually many more potential interactions between the different parts. To have confidence that the concurrent program is correct, we need to follow a sequence of unambiguous design rules.”

The word correct above (emphasis mine) refers to this type of correctness, and shows a side of reasoning about computer programs that is underrepresented in much modern programming language literature. The authors’ pursuit of declarativeness, for example, is toward the end of creating programs which are simpler, thus easier to reason about.

Beginning with a specification and ending with a period of testing and iteration, the methods that the authors advocate might not be revolutionary, but its simplicity and practicality is noteworthy. Here are the steps suggested:

Create an informal specification which serves to communicate what the system should do, at varying levels of granularity for various purposes.

Enumerate all forms of concurrent activity in a specification, and label each one as a component. These components can be connected and turned into a diagram to understand the flow of data in the specification.

Once components are in place, message protocols are required to nail down the implementation of what data will be exchanged and what form the data will take. Augment diagrams with message protocol information.

For each component, create a state diagram which describes for each state how the component will react to incoming messages. This should be handled with extreme care.

Finally, the authors sugges to implement the program in your “favorite programming language,” and to test and iterate until the program meets the specification.

There is a demonstration of this method for a design problem given in the chapter, and although the problem itself is fairly simple, it is illustrative in how detail oriented and correctness-obsessed the solution needs to be to truly implement the specification.

A Look at Erlang and OTP

Perhaps unsurprisingly given the subject of this chapter, the authors seem to find an affinity in the existing software surrounding Erlang and its OTP framework. The Erlang equivalent of the book’s Thread is an Erlang process, which is created with a port object and a mailbox to form a very powerful and flexible unit of concurrent execution. I won’t dive too deeply into the chapter’s treatment of Erlang and OTP, but like the examination of Haskell earlier in the book, I find an important link between the ideas the authors present and the concrete implementation of Erlang.

The computation model is discussed, and Erlang is characterized as having a “Functional core” with a “Message-passing extension.” The term “process independence” is applied to the idea that in Erlang, each running process theoretically has no impact on any other, and nothing is shared between them. According to the authors, “Process independence makes it easier to build highly reliable systems.”

Erlang’s combination of pattern matching for consuming messages from a process’s mailbox, a dedicated location for persistent data and the scheduler to map processes onto system threads provides an excellent framework for the kind of system the authors advocate for. Erlang has declarative properties and favors a functional approach, and there are demonstrations in the text of converting Erlang code to the kernel language that aren’t as outrageous as you’d expect.

As far as production-ready systems go, there are a lot of the ideas of the ideal kernel language present in Erlang and the OTP framework. Erlang is known for being highly concurrent and much like the kernel language, it is crucial to understand the impact that the implementation of the units of concurrent execution have on the semantics of the language. The authors present the idea that message-passing is a powerful approach to concurrency and Erlang and its success lends credence to this idea.

My take on it is that as far as all of the options go, if you’re looking for a programming language with interesting ideas that you could actually make money using, Erlang is a good choice. Erlang has a good mix of the practical and theoretical, and while there is perhaps a strange beauty to its syntax, there is undoubtedly a ton of elegance in the abstractions it makes available.

Conclusions

The expansion from declarative concurrency to message-passing concurrency in the kernel language serve to do far more than illuminate the reader about a new mechanism to deal with the challenges of concurrent programs. Rather the chapter gives rare insight into the complexity of concurrent programs at even a small scale. The towers of code and state built from small, mostly declarative units illustrate the interconnectedness of even well thought out programs and the intricacies of timing in a distributed system.

As the limitations of the declarative concurrent system previously presented were extensively discussed in the previous chapters, it is encouraging that the authors are capable of alleviating some of those unappealing aspects by making trade-offs between functionality and expressiveness, declarativeness and statefulness, and so on. Various design solutions and abstractions are introduced that reinforce, to my mind, the importance of the careful consideration of correctness and design in creating flexible, powerful, concurrent software.

Works Cited

All quotes unless otherwise cited from Van Roy and Haridi.

[1] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

[2] Hewitt, Carl Viewing Control Structures as Patterns of Passing Messages Journal of Artificial Intelligence, June 1977

What It Really Means To Be Lazy

Tue, 09 Jul 2013 00:00:00 +0000

This post is part of a series exploring Concepts, Techniques, and Models of Computer Programming by Van Roy and Haridi. Check the blog index for more.

What It Really Means To Be Lazy

Two extensions to the declarative computation model are made in Chapter 4 of Concepts, Techniques, and Models of Computer Programming (CTM). The first is invoked with the instruction thread, and enables concurrency. The second, which is the topic of this post, adds the instruction ByNeed, and enables an alternative execution order for statements in programs that is known as lazy execution or demand-driven computation.[1]

Although more or less restricted to one subsection of Chapter 4, the discussion of laziness is extremely rich. Additionally, laziness is a subject that comes up often in the lexicon of professional programming. Haskell and Clojure for example, both rely on laziness for certain language features. Basically, this material was too complex to stuff into the post about concurrency, and too good to not cover.

We’ll define laziness in the concurrent declarative computation model and try to be thorough about clarifying its semantics - interesting questions arise like when is a variable truly needed? We’ll discuss the impact of alternate execution orders, see how this computation model lends itself to various real world applications, and bring to a close, for now, the posts about declarativeness.

What Is Laziness?

What do the authors mean when they talk about laziness? How does it relate to the term laziness as it is used in statements like:

“Haskell is a lazy programming language”

or, for any given value of Programming Language N:

“The modules in Programming Language N are lazily loaded”

The laziness this post covers is of a general nature, but subsumes both types of laziness above. Laziness can be thought of as a fundamental rethinking of how statements in a given computation are executed. As usual I’d like to let the authors do as much of the defining as possible, and this happens to be one of my favorite quotes in the book so far:

“Up to now, we have always executed statements in order, from left to right. In a statement sequence, we start by executing the first statement. When it is finished we continue to the next. This fact may seem too obvious to require mentioning. Why should it be any other way? But it is a healthy reflex to question the obvious! Many significant discoveries have been made by people questioning the obvious: it led Newton to discover the spectrum of light and Einstein to discover the postulate of relativity. Let us therefore question the obvious and see where it leads us.”

It turns out that this exploration has led some very smart theorists and language designers down some very interesting paths. For a definition, we can use the following:

In lazy or demand-driven evaluation, statements are only executed when their results are needed elsewhere in the program. This is in contrast to eager evaluation where if a statement is reached, it is executed.

Let’s explore the following program fragment to get an intuition for how laziness works:

fun lazy {F1 X} 1+X*(3+X*(3+X)) end
fun lazy {F2 X} Y=X*X in Y*Y end
fun lazy {F3 X} (X+1) * (X+1) end

A={F1 10}
B={F2 20}
C={F3 30}

D=A+B

F1, F2, and F3 are lazy functions (lazy is a syntactic construction on top of the ByNeed instruction we mentioned earlier, and is explained below). When these functions are called, as they are with the assignments to A, B, and C, a stopped execution is created. The values of these variables are not yet needed, so those stopped executions are not resumed and completed. The final line of the fragment shows that the assignment to D needs the values of A and B. This call triggers the execution of A and B, and D is assigned the sum of {F1 10} and {F2 20}. C, which is still stopped, is never executed, because its value is never needed. In other words, though this program can be read and understood in a sequential way, it is in fact not executed in the traditional eager manner.

Laziness changes the way things happen in a program and can be applied in the small, as in the above code snippet, or in the large as in the “lazily loaded code modules” example above. An entire language can be lazy if that is the default means by which execution within computation occurs. This is not the case for our model, which chooses to extend the kernel with the ability to declare computations as lazy at the discretion of the developer. With the combination of concurrency, laziness, and declarativeness, we have a powerful set of ideas to explore.

For the sake of completeness it’s interesting to see how the lazy linguistic abstraction seen here:

fun lazy {Generate N} N|{Generate N+1} end

Is turned into a normal function here:

fun {Generate N}
  {ByNeed fun {$} N|{Generate N+1} end}
end

The function is wrapped in a new zero-argument function that only gets called when the value of Generate N is needed.

The Semantics of Laziness

This section builds on the abstract machine discussed in this earlier post in the series.

Implementing laziness in the kernel language requires a fundamental change to the system on the same order of impact as adding concurrency semantics. A new instruction, ByNeed, is added to the kernel language, and the concept of a by-need trigger is added to the computation model. ByNeed can be understood to be similar to thread, with the difference being the scheduling of how the contained computation is executed. thread calls will always be executed eventually, where ByNeed calls will only be executed if the contained computation is needed. In order to define the semantics of laziness, we need to do the following:

Extend the execution state with a trigger store ($\tau$)
Define trigger creation
Define trigger activation
Define what it means for a variable to be “needed”

Prior to the introduction of the trigger, the execution state of the (declarative, concurrent) abstract machine is represented by the pair (MST, $\sigma$): the multi-set of semantic stacks and the single-assignment store. We add the by-need trigger which is a pair trig(x,y) of a variable y and a one-argument procedure x. To accommodate these triggers, we add a trigger store ($\tau$) ** alongside our **single-assignment store ($\sigma$), which contains all of the triggers for the computation and is initially empty. The execution state then becomes the triple (MST, $\sigma$, $\tau$): the multi-set of semantic stacks, the single-assignment store, and the trigger store.

Trigger creation is represented by the semantic statement ({ByNeed x y}, E) where x is the procedure as noted above, y is a dataflow variable, and E is the environment. During execution, if y is not determined, or unbound, then the trigger trig(E(x), E(y)) is added to the trigger store, where E(x) and E(y) represent the value when applied to the given E environment. If y is determined, or bound, then a new thread is created with the initial semantic statement ({x y}, E).

If the trigger store contains trig(x,y) and a need for y is detected, then trigger activation consists of removing the trigger from $\tau$ and creating a new thread to execute the computation.

Finally, we can say that a variable is needed if there is a suspension on the variable or if the variable is determined. Additionally, the authors mention that being needed is monotonic: once it is needed, it is needed forever. This is interesting because the monotonicity of being needed turns out to be an essential aspect of the ability to prove that the lazy concurrent model is declarative.

The mechanics of ByNeed show how dataflow variables can enable laziness, and this simple implementation strikes me as beautiful in its elegance.

A Picture Of Declarative Computation Models

In an interesting but clarifying diversion, section 4.5.2 is devoted to breaking down the declarative computation models that have been discussed through Chapter 4. A contrast is made between concurrency and laziness:

“Concurrency can make batch computations incremental. Laziness can reduce the amount of computation needed to get a result. A language can have neither, either, or both of these concepts.”

Starting from a purely functional kernel, the following elements have been added to the model during the course of the first four chapters:

Dataflow variables
Concurrency
Laziness

From these three additions, there are six possible combinations. They’re outlined very nicely in this diagram:

Six practical declarative computation models.[1]

It’s a very interesting spectrum of languages to consider: take for example the differences between the dataflow enabled non-sequential laziness of our concurrent, lazy, declarative model and Haskell’s sequential laziness. Because dataflow variables allow the separation of declaration (1) and specification (2), a maximum amount of expressiveness can be achieved. The complexity of managing these features however, is not lost on the authors:

“One way to understand the added expressiveness is to realize that dataflow variables and laziness each add a weak form of state to the model. In both cases, restrictions on using the state ensure the model is still declarative.”

Declarativeness is still paramount, and the ability to use these features in a declarative context is fascinating and has deep-reaching consequences.

Applications and Theories of Laziness

Laziness as a language construct has the ability to shift the burden of deciding how much computation is necessary from producer to consumer. Concurrent stream communication, for instance, in an eager setting would default to the producer deciding the rate and amount of computation. Laziness allows resource intensive computation to be executed on a by need basis, which can have advantages in computationally limited settings: while laziness may use the same amount or more overall resources, it can limit usage in a point in time manner. Particularly well-known techniques of this ilk include lazily reading lines from a log file, consuming data from an external feed, etc.

The authors also cover other material to show the usefulness of declarative laziness in various settings. Algorithm design is explored quite deeply, with a focus on how laziness can enable the design and implementation of persistent data structures which have similar time/space complexity of their ephemeral, imperative counterparts. Based on the work of Chris Okasaki, various data structures are covered that have surprisingly favorable performance comparisons to implementations in imperative settings. The implementation of list comprehensions (introduced as “a notational convenience for defining streams”) in the declarative model are also given a mind-bending deepdive. Traditional list comprehension use cases are translated to the declarative model using functions like LMap and LFlatten (lazy map and lazy flatten) that were built up in an earlier subsection.

The “Advanced Topics” section at the end of Chapter 4 also romps through some quite heavy material which hints at the depth of the ideas wrapped up in laziness. Lazy evaluation exhibits the behavior of normal order reduction, but there are other possible reduction orders. “Reduction orders” here refers to the order in which remaining suspended computations are calculated. A thrilling explanation of the Church Rosser theorem follows, along with, believe it or not, one of the most concise and comprehensible explanations of Monads and their use in Haskell that I’ve ever seen. And I’ve seen way too many.

Conclusions

Laziness is a quality of a programming language’s semantics that is often misunderstood by developers. It sounded bizarre and impossible to me when I first encountered it, as a complete novice functional programmer coming from an imperative background. As I began to understand a bit more about its application and philosophy I still did not understand its implementation. After thorough investigation it turns out that laziness, like concurrency, is a tool that can help a developer get the most out of the computational power they have available to them. It is no surprise that it is a pattern to lean on when the simplest plans do not work out, not a solution to all of our engineering problems. Laziness can help us build new and interesting structures, but its application is limited.

Adding laziness to the concurrent declarative model also provides insight into how complexity is introduced into a programming language and its computational model. This language is small and provably correct, but considerably less expressive than a typical production language with many of the same characteristics. Adding laziness meant adding an entirely new concept to the abstract machine, new instructions to the kernel language, new linguistic abstractions, new ways of thinking about time and space complexity, and more. The relationship between laziness and state is also quite compelling. Reading that “laziness is state” is one thing, but seeing the mechanics of how that state is added and passed around in the abstract machine is a whole beautiful other.

Special thanks to Mauricio Martínez Jimenez for submitting a correction to this post via email.

Works Cited

All quotes unless otherwise cited from Van Roy and Haridi.

[1] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

Concurrency Made Simple

Mon, 01 Jul 2013 00:00:00 +0000

This post is part of a series exploring Concepts, Techniques, and Models of Computer Programming by Van Roy and Haridi. Check the blog index for more.

Concurrency Made Simple

Chapters 3 and 4 of Concepts, Techniques, and Models of Computer Programming (CTM) cover the definition of declarativeness and its application in concurrent programing. As noted in the previous post in this series which mostly covered Chapter 3, one of the limitations of the declarative paradigm is its inability to deal with the “real world.”[3] Declarative programming is pure, but not always realistic. The “real world” includes concurrency, the basic definition of which is “A set of activities that execute independently.” In other words, in the real world, things operate both in tandem with and independently from each other. Not everything can be neatly partitioned as declarative programming can sometimes require. Dealing with concurrency in programs is notoriously challenging, but not by necessity. The authors have developed a simple mantra:

“Concurrency can be simple.”

They demonstrate the basic means by which a program can be made concurrent as follows. Consider this simple program in the declarative kernel language:

fun {Gen L H} 
	{Delay 100} 
	if L>H then nil else L|{Gen L+1 H} end
end

Xs={Gen 1 10} 
Ys={Map Xs fun {$ X} X*X end}
{Browse Ys}

A function Gen takes two variables L and H. It waits 100 milliseconds. It terminates if L is greater than H, else it recurs on a constructed list with L as the head and a new list beginning with L+1 as the tail. When run with {Gen 1 10}, it would return a list [1 2 3 4 5 6 7 8 9 10] Below the function definition, the 1..10 list is bound to Xs, and Ys is bound to the results of mapping over the elements of Xs and squaring them. The results of the list are then shown with Browse.

The concurrent or threaded version uses the same definition for Gen, and only adds the thread keyword:

thread Xs={Gen 1 10} end
thread Ys={Map Xs fun {$ X} X*X end}
{Browse Ys}

The result is that the results are incrementally displayed in Browse as they are calculated. The reason to put a Delay call in Gen is to illustrate the point that concurrency has the effect of making ‘atomic’ calculations ‘incremental’. Note that nothing else about the program needed to change in order to make it concurrent: just add threads and you’re good. This is the dream of concurrent programming.

This post will cover the parts of Chapter 4 which define concurrency and its application to declarative programming. We will see the semantics of threads in the extended kernel language, get a glimpse at programming with streams, and I will attempt to tie the lessons of declarative concurrency to the current state of programming tools. I’m not covering everything in this chapter by a long shot - it is very dense. I’m intentionally not covering error handling or exceptions, which might rankle some, especially in the context of streams. If you want to know more about this, I suggest that you read the chapter: the authors have some good answers for you. Our next post will finish out Chapter 4 by taking a look at Declarative concurrency and its interaction with a fascinating language construct known as laziness.

What is Concurrency?

Declarative programming has been shown to be a surprisingly rich and capable paradigm which can be applied in a variety of circumstances if the developer is aware of the involved tradeoffs. The declarative kernel language is small and flexible enough that adding the ability to execute code concurrently ends up being simple, at least from a programming language design standpoint.

“Our approach to concurrency is a simple extension to the declarative model that allows more than one executing statement to reference the store.”

A new instruction, thread, is added to the existing kernel, which gives us the following language:

The entire data-driven concurrent kernel language, including the new thread instruction. [3]

The ability for the kernel language to express concurrent programs means that under the right conditions, things can be happening at the same time. The authors do an excellent job of describing what “at the same time” really means, and it’s worth going through the terminology so that we can thoroughly understand the implications of declarative concurrency.

It is said that all operations in a program have a global sequence and that threads do an interleaving execution over this sequence. A scheduler is responsible for choosing work for threads. Concurrency in the kernel language relies on the existence of single assignment or dataflow variables, the properties of which provide both immutability guarantees and a coordination point for external computation with respect to their bound/unbound state.

An execution is nondeterministic if the choice of which thread to execute becomes visible to the programmer. In the concurrent declarative kernel language, nondeterminism is never visible. Simply put, this lack of visible nondeterminism can be attributed to the fact that there is simply no choice but to wait if a variable is not yet bound. There is no chance that two threads will write to the same variable, since variables can only be assigned a value once. Therefore all “reads” will be deterministic and at worse you cannot predict exactly when “writes” will happen.

The scheduler which runs the concurrent kernel language is assumed to fairly select threads to do work, and never lets any thread starve. Threads can be either be ready or suspended, depending on if they have work to do, or are waiting on other calculations to be complete before their work can proceed. Let’s take a closer look at how threads integrate into the semantics of the kernel language and how it executes in our abstract machine.

The Semantics of Threads

This section builds on the abstract machine discussed in this earlier post in the series.

Adding threads to the existing declarative kernel language is a matter of adding a thread operation. This operation extends the abstract machine by allowing multiple semantic stacks to have access to the same data store.

Intuitively, the difference between a sequential and a concurrent program is reflected in how we introduce concurrency into the abstract machine: before, one total order was possible because only one environment was executed. After, a causal order is guaranteed by the scheduling algorithm and the existence of multiple stacks, each represented by a thread.

We still have the concepts of a single assignment store ($\sigma$), an environment ($E$), a semantic statement ($< S >, E$), and a semantic stack ($ST$). We extend the concept of an execution state and computation to allow for multiple semantic stacks. An execution state becomes a pair ($MST,\sigma$) where MST is a multiset (which we can think of as a list for our purposes) of semantic stacks and $\sigma$ is the single assignment store. A computation becomes a sequence of execution states which starts at the beginning of the list of semantic stacks, and ends at its end. The choice of which semantic stack to evaluate is up to the scheduler (the implementation of the scheduler is another fascinating subject that this post does not do justice).

Consider the following small sample concurrent program:

local B in
	thread B=true end
	if B then {Browse yes} end
end

This program only has one place to begin, on the first line. B is introduced into an environment in a semantic stack and then the next line is reached. thread never blocks, but peels off the work into its own semantic stack to be executed at the scheduler’s convenience. The next line is reached. If B has been bound by the work done in the computation in the previous step, then this line will display yes and the program will be complete. There are no circumstances under which this program would behave in any other way under the given semantics.

What is Declarative Concurrency?

It has been shown that the “threadless” programs in the preceding chapters are declarative, and we have defined what concurrency means. So what does it mean for a program to be both declarative and concurrent? We know that the basis of declarativeness is that the output of a program should be a mathematical function of its input. Two issues arise with concurrent programming, however - termination is not guaranteed thus output is difficult to measure, and definition as functional is challenging, because programs inputs and outputs can both contain unbound variables. Simply put, we can think of declarative concurrency as concurrency with no observable nondeterminism, but lets spend some time with the technical definition:

“A concurrent program is declarative if the following holds for all possible inputs. All executions with a given set of inputs have one of two results: (1) they all do not terminate or (2) they all eventually reach partial termination and give results that are logically equivalent.”

For programs which do not terminate, we don’t need to illustrate much – they just won’t terminate, and that is true of any time you run them, regardless of externalities.

What does partial termination mean? Since we cannot rely on total termination in a programming paradigm with streams that can grow indefinitely, we can rely on a partial termination, defined as a stopping point when calculations are temporarily exhausted. Consider the following program:

fun {Double Xs}
	case Xs of X|Xr then 2*X|{Double Xr} end
end

Ys = {Double Xs}

This program will run as long as there are values in Xs to be processed. If there are none, the program will simply be in a state of partial termination. How can we assert the logical equivalence of the results of a program such as this? We can essentially rely on our intuition, but a technical explanation exists. Each state of partial termination contains enough information for us to quantify in terms of constraints. These constraints allow us to compare states in terms of bound and unbound variables and the values to which they may be bound. Since equivalence on these constraints is defined (but outside the scope of this post), we can assert equivalence on partial termination states even if the total order present in the sequential version doesn’t hold. As long as the causal order is maintained, and no nondeterminism is visible, our concurrent programs can be considered declarative.

Programming With Streams

Streams are given extensive treatment by the authors in Chapter 4 and are referred to as one of the most convenient and reasonable ways to program with declarative concurrency. A special case of stream programming is covered, known as deterministic stream programming, where “each stream object always knows for each input where the next message will come from.” This is a declarative means of stream programming - non-deterministic stream programming is covered in Chapter 5, Message-Passing Concurrency. Though this sounds limiting, some very interesting examples are covered.

The concept behind streams are that they are convenient ways to declaratively express computation as communication between threads. The authors description, as usual, is compact and thorough:

“A stream is a potentially unbounded list of messages, i.e., it is a list whose tail is an unbound dataflow variable. Sending a message is done by extending the stream by one element: bind the tail to a list pair containing the message and a new unbound tail. Receiving a message is reading a stream element.”

Streams and dataflow variables have a natural affinity and the authors capitalize on this insight. They give a nice definition and demonstration of basic use of streams, and begin to explore patterns useful in stream programming, even using streams to perform digital logic simulations.

Declaring a stream looks like this:

declare Xs Xs2 in
Xs = 0|1|2|3|Xs2

Here is how you incrementally extend a stream:

declare Xs3 in
X2 = 0|1|2|3|Xs3

And a program that combines these concepts to form a basic producer/consumer pattern:

fun {Generate N Limit}
	if N<Limit then
		N|{Generate N+1 Limit} 
	else nil end
end
fun {Sum Xs A}
	case Xs
	of X|Xr then {Sum Xr A+X}
	[] nil then A
end
end
local Xs S in
	# Producer thread
	thread Xs={Generate 0 150000} end 
	# Consumer thread
	thread S={Sum Xs 0} end 
	# Display results				  
	{Browse S}
end

Producer and consumer each run in their own threads, operating on the shared dataflow variable Xs. As data is appended to the stream by Generate, it is consumed by Sum. The case statement in Sum blocks when there are no available values to bind, and calculates values when they are there to be read. As the authors say, “Waiting for a dataflow variable to be bound is the basic mechanism for synchronization and communication in the declarative concurrent model.”

You can use this same basic program to have multiple consumers and producers, or create a pipeline of filters:

local Xs Ys S in
	thread Xs={Generate 0 15000} end
	thread Ys={Filter Xs IsOdd} end
	thread S={Sum Ys 0 } end
	{Browse S}
end

And when streams are generalized to “a recursive procedure that executes in its own thread and communicates with other stream objects through input and output streams,” things really start to get wild: simple declarative concurrency primitives when combined can produce powerful results.

Conclusion

The authors have shown us yet again that simple extension of the semantics of the kernel language can make very sophisticated programs possible with seemingly limited resources. Concurrency and its application in modern software design is of paramount importance as our “free lunch” has long been over but we struggle in denial against its reality, often wielding primitive, wasteful tools.[2] Our tools rely on syntactic and semantic constructions which force us to amortize the costs of writing our programs over a very long period of time - they are easy to create, but difficult to maintain. Some of these constructions include (faux, overloaded) Object Orientation, global state, and mutability.

Mutability refers to the ability of the programmer to change the value of an object after it has been instantiated (object is an abstract name given to any assignable quantity that developers have access to). Much discussion has been raised specifically surrounding the connection between mutability and the inability for our tools to properly handle concurrency in a reasonable way, and in fact we are making progress.[1] Several active programming languages, including the Mozart platform that the CTM book constructs, Clojure, which runs on the Java Virtual Machine, and Haskell all have embraced the value of immutability and the leverage it gives programming language designers in providing simpler concurrency.

Beginning professional developers who jump right in to programming with mutable, global state, objects, and concurrency rightfully believe that “concurrency in LANGUAGE X is Hard.” Unfortunately, many people extend that to “programming with concurrency is Hard.” They don’t seek other paradigms because they think that the complexity is necessary. They think it is capital-H Hard and the truth is that it doesn’t have to be.

Works Cited

All quotes unless otherwise cited from Van Roy and Haridi.

[1] Hickey, Rich. Index of Published Talks HTML Page

[2] Sutter, Herb. A Fundamental Turn Toward Concurrency in Software Dr. Dobbs Software Journal. Available here.

[3] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

What Is Declarative Programming?

Thu, 20 Jun 2013 00:00:00 +0000

This post is part of a series exploring a brain-melting epic known as Concepts, Techniques, and Models of Computer Programming. Check the blog index for more.

What Is Declarative Programming?

Van Roy and Haridi have a way with definitions in Concepts, Techniques, and Models of Computer Programming (CTM), but interestingly, declarative computation and declarative programming get thrown around a lot in Chapter 2 before they are properly or even pithily defined. This post provides this definition and also looks closely at declarative programming as a paradigm: its benefits, its limitations, and how it impacts professional developers. We will also see how declarativeness relates to what is commonly understood as functional programming, and consider ways in which we can reach for declarativeness as a tool in our otherwise imperative code.

A Definitional and Observational View

In Chapters 1 and 2, before the practical application and programming techniques of declarative programming are discussed, we learn the paradigm through the semantics of a simple kernel language which is defined by its ability to “evaluate functions over partial values.”[1] Chapters 3 and 4 cover declarative programming in depth while rounding out the foundational Computer Science knowledge developed in the first few chapters. Declarative programming as a paradigm is defined, extended to include concurrency primitives (the subject of our next post), and then used to bridge the gap to a chapter on message passing concurrency.

Back to that definition - what does it mean to be declarative?

“We say the operation is declarative if, whenever called with the same arguments, it returns the same results independent of any other computation state.”

To break that down, declarative computation can be described as having the following properties:

It is independent: it computes irrespective of other computation state
It is stateless: each action is unrelated to any previous action
It is deterministic: the same inputs will return the same outputs

Finally, declarative programming is described as being necessarily referentially transparent if “certain rules are followed.” Referential transparency is a quality ascribed to code which has mathematical origins. An expression is referentially transparent if it can be replaced with its value without changing the meaning of the program. In mathematical functions, all applications are referentially transparent: a function with a certain input will always produce a certain output, and therefore, it can be replaced by its value. This is a powerful property that allows mathematicians to reason about their “code” in interesting ways. If our code is declarative, it can be referentially transparent, and programmers can leverage the same benefits: ability to easily reason about structure and function, ease of composability, and more.

Reading these definitions of declarativeness sequentially, you may start to see the hazy picture of how the roots of functional programming are formed in the semantics of declarative programming. The authors are somewhat well known for the following diagram, which puts programming languages on a continuum from “more declarative” to “less declarative” and is one of the keys to understanding the system as laid out by CTM.

The massive paradigm chart - "More is not better (or worse) than less, just different." (source)

More declarative programming languages, including Prolog, Haskell, Oz, and SQL, are part of a sub-group of declarative languages which have the property of using “Unnamed state (sequential or concurrent).” More declarative languages are also shown to belong to categories like “Lazy functional programming,” “Constraint (logic) programming,” and “Declarative concurrent programming.”

Less declarative languages use “Named state,” and prominent examples include Erlang, Pascal, C, and Java. Interestingly, Prolog also appears in this list (due to a Prolog programmer’s ability to forgo the maintenance of declarativeness through the use of certain operators) as does Oz, which is a multi-paradigm language. Categories of less declarative languages include “Stateful functional programming,” “Concurrent object-oriented programming,” and “Message passing concurrent programming.”

There is a lesser known and more narrowly scoped illustration in Chapter 3 of CTM which I have recreated here because I think it helps round out the picture of what declarative programming is and how it tends to work:

What is Declarative Programming? Diagram based on</br>Van Roy and Haridi's, recreated with stolen imagery.

In other words, declarative programming is predictable, timeless, and knowable, and the rest of programming is a swirling, galactic vacuum of energy and space. Why is this? According to the paradigm diagram, declarativeness is a continuum resting on a fulcrum of statefulness. The more state we have, the more expressive we can be. Because engineering is always about tradeoffs, however, it is important to remember that state also controls how much more more imperative we become. But declarativeness is more than an intuition - it’s a knowable entity. We should be able to prove that declarative computation is more knowable and repeatable, and indeed we can. Here’s the process:

“Given any statement in the declarative model, partition the free variable identifiers in the statement into inputs and outputs. Then, given any binding of the input identifiers to partial values and the output variables to unbound variables, executing the statement will give one of three results: some binding of the output variables, suspension, or an exception.”

Let’s try to apply that process to the following statement:

if X>Y then Z=X else Z=Y end

This statement can be shown to be declarative by considering the input identifiers X and Y and the output identifier Z. If the input identifiers are bound to any partial values, two possibilities exist: the statement will either block or bind Z to a value. Since both of those possibilities are subsumed by the possibilities mentioned above, this illustrates the mechanism by which a statement can be assigned the property of declarativeness and lends concreteness to the notion that independent, stateless, deterministic computation is declarative.

In case you’re confused about why it is taken for granted that the statement will block if no binding is made, the technical reason lies in the specification of the kernel language’s single assignment store. I covered this mechanism a little bit in this post, but to make it explicit, the semantics of variables in this store are such that:

Single assignment variables can only be bound once
If single assignment variables are needed in a computation before they are bound, the computation will block until an assignment is made

The strict scoping rules of the declarative language also come heavily into play here in allowing units of declarative code to be composed in a way which leverages their referential transparency. Since calculating with environments in declarative code has been shown to be extremely straightforward, and it has so many benefits, the natural question arises - when is it not applicable?

The Realm Of The Nondeclarative

What kinds of problems cannot be solved declaratively? It is instructive to look above at the list of qualities which declarative code must have, because if you need any of these qualities to disappear, you know you must sacrifice at least some declarativeness. As the authors state in section 4.7.4 with the somewhat heavy name The Real World,

“The real world is not declarative. It has both state (entities have an internal memory) and concurrency (entities evolve independently).”

The classic examples offered include client server applications and video display applications. The inability to easily maintain state in the declarative model available at this point in the book shows that solving these problems in a purely declarative way just isn’t realistic. Instead of attempting to pull code from the middle of something more complex, I’ll use an example from the book to illustrate a typical challenge posed by purely declarative code.

Example: Code Instrumentation

Consider a defined component in the kernel language to which a developer would like to add instrumentation, or a means of measuring aspects of program execution. Given a reusable component SC:

fun {SC ...}
  proc {P1 ...}
    ...
  end
  proc {P2 ...}
    ...
    {P1 ...}
    {P2 ...}
  end
  proc {P3 ...}
    ...
    {P2 ...}
    {P3 ...}
  end
in
 `export`(p1:P1 p2:P2 p3:P3)
end

We can see that SC encapsulates three operations, P1, P2, and P3, and that the procedures call each other and themselves. Because this code is declarative, it can be used elsewhere in an understandable way, but in order to protect this quality, we sacrifice expressiveness. Demonstrably, in order to do something as simple as add an accumulator to this component for instrumentation purposes, we must add two arguments to each procedure: this means that we must change three procedure definitions and four procedure calls. Here’s what the altered code looks like:

fun {SC ...}
  proc {P1 ... S1 ?Sn}
    Sn=S1+1
    ...
  end
  proc {P2 ... T1 ?Tn}
    ...
    {P1 ... T1 T2}
    {P2 ... T2 Tn}
  end
  proc {P3 ... U1 ?Un}
    ...
    {P2 ... U1 U2}
    {P3 ... U2 Un}
  end 
in
  `export`(p1:P1 p2:P2 p3:P3)
end

In a stateful language, the same could be achieved by changing no procedure signatures. Incrementing would be done at the call site, and a variable would be made available globally upon program instantiation. Consider a bit of Ruby code:

module Work
  def part_one
    ...
  end

  def part_two
    ...
    part_one(...)
    part_two(...)
  end

  def part_three
    ...
    part_two(...)
    part_three(...)
  end
end

And the instrumented version:

module Work
  @@counter = 0

  def part_one
    @@counter += 1
    ...
  end

  def part_two
    ...
    part_one(...)
    part_two(...)
  end

  def part_three
    ...
    part_two(...)
    part_three(...)
  end
end

No method signatures change, and significantly, the only places you see change are where you’d expect them.

Example: SQL Queries

Professional developers who write applications which interact with traditional databases are familiar with the pattern of embedding declarative nuggets in the screaming abyss of their imperative Java, Ruby, Python, or PHP code. SQL is the “what” without the “how” and has a way of being understandable on its own without needing to understand its surrounding context. Queries which select certain fields from certain rows in a certain way will always do just that.

There is an interesting correspondence between how queries embedded in application code crystallize and come to represent the data they retrieve and the referential transparency of mathematical functions. Other tokens in stateful Object Oriented languages, for example are not nearly as reliable. So much more context is necessary to decipher even highly localized pieces of code in what have become the standard for modern Object Oriented applications, particularly in web platforms. Declarativeness is a tool that can be reached for more, a point which the authors drive home quite heavily at the end of Chapter 4.

Example: Haskell

One of the more interesting aspects of Chapters 3 and 4 is the extended treatment of the Haskell programming language. According to the authors,

“Haskell is perhaps the most successful attempt to define a practical, completely declarative language.”

The introduction to the language at this point in the book is very interesting, because it serves as a contrast to the available declarative kernel, and illustrates deep connections between Haskell’s design and the goals of declarative computation. By showing how compact and powerful Haskell code is, the authors provide another alternative for developers “stuck” in the imperative world - look for services or pieces of your data flow which can be replaced by something declarative - you’ll be happy you put in the effort. I also found it fascinating as someone who came to CTM with an understanding of what functional meant, but not a very solid understanding of what it means to be declarative. Pure functional programming is a specific case of Declarative programming, a fact that has far reaching implications and is precisely the kind of insight that has gotten me so hooked on this book.

Conclusions

Declarativeness is a property of code which subsumes what professional developers typically refer to as functional and purely functional code. While real world applications don’t allow us to solely rely on declarative code to solve all of our problems, we can embrace its positive qualities and learn to overcome its shortcomings by following two basic approaches: embedding and extending. As we have shown, we can embed declarative code within our imperative codebases if we are sure to protect the boundaries between the two.

Extending the declarative model to include properties for concurrency, for example, can help make declarative code more realistic for application in real world programs. Declarative concurrency is a fascinating subject which has a deep history, and in covering it with the next post, will pause our in depth study of the declarative paradigm as we move on to Message Passing Concurrency in Chapter 5.

Works Cited

All quotes from:

[1] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

Declarative Computation, Syntactic Sugar, and the Abstract Machine

Mon, 17 Jun 2013 00:00:00 +0000

This post is part of a series exploring the incredible book known as CTM. Check the blog index for more.

Declarative Computation, Syntactic Sugar, and the Abstract Machine

The first kernel language that is explored and expanded upon in Van Roy and Haridi’s Concepts, Techniques, and Models expresses what the authors call the Declarative Computation Model.[3] Starting with Chapter 2, there is a large amount of material about this model in the book, and I hope to cover various aspects of it over the course of a few posts.

Leveraging the simplicity of the declarative model, in Chapter 2 we see up close how syntactic sugar enhances expressivity, and are treated to an in-depth look at the mechanics of the abstract machine, a device to reason about the execution of our code. I’ll dive a little bit into each in this post to lay the ground work for expanding the declarative model to include, for example, the primitives for “declarative concurrency.”

We aren’t given the best definition of declarative computation to start, only that it is “evaluating functions over partial data structures” and is sometimes known as “stateless programming.” The few examples here begin to flesh it out, but it is more interesting to grasp the basic mechanics here than to attempt to fill out a larger picture at this point.

Note: All quotes directly from CTM unless noted. Most code adapted from the text.

Syntactic Sugar

The declarative kernel language in the second chapter of CTM is very simple. There are few moving parts, and programs can be read and understood in a linear, predictable way. Nearly the entirety of the syntax of the language (excluding the value expression syntax) including everything we need to declare variables, assign values, execute statements, use conditionals, execute procedures, etc. fits in this small box:

This predictability of the language comes at the cost of reduced expressiveness, but the idea is not to produce a perfect, industrial programming language. The point is to start from the basics, understand their formal foundations, and then progressively expand the expressive capabilities (and therefore the complexity of the underlying semantics) of your kernel language.

For our purposes, we need to know that:

local X in
end

declares an X variable. In other words it makes a variable X available to be bound to another variable or value within the scope of in…end. We also need to understand basic assignment:

X = 2

In our data store X will now be forever bound to 2, and will be indistinguishable from its value. Finally, we have the concept of a procedure (proc). A procedure is a building block for a function in kernel languages. Take the following simple Simple proc form for example:

local Simple in
  Simple = proc {$ X ?Z}
    Z=1
  end
end

This proc will take a variable X and then regardless of its value, bind ?Z to 1. So:

local W in
  {Simple 1999 W}
end

Will end with 1 bound to W. Variables used within a proc do not have to be declared outside of it, but should be bound inside of it to ensure consistent behavior due to static scoping.

Using no syntactic shortcuts, here is how we we could define and use a Max procedure (as it is shown in the book), which given two inputs returns the one that is of greater numerical value. For example, {Max 3 5 X} should bind X to 5.

local Max in
  local A in
    local B in
      local C in
        Max = proc {$ X Y ?Z}
          local T in
            T=(X>=Y)
            if T then Z=X else Z=Y end
          end
        end
      A = 3
      B = 5
      {Max A B C}
      end
    end
  end
end

Using some shortcuts, the authors then show the following version. This can be unpacked to create the code above, or it can be used as it appears here:

local Max C in # declare variables Max (to hold proc) and C (to hold result)
  proc {Max X Y ?Z} # declare the proc (two inputs and one output), store it in Max
    if X>=Y then Z=X else Z=Y end # the Max algorithm
  end
  {Max 3 5 C} # call Max with 3 and 5, store the result in C
end

Syntactic Sugar is a code transformation which preserves semantics but eases use through clarification of intent. The following examples of syntactic sugar have been applied to the code in this example:

Multiple variables declared per line (local Mac C in)
‘In-line’ values instead of variables (proc {Max X Y ?z})
Operations as expressions (if X>=Y)

This illustration of syntactic sugar was an extremely effective way of understanding how the transformations the authors refer to actually work. It also showed me a way to keep programming languages simple: instead of changing semantics or introducing new operators, language authors can find more compact means of expressing the same semantics. The beauty of this reminds me of the tendency toward spare efficiency in good code. Code that can keep functions tight and packed with meaning by providing APIs that are composable is preferable because it easier to understand and reuse.

The Abstract Machine: Definitions

While syntactic sugar is one of the main mechanisms used to extend language functionality, the heart of understanding how the kernel languages operate is in how they execute in the abstract machine. As the authors say:

“We will define the kernel semantics as an operational semantics, i.e., it defines the meaning of the kernel language through its execution on an abstract machine.”

In other words, the abstract machine is the device, or the main metaphor, that is used to explain how the language works. An abstract machine executes the instructions that we give it in a predictable, repeatable way, and gives formal credence to our programmer’s intuition for how some basic programs work.

The abstract machine has a few components that need defining before we can examine how a relatively simple bit of code is executed.

Single assignment store ($\sigma$): a set of variables. Variables can be unbound (declared but not assigned a value) or bound (declared and assigned a value). Bound variables can be assigned the value of a number, a data structure, or another variable, but not rebound.

Environment ($E$): a mapping from variables to members of the single assignment store $\sigma$.

Semantic Statement ($ST$): a pair <$S$, $E$> where $S$ is a statement to be executed in environment $E$.

Execution State: The current state of the abstract machine.

Computation: The work done by the abstract machine:

“A single transition in a computation is called a computation step. A computation step is atomic, i.e., there are no visible intermediate states. It is as if the step is done ‘all at once.’”

With these definitions in place, we can proceed with program execution.

The Abstract Machine: Program Execution

An illustration of a small bit of declarative code (which actually does not have much syntactic sugar applied to it) as executed in the abstract machine can appear simple but still be quite instructive. The following piece of code and the accompanying prose that explains how it is executed in the abstract machine will illustrate how the declarative kernel language’s notion of static or lexical scope is calculated. Note the new Browse function, which displays the value given to the user, and that the <s> statement comments are referred to in the steps below.

local X in #<s> begin
  X = 1
  local X in #<s>1 begin
    X = 2
    { Browse X }
  end #<s>1 end
  { Browse X } # <s>2 begin/end
end #<s> end

Because the code and the kernel language in this state are so simple, you should be able to easily reason about it and predict how it will execute. As an exercise, answer the following question: What values will the Browse functions display, and in what order?

Luckily, we don’t have to just guess or work it out in our heads. The formalization of the abstract machine allows us to “calculate with environments,” or express its execution in a series of steps:

(1) $(ST, \sigma)$

The execution state is represented by a stack of semantic statements $ST$ and a value store $\sigma$.

(2) $([(\lt s \gt, \emptyset)], \emptyset)$

The initial execution state of our program in the abstract machine. $E$ has been replaced with $\lt s \gt$, a statement which represents the outermost $local$ statement in the program, and $\emptyset$, meaning there are not yet any mappings present. The value store has also been replaced with $\emptyset$ - it has no members.

(3) $([(\lt s \gt_1 \lt s \gt_2,\{X \rightarrow x\})], \{x = 1\})$

The outermost $local$ statement and $X = 1$ are executed. There is a sequential composition $\lt s \gt_1\lt s \gt_2$ remaining, the environment reflects that $X$ refers to the store variable $x$, and one binding ($x=1$) is added to $\sigma$.

(4) $([(\lt s \gt_1, \{X \rightarrow x\}), (\lt s \gt_2,\{X \rightarrow x\})], \{x = 1\})$

After executing the sequential composition we see that each statement ends up with its own environment; they are now on the execution stack. We will sequentially execute each statement next.

(5) $([(X=2 \{Browse \space X\}, \{X \rightarrow x’\}), (\lt s \gt_2,\{X \rightarrow x\})], \{x’, x = 1\})$

Executing $\lt s \gt_1$ binds a new variable $x’$ and calculates a new environment. This has a clear semantic specification; given an initial $E$ of $\{X \rightarrow x\}$, adding $\{X \rightarrow x’\}$ to it will yield $\{X \rightarrow x’\}$. The second mapping of $X$ overrides the first.

(6) $([(\{Browse \space X\}, \{X \rightarrow x’\}), (\{Browse \space X\}, \{X \rightarrow x\})], \{x’ = 2, x = 1\})$

$\lt s \gt_2$ is a $Browse$ statement, and $X = 2$ is bound. From this we can tell that we will see $Browse$ run twice - the first will show 2, and the second will show 1.

As demonstrated, the programmer’s intuition for scope can be reduced to a mechanical calculation. Awesome.

Conclusions

Most of us spend our time using programming languages which are considerably more expressive and complex than the declarative kernel language. The power of the kernel language approach is in its ability to illustrate how complexity and expressiveness can be derived from simple tools. As more elements are added to the language, the rules involved in translating the results from the computation steps in the abstract machine will also become more complex.

It’s an interesting exercise to consider how the programming language constructs you rely on could be composed of smaller, more atomic elements. Are all of the reducible? Working on a language like Ruby would be quite different than trying to reduce JavaScript or Scheme. Other paradigms for teaching computation from the ground up, including the interpreter approach championed by Friedman and Sussman advocate extending interpreters for smaller languages in order to run more complex ones. [1][2] This is an equally compelling approach that I intend to investigate in the future.

Now that we have seen how to define and execute a small language, we can begin to extend it. The variables in the Single Assignment Store have interesting properties that the authors leverage for various purposes, and before long a useful, beautiful language emerges.

Works Cited

[1] Harold Abelson, Gerald Jay Sussman, and Julie Sussman. Structure and Interpretation of Computer Programs, Second Edition. The MIT Press, Cambridge, Mass, 1996.

[2] Friedman and Wand. Essentials of Programming Languages. MIT Press, hardcover, ISBN-10: 0-262-206279-8, April 2008

[3] Van Roy and Haridi. Concepts, Techniques, and Models of Computer Programming MIT Press, hardcover, ISBN 0-262-22069-5, March 2004

Matter, Anti-Matter, and the Unified Theory of Garbage Collection

Wed, 12 Jun 2013 00:00:00 +0000

Matter, Anti-Matter, and the Unified Theory of Garbage Collection

Introduced in 1960 by John McCarthy and George Collins respectively, tracing and reference counting are both forms of the type of automatic memory management known as garbage collection (GC).[4] [2] This name is derived from the fact that GC algorithms exist to seek out and reclaim memory that has been allocated, but is no longer needed.

Reference counting and tracing are often viewed as fundamentally different, incompatible forms of memory management. An extraordinarily insightful 2004 paper by Bacon, Cheng, and Rajan, A Uniﬁed Theory of Garbage Collection draws incredible parallels between these two approaches to GC.[1] It begins by introducing the insight that tracing and counting are related and goes so far as to formalize and generalize the work done by GC algorithms in order to prove their point: these approaches are in fact “algorithmic duals” of each other which take the same approach from different starting points. In other words, the difference is that:

“…tracing operates on live objects, or ‘matter’, while reference counting operates on dead objects, or ‘anti-matter’. For every operation performed by the tracing collector, there is a precisely corresponding anti-operation performed by the reference counting collector.”

The Technique

I’m very interested in this deep insight and thought it would be fun to share the technique that they use to demonstrate their point. While the heart of the paper is actually based on a mathematical formalization known as a fix-point formulation, the basic intuition is easily expressed. It is more or less as follows:

Explain the basics of naive implementations of tracing and reference counting
Introduce small, commonly applied optimizations to each algorithm which do not impact their complexity
Illustrate how these optimizations “pull” each paradigm toward a common center-point
Examine real-world GCs and illustrate that they are in fact, hybrid counter-tracers

A naive implementation of reference counting is very simple. Objects on the heap have an associated reference count. References are maintained by a write barrier through which all destructive actions are routed. Here’s a Ruby pseudocode version of the algorithm, adapted from Jones et al.’s The Garbage Collection Handbook.[3]

# The program interface to the heap
def new
  ref = allocate
  if ref.nil?
    raise "Out of memory"
  end
  ref.ref_count = 0
  ref
end

# The write barrier - all writes are redefined to include administrative
# duties for reference counts.
def write(src, i, ref)
  add_reference(ref)
  delete_reference(src[i])
  src[i] = ref
end

def add_reference(ref)
  rc.ref_count = rc.ref_count + 1
end

def delete_reference(ref)
  ref.ref_count = ref.ref_count - 1
  if ref.ref_count == 0
    ref.pointers.each do |field|
      delete_reference(field.address)
    end
    free(ref)
  end
end

The naive mark and sweep implementation is even simpler than reference counting in the respect that no write barrier is involved. When object allocation is requested, if no memory is available, the GC is invoked. Starting from a set of objects (the matter in Bacon et al.’s formulation) known as the roots, the entire reachable heap is scanned, and objects which are referenced are marked as alive. The full heap is then scanned and objects which aren’t marked are presumed dead, and collected. Here’s Ruby pseudocode for a naive mark and sweep, again translated from Jones.

# The program interface to the heap
def new
  ref = allocate
  if ref.nil?
    # The following two methods are the GC
    mark_from_roots
    sweep(HEAP_START, HEAP_END)
    ref = allocate
    if ref.nil?
      raise "Out of memory"
    end
  end
  ref
end

def mark_from_roots
  worklist = Worklist.new
  heap_roots.each do |root|
    ref = root.address
    if ref && !ref.is_marked?
      ref.set_marked
      worklist << ref
      # performs mark on the descendants of this ref
      recursive_mark(ref.children, worklist)
    end
  end
end

def sweep(heap_start, heap_end)
  object_cursor = heap_start
  while object_cursor < heap_end
    if object_cursor.is_marked?
      object_cursor.unset_marked
    else
      free(object_cursor.ref)
    end
    object_cursor.next
  end
end

The Tweaks

There are situations in which the way that naive reference counting algorithms allocate and free memory are desirable, but for the most part, it is necessary to offload some of the tax on the mutator, or main thread of program execution. While not a recommended optimization, in order to push the two paradigms toward each other, the authors suggest a simple tweak to the reference counting algorithm: buffer decrement operations. Under this algorithm, when space is needed during an allocation, decrement operations which have been buffered in a work-list are recursively iterated over. Objects with a zero reference count are collected and objects which reference them are handled appropriately. This work-list contains anti-matter. I have translated both the tracing and reference counting modified algorithms to Ruby for comparison with the naive implementations above.

def new
  ref = allocate
end

def collect
  scan_by_counting
  sweep_for_counting
end

def scan_by_counting
  work_list.each do |work|
    work.ref_count = work.ref_count - 1
    if work.ref_count == 0
      recursively_scan(work)
    end
  end
end

def sweep_for_counting
  object_cursor = heap_start
  while object_cursor < heap_end
    if object_cursor.ref_count.zero?
      free(object_cursor.ref)
    end
    object_cursor.next
  end
end

def dec(x)
  work_list << x
end

def inc(x)
  x.ref_count = x.ref_count + 1
end

def assign(a, p)
  old = a
  a = p
  dec(old)
  inc(p)
end

The mark and sweep algorithm is altered in a similarly interesting way: instead of maintaining a mark bit for whether the object is live or not, a true reference count is maintained. This doesn’t impact the algorithm in complexity or really change it much conceptually, but it does allow the Ruby pseudocode for tracing and counting to look eerily similar.

def new
  ref = allocate
end

def collect
  initialize_for_tracing
  scan_by_tracing
  sweep_for_tracing
end

def initialize_for_tracing
  work_list << heap_roots
end

def scan_by_tracing
  work_list.each do |work|
    work.ref_count = work.ref_count + 1
    if work.ref_count == 1
      recursively_scan(work)
    end
  end
end

def sweep_for_tracing
  object_cursor = heap_start
  while object_cursor < heap_end
    if object_cursor.ref_count.zero?
      free(object_cursor.ref)
    else
      object_cursor.ref_count = 0
    end
    object_cursor.next
  end
end

In this modified form, the heart of each algorithm is very similar. As the authors say:

“By changing two characters in the heart of the algorithm, we have changed from tracing to reference counting!”

And indeed I applaud their use of an exclamation point. This is truly an amazing insight and the mechanics of explaining it are made very plain in the original paper, which as always I suggest you read. The work-list in one algorithm is full of matter, and in the other, anti-matter. The heap is swept, counts are maintained, garbage is collected. What do we have to learn from the similarities of these two algorithms? Most compellingly, the authors demonstrate that “realistic” Garbage Collectors that we use every day are all hybrids.

Real-World Hybrids

“…all high-performance garbage collectors are in fact hybrids of tracing and reference counting techniques.”

Generational GC, which I have written about elsewhere, is an excellent example of a class of GC algorithms which are hybrids. Generational GC is based on the hypothesis that most objects “die young,” so objects which have survived multiple generations should not be swept before objects which have not. These “generations” are maintained by dividing the heap into a young and mature space which are each collected at their own rate. There are issues with GCing objects when older generation objects refer to younger generation objects, but not vice versa. In order to avoid these safety issues, generational GCs maintain a remembered set of objects in the young generation which are pointed to by objects in the mature space. From the paper:

“The remembered set is maintained by a write barrier which is executed at every heap pointer update. The write barrier checks whether the pointer crosses from mature space to the nursery, and if so adds it to the remembered set. By now, the analogy with previous collectors should be somewhat obvious: the write barrier is the assign function, which we have observed is correlated to the reference counting portion of a collector.”

In other words, Generational GCs are hybrid tracer-collectors. Other compelling examples are offered in the paper. Additionally, the cost calculation formula that the authors devised is thoroughly discussed, cycle collection is treated in-depth, and more.

Conclusions

This post merely attempts to expose the central most interesting technique used in this paper to illustrate the point of the “algorithmic duality” between reference counting and mark and sweep, or tracing, garbage collection. After implementing various real-world GCs, the authors had formed a deep intuition for the heart of GC algorithms and this insight sprang from this experience. After studying GC heavily in preparation for this blog and a conference talk, I felt that nothing made the abstract concepts at play click for me as well as this paper did. I hope I have been able to share some of this paper’s insight if not some of its enthusiasm.

After demonstrating these similarities, the authors also posit that the design of GCs could be “made more deliberate.” Instead of choosing between reference counting and tracing, designers should consider the following choices:

“Partition: divide memory (heap, stack, and global variables) into a set of partitions, within which different strategies may be applied; Traversal: for each partition, decide whether the object graph will be traversed by tracing or reference counting; and Trade-offs: for each partition, choose space-time trade-offs such as semi-space vs. sliding compaction, pointer reversal vs. stack traversal, etc.”

This shows that the GCs which are successful in real-world application exhibit either a great degree of flexibility through tunable parameters or specialization in a field like Real-Time GC, but are always hybrids. In other words, successful GCs are those which are chosen based on a fine-grained understanding of their operational characteristics and how they will support the mutator most efficiently.

On a parting note, the comparison of two long-standing approaches to a common challenging problem reminded me of the tensions between the successes and failures of Object-Oriented and Functional Programming. Though not “algorithmic duals” by Bacon et al.’s definition, there are many texts which, using a Lisp implementation for example, stretch a functional system until it appears Object Oriented, or torture the Object Oriented principles until they appear functional. More than a similarity in expositional style, however, these two pairs of systems share parity in other interesting ways. A distant, vague connection, perhaps, but that’s enough for me.

If you like this kind of content, follow me on Twitter for more.

Works Cited

[1]David F. Bacon, Perry Cheng, and V. T. Rajan. A Uniﬁed Theory of Garbage Collection In OOPSLA 2004 [OOPSLA04], 2004, pages 50-68

[2] George E. Collins. A method for overlapping and erasure of lists. Communications of the ACM, 3(12):655-657, December 1960.

[3]Richard Jones, Antony Hosking, and Eliot Moss. The Garbage Collection Handbook: The Art of Automatic Memory Management. CRC Applied Algorithms and Data Structures. Chapman & Hall, August 2012.

[4]John McCarthy. Recursive functions of symbolic expressions and their computation by machine, Part I. Communications of the ACM, 3(4):184-195, April 1960.

To Know A Garbage Collector: GoRuCo 2013

Mon, 10 Jun 2013 00:00:00 +0000

To Know A Garbage Collector: GoRuCo 2013

After being an attendee of the Gotham Ruby Conference (better known as GoRuco) for five consecutive years, I was very pleased to get the chance to speak this year, which was my sixth. I have learned a great deal both about programming and community from the New York City Ruby groups that I have been a part of, and GoRuCo is always a highlight of my year conference-wise. As you can see here, here, and here, I have tried to be thorough in my research in preparing, and have had a great time learning so much about such a fascinating topic.

The talk was recorded to be released later, and I will add a link here when that happens. In the meantime, I have written up a short summary and embedded the slides below.

Influences

After introducing myself and making some corny jokes, I spoke a bit about three main influences, material-wise, for my talk.

*Jones et al.’s *The Garbage Collection Handbook ** - I’ve written about this book a few times before, but the gist of it is: if you want to know anything about the art of automatic memory management, this is your source. I focused on this a particular quote as my entry point into the wild world of Garbage Collection:

“The undecidability of liveness is a corollary of the halting problem”

I couldn’t possibly do the depth of this quote any justice here but it put the discipline in perspective and made me realize that GC is truly one of the quintessential Computer Science problems.

Cooper et al.’s Teaching Garbage Collection without Implementing Compilers or Interpreters - I wrote about this paper in a blog post, so I won’t say more than that I specifically mentioned how the framework described in this paper allowed me to see GC algorithms in a concise and clear way that wasn’t possible elsewhere.

Bacon et al.’s A Unified Theory of Garbage Collection - I made sure to play up the important role that this paper played in how I intended to describe some interesting characteristics of the relationship between various Garbage Collection algorithms. I’m going to write a post on this paper, so I won’t cover it here, but I highly recommend it.

Two Goals

I framed the talk in terms of two goals that I had for the audience. These were to get excited about GC and to think about the connection between programming languages and GC. I found that after many attempts to form the exposition in other terms, the idea of being very clear up front with the audience with the goals helped focus me and made editing considerably easier.

Get excited about GC - This was the most challenging aspect of the presentation to put together because I intended to do the following in 30 minutes:

Define GC in simple terms
Define some technical terms relating to GC
Re-Define GC in technical terms
Talk a little bit about the academic history of GC
Describe Mark & Sweep (Tracing) GC Algorithms using Ruby pseudo-code
Describe Reference Counting GC Algorithms using Ruby pseudo-code
Discuss a few pros and cons about each type of GC
Introduce the Unified Theory
Gradually refine the Unified Theory

And that’s it. Teaching the basics of 8 technical terms, 2 algorithms, and comparing and contrasting them in that short a period of time is an extremely fun challenge. (Note that I sort of cheated on this challenge because as a former Middle and High School teacher I often had to teach at a similar pace.) Looking at the talk as a “lesson” for a group of people that are probably smarter than me guided me; I aimed to satisfy those that had some knowledge or intuition about how Ruby interacts with its GC.

Think about the connection between programming languages and GC - I asked some questions about how programming languages are designed and created, segueing from a point in Bacon et al.’s paper about how design of GC can be “made more deliberate.” Various Ruby implementations, Java, and Haskell were discussed briefly before I summarized, concluded, answered some questions (great ones! About how Linux copy-on-write behavior impacts GC, what generational GC is, etc.), and said goodbye.

In Conclusion

This talk wasn’t a “how to guide” for tuning your application’s GC. The idea was to give people the technical and critical tools to understand that like most complex problems in programming, the more you make GC selection and tuning decisions based on your domain and requirements, the more successful you will be.

Garbage Collection is designed to be transparent. Your goal should be to control the GC completely independently of your application code, and allow developers to just express business logic with application code, as it was meant to be. I spent some time during the talk discussing the following quote, which I feel is appropriate:

“GC is not a generic solution for memory leaks, but a (correct) GC is a generic solution for ‘dangling pointers’. Just as there is no general solution for ‘loops’ (due to undecidability), there is no general solution for ‘leaks’.” – Henry Baker

Finally, I’d like to thank all of my friends and colleagues for their help preparing for this presentation. During research they helped me with guidance and critiques, and were instrumental in me providing a great experience for the GoRuCo attendees.

The Video

</param></param></param></embed>

The Slides

Real-Time Garbage Collection Is Real

Mon, 03 Jun 2013 00:00:00 +0000

Real-Time Garbage Collection Is Real

Real-Time Garbage Collection (RTGC) tests the limits of what we typically think is possible in production software. You might hear “Real-Time” and “Garbage Collection” together and think “If you truly need real-time guarantees, why would you use a garbage collected language?” or “How is real-time Garbage Collection even possible?” Or if you’re like me, you might even want to know “What is Real-Time?” It turns out that RTGC is very real, and research on it is very active. If you want to learn more, there is fantastic coverage of the subject in Jones et al’s The Garbage Collection Handbook, which heavily influenced the writing of this post: unless otherwise noted, quotes are from Jones.[5]

This post is a brief introduction to the concepts of RTGC with a focus on how the work in such a system is scheduled. Along the way, the casual observer can maybe learn a lesson about the ever-present trade-offs that stretch our limits as programmers and thinkers, and make insane ideas like RTGC possible.

What is Real-Time?

Since Garbage Collected languages have conquered the enterprise and are thoroughly mainstream, their benefits are desired in real-time scenarios. These days, Real-Time Garbage Collected software can be found running everywhere from dedicated server applications to embedded devices.[6] In order to be able to discuss some interesting aspects of implementing RTGC, it would be helpful to have some definitions. Here are a few key terms:

Garbage Collection (GC) is a form of Automatic Memory Management which gives a program the appearance of infinite memory by reclaiming allocated objects which are no longer in use. The implementation details of GC include how memory is allocated, stored and referred to during program execution, and eventually reclaimed and freed.
A real-time system is “a hardware or software system that is subject to deadlines from event to system response.” Some soft real-time systems such as video displays can tolerate the occasional failure; a dropped frame isn’t the end of the world. There are also hard real-time systems such as those running internal combustion engines, which rightly consider even one missed deadline a total system failure: missing a deadline can mean damaging an engine.
The mutator is the part of a running program which executes application code, so called because “from the collector’s point of view it simply mutates the graph of objects”
The collector is the part of the running program which handles the duties of GC
Mutator utilization is the “fraction of CPU time used by the mutator, as opposed to by the collector”

With those in place, Real-Time Garbage Collection is Automatic Memory Management capable of:

Guaranteeing a certain amount of mutator utilization in a given time window
Accounting precisely for all mutator interruptions
Ensuring that space bounds are not exceeded

Which is a pretty amazing feat, even once you understand that RTGC isn’t a magical formula that somehow makes the trade-offs present in regular GC go away. In order to put a finer point on the requirements listed above, consider the following illustration of program execution and GC behavior:

Mutator (white) and Collector (non-white) timings
during normal execution and two GC runs.
Top: Stop the World, Middle: Incremental, Bottom: Real-Time [3]

The top band shows two long GC pauses which reflect normal behavior of Stop the World GC such as vanilla Mark-and-Sweep. While the pauses appear consistent, they are each too long. For this reason, RTGC is not often pursued in a non-incremental context: it is too difficult to keep pause times below a desired threshold. The middle band, which shows shorter but more irregular GC pauses over two collections, illustrates a normal incremental GC such as a Baker copying collector. While the lengths are better, the pauses are simply too irregular. This prompted Jones to say, in the last chapter of a textbook which covers every classic and most well-known modern collectors:

“…as real-time systems are now understood, none of the previous algorithms live up to the promise of supporting true real-time behavior.”

The bottom band above shows ideal behavior for RTGC. Pauses are short and regularly spaced out - in a word, predictable. I was drawn to this means of visualizing GC behavior by the authors of a very interesting approach to RTGC, known as The Metronome.[1] [3] The algorithm is interesting for a number of reasons, but I found one detail particularly striking, as a tweak to the algorithm is made in an area not often explored in GC. The authors suggest that a different means of scheduling the collector execution is key to guaranteeing sufficient mutator utilization.

Scheduling Real-Time Garbage Collection

In order to satisfy the guarantees described above, RTGC algorithms need to pay special attention to how the collector will be called to do its work. There are three primary ways of describing how RTGC can schedule the work for its collector:

Work based scheduling executes small amounts of collector work each time the mutator does work
Slack based scheduling runs the collector when the mutator is not working
Time based scheduling periodically schedules and runs the collector according to an interval

The Metronome authors made some headway in the area of RTGC scheduling. As they state:

“We show both analytically and experimentally that time-based scheduling is superior, particularly at the short intervals that are typically of interest in real-time systems. Work-based algorithms may achieve short individual pause times, but are unable to achieve consistent utilization.” [2]

I found this to be fascinating - it’s true that work and slack based scheduling approaches can provide predictably short pause times, but they do not have the ability to make the decision to also guarantee an acceptable amount of mutator utilization. Jones brings up another very interesting paper, A hard look at hard real-time garbage collection where the author concurs:

“I see a growing consensus that time-based methods, in which the garbage collector is treated as a separate task in the real-time scheduling problem, allow more realistic timing analysis of the real-time code, by decreasing GC “taxes” on mutator operations.” [4]

That’s just how the Metronome functions when it is run with the time-based scheduler. During programming execution, many times per second, a heartbeat runs which causes the system to schedule small units of collector work if necessary. The algorithm can target a particular percentage of mutator utilization and guarantee that it will be preserved over a sliding window, scheduling even sized slices of time for either the mutator or the collector.

A Trade-Off and a Send-Off

Just to remind you that I’m not lying, Real-Time Garbage Collection is real, and this is not magical: there are trade-offs to be made. In order to be able to provide a consistently scheduled amount of memory, we have to ease off on our requirements for space bounds - our running programs may exceed the amount of space we want them to take (but not the amount of space that we estimate they could possibly take).

While reading the literature about RTGC, I was struck again and again by how the basic principles that non real-time software developers operate by are invalid in a world of such strict guarantees. In order to consider deploying real-time code, an enormous amount of testing, measurement, prediction, proof, etc. has to occur. The Garbage Collector can operate with the knowledge that it can expect a consistent amount of work, within a reasonable degree. Jones points out that work-based scheduling can guarantee our requirements as long as the amount of work done can be properly estimated. Time-based scheduling can provide stronger time guarantees but will have to use more space during times of heavy use if the system is not expecting it. Trade-offs. Always trade-offs.

Thanks to James Golick for his proofreading and Computology expertise.

Works Cited

[1] David F. Bacon, Perry Cheng, and V.T. Rajan. The Metronome: A simpler approach to garbage collection in real-time systems. In On The Move to Meaningful Internet Systems 2003: OTM 2003 Workshops, volume 2889 of Lecture Notes in Computer Science, pages 466-478, Catania, Sicily, Italy, November 2003. Springer-Verlag

[2] David F. Bacon, Perry Cheng, and V.T. Rajan. A real-time garbage collector with low overhead and consistent utilization. In POPL 2003 [POPL03], 2003, pages 285-298.

[3] Perry Cheng. A Mostly Non-Copying Real-Time Collector with Low Overhead and Consistent Utilization PowerPoint presentation. Available Here

[4] David Detlefs. A hard look at hard real-time garbage collection. In ISORC 2004 [ISORC04], 2004, pages 23-32.

[5] Richard Jones, Antony Hosking, and Eliot Moss. The Garbage Collection Handbook: The Art of Automatic Memory Management. CRC Applied Algorithms and Data Structures. Chapman & Hall, August 2012, pages 375-416.

[6] Tony Printezis. On measuring garbage collection responsiveness. Science of Computer Programming, 62(2):164-183, October 2006

A Generation Ago, A Thoroughly Modern Sampling

Tue, 28 May 2013 00:00:00 +0000

A Generation Ago, A Thoroughly Modern Sampling

The Artificial Intelligence Laboratory at the Massachusetts Institute of Technology was a keystone institution responsible for a sizable share of what we now think of as hacker culture, fusing deep Mathematics and Computer Science knowledge with a culture of progress, exploration, and documentation. An astonishing quantity of written material came from the Lab, including the “A.I. Memos,” a series of papers which cover research topics engaged in by a variety of many of the most brilliant and imaginative minds in Computer Science history.

The origins of Lisp are in these memos, including many written by our field’s pioneers like John McCarthy, who was communicating the accomplishments of the language via beautiful, roughly typeset pages with headings like this:

McCarthy, Minsky, Sussman, Steele, and more all contributed fascinating, groundbreaking insights into their research or the research of their students, and things remained interesting throughout the long history of the organization. As they tried to build faster, more efficient, and more powerful machines, they continued to publish their findings, curiosities, and musings as memos.

This post couldn’t possibly serve as an overview of the breadth or depth of the work done in these memos, but recent research made it possible for me to have to read one of them while studying Generational Garbage Collection, and I found a lot of insights and surprises beyond the known bit of innovation I was seeking. I was surprised by many other things, including the influence that Lisp had on the design of the algorithm, the design considerations for modern-for-1981 computers, how instrumenting and tuning the generational GC was invented right alongside the algorithm itself, and how Lieberman and Hewitt are interested in memory management’s impact on code style.

A Tome From Two Great Minds

“A Real Time Garbage Collector Based on the Lifetimes of Objects”(pdf link) by Henry Lieberman and Carl Hewitt introduced the concept of generational garbage collection, the algorithm that is used in what is widely considered the fastest and most modern virtual machine for running multi-purpose, object-oriented code, the Java Virtual Machine (JVM).[1] [2]

A multi-sectioned, appendix-heavy monster, A.I. Memo No. 569a was published in October of 1981 with the Keywords:

“Real time garbage collection, reference counting, Lisp, object-oriented programming, virtual memory, parallel processing”

and touts an algorithm with the following properties:

“Makes storage for short-lived objects cheaper than storage for long-lived objects; Operates in real time - object creation and access times are bounded; Increases locality of reference, for better virtual memory performance; Works well with multiple processors and a large address space.”

At this point it should not be novel for me that authors on the cutting edge of technology one year after my birth, a generation ago, should be concerned with more or less the same exact things that I am, but for some reason, it is. The origin of deep ideas beget generations of implementations, each of them nuanced and purposeful. Plus, I love the style and depth of this paper. I love the way the audience is treated, as knowledgeable and excited, which the authors clearly are.

There’s Lisp In The Generational GC

Hewitt and Lieberman’s main motive in working on a Garbage Collection algorithm was to improve the performance and scalability of Artificial Intelligence programs. While the authors are primarily concerned with Lisp, as their proclivity for name-dropping other obscure programming languages (AMORD, ETHER, KRL, and OMEGA to name most of them) proves, they also had their sights set on other pastures.

Artificial Intelligence applications are characterized as long-running and processor intensive, and the authors recognize that existing algorithms for automatic memory management tend to penalize programmers for code that should be faster. A metaphor of “renting vs. buying” is invoked, where there shouldn’t be the same amount of overhead paid for short-lived objects as for long-lived ones, and there should be many ways for the system to know if an object will be long lived.

The algorithm is described as an extension of Baker’s copying algorithm, which works roughly like this:

The heap is divided into two spaces, the from space and the to space
Memory is allocated in the from space
When the from space runs out of memory, the space is scanned in a scanning process and objects are evacuated to the to space
A flip operation occurs, and the from space is now the to space and vice versa
Objects with references to objects in the from space are updated in a scavenging process

Hewitt and Lieberman’s algorithm strives to make short lived object allocations cheaper, after the observation that most objects die young, and that it is expensive to scan objects on the heap which are known to be long-lived. Adding the consideration for the lifespan of objects is seen as the central contribution of this paper. It turns out that relying on the so-called “weak generational hypothesis” in and of itself is not enough, but the hypothesis served as a starting point for a direction that is still seen as fruitful today, over 30 years later.

The generational algorithm starts by considering a Baker like algorithm which substitutes many smaller regions for the Baker semi-spaces. The same operations scanning, evacuating, and scavenging occur, but on a smaller scale, per region. The paper explains their primary function:

“We will use these fine divisions of the address space to vary the rate of garbage collection for each region, according to the age of that region. Recently created regions will contain high percentages of garbage, and will be garbage collected frequently. Older regions will contain relatively permanent data, and will be garbage collected very seldom.”

One of the challenging aspects of copying or moving garbage collectors is dealing with objects that hold references to other objects, particularly if older generation objects hold references to younger generation objects. This is an issue because we want to optimize for the fast creation and destruction of young objects, and if older objects hold references, we will have to make sure that they don’t get out of sync. The authors propose an “entry table,” where references to young objects from the older generations would be stored in an intermediary table that could be updated, without having to scan all of the older generation objects. This concept is the precedent to the modern concept of the “remembered set” and “write barrier” techniques in modern generational garbage collectors and is a key “hook” or insight that the authors had.

As clever and shockingly effective as the algorithm is, the salient part is that it was directly inspired by Lisp. The concept arose from software created on a completely different architecture and used and measured in radically different ways, but remains a driving force in practical application of garbage collection in today’s virtual machines.

The authors say:

“We intend to exploit some empirically observed properties of heap storage. Most pointers point backward in time, that is, objects tend to point to objects that were created earlier. This is because object creation operations like CONS can only create backward pointers, since the components of the object must exist before the object itself is created.

Garbage Collector Tuning: Instrumentation and Heuristics

The simple fact that the age of objects are taken into account helps the authors feel confident that their algorithm will be capable of providing efficient performance for a wide variety of programs. However, they are concerned with measuring and demonstrating this efficiency, lamenting that “judging garbage collection algorithms is tricky.” Not ones to shy away from a difficult problem, the authors provide a set of guidelines that we can use to measure aspects of our Lisp programs that could then be applied to garbage collection research. This is a really interesting list:

How fast are objects created?
How fast do objects become inaccessible?
How often do pointers point to objects that are younger than themselves, versus pointing to older objects?
How much locality is there in the program?

After adding that “Certainly the trends are in the direction of programs with increased locality and toward programs that rely on object creation rather than modification,” the authors lay out several ways in which we might be able to tune how our running programs manage memory.

“Often a sophisticated user is in a position to know whether a particular object is likely to be relatively temporary or more permanent. The system should be able to take advantage of such knowledge to improve the performance of the program. It might be advantageous to supply the user with several different flavors of object creation operations, so that the system can choose the best allocation strategy appropriate for that kind of object.”

One example, above, is given for how changing the programming language itself might make things easier. We are also offered examples of a few numbers that can be tweaked, like region size, in order to accommodate certain programs. Both of these concepts, which are echoed in modern day compiler hinting and GC tuning are very important to how people think about code and running programs to this day. It is amazing to see that they were introduced alongside the algorithm itself, and thrilling to see the authors discuss these ideas as they apply to a variety of different paradigms, as everything is open, possible, and hackable.

Multiple Processors and a “Large Address Space”

1981 was clearly an intense time to be a researcher in Artificial Intelligence. It’s interesting to try and imagine the disparity between the depth of the technology at the Lab and the speed of information transfer in the world at large. Clearly the authors of this paper had an imperative to consider what the next few computers might look like while developing algorithms that would suffice on existing ones, because the idea that the generational algorithm would function well for computers with multiple processors and a “large address space” comes up multiple times.

“Since processors are continually getting cheaper, multiprocessor machines will soon appear. The incentive for using multiple processor machines is especially important for AI applications. Our garbage collection scheme has been designed to be suitable for implementation on multiple processor machines.”

The authors describe a very interesting sounding system which eschews shared memory in favor of processes with their own heaps which get information passed to them via messages. They have considered the challenges in sharing changing object references and come up with some ideas for tables that can be maintained on the borders between processors to make updating these references simpler.

I was struck by this because of the prevalence of message passing semantics and its role in modern programming languages and garbage collection technology. Languages like Go rely on message passing to reduce shared memory, which relieves some pressure on the garbage collector. Erlang also uses message passing semantics and has per-lightweight-process GC Heaps, making system wide GC pauses very rare. Even though I know that these languages are based on old ideas, it’s always cool to see how concepts applied in modern software originate in the primordial swamp of early Computer Science.

Throughout the paper the concept of a system with a “Large Address Space,” (translation: “a lot of memory”) are discussed. There are a variety of challenges associated with running programs at scale, and the lab was a place where that was possible. Always trying to build and be prepared for the next wave of hardware capability was a crucial skill for an early hacker, a spirit that many 21st century programmers can empathize with.

Style Matters

The final section of the paper, “Cheaper short term memory may improve programming style,” begins:

“It is our hope that making the use of short- lived objects cheaper will lead to improvements in program clarity. Often, complications in program structure are motivated by the need to avoid creating short-lived objects for intermediate results.”

We see this habit in modern programming languages as well, and it is known to hurt readability. The example that the authors provide (reproduced below, in lowercase, as an appendix) shows how the ideal functional implementation of matrix multiplication would allow for a more modular design, while the program that is “considered faster” is more compact, less reusable, and harder to read and understand.

The design of a programming language and the design of the memory management system that supports it have an interesting relationship. It is clear in the provided example that inefficient memory management is used as reasoning for writing programs which are harder to maintain. One of the main reasons for providing a mechanism to create cheap short-lived objects is stylistic, to allow for a more expressive ecosystem of usable code. Other ideas including program transformation and a “smart compiler” are tossed around for making it possible for more expressive programs to be more efficient.

Generational GC A Generation Later

Modern tunable Garbage Collectors have clear roots in the work Hewitt and Lieberman as presented in this paper. Developers who rely on programming languages with automatically managed memory can learn valuable lessons about the tradeoffs present in their tools by understanding where the ideas they rely on came from, but more than a touchstone for a modern technology workhorse, this paper represents an archeological sampling of a wide swath of deep thought in applied Computer Science that encourages, surprises, and enlightens.

The AI Memos are an extremely fertile ground for modern research. While it’s true that what this group of pioneers thought was impossible then may be possible now, it’s even clearer that some things we think are impossible now have been possible all along.

# Special thanks to Maciej Katafiasz for correcting some incorrectly transcribed LISP code below.

Works Cited

[1] “A Real-Time Garbage Collector Based on the Lifetimes of Objects” Henry Lieberman & Carl Hewitt, 1981

[2] “Garbage collection: algorithms for automatic dynamic memory management” Richard Jones & Rafael Lins, 1996. p. 166

Lisp Code Appendix

The fact that code appeared in this paper is one of the reasons that made me want to write about it. I have become very interested in it as I read the paper over and over again, so I decided to type it out to preserve it, in a non-image format, here. The first section is, according to the authors, clearer but less efficient, both due to the use of creating extra lists and functions for clarity and expressiveness. The functions are composable, which the authors also feel lends to their comprehensibility.

(defun dot-product (left-vector right-vector)
  (cond ((or (null left-vector) (null right-vector)) 0)
        ((+ (* (car left-vector) (car right-vector))
          (dot-product
          (cdr left-vector)
          (cdr right-vector))))))

(defun transpose (matrix)
  (cond ((null (car matrix)) nil)
    ((cons (mapcar 'car matrix)
    (transpose (mapcar 'cdr
    matrix))))))

(defun matrix-multiply (left-matrix right-matrix)
  (let ((columns (transpose right-matrix)))
    (mapcar
      '(lambda (row)
               (mapcar
               '(lambda (column)
                        (dot-product row
                         column))
                columns))
                        left-matrix)))

The second code sample is more efficient because it allocates fewer lists and functions, and performs fewer passes. For this reason, two operations are conflated, making this harder to read, and less reusable.

(defun matrix-multiply-without-transposing (left-matrix right-matrix)
  (mapcar
   '(lambda (row)
     (let ((column-index 0)
           (mapcar
             '(lambda (column)
               (prog1 (dot-product-column
                       row
                       right-matrix
                       column-index)
                 (setq column-index
                      (+ column-index 1))))
             (car right-matrix)))))
   left-matrix))

(defun dot-product-column
(row matrix column-index)
  (cond ((null row) 0)
        ((+ (* (car row)
               (nth column-index (car matrix)))
            (dot-product-column
              (cdr row)
              (cdr matrix)
              column-index)))))

Adventures in Garbage Collection Pedagogy and an Introduction to Racket

Mon, 20 May 2013 00:00:00 +0000

Adventures in Garbage Collection Pedagogy and an Introduction to Racket

While preparing for a conference talk on Garbage Collection algorithms, implementations, and history, I have made a habit of scouring the internet for information on how to communicate the challenging ideas that Garbage Collection (GC) encompasses. During one of these frenzies I came across a fascinating paper, Teaching Garbage Collection without Implementing Compilers or Interpreters, which attempts to provide a framework to teach Computer Science students the finer points of GC through the constructivist means of having them actually implement the algorithms. This framework is necessary because according to the authors:

“Many students appear to have a poor grasp of garbage collection.”

While it is interesting to me how students learn, I am much more interested in how professionals learn. It’s no surprise that this concept isn’t discussed much in this paper, as it is clearly beyond its scope, but it is posited that:

“Beyond the pure pedagogic value, this (approach) could have a salutary impact on the software students write in their careers.”

I believe it can. Having tried to draw connections between academic and professional practice before with some success, I’m considering taking something very similar to the approach outlined in this paper in the talk I’m preparing, in order to illustrate the concepts at the core of Garbage Collection and leave a lasting impression.

According to the paper and bolstered by my own experience as a Computer Science educator and recruiter for a startup, GC is taught at the Undergraduate and Graduate levels in varying degrees of granularity, for a wide range of purposes. Some programs merely place GC in the hierarchy of important problems and work areas in Computer Science, acknowledging that it is relied on, but not necessary to thoroughly understand. Other programs attempt to teach students how to implement GC in the context of Programming Language or Virtual Machine design, accepting the burden and relying on a host of prior knowledge from the student for the material to be comprehensible.

The authors propose an interesting way to get at the heart of the issues that GC represents, allowing students to focus on the implementation details of various GC algorithms and their impact on runtimes without having to endure the pain of “curricular dependencies” (parsing, lexing, program transformation, etc.) normally associated with Virtual Machine construction.

This post on the Brown University PLT Blog outlines the technique, which uses the Dr. Racket IDE and Racket programming language to teach the material. The central concept behind this pedagogy is that students should focus on the implementation details of the collector and mutator routines in their runtimes by actually implementing them, because…they are learning about Garbage Collection. The authors repeatedly refer to “curricular dependencies” as a deterrent to being able to teach GC thoroughly and deeply, and they cite evidence that their approach, which allows students to shed a number of these dependencies, leads to students who not only understand memory management and GC, but are excited to use them in practice. The authors make some of their course work available, including a sample collector that simply throws an error when the heap is out of space (shown in a modified form below), and their approach and implementation are both very compelling.

The flexibility and feature-set of Racket makes it possible for students to implement a collector in ten functions in order to be able to interface with a mutator that is provided by the framework.

Function	Arguments	Returns
init-allocator	none	void
alloc-flat value	none	location?
cons	first-addr rest-addr	location?
cons?	addr	boolean?
first	cons-addr	location?
set-first!	cons-addr, first-addr	void
rest	cons-addr	location?
set-rest!	cons-addr, rest-addr	void
flat?	addr	boolean?
deref	addr	heap-value?

Here’s a breakdown of how the framework functions in practice:

Students implement a Garbage Collection algorithm by implementing the functions above. These functions serve as a specification that abstracts GC so that each algorithm, or collector, can be implemented. Note that mark and sweep, compacting, and generational collectors are explored. Concurrent and real time systems are beyond the scope of this work.
Students write limited Racket programs which are translated by the runtime and executed by a mutator which interacts with the system memory through the student implemented collector.
The provided heap is visualized, giving students access to an interactive map which includes the ability to follow pointer references and see the memory layouts students have implemented in a realized form.

Racket is an excellent choice for this kind of experiment and the authors of the framework have worked diligently to provide a smooth experience for students, leveraging some awesome language features along the way. An awesome side effect of diving into this material has been an introduction to this great language and framework up close, which I had only read about in a passive way. One of the most compelling implementation details of the framework is how the authors provide automatic A-normal form program transformation for input programs. Combined with the use of continuation marks and other language features, the framework is capable of automatically “re-routing” memory allocations to the student designed heap and collector. I have provided an example of this below. Contracts are also employed to ensure that any of the implemented functions cannot “cheat” by, for example, allocating complex data structures on the student heap directly without laying them out on their own.

;; This small program
#lang plai/mutator ;; Tell Racket to use the 'mutator' language
(allocator-setup "mrbcollector.rkt" 128) ;; Set up an allocator with 128 spaces
(define (incr-list lst) ;; A simple function and function call
 (if (empty? lst) empty
 (cons (+ 1 (first lst)) (incr-list (rest lst)))))
(incr-list '(1 2 3 4 5))

;; Expands (partially, at least) to the below. Notice the 'set-collector'
;; calls and the invocation of the gui in particular.
(module mutator plai/mutator
  (#%module-begin
   (begin
     (require "mrbcollector.rkt")
     (set-collector:deref! gc:deref)
     (set-collector:alloc-flat! gc:alloc-flat)
     (set-collector:cons! gc:cons)
     (set-collector:first! gc:first)
     (set-collector:rest! gc:rest)
     (set-collector:flat?! gc:flat?)
     (set-collector:cons?! gc:cons?)
     (set-collector:set-first!! gc:set-first!)
     (set-collector:set-rest!! gc:set-rest!)
     (init-heap! (#%datum . 128))
     (when (gui-available?)
       (if (<= (#%datum . 128) 500)
         (set-ui! (dynamic-require `plai/private/gc-gui 'heap-viz%))
         (printf "Large heap; the heap visualizer will not be displayed.\n")))
     (init-allocator))
   (mutator-top-interaction define
    (incr-list lst)
    (if (empty? lst) empty (cons (+ 1 (first lst)) (incr-list (rest lst)))))
   (mutator-top-interaction incr-list '(1 2 3 4 5))))

An example of how Racket macro expansions are used to insert
and reroute allocations to student written collectors

After executing the above program, the heap layout above is displayed

The paper reports a great deal of success with this approach, which leads me to wonder - can I turn it around a bit to help make various schemes clearer to professional developers in a conference setting? There is clearly great power in the simplicity in writing a very small lisp program and seeing your custom memory layout visualized, but will it translate when it is someone else’s? Is the power in this approach in the constructivism that the authors worked so hard to achieve, or is the interface they provide an equally powerful tool?

;; type | value | inuse | x
(define cell-size 4)

(define (gc:alloc-flat p)
 (begin
   (when (> (+ heap-ptr cell-size) (heap-size))
     (map (lambda (s) 
            (print (read-root s))) (get-root-set))
     (error "out of memory"))

In progress work on implementing a mark and sweep collector in #lang plai/collector

Knowing the basic concepts behind each of the major GC algorithms is not the same as implementing them or even pondering their implementation. Developers who rely on Garbage Collected programming languages, including Java, Ruby, and JavaScript to name a few, have a lot to gain by understanding the behaviors relative to the workloads they impose on their systems. It is one thing to know that head space is needed in a heap of a certain layout under certain workloads, and quite another to see it visualized and be able to interact with it.

It is relatively simple to get up and running with this system. Download Racket, follow the instructions on this course page, begin implementing various GC algorithms, and see your favorites in action (an example Mark and Sweep collector can be found here). If you don’t learn something new, at the very least you’ll appreciate the implementation of the GC you rely on daily.

Thank you to the authors of the original paper for their encouragement and links to further resources, and to @jdmaturen for the correction.

Note: An earlier version of this post linked directly to the PDF. A correction was made to link to the information page at the request of the authors.

April 17 2015: Thanks to Nikita Baksalyar for linkrot corrections.

A Nice Feature in core.logic and Thoughts About A Data Structure

Sun, 12 May 2013 00:00:00 +0000

A Nice Feature in core.logic and Thoughts About A Data Structure

core.logic is a library for the Clojure programming language that allows the functionality of Relational or Logic Programming to be embedded in your regular Clojure code. The driving force behind core.logic is David Nolen, who puts a lot of effort into maintaining and improving the codebase of this fascinating library. Recently, David responded on Stack Overflow to a very good question:

How do I de-structure a map in core.logic?

David gives a very nice answer which has a few interesting pieces to it, but I just want to focus on one - the relation called featurec. Here’s the documentation for it:

clojure.core.logic/featurec
([x fs])
  Ensure that a map contains at least the key-value pairs
  in the map fs. fs must be partially instantiated - that is,
  it may contain values which are logic variables to support
  feature extraction.
nil

I asked David where the name came from, as it seemed kind of strange. He responded with a link to this tweet:

Does no Prolog support feature structures (attribute value matrices) as terms?! @swannodette stackoverflow.com/questions/1280…
— Chung-chieh Shan (@ccshan) October 9, 2012

Which takes us to yet another Stack Overflow question, this time asked by David himself. It is summed up by Chung-chieh Shan as “Does no Prolog support feature structures (attribute value matrices) as terms!?” The short of it is this: there is a dearth of material concerning Prolog or other logic programming environment’s treatment of records, dictionaries, hashes, or whatever you prefer to call them.

After some back and forth in the thread, some attention was drawn to the LiLFeS project and its concept of Feature Structures. LiLFeS is extremely obscure, but it is definitely legitimized by its logo:

LiLFeS is built to be a fast Logic Programming system where a dictionary like data structure, called a “Feature Structure” is a first-class citizen. I found it interesting that this was unique amongst Logic Programming enthusiasts and academics, as dictionary-like structures are the High Fructose Corn Syrup of modern dynamic languages.

This episode highlights one of the differences between the work being done on core.logic, and the “Academic” work being done on the inspiration for core.logic, miniKanren. Both push boundaries in different and interesting ways, but core.logic’s integration with Clojure makes treating maps and other import Clojure Data Structures properly that much more important. Developers who come from dynamic programming backgrounds want feature rich support for associative data structures, and featurec is an example of how that can be provided. Let’s take a look at some of the tests for it from the core.logic test suite.

The first test is a very simple three line core.logic program:

(is (= (run* [q]
         (featurec q {:foo 1})
         (== q {:foo 1 :bar 2}))
       '({:foo 1 :bar 2})))

Let’s look at it line by line.

(run* [q]

run* starts the core.logic engine with the understanding that the program will return as many results as possible given the conditions provided, and the logic variable or lvar q is instantiated for use.

(featurec q {:foo 1})

The first condition on q is provided in the next line, above. It states that q is a map which contains the key value pair {:foo 1}. We now know two things about q that we didn’t after the first line ran - that q is a map, and that it contains {:foo 1}.

(== q {:foo 1 :bar 2}))

The last line unifies (the == operator stands for unification in this context) the partially defined map q with the value {:foo 1 :bar 2}. Essentially, this program asks the following question:

When q is {:foo 1 :bar 2} and q contains {:foo 1}, what are all of the possible values of q?

Let’s look at another test which puts a finer point on the above:

(is (= (run* [q]
         (featurec q {:foo 1})
         (== q {:bar 2}))
       ()))

We don’t need to break this one down quite as far. Let’s talk about the two important lines - the featurec and the == clauses. The featurec constrains q to maps which contain the map {:foo 1}, and the == unifies q with the value {:bar 2}. This program asks if there are any possible values of q where q is {:bar 2} and q contains {:foo 1}. Since the answer is no, this program returns the empty list - there are no possible values of q which satisfy these conditions.

(is (= (run* [q]
         (fresh [x]
           (featurec x {:foo q})
           (== x {:foo 1})))
        '(1)))

The last test shows more advanced usage of featurec, bringing us full circle to map de-structuring. Since the maps passed to the second position of featurec do not need to be fully bound, featurec can constrain x to maps which contain the key value pair {:foo q} even though q is the output variable provided to run* and at this point has no specific value. Simply stated:

If x is a map which contains a key :foo and a value q, and x is also a map with the value {:foo 1}, then the unbound variable q must have the value of 1.

featurec is an example of a very useful constraint relation that can serve to declaratively describe loose structure in core.logic programs and replace clumsier attempts to use matche and pattern matching to unify maps. featurec is built on top of the concept of a partial map (PMap in the core.logic source) that encapsulates the functionality of maps which play nicely with purely relational functions. This is nicer than pattern matching for the same purpose because you do not need to explicitly specify all of the keys using featurec as you do with matche.

Clojure programmers often resort to a lot of trickery in core.logic to massage their data structures into lists for compatibility with the classic examples of Prolog, miniKanren, etc. Rich support for maps is one of the things that makes core.logic such an exciting library that is edging ever closer to wider adoption amongst Clojure programmers in non-trivial applications. Drawing on inspiration from a wide array of sources is what makes Clojure so fascinating and vital, and core.logic is clearly following in these footsteps. I hope this blog post helps get you excited enough to run* your own.

A List of Lists: NYC.rb 2013.03.12

Wed, 13 Mar 2013 00:00:00 +0000

A List of Lists: NYC.rb 2013.03.12

I had the pleasure of giving a small talk to a great crowd of developers at NYC.rb recently. It’s different than the typical meetup talk - instead of the group’s topic, I talked about some things I was excited about at the time. I decided to turn the meager ‘slides’ that I had into a blog post - I hope you learn about something you’ll enjoy!

List One: Three Books You Should Read

Books are my favorite things. I am constantly pursuing them, acquiring them, and digesting them, and I always want more. I love getting people excited about books that I’m immersed in, and in this case I chose three titles that might be attractive to programmers. The headings for each title are terms I used on the slides to give the main ‘thrust’ behind each recommendation.

Rethink Everything

Concepts, Techniques, and Models of Computer Programming by Van Roy and Haridi

I’ve already written about how important I feel Van Roy and Haridi’s work is on this blog. I chose “Rethink Everything” because it’s clear that existing developers can learn quite a bit from reading this book - in addition to supplying new information, it helps you understand things you already know in a deeper and broader way. Van Roy and Haridi push you to investigate the complex relationships between various programming paradigms and is truly enlightening.

Click here for more information.

Revisit Ruby’s Roots

Functional Programming for the Object-Oriented Programmer by Brian Marick

Recently a group of developers at Paperless Post have been going through this book as part of an internal book club. It’s been awesome to see how both new day to day Object Oriented programmers and those with deeper OO roots react to solving problems in a functional way. It made me think that the group at NYC.rb would enjoy it too. “Revisit Ruby’s Roots” is a call to think about using Ruby functionally and to understand the deeper ideas behind powerful data structures with expressive interfaces. This book, like CTM and The Art of the Metaobject Protocol before it, does a lot to draw connections between Functional and Object Oriented programming.

Click here for more information.

Get Under The Hood

The Garbage Collection Handbook by Jones, Hosking, & Moss

If you want to know Ruby deeply, to “Get Under The Hood,” you have to try to understand its Garbage Collector. If you want to understand Garbage Collection, you have to read this book. The amount of research that has been done into this area of Computer Science is staggering, and there are still advancements being made - some which blur the line between hardware and software. Jones, Hosking, and Moss explain an extremely complex area of study in a straightforward, digestible way. You don’t need to read the whole thing, but the first few chapters alone are worth it.

Click here for more information.

List Two: Ten Minimalist Albums You Should Program To

A broad, flexible term to refer to a lot of different kinds of music, ‘minimalism’ is not a single genre. Minimalism is an underlying philosophy in the composition, performance, and structure of music and sound, and it typically has a few specific characteristics, including repetition in structure, compositions with extended durations, and effects to warp time. I’ve chosen ten albums I think people would enjoy, and assigned each a tongue-in-cheek ‘occasion’ that might go well with the suggested recording. Where possible I’ve provided a YouTube link to the music, but of course, any albums still in print are worth purchasing.

The Feature Push - Charlemagne Palestine “Strumming Music”

You need to get that code out the door. You can’t wait to :shipit:. Might I suggest 52 minutes of alternately dissonant and melodic pulsing frantic piano playing? I might.

Click Here for the music on YouTube.

The Careful Refactor - Aphex Twin “Selected Ambient Works Vol. 2”

The dark art of the code refactor requires the calming, warm embrace of analogue techno music. In this case, there is no one I would rather hug than Richard D. James.

Click Here for the music on YouTube.

Architectural Decisions - Steve Reich “Music for 18 Musicians”

You can’t believe your luck. You’re going to fucking design something. Holy shit. You better take some Steve Reich with you, you’d be lost without it.