Broken Promises

James Coglan wrote a lengthy article about Promises in node.js. In his article James quotes the first of many talks I gave last year tracing the causes of node's success. James makes a convincing argument that Promises do, in fact, have features and that node does not ship with several of those features.

The value of a platform is measured by the value you can pull from the ecosystem to write your program. More than any other metric the success of the node.js ecosystem has been the measure of node's success overall. That success is measured in two parts: the absolute number of modules and the degree of compatibility between those modules.

A platform should provide enough API to ensure compatibility, and no more. A healthy community is a productive one. A platform with more people creating value than that which ships with the platform itself is the mark of a healthy community. Although counter-intuitive, doing less encourages more value to be created so long as people are confident that what they create will be compatible and useful to the largest number of people.

People seem to forget that node originally had Promises. We can argue all day about whether or not those were the right Promises, there was certainly plenty of that before Ryan took them out of core, but the effect of removing them was that flow control was no longer defined by node. This lead to several Promise libraries being published along with a host of other flow control approaches.

The proliferation of JavaScript flow control styles wouldn't have happened if node took Promises in to core. Ryan Dahl, on his best day, would not have been able to design a suitable flow control library because Ryan Dahl writes servers in C for fun. Node's success has as much to do with what Ryan and Isaac have been smart enough to leave out than what they put in.

In the time since Promises were removed from node, a greenfield of opportunity for flow control, the ecosystem has mostly rejected Promises and other similar abstractions in favor of "callback managers" like async. Why? Simple, if you stay at the callback layer you retain the most compatibility with the rest of the ecosystem.

Think of the node module ecosystem as a market experiment. For many years node.js has not shipped with flow control determined by node. The module system has exploded and is on track to eclipse Python's by the end of the year. Many flow control libraries are in the npm registry and none have seen significant adoption because the compatibility trade off is too great. Flow control, as a problem that needs to be solved, is not greater than compatibility and people have managed the problem with the least intrusive approach available which is callback management like async.

An ecosystem is built by multiplying the value of one module by another. Python and Ruby have incredibly interesting approaches to alternate concurrency but their biggest failing is that they have several interesting approaches to concurrency and compatibility between modules written on top of them vary. The majority of value available to a Ruby or Python developer is written for the platform's default IO pattern which is blocking. To choose an alternative concurrency pattern is to limit the amount of value available to you. Similar, although not quite as severe, penalties arise when you build libraries in node that use an alternative to API patterns provided by node core.

A lot can be said, and opinions can differ widely, on what the best API is and what is simpler by any individual's definition of simplicity. I think that my views are best summed up by James' own example comparing a Promise to async.

async.map(inputs, fn, function(error, results) {});

is equivalent to:

list(inputs.map(promisify(fn))).then(
   function(results) {},
   function(error) {}
);

These are both fine APIs but if you walk through the code you do need to understand more about the Promise's API than async's. Without even knowing what async.map does I can assume that this array of inputs is manipulated by this function and then, finally, I will receive the final callback and results. To understand the second API I need to know what inputs.map returns and then what list does to that return value. I can figure out what .then() and its callbacks do without learning all about Promises but inputs.map and list require me to do some research. This means that if I publish a library I can expect that anyone who wants to contribute will need to learn a bit about the Promise API but if I used async there's a good chance they could figure out how things worked without reading through that project's documentation.

I don't think people will have a very hard time learning the second API but hiding the semantics behind syntax complicates it and adds to the cognitive load I need to retain in my own head every time I sit down to write a line of code.

In the very early days of node I recall Ryan saying something that stuck with me: "Things that happen in the future should look like they happen in the future." The best thing about node's callback API is that it requires little to no explanation. Passing a function implies it will be called in the future where composing objects and stacking them together do not.

Two kinds of developers will take syntax > semantics, brand new developers and developers who build the abstractions that hide those semantics. A new developer doesn't understand the semantics of either approach and therefor cannot evaluate what is visible and what is not. Most developers, once comfortable with an API, will take a syntax that hides less of the semantics they'll have to keep in their head to write code effectively.

An abstraction should remove the visible semantics when those semantics don't need to be understood by the end developer. But flow control fails this test because the semantics which might be removed, literally the flow that is being controlled, are operations provided by the final consumer of that API.

The person who writes an API is in the worst possible position to evaluate its simplicity. The author necessarily understands all the semantic complexity hidden by their library and can rarely separate what they are hiding that needs to be understood by the consumer of their library and what doesn't.

The platform must make decisions that encourage compatibility and discourage incompatibility in the ecosystem. There's a list of people I see that attack node for having a "mono-culture" whenever they are given a chance. What they take issue with are decisions that core has made or patterns being advocated in the community that encourage compatibility. Compatibility, in many cases, is at odds with diversity but a culture that encourages incompatibility is no longer a productive one.

Being compatible means that people who have a different opinion about the API or pattern you use for compatibility get pushed out. The key to node's success is to make as few of these decisions as possible and let the community work the rest out. So far, a higher level API for flow control hasn't been important enough for the community to get traction around an alternative to callback management. It turns out that this particular problem is not big enough to build an ecosystem on top of.

Individuals that invested in solving the problems node has standardized will always take issue with those decisions. The fact remains that node standardized far less than its contemporaries and ships with a standard library a fraction the size of any competing platform. What has been standardized is certainly different from what other platforms define but the volume is far less and the diversity of libraries in the community for what would have been the standard library is much greater.

Complaining about callbacks in node is like complaining about significant whitespace in Python. It's a decision that's been made, it's part of the platform, and node's growth seems to indicate that people are doing well with what is available and have less of a problem than James and others would suggest. I see little evidence that this is holding back adoption and the failure of alternative styles of flow control to gain wider adoption supports the decision not to include it as part of core.

If Promises were a "missed opportunity" I'd wonder how much more opportunity node could have handled while it grew faster than any platform I've ever seen. Instead, I think that Promises are one of those things that a vocal minority of intelligent people like to talk about while everyone else continues to build a healthy and diverse ecosystem just fine without.