There’s never been a committee, board, or any other official body that directed node.js. But, in the first year of node.js, whenever a problem seemed too insurmountable to handle through email, a small group of us that, for a short time, all lived in the Bay Area would meet. I labelled our group the “Node Illuminati” as a slam against our clearly undemocratic process and lack of transparency.
At the time CouchOne had a beautiful office in recently renovated “Old Oakland” where I maintained a desk and an iMac I seldom used in favor of the IKEA rocking chair and a laptop.
In early October Matt Ranney called a new meeting, mainly to talk about Streams, which were a topic at every meeting, and some recent binary issues. When assembled this meeting consisted of me, Ryan Dahl, Paul Querna, Tim Caswell, Isaac Schlueter and Matt Ranney. Paul and Matt were the adults. They had more experience than all of us and were tremendously respected. Tim, Isaac and I were younger and had taken on a lot of the early node.js experimentation. Ryan was Ryan.
The 800 pound gorilla was "what to do about binary." Buffers (static allocations of memory outside of the v8 heap) were added in April; prior to that we were using binary encoded strings which were slow and meant we easily exceeded the then-max heap size of 1gig. From April through October, Buffers had slowly cemented themselves in to all of the APIs in node.js and their shortcomings were starting to become clear.
“What you really want is a string, most of the time,” Ryan said. The issue, of course, remained that data from the kernel had to be written to something like a Buffer and then converted to a string, requiring among other things a memcopy. Even worse, when the string ends up needing to be written back out to a file or socket it’ll require another conversion.
“Can we get at the raw memory address the underlying string is in so that we can write it out using a writev()?” I asked.
Ryan explained, “The problem is that the string is in the heap, which means at any time it can be compacted and that address in memory won’t be valid anymore and when we give that address to the kernel (this was before libuv, now we hand the address to libuv) it doesn’t write it immediately, it'll do it some time in the future.”
For a while we discussed how we might get hooks into the garbage collector to mark strings to not be compacted, as well as other possible optimizations. Then we started talking about what the rest of JavaScript was doing with binary.
At this time there were some competing standards for defining binary interfaces in JavaScript. The W3C group working on WebGL had a standard called Typed Arrays which eventually won but there was also a binary data proposal being considered for Harmony at ECMA. Tim and I were the most versed in those standards and started asking about how we should converge our work with theirs.
After a few minutes the tone of our conversation changed and we all began to speak as though it was someone else’s job to solve this problem, and that we needed to line up behind their work. Ryan became enraged, “The experts on working with binary data in JavaScript are in this room!”
It wasn’t an exaggeration. The most anyone had done with Type Arrays were gaming demos and the ECMA standard was barely a wiki page. Meanwhile, Matt was pushing voice messages for a few hundred thousand users of Voxer through node.js and in a few months this would increase to millions so his experience alone put our group ahead of anyone working on standards at the time. Ryan’s protest changed our direction and it enabled us to be bolder in solving our own problems.
The eventual solution was much simpler than we had anticipated. As it turns out the slower part of Buffers at that time were waiting on malloc and so we moved to SlowBuffer and FastBuffer. SlowBuffer is the old C code for Buffer that does malloc, and FastBuffer is pure JavaScript code that cuts up a SlowBuffer into smaller chunks. Today, the Buffer object in node.js is FastBuffer.
The longer the project goes on without Ryan being involved day to day the more you can see what his real legacy is. He left the project with a lot more than code, and it’s moments like this that I remember the most. Ryan had no patience for roadblocks. Anything (standing?) between people using node and succeeding in accomplishing the goals of their application was immediately removed. Even worse were committees and standards bodies which are the slowest to accomplish their goals and were to be avoided and even actively ignored.
Proclamations and guiding words have a short lived effect on the community. Ryan rarely commented on stuff like this publicly. What he did instead was far more effective: he changed the way the rest of us felt. Isaac and I were especially impressionable those first few years and Ryan altered our methods considerably. Isaac and I actually met through CommonJS and worked on a few standards together only to later thumb our noses and leave CommonJS to focus more exclusively on node. Now Isaac and others stand around core, greatly influencing the culture which remains very true to Ryan's early ambitions and beliefs.
If you want to leave your mark, and if you want to change something for the better, you need to change people more than code. Write all you want, prove you’re right a hundred times over, nobody will be affected unless you change how they think and enlighten them about a better way.