Stream programming

In my recent dalliances with Brain Things I've finally had a chance to play around with array programming, a kind of programming where instead of just adding numbers, you can add vectors, arrays and even matrices. I was mostly using NumPy with a bit of GLSL at the end, and I have to say it's pretty nice.

I actually had to train myself out of my old ways of thinking; I kept wanting to take my big nested arrays and iterate over them, which led to a lot of ugly nested looping code. Later on when I figured out how to array properly, all of that code just disappeared and left me with disturbingly simple one-line mathematical expressions. Crazier still, that shorter and prettier code was also much, much faster.

Which got me thinking of a feature dear to my heart: streams. Streams are sequences much like arrays or vectors, with the difference that you can't necessarily access the whole sequence at once. They particularly suit communication-heavy applications like databases, web servers and, y'know, just about everything we make these days. That's not a coincidence; modern applications tend to be distributed and communicative by nature because they need to scale up and individual CPU cores aren't getting any faster. Streams are so useful that they're baked into Node in a more fundamental way than most other languages.

But I think maybe not fundamental enough. I'd like to see an implementation of streams baked into the language itself, so that operations for manipulating streams of numbers are as primitive as operations on numbers themselves. Imagine if you could add streams, multiply streams, filter streams, connect streams together – all with a simple syntax. And it could be more than a syntactical improvement: being able to describe the stream operations in a semantic way might yield some really cool performance improvements, such as doing stream operations in highly-optimised runtime code or even the kernel.

Being in JS-land for so long has made me forget how cool it is to have custom syntax for things that deserve it. Array programming is definitely a worthy addition to any language, and I think stream programming could be too.