On working through difficult problems
Derw status: March 2022
In the last few years as a programmer, I’ve run into situations where I didn’t know what to do. The problem manifests in two particular ways: either I know too many ways of implementing something, or I can only think of non-trivial ways of implementing it. I’m a big fan of shower debugging. Coding and unsure what comes next? Get an overview of the problem then go shower. It’ll come to you in the shower what you should do. I used to get the same result from sleeping, where you spend all day focusing on a problem, so much that you dream of it, and in the dream you find the solution. That doesn’t happen as much for me any more, but still getting a good night’s sleep is just as important.
I used to pride myself on my ability to create weird programming experiments, at the language level. A simple idea grows from the question how can I implement this? Often I find myself thinking about the fastest way to write something, to prove that it’s possible. Start there, then iterate onwards. When you’re working on a language like Derw, there’s often decisions about code generation which have pros and cons. Maybe if you generate code this way, it’ll be slower. Maybe if you generate it this way, it will break if users use a keyword. The thought process is not merely what is the quickest way to implement this? but also what is the most consistent way to implement this?
That brings me to the problem that I’ve been working on, on and off, for the last month. I want list unpacking to allow for gaps in arguments which can be collected as a single argument. Imagine you want to match against elements inside of parenthesis, and you have a collection of values made up of Paren, and Char. It’d look something like this:
In order to implement this in generated TypeScript, it involves writing a loop that will collect all elements, comparing the start and end element to Paren. Not a difficult task, but what comes up when you’re implementing it are questions like: what about nested cases? What about when there are multiple different collection elements in the list unpacking? I typically refer to what other languages have done, because copying someone else’s research is typically more efficient than performing your own.
But to understand this problem best, I spent a while coding how I thought the syntax should work, then several options on how the code it generates should look. Still not settled on a solution, I took it further by writing documentation on how it should behave, and more importantly, why. I did this on my remarkable - where I am gathering all handwritten notes about language design and implementation. The idea was to step away from the computer and write on something (almost) physical.
With the documentation ready, I now knew how I wanted to implement it. I wrote some tests, and got half of them working, before I ran into an edge case that complicated the matter - specifically a case where the list unpacking syntax is used heavily.
Run into another thinking wall again, but not wanting to delay progress, I committed the functioning code, and left the failing tests to wait there for me. And I did what I always do when I feel like one problem is taking too much time: I moved onto other ideas and problems. And thanks to that, I’ve implemented a basic language server for Derw, some refactors, and English to Derw generation. I’ll be walking or cooking sometime in the future and figure out a preferable way to handle the unpacking problem, but until then, I’m happy to say I’ve got things done.
English to Derw generation can be tested at the Derw playground. The idea behind it is what if the compiler could explain in plain English what all this unfamiliar syntax is doing? It’s still experimental, so I’m not sold on it, but we’ll see how it goes.