Derw: a language to make web programming sane, April 2022 status
The last month has been a month of working on real world projects from work, in Derw. These projects help me ensure that using Derw makes sense, and that any bugs get fixed, and any gaps get plugged. The theme for this update therefore is bringing sanity to Derw. That means making the development cycle easier through tooling like language servers, it means writing APIs for real world use, and cleaning up some bugs in the parser.
Changelog
The language server
I mentioned it in last month’s change log, but I’ve since fixed a few bugs with the language server to make it more robust. It provides the editor with some of the errors with the correct line highlighting, but there’s features like specific token highlighting to be worked on it. It also crashes sometimes, mostly due to some of the parsing still being string-based rather than token based. As I continue on improving the parser, these errors should go away. I’m no language server expert yet though, so I’m still learning how to put it all together in a useful way. Short term goals include adding go-to definition support.
Web API support
In order to make Derw useful for real-world projects, it was natural to add support for some web API, which can be found in the web package. At the moment I’ve added things I’ve needed for projects I’m working on: localstorage, indexeddb, file reader, geolocation, history API. Some of these are only partially implemented, but anyone interested in extending what’s already there should have no problem writing the necessary kernel code. Both pull requests and issues are welcome. As there is a lot of web APIs, and they have a lot of implementation details to think about, implementing them can be slow. For example, while a good read, the IndexedDB spec is pretty heavy if all you want to know how to do is to store images locally.
Stdlib
Stdlib had a few improvements, specifically bug fixes in the list code, adding some useful methods like sorts for lists, and a random module.
Coed & problematic-projects
I ran into a problem where it seemed like the diffing algorithm in Coed, the kernel code for Derw’s Html package, was not correctly being applied. Debugging virtual dom behaviour is pretty complicated since there’s a lot of states involved, typically: current vdom, next vdom, and actual DOM. Typically when I have been debugging these issues both in Elm and in Javascript projects, I find it helps to find the minimal code base for reliably causing the problem, which is sometimes easier said than done. With that in mind, I made the problematic-projects for documenting bugs and the commits they were fixed in. In this case, the bug in Coed was simple: node attributes weren’t being removed by the patching algorithm. For future proofing, I also made sure that the patching algorithm would short-circuit and do a simple replace if two node IDs didn’t match instead of attempting to patch the attributes and children.
Compiler and parser
Type checking now checks the types of all arguments in a function definition, giving errors if they don’t exist in the current namespace. globalThis can now be used for importing global types into the current namespace.
Derw generation now uses less newlines between imports, and sorts them - first by local vs global imports, then alphabetically.
Lots of fixes for type argument generation.
Fix object literal parsing inside lists
Inference of object literal types
Templates for web apps via derw template --path src/Main.derw --template web
Only wrap constructors in parenthesis if they take arguments
Fix parsing of multi-line predicates in case..of
Fix parsing of names beginning with type
Fix parsing of negative numbers
Add an error message when the default argument is not last.
Fix parsing of list prepending with spaces
Fix operator parsing inside object literals
Add support for accessors so that now it’s possible to do List.map .name [ { name: “Noah” } ]
Type checking support across modules
Fix bug where parsed files with errors were not added to parsed files list
Keep running esbuild on watch mode even if esbuild crashes due to a bug in generated code
Add syntax highlighting to Github/Gitlab through .gitattributes (thanks to toastal)
Give better error messages when a default is or isn’t needed in case..of statements
Switch case..of and if statements over to using more token-based parsing to prevent runtime errors in the parser
Acknowledge true and false as known global values when using derw compile —names
Others
Update homepage with changelogs and playground links
Add Derw to cloc
Add a Zipper implementation in Derw
Thought cauldron
Evangelicalism
Derw is now at the point where other people could start using it. So I’m going to do what used to do for Elm: give talks at meetups, and give talks online. I’ll go over how to use Derw — and where you might want to use it. Firstly I plan to give a talk internally, to other Schibsted teams. Then I’ll repeat the talk publicly. The goal is to get people past the stage of looking at it and thinking “oh that looks interesting” to the point of actually downloading and trying to compile something with it. Since there’s so many similarities to Elm, those experienced with Elm will have the best success picking it up. But until I write a guide, there’s a few example repos I’ve written which can be used for inspiration along with a template generator and the Derw README covers the syntax for various features. If you’re interested in joining the talk online, I’ll share more details in a future newsletter and tweet once I’ve figured out the logistics.
Web assembly
There’s a lot of excitement about web assembly and what it could unlock for both imperative and functional languages that could be much faster than Javascript in theory if they could generate more efficient code. Algorithms like those dealing with immutable values could provide a lot of optimizations for Elm and Derw, for example. There were some missing questions like how garbage collection was done, or how web assembly code could talk to JavaScript APIs like DOM interactions. Typically to take full advantage of Web Assembly, you’d want to write your own garbage collection that fits the memory model of your language. JavaScript API communication however is going to be pretty similar between most languages. You might want to have your virtual-dom implementation to be in web assembly, then passing out the diffs to be applied in JavaScript land. I’ve yet to see any evidence that this will be beat the highly optimized JavaScript engines, though. Serialization/deserialization tends to have a significant overhead when working with tight loops. Derw’s selling points focus much more on the developer experience, rather than performance. That isn’t to say that Derw is slow, or bulky - so far in my benchmarks it performs competitively with React. But that’s not the focus.