Derw: hydrating the web language landscape, May 2022 status

Better in-browser testing, hydration, new packages for general use, improved contribution documentation, do-notation

May for Derw has been all about expanding the libraries for different use cases, encouraging people to try it out, and testing. A lot of my work so far has been to get feature parity with Elm: now the goal is to go further. Improve the story around testing, web framework features like hydration, add a better way to do effects, server side code, web API support like websockets and localstorage, etc.

With this month I’ve published Derw as 0.0.4, so go grab it if you want to give a test. The README is a great place to get an overview of the syntax, and the examples repo is a good place to see some real Derw code. At this point Derw is in production for at least 4 apps, running well and without failure!

Follow Derw on Twitter to stay up to date: https://twitter.com/derwlang

Changelog

Making it easier for people to contribute

One of my goals now that Derw has reached a significant level of usability is to make it easier for other people to use and contribute. So in the interest of giving people an insight onto how the compiler works and how to run it locally from the repo, I’ve documented as much as I can in terms of structure, working on it locally, and how pieces work. I’m also planning to give a talk in the near future online about Derw, having already given that talk internally at work. Documentation has also been added to some packages, showing how to use them from real-world code: web and websocket.

In-browser testing

I have a love-hate relationship with tools for automating in-browser tests like Capybara, Cypress, and Selenium. They’re typically clunky, slow and the APIs are often just weird. But there’s a clear need for them: they are the most reliable way of doing integration testing in actual browsers, and I’m at the point where there’s so many examples I have that it would be good to automatically run tests to ensure that the state is reasonable. So I wrote a test runner aimed at solving my particular need. At the moment it only supports clicking, changing values, and testing attribute values, but I have planned to add some controls that allow for waiting for async events to complete (e.g network requests). There is a test runner that you can embed directly onto your page just by adding a div with the id `html-test-root`. If it fails to find such a div, it will run in the console.

A plan looks roughly like this:

Hydration

Hydration is the concept of sending static html to the client on first load, then re-attaching event handlers so that the virtual-dom can be used for subsequent changes to the page based on the event loop being triggered. I’ve implemented it a couple of times before, and recently at work we were playing with hydration in a React app, so I got inspired to add it to Derw. In the gif below, the page is loaded first, events re-attached when the js is loaded, then I click on the button triggering one of those events. A couple of examples can be found here and here.

Do notation

I planned for a while to add do-notation to Derw, but wasn’t quite sure how it should look. Now I’ve added it to functions, with the idea of being able to do side-effectful things in a particular place rather than anywhere. For now do-notation doesn’t behave that differently from let..in, except it allows you to have top level expressions which run without assigning a value. You can also combine it with let..in. The long term plan is to run everything in a do..return inside an async/await function, but I haven’t yet decided how that should work. So to begin with, I’ve just added the syntax to see how much I like it before moving on with the underlying code.

Adding a template generator

In favour of laziness, I almost always copy code from existing examples to get started with a project. Usually my own examples as opposed to anyone else’s, but enough to cover the basic setup. There’s been many tools for doing this in the JavaScript world, like create-react-app and yeoman. I’ve never been a fan of these tools: they’re slow, they generate way bigger projects than I need, and they tend to be over enthusiastic on the number of dependencies you need. But still, they serve a purpose. So Derw now includes a template command, which can currently generate a web app for you. All you need to do is run `derw template —template web —path src/Main.derw`, and it’ll install the html package, and a file with your typical model/msg/update/view/root setup.

Passing down the parent scope type arguments

When you define a function via a let within another function, the parent function’s type arguments will be used in the nested function. To put that simply:

isTrue: Maybe a -> boolean
isTrue value =
    let
        innerFunction: a -> boolean
        innerFunction y = ...
    in
        ...

The type of `a` for `innerFunction` would be the same as the `a` for `isTrue`. This helps make well-typed functions within lets.

Repl + Loop

I had a need at work to build a TUI to modify some our JSON configuration files. I’d like Derw to be not just used for web apps, but for tooling and server code too. So I wrote a wrapper around readline in a package called repl. There’s an example usage there, but basically it follows the old Elm StartApp structure that I used when writing server-side Elm: model, update, view. As I was writing repl I realized that I also needed a generic loop that could be used for wrapping more use cases in the future, so I made loop. Loop is aimed at providing a structure to start from when working with an immutable workflow, along with a send function for doing async stuff.

Test runner improvement: only fails

Derw’s compiler currently has 1272 tests, over 106 files. Bach, the test runner I use, defaults to logging when each test is run and a table at the end of each test file and how many tests failed or passed within that file. When working with failing tests, it can be hard to spot which of the files is failing. So I added a flag `—only-fails` that will only print failing tests and only show failing tests in the table. This also works from `derw test`

Canvas support

A typical thing for me to implement in every language is Conway’s game of life. The algorithm is simple, and predictable. I know the pattern that my profile picture generates off by heart. To implement it in an efficient way, I needed a quick renderer - best suited to canvas in html. So, I added some drawing directives to the web package that support some basic canvas operations like drawing rects.

Websocket

I wanted to use websockets on a project, so I added a websocket server and client to Derw. There’s an example to get people started with, but it needs some real world testing before I would mark it as ready.

Calling new

There are some constructors in JavaScript that still require the usage of `new X(..)`, and Derw doesn’t have any support for writing `new`. So I added a helper to the stdlib that would allow you run `new` on functions within the `globalThis` namespace.

Compiler fixes

• Parsing of type arguments in union types won’t crash the parser

• Functions can be arguments given to union types

• The command line will show possible commands when asking for help

• Lambdas within object literals and function calls are better parsed

• When two object literals are given as an argument to a function, the parsing is correct

• if..else, case..of, etc now correctly work within lambdas

• object literals now correctly work with list prepending

• fix handling of ... within strings within lists

• let..in now works within consts

Misc

• only download packages of a particular version once when doing `derw install`

• bump bach to improve testing experience

• bump derw version in the playground

• stricter types in dict

• add some number functions to stdlib

• only re-set attributes if they don’t already exist on the vdom to prevent re-rendering draw calls on elements that haven’t changed

• improve the readme for the examples repo

• playground shows errors when compiling to user

• better error messages for when install is given a package name

Thought cauldron

Issue Tracking

Up until now I’ve mostly worked on whatever in my head, fixing problems as I come across them. As I begin to scale up the number of projects I have going on, there’s a clear need to track some things for the future, rather than fixing immediately. I’d like my personal workflow to follow the issues I prioritize, and track things like making a blog post or an example without needing to make a Github issue for it which is publicly visible. So I’ve drawn up a division: Github will have public issues for people to report or comment on, along with commits, so they’re all visible to the public. But my own personal prioritization will be in a Pivotal Tracker instance, which I’ve had good times using previously.

Package documentation

Up until now, most of the packages I have published for Derw have been intentionally free of documentation: if you were to use Derw during these early stages, you should be reading the code so that you get a feeling of how Derw looks and works. As I open it up for other people to start using, there’s obviously a need for proper documentation across every package. So my plan is to start writing + generating documentation for Derw code, so that there’s less need to jump to the source. I also plan on adding a :help command to the derw repl so that it’s possible to see documentation + source without needing to find the file itself.