An important (but not the only) function of Well is to run programs on the server and deliver the output to the browser on the client. If all program run times and all network transfer times were instant, the Well client could simply dump a program’s output into a <div> element as soon as it’s received . This is the approach of most websites when using AJAX to pull sub-sections of a page. If you’ve ever used one of those sites over a slow connection, you’ll know how irritating that experience can be.

From the very beginning with Well, I decided I wanted to display any output as soon as it was available. I wanted this for 2 reasons:

1. To support interactive and long-running console programs
2. To make Well feel as zippy and and as responsive as possible

The first is largely out of a desire to support existing (non-Well-aware) console programs. (Well-aware programs do interaction differently.) This case can be handled pretty easily since these programs only output text (Well doesn’t do VT100-style emulation just yet). It’s not super simple because simply dumping into a <pre> has its problems: text still needs to be split up (if only to gauge when the maximum buffer size has been hit) and a minimum of formatting applied. However, progressive rendering can be achieved for the most part by simply adding the new output lines after the old ones.

The second reason I gave above (speed) is the very same reason browser’s render web pages progressively, rather than waiting for all the page data to download first. It was quite a bit harder to handle progressive HTML than progressive text, though the eventual approach I took was quite simple.

Progressively rendering HTML in AJAX

When Well receives a piece of HTML output from a Well-aware program, it first strips the piece of output of any incomplete tags, or any open <script> and <style> tags. For example, the intentionally(!) incomplete HTML:

<div>
<p>This is a <script>document.write('test

is stripped back to:

<div>
<p>This is a 

This is then passed in to a temporary <div> object using div.innerHTML. The browser will parse this and yield an object hierarchy. Well then places a marker in the form of an empty tag that doesn’t interact with content (specifically, <script type=”text/well__dummy” class=”well__endmarker”></script> but I will use <span well__endmarker> to avoid confusion) at the end of the hierarchy, so that it looks as follows:

• [div]
  • [div]
    • [p]
      • This is a [span well__endmarker]

This is immediately sent to the output <div>.

The next time Well receives a piece of output, it will combine it with any unused previous input plus the parents of the marker along with a new start marker and place it into a new, blank <div> object. For example, if the new output was as follows:

');<script> and this is the
rest. <strong>There is</strong> more
to

Well would combine this with the previous unused output to give:

<script>document.write('test');<script> and this is the
rest. <strong>There is</strong> more
to

and then add the end marker’s parents to the start, along with a start marker to give:

<div><p><span well__startmarker></span><script>document.write('test');<script> and this is the
rest. <strong>There is</strong> more
to

Well then passes this to a temporary <div> object to parse, giving the hierarchy:

• [div]
  • [div]
    • [p]
      • [span well__startmarker]
      • [script]
        • document.write(‘test’);
      • and this is the rest.
      • [strong]
        • There is
      • more to

In this way, the start marker in the new output matches up exactly with the end marker in the old output and Well can simply start inserting the new nodes into the existing output (i.e. display what new output has been made available) right at that point, and then strip out the end marker.

If a closing tag appears in the new output, e.g.:

say.</p>
<p>Much, much more

The object hierarchy will handle it without hassle because the right parents are already there:

• [div]
  • [div]
    • [p]
      • [span well__startmarker] say.
    • [p]
      • Much, much more

Note that, in the above, the second [p] is at the same level as the original [p] because of the closing tag that came before it. Of course, [p] has peculiar parsing behaviour in HTML5, so this would be true even if there were no closing tag — but this is handled automatically by the div.innerHTML parsing! (Which is why I pass the current parent elements in as text, rather than creating them as javascript objects.)

This approach is surprisingly fast — it is about as fast as simply writing the output progressively into an iframe (which has all manner of problems I won’t go into, though it does come in useful occasionally) and is significantly faster than a naive div.innerHTML += newOutput approach.

The code that does this progressive rendering (using this and several other approaches, including naive div.innerHTML, hidden iframes and visible iframes — which gives you an idea of what I tested!) is available here.

My last (and first) post two months ago outlined my intentions with Well, and since then I have been remiss with providing updates — though that may be because there have been no updates to provide. It’s the usual story of other projects and concerns stymieing my progress.

In the meantime, I have discovered (via this post by Mozilla’s Kevin Dangoor) a very similar project that hosts a command line within the browser, called TermKit. That project seems to have sufficient polish to attract the interest of committed developers to the task, which would be fantastic. (I hope to explain why I think it would be fantastic in future entries.)

Of course, there have been a few attempts over the years at webby command lines. The one that excited my initial interest, XMLTerm, appeared 11 years ago (!) and used a Unix shell program as a backend. While things then went quiet for a while, new programs have begun to appear over the past few years, some of which are (or were) straightforward terminal emulators (e.g. JsTerm, MeshCentral VT100 emulator, VT Ajax Telnet) but others (such as BeSpin and now TermKit) went beyond command line traditions and explored the possibilities.

I started Well as a hobby over 2 years ago, largely in the spirit of an XMLTerm-style command line. I considered it a bit of a specialist project initially, of interest only to command line aficionados who wanted some new tools and ideas to play with. However, over time (and without much effort given my time constraints!) it evolved into something very different. If projects like Well and TermKit continue to evolve — and I think it inevitable — the way we use our computers to create (not just consume) may be subject to the most profound change since the development of graphical user interfaces.