Webian Shell and Ownership

I was hoping to have a new version of Well ready by today, but real life keeps intervening, so it’s maybe a week or two away still. One must have patience, I guess.

Anyway, I noticed Mozilla released Webian Shell (introduction on Mozilla Labs). Despite sounding a lot like Well (especially since the word Well comes from Web Shell), Webian’s focus is firmly on the client side of the equation. Webian aims to do away with the traditional “heavy-weight” OS interfaces that we’re all familiar with for people and devices that don’t need it.

Chrome OS is a very similar project with the rather major difference being that it includes an actual OS and can thus run standalone. Nonethless, my guess is that Webian is hoping that s/Shell/OS/ at some point in the future. (It can’t just be coincidence that Webian rhymes with Debian.) Thus, I consider these two projects to have the same goals in mind, albeit with slightly different starting approaches.

I started working on Well around September of 2008, treating it mostly like a fun hobby. When Google announced Chrome OS in July of the next year, it spurred me into more focused action. What spurred me specifically was my concerns with Chrome OS, which were the same then as my concerns with Webian today. Both projects propose that all the servers, all the data and (ultimately) all the control live in “the cloud”. That sounds nice, but the cloud isn’t as disparate nor as fluffy as it sounds. When we think of “the cloud”, we tend to think of millions of servers at our disposal. And that is true, but in reality most of those servers are run by only a handful of companies and organisations — those few that have the resources to both create excellent software and host massive amounts of data. My guess is that 90%+ of the important data that lives in the cloud and on 3rd party servers lives on servers run by no more than about 20 companies and organisations. As web applications improve, we might expect this number to increase, but it won’t. Possibilities for mergers combined with the barriers to entry (premium skills in software design, systems engineering, extraordinarily deep pockets and extraordinary brand recognition) are such that this number will likely never grow.

At least, not unless we change our approach to the cloud. Well is one such attempt. My hopes for Well (or a Well-like project) is to transform the cloud into something less fuzzy and more certain, while being just as (if not even more) convenient. The aim is to give people ownership of their own data and to give them the ability to decide who will serve it and who will guard it. Your documents, your emails, your programs and apps and games, your work, your photos and videos, your personal information. You should own all of it — and in the fullest sense of the word “own”.

Well allows you to do just that.

Or it will, once I actually release it.

Tags: , , , , ,

Progressive Rendering

An important (but not the only) function of Well is to run programs on the server and deliver the output to the browser on the client. If all program run times and all network transfer times were instant, the Well client could simply dump a program’s output into a <div> element as soon as it’s received . This is the approach of most websites when using AJAX to pull sub-sections of a page. If you’ve ever used one of those sites over a slow connection, you’ll know how irritating that experience can be.

From the very beginning with Well, I decided I wanted to display any output as soon as it was available. I wanted this for 2 reasons:

  1. To support interactive and long-running console programs
  2. To make Well feel as zippy and and as responsive as possible

The first is largely out of a desire to support existing (non-Well-aware) console programs. (Well-aware programs do interaction differently.) This case can be handled pretty easily since these programs only output text (Well doesn’t do VT100-style emulation just yet). It’s not super simple because simply dumping into a <pre> has its problems: text still needs to be split up (if only to gauge when the maximum buffer size has been hit) and a minimum of formatting applied. However, progressive rendering can be achieved for the most part by simply adding the new output lines after the old ones.

The second reason I gave above (speed) is the very same reason browser’s render web pages progressively, rather than waiting for all the page data to download first. It was quite a bit harder to handle progressive HTML than progressive text, though the eventual approach I took was quite simple.

Progressively rendering HTML in AJAX

When Well receives a piece of HTML output from a Well-aware program, it first strips the piece of output of any incomplete tags, or any open <script> and <style> tags. For example, the intentionally(!) incomplete HTML:

<p>This is a <script>document.write('test

is stripped back to:

<p>This is a 

This is then passed in to a temporary <div> object using div.innerHTML. The browser will parse this and yield an object hierarchy. Well then places a marker in the form of an empty tag that doesn’t interact with content (specifically, <script type=”text/well__dummy” class=”well__endmarker”></script> but I will use <span well__endmarker> to avoid confusion) at the end of the hierarchy, so that it looks as follows:

  • [div]
    • [div]
      • [p]
        • This is a [span well__endmarker]

This is immediately sent to the output <div>.

The next time Well receives a piece of output, it will combine it with any unused previous input plus the parents of the marker along with a new start marker and place it into a new, blank <div> object. For example, if the new output was as follows:

');<script> and this is the
rest. <strong>There is</strong> more

Well would combine this with the previous unused output to give:

<script>document.write('test');<script> and this is the
rest. <strong>There is</strong> more

and then add the end marker’s parents to the start, along with a start marker to give:

<div><p><span well__startmarker></span><script>document.write('test');<script> and this is the
rest. <strong>There is</strong> more

Well then passes this to a temporary <div> object to parse, giving the hierarchy:

  • [div]
    • [div]
      • [p]
        • [span well__startmarker]
        • [script]
          • document.write(‘test’);
        • and this is the rest.
        • [strong]
          • There is
        • more to

In this way, the start marker in the new output matches up exactly with the end marker in the old output and Well can simply start inserting the new nodes into the existing output (i.e. display what new output has been made available) right at that point, and then strip out the end marker.

If a closing tag appears in the new output, e.g.:

<p>Much, much more

The object hierarchy will handle it without hassle because the right parents are already there:

  • [div]
    • [div]
      • [p]
        • [span well__startmarker] say.
      • [p]
        • Much, much more

Note that, in the above, the second [p] is at the same level as the original [p] because of the closing tag that came before it. Of course, [p] has peculiar parsing behaviour in HTML5, so this would be true even if there were no closing tag — but this is handled automatically by the div.innerHTML parsing! (Which is why I pass the current parent elements in as text, rather than creating them as javascript objects.)

This approach is surprisingly fast — it is about as fast as simply writing the output progressively into an iframe (which has all manner of problems I won’t go into, though it does come in useful occasionally) and is significantly faster than a naive div.innerHTML += newOutput approach.

The code that does this progressive rendering (using this and several other approaches, including naive div.innerHTML, hidden iframes and visible iframes — which gives you an idea of what I tested!) is available here.

The Command Line Renaissance

My last (and first) post two months ago outlined my intentions with Well, and since then I have been remiss with providing updates — though that may be because there have been no updates to provide. It’s the usual story of other projects and concerns stymieing my progress.

In the meantime, I have discovered (via this post by Mozilla’s Kevin Dangoor) a very similar project that hosts a command line within the browser, called TermKit. That project seems to have sufficient polish to attract the interest of committed developers to the task, which would be fantastic. (I hope to explain why I think it would be fantastic in future entries.)

Of course, there have been a few attempts over the years at webby command lines. The one that excited my initial interest, XMLTerm, appeared 11 years ago (!) and used a Unix shell program as a backend. While things then went quiet for a while, new programs have begun to appear over the past few years, some of which are (or were) straightforward terminal emulators (e.g. JsTerm, MeshCentral VT100 emulator, VT Ajax Telnet) but others (such as BeSpin and now TermKit) went beyond command line traditions and explored the possibilities.

I started Well as a hobby over 2 years ago, largely in the spirit of an XMLTerm-style command line. I considered it a bit of a specialist project initially, of interest only to command line aficionados who wanted some new tools and ideas to play with. However, over time (and without much effort given my time constraints!) it evolved into something very different. If projects like Well and TermKit continue to evolve — and I think it inevitable — the way we use our computers to create (not just consume) may be subject to the most profound change since the development of graphical user interfaces.

What is Well?

Even though my ideas on Well are still evolving, I thought for this first post I would try to explain what Well is along with its purpose, its use and what it could be in the future.

Well is a web-based shell. It’s a little bit like the *sh shells from Unix in some respects, but in other respects is very different. It has a Python-based webserver back-end and HTML/etc. front-end. It would not be too far from the truth to call it a graphical, network-based command runner, but it aims to do a little bit more by default to make the user’s life a lot easier. It is also a software platform, but one that aims to be as open, as compatible and as interoperable with every other software platform as possible. The aim of Well is peace, not war.

It often helps to explain a new idea (or new software) by explaining what it is similar to. As a platform, Well is similar to:

  • sh
  • ssh
  • VNC
  • the human interface part (i.e. the GUI and command line) of an operating system
  • a web server
  • an integrated development environment

With the default apps (which are tightly integrated), Well currently (and in alpha form) shares features with:

  • file managers (like Windows Explorer, Mac Finder, Nautilus, etc.)
  • ftp servers and ftp clients
  • database managers
  • media viewers and streaming servers
  • text and html editors

Like any other platform, Well can potentially do anything but unrealised potential is of no use to a user so my hope is that this list expands with time. I believe the key to creating a successful and useful platform is to make life as easy as possible for both developers and users — which means a simple, comprehensive API for developers and a seamless, friendly but still powerful experience for users. Well isn’t quite at that stage yet, but that’s certainly the aim for the future.