24 November 2009

An open platform for publishing applications

HTTP, HTML and CSS are inadequate for developing complex end user applications that have hard latency requirements, depend on more sophisticated input-output facilities or need to integrate closely with the clients computing environment. Examples include games, multimedia or attempts to reproduce modern desktop applications on the web.

The failure to deliver browser-based applications in those domains is in my opinion caused by:

• the weakness of the request-response scheme of HTTP as a basis for a programming model - it works well for the operations on hypertext documents it was designed for, but it is a poor basis in almost all domains that do not naturally deal with documents, especially those requiring sending excessive amounts of information back and forth
• poor capabilities of the browser itself - what is being made available to the programmer is insufficient for any kind of more sophisticated development, as no access at all is given to important parts of the user environment - neither to the hardware (graphic cards, sound cards, video cameras, gaming devices) nor to the software resources (files on the client computer)

Many technologies were created to deal with the two problems. AJAX, continuations-based web servers or Comet were attempts to workaround the first issue, but they all have been built on top of the flawed model, while also suffering problems with the web browsers, without trying to directly fix them. The second problem was meant to be solved by the likes of Flash or Silverlight, which do not integrate well with the rest of the web (for example search engines), making it questionable why do they at all run inside the web browser, experiencing efficiency and stability issues. Java the platform was the closest to what I want to propose, but even through it had the necessary pieces, it wasn’t clearly steered in this direction and suffered from slow start-up times, slow and ugly GUI tool-kits and weaknesses of Java the programming language.

The author is seeing much potential in an integrated attempt to solve both of those problems with a new set of standardized technologies. We already have excellent protocols for hypertext, efficient file transfer or sending messages, why shouldn’t a protocol for sharing applications exist? Links could be put to apps://musicplayer.com/ on traditional web pages in the same way as links to e-mail addresses or file servers, leaving the browsers with what they do best - processing documents.

The proposed solution that will be described here is called OAPP, standing for Open Application Publishing Platform. The following things should be specified as a part of OAPP:

1. A virtual machine of predefined capabilities running on the client computer
2. A way of sending executable computer programs to the virtual machine from the server
3. A way of singalizing events on the virtual machine to the server
4. A programming model based on points 1, 2 and 3

Having each aplication run in a separate virtual machine creates a common denominator for all the client platforms, while giving the possibility to privilege access to the sensitive parts of the clients environment. The virtual machine should not only deliver means of executing programs, but also give abstractions (libraries) over users hardware and software resources. In fact, applications could require additional libraries that would also be delivered on demand to the client from different OAPP servers. This forms a rich and fully programmable platform developers can target. This virtual machine, together with the libraries and an interface for choosing from the available programs, is in OAPP terms an application browser.

For a practical demonstration of the benefits of such approach, the V8 virtual machine was chosen, both for the server and for the client. JavaScript is already well-known by web developers, while being a powerful programming language, in fact turning out to be surprisingly well suited for the problem. The quality of the virtual machine may decide about the success of the project and with its beautiful design, very quick start-up time and some decent libraries already available seems to be a good candidate. The package Node.js which provides a modified version of will be used, as it contains networking support - to run one of the examples, just use the ‘node’ executable with the name of the source file as an argument. Links to the source files can be found at the end of the article.

The OAPP communication protocol consists of packets in the form of JSON literals. For simplification of the examples packets are separated simply by three consecutive newline characters, while in practice a thin transport layer should be implemented, with the ability to store different applications under the same host name and to compress and encrypt the information being exchanged.

After a connection is established, the OAPP server sends the first packet with the initial code to the client. For example, a server delivering an application showing the start-up time looks like this:

var sys = require('sys');
var tcp = require('tcp');

function remoteClosure(fun, variables) {
  return "(" + JSON.stringify({
    proc: "(" + fun.toString() + ")",
    data: variables
  }) + ")\r\n\r\n\r\n";
}

function Application() {
  var sys = require('sys');
  sys.puts("Server was started at: " + this.date);
};

var date = new Date();

var server = tcp.createServer(function (socket) {
  socket.addListener("connect", function (conn) {
    socket.send(remoteClosure(Application, {date: date.toString()}));
  });

  socket.addListener("eof", function () {
    socket.close();
  });
});

server.listen(7000, "localhost");

The Application function is the code that will initially be sent to the client, which in this example is rather trivial. The JSON object transferred is built from two fields:

• proc stores the text of the Application function, retrieved using the toString() method of the function object
• data is any data dynamically computed on the server, in this particular case it is the date the server saved at startup

The extra parenthesis are just syntactic sugar for easier evaluation on the client side. This is how the sent JSON actually looks like:

({
  "proc": "(function ClientApp() {
              var sys = require('sys');
              sys.puts("Server started at: " + this.date);
            })",
  "data": {
    "date": "Sat Nov 21 2009 19:00:18 GMT+0100 (CET)"
  }
})

The client role is very simple - it is supposed to fetch consecutive packets and evaluate the code that is represented by them. When a valid packet arrives, the client parses the received JSON literal. By using eval on the proc field, it gets an identical Application function to the one that was defined in the code of the server, on the client side. It executes the function by using the apply method, making all the usages of this inside the function code refer to what is in the JSON literal under the data key. In this way, the function is executed in a dynamic context specified by the server. Otherwise, it would not be possible to have the entire source code of the application in one source file in normal form. I call it a remote closure, because I look at it as an analogy to the closure concept you may know from programming languages. That’s how the code for the client looks like:

var tcp = require('tcp');

var jsonBuffer = "";
var connection = tcp.createConnection(7000, "localhost");

connection.addListener("receive", function(data) {
  jsonBuffer += data;
  if(jsonBuffer.indexOf("\r\n\r\n\r\n") > -1) {
    var remoteClosure = JSON.parse(jsonBuffer);
    remoteClosure.proc = eval(remoteClosure.proc);
    remoteClosure.proc.apply(remoteClosure.data);
    connection.close();
  }
});

For the communication to be complete, a way of signalizing the events on the clients virtual machine to the server has to be established. For security reasons, it seems that it is better to use simple data objects in this direction, instead of the code evaluation scheme just presented. Each event sent will also be a JSON packet.

To clarify the code of the next example, a library was built around the concepts discussed so far:

var tcp = require('tcp');

function packet(object) {
  return "(" + JSON.stringify(object) + ")\r\n\r\n\r\n";
}

function packetBuffer() {
  var buffer = "";

  return {
    append: function(data) {
      buffer += data;
    },
    packetReady: function() {
      return buffer.indexOf("\r\n\r\n\r\n") > - 1;
    },
    fetchPacket: function () {
      var packetEnd = buffer.indexOf("\r\n\r\n\r\n");
      var packet = buffer.substring(0, packetEnd);
      buffer = buffer.substring(packetEnd + 1, buffer.length);
      return JSON.parse(packet);
    }
  };
}

function event(name, data) {
  env.connection.send(
    packet({
      name: name,
      data: data
    }));
}

function closure(fun, variables) {
  return packet({
    proc: "(" + fun.toString() + ")",
    data: variables
  });
}

function publish(application, host, port) {
  var handlers = {};
  var buffer   = packetBuffer();

  var server = tcp.createServer(function (socket) {
    socket.addListener("connect", function (conn) {
      socket.send(closure(application, {}));
    });

    socket.addListener("receive", function(data) {
      buffer.append(data);

      while(buffer.packetReady()) {
        var event   = buffer.fetchPacket();
        var handler = handlers[event.name];
        socket.send(handler(event.data));
      }
    });

    socket.addListener("eof", socket.close);
  });

  server.listen(port, host);

  return {
    addListener: function(eventName, listener) {
      handlers[eventName] = listener;
    }
  };
};

exports.packetBuffer = packetBuffer;
exports.packet       = packet;
exports.event        = event;
exports.closure      = closure;
exports.publish      = publish;

The library introduces two major new elements:

• an event function to send event objects from the client to the server
• a set of event handlers for the application server, each reacting to a particular kind of event

To illustrate how the full communication scheme works in practice, a slightly more complex program should be considered: a calculator. It is the simplest possible example of the general two-way communication scheme that is the basis for all OAPP applications:

var sys  = require('sys');
var tcp  = require('tcp');
var oapp = require('./oapp');

function Calculator() {
  function parse(input) {
    var position = 0;

    function peek()    { return input[position]; }
    function consume() { return input[(position += 1) - 1];  }

    function token() {
      var token = consume(), c;
      if(token < '0' || token > '9')
        return token;

      while((c = peek()) && c >= '0' && c <= '9')
        token += consume();
      return parseInt(token);
    }

    function binary(left, right, ops) {
      var token, a = left();
      if((token = peek()) && (ops.indexOf(token) == -1))
        return a;
      var op = consume(), b = right();
      return [op, a, b];
    }

    function term() { return binary(token, term, ['*', '/']); }
    function expr() { return binary(term, expr, ['+', '-']); }

    return expr();
  }

  process.stdio.open();
  process.stdio.addListener("data", function(input) {
    oapp.event("expressionEntered", { expr: parse(input) });
  });
};

var app = oapp.publish(Calculator, "localhost", 7000);

app.addListener("expressionEntered", function(event) {
  function compute(expr) {
    if(!(expr instanceof Array))
      return expr;

    var op = expr[0];
    var a  = compute(expr[1]);
    var b  = compute(expr[2]);

    switch(op) {
      case '+': return a + b; case '-': return a - b;
      case '*': return a * b; case '/': return a / b;
      default:  return null;
    }
  }

  return oapp.closure(function() {
    sys.puts("Result is: " + this.result);
  }, { result: compute(event.expr) });
});

The following scenario is executed when a client connects to the server:

• The Calculator application is sent to the client and installed. In JavaScript terms the application is a function, and installation is just function evaluation.
• The client application waits for a line of input from the user
• The input is parsed, forming a tree from the infix expression the user entered, for example parsing 9*8+2*3 returns ['+', ['*', 9, 8], ['*', 2, 3]]
• The resulting expression tree is sent to the server with the expressionEntered event
• The server computes the answer and sends back a closure displaying the result to the client

As one can see, the abstractions introduced make the application code look almost indistinguishable from an event-driven client side program, for example one built with a toolkit like Swing. For a library that is not even 2 kilobytes large, this is not a bad result and a big compliment for the techniques and technologies used.

The slightly extended client code looks like this:

var sys  = require('sys');
var tcp  = require('tcp');
var oapp = require('./oapp');

var buffer = oapp.packetBuffer();
var global = this;

env = {};
env.connection = tcp.createConnection(7000, "localhost");

env.connection.addListener("receive", function(data) {
  buffer.append(data);

  while(buffer.packetReady()) {
    var remoteClosure = buffer.fetchPacket();
    remoteClosure.proc = eval(remoteClosure.proc);
    remoteClosure.proc.apply(remoteClosure.data);
  }
});

The two examples considered were of course trivial, meant to illustrate the techniques used for developing OAPP programs. To check if the idea for the platform is at all bearable and sensible, one should consider how more complex kinds of applications would look and work like, especially if the virtual machine would be extended with additional capabilities, for example with a graphics library. This will perhaps be a topic of another article in the future.

I think that the set of ideas presented, if complemented with a much more mature implementation, would offer an interesting alternative to the way we develop web applications today. A real-life project based on this would of course have to deal with the topics of latencies, security or ease of debugging and be tested much more extensively. I leave the evaluation of the usefulness of such a project to the reader, at the very least I hope you have just seen an interesting usage of the JavaScript programming language.

Code of the first example (server start-up time):

server.js
client.js

Code of the second example (calculator):

oapp.js
server.js
client.js

Github repository

---

, student of Polish-Japanese Institute of Technology
Idea sparked by conversations with Magdalena Wójcik and Karol Kozub

Comment