The previous installment of the Dive Into Dojo series shows how easy it is to Dive Into Dojo Charting to get started with Dojo’s charting library. It comes with dozens of stylish themes you can effortlessly plug into any chart. But what if you want your charts to match your website’s design or business’ branding? No worries: Dojo’s charting library allows you to create custom themes!

Setting Up Your Namespace

All of Dojo’s charting themes live with in the dojox.charting.themes namespace. As is the best practice with custom class creation with Dojo, we’ll create our own namespace for our custom chart themes. Let’s give our custom theme the davidwalsh.charting.themes namespace. davidwalsh serves as my namespace for all custom Dojo classes and I’ve chosen to mimic the path dojox uses to themes.

/* declaring charts within my namespace */
dojo.provide("davidwalsh.charting.themes.SitePen");
dojo.provide("davidwalsh.charting.themes.SitePenFTW");
  Quick Peek at a Simple Theme

The levels of complexity within Dojo themes can vary greatly depending on your desire to use simply define colors or implement advanced features like gradations, markers, scrolling, panning, and chart events. Let’s start by looking at a basic theme called Dollar:

dojo.provide("dojox.charting.themes.Dollar");
dojo.require("dojox.charting.Theme");
(function(){
        dojox.charting.themes.Dollar = new dojox.charting.Theme({
                colors: [
                        "#A4CE67",
                        "#739363",
                        "#6B824A",
                        "#343434",
                        "#636563"
                ]
        });
})();
 

Dollar illustrates how simple creating a theme can be; provide a list of colors and you’ve created a custom theme!

Basic Custom Theme: SitePen

Now that we’ve seen how simple it is to make a basic theme, we can easily create a custom SitePen theme base on the SitePen logo colors:

dojo.provide("davidwalsh.charting.themes.SitePen");
dojo.require("dojox.charting.Theme");
(function(){
        davidwalsh.charting.themes.SitePen = new dojox.charting.Theme({
                colors: [
                        "#f2f2f2",
                        "#bed3d9",
                        "#7fc25d",
                        "#60b32b",
                        "#277085",
                        "#333"
                ]
        });
})();

Your colors array can have any number of colors. Colors are repeated in sequence within the chart, if necessary. Check out a few different charts using the new SitePen theme:

Advanced Chart Theming

With the basic SitePen chart theme, we simply defined a series of colors we’d like used in the chart. With advanced chart theming, you can customize everything from default plotareas, axis, series, and marker colors, fonts, and strokes. The amount of control you can have over your charts by creating your own theme is truly incredible. Using Tom Trenka’s “Tom” theme as a template, let’s further customize and enhance the SitePen theme.

/* requires and provides */
dojo.provide("dojox.charting.themes.SitePenFTW");
dojo.require("dojox.gfx.gradutils");
dojo.require("dojox.charting.Theme");

/* define the theme */
(function() {

/* create shortcut references to dojox classes */
var dc = dojox.charting, themes = dc.themes, Theme = dc.Theme, g = Theme.generateGradient,
       
        /* fill settings for gradation */
        defaultFill = {type: "linear", space: "shape", x1: 0, y1: 0, x2: 0, y2: 100};

/* create theme */
davidwalsh.charting.themes.SitePenFTW = new dc.Theme({

        /* customize the chart wrapper */
        chart: {
                fill: "#333",
                stroke: { color: "#333" },
                pageStyle: {
                        backgroundColor: "#000",
                        color: "#fff"
                }
        },

        /* plotarea definition */
        plotarea: { fill: "#000" },

        /* axis definition */
        axis:{
                stroke: { // the axis itself
                        color: "#fff",
                        width: 1
                },
                tick: { // used as a foundation for all ticks
                        color: "#fff",
                        position: "center",
                        font: "normal normal normal 7pt Helvetica, Arial, sans-serif",  // labels on axis
                        fontColor: "#fff" // color of labels
                }
        },

        /* series definition */
        series: {
                stroke: { width: 2.5, color: "#fff" },
                outline: null,
                font: "normal normal normal 8pt Helvetica, Arial, sans-serif",
                fontColor: "#fff"
        },

        /* marker definition */
        marker: {
                stroke: { width: 1.25, color: "#fff" },
                outline: { width: 1.25, color: "#fff" },
                font: "normal normal normal 8pt Helvetica, Arial, sans-serif",
                fontColor: "#fff"
        },

        /* series theme with gradations! */
        //light => dark
        //from above:   g = dojox.charting.Theme.generateGradient
        //defaultFill object holds all of our gradation settings
        seriesThemes: [
                { fill: g(defaultFill, "#fff", "#f2f2f2") },
                { fill: g(defaultFill, "#d5ecf3", "#bed3d9") },
                { fill: g(defaultFill, "#9ff275", "#7fc25d") },
                { fill: g(defaultFill, "#81ee3b", "#60b32b") },
                { fill: g(defaultFill, "#4dcff4", "#277085") },
                { fill: g(defaultFill, "#666", "#333") }
        ],     

        /* marker theme */
        markerThemes: [
                {fill: "#bf9e0a", stroke: {color: "#ecc20c"}},
                {fill: "#73b086", stroke: {color: "#95e5af"}},
                {fill: "#216071", stroke: {color: "#277084"}},
                {fill: "#c7212d", stroke: {color: "#ed2835"}},
                {fill: "#87ab41", stroke: {color: "#b6e557"}}
        ]
});
})();

You wont need to define every property created above; if a given property isn’t defined, a default value will be used. On the flip side, you may also override each and any of these settings when creating chart instances. This custom theme acts as a middle-ground between chart defaults and chart instance settings.

Take a look at some of our enhanced theme examples:

Using Gradients

The advanced SitePen theme we created above made use of gradients thanks to the dojox.gfx.gradutils class and dojox.charting.Theme.generateGradient method introduced in Dojo 1.5. Gradation instances can be passed to any property that desires a color (usually the fill property.) The signature of the new generateGradient method is:

generateGradient: function(
        fillPattern,
        colorFrom,
        colorTo
}
 

The fill pattern holds the gradient settings:

{
        type: "linear",         //or "radial"
        space: "shape",
        x1: 0,                  //gradation direction
        y1: 0,                  //gradation direction
        x2: 0,                  //gradation direction
        y2: 100                 //gradation direction
}
 

Feel free to experiment with the gradation settings to find just the right gradient for you.

New to Dojo 1.5: Gradients, Themes, and Extended Support!

Dojo 1.5 introduces great new features to the charting library, including:

• New Themes
• Gradients
• Experimental support for the SVGWeb plugin in addition to the existing support for SVG, VML, Canvas, Flash, and Silverlight.

Great Charting Resources

• Dive Into Dojo Charting
• Dojox Charting Reference Guide
• Nightly Tests
• Theme Preview
• Introducing Dojox Data Chart
• A Beginner’s Guide to Dojo Charting, Part 1 of 2
• A Beginner’s Guide to Dojo Charting, Part 2 of 2
• Zooming, Scrolling, and Panning in Dojo Charting

Create Your Theme!

While Dojo’s charting library comes with a variety of stylish themes, don’t feel as though you need to choose an existing theme; take a few minutes to create an eye-catching custom theme to match your branding!

Related posts:

1. A Beginner’s Guide to Dojo Charting, Part 2 of 2
2. Zooming, Scrolling, and Panning in Dojo Charting
3. Dive Into Dojo Charting

Dojo Toolkit 1.5 is now available for immediate download. Dojo is a JavaScript toolkit that is lean enough for use on a simple blog, yet powerful enough to scale to solve the most advanced web application engineering challenges, allowing you to use just the features and flexibility needed for your application. The 11th major Dojo release, version 1.5 offers many important improvements and enhancements and remains as IP-safe, freely-licensed, and free to use as the first release over five years ago.

Substantial user interface improvements

The new version of Dojo offers substantial user interface improvements in the form of the beautiful Claro theme. Claro delivers a modern and engaging visual style for rich internet applications with Dojo’s Dijit library, with the visual enhancements of transparent gradient background images, drop shadows, and appropriate CSS animations (on Webkit and Mozilla-based browsers). It delivers a ‘fit and finish’ that surpasses previous themes and releases, and significantly improves ease of customizing the theme CSS, so you can easily create your own theme by styling elements such as padding and color, without needing to design new background images. All of this while being fully accessible and internationalizable across a growing collection of user interface widgets! Learn more in our Dive into Dijit article.

The portal layout mechanism found in sites such as the personalized Orange home page is provided with Dojo, making it easy to create user-customizable application interfaces. Dojo’s very powerful native vector graphics, charting, and drawing components have received many improvements, including new themes, gradients, and experimental support for the SVGWeb plugin in addition to the existing support for SVG, VML, Canvas, Flash, and Silverlight. This guarantees that your graphics will work natively when possible, but will work everywhere with the simple dojox.gfx APIs.

Stable, backwards and forwards compatible Core

The core Dojo libraries are remarkably stable, making it very easy to upgrade your application from earlier versions of Dojo while receiving a number of improvements to make development easier. dojo.Stateful was added, and dojo.Deferred is improved through an underlying Promises-based API. The core libraries provide everything you need for simple and advanced web sites and apps.

HTML5 & CSS3

Dojo supports many HTML5 features, and has supported many capabilities before they were supported in any browser such as local storage. The Dojo approach is to wrap native support where possible, fixing any glitches, adding API capabilities or simplifications, and offering compatible solutions for older browsers. Whether it’s improved browser history, placeholder hint text for all TextBox-based widgets, a new dojox.style extension to support the CSS transform and transform-origin properties, local storage, rich-text editing, multi-file uploading, or gfx for Canvas/SVG support, we have you covered now, with much more to come in future releases. Check out the presentation and Gorillas examples for a glimpse at what’s possible!

Dojo Mobile

The mobile app space is evolving far too rapidly for the API stability promised by Dojo’s core libraries, so we have many initiatives underway to solve your mobile app development challenges now, and converge on more stable solutions soon. We strive to address challenges for both mobile web apps, and mobile installed apps that embed a web browser. A number of new Dojo Toolkit and Dojo Foundation initiatives are underway:

• dojox.mobile: lightweight mobile web widgets
• dojox.mobile.app: mobile application development framework
• embedjs: Dojo API, optimized for mobile
• TouchScroll: scrolling layer for WebKit Mobile
• wink: mobile web app toolkit

Browser Support

Dojo supports a wide variety of modern browsers. The complete list of officially supported and tested browsers: Chrome 5, Firefox 3.5 and 3.6; Internet Explorer 6, 7, and 8; Opera 10.6 (Dojo Core only); Safari 4.1 and 5. Other browser versions may work fine with Dojo even if they are not officially supported by the project.

Performance and Stability

Not only is Dojo consistently rated among the best performing Ajax toolkits, we also provide solid tools for scaling and growing your application. It’s easy to run into performance problems with any development tool, but the community and various vendors offer solutions to diagnose and solve all of your performance obstacles.

Integrations

It’s always easier to get started when you start with something you already know. Whether you use Zend Framework, Spring, Persevere, Node, Narwhal, cometD-Jetty, DWR, Compuware Uniface, Ruby on Rails, Django, WebSphere, Apache, IIS, or any other HTTP-compliant server-side environment, there’s a simple drop in integration solution to get you up and running quickly. Dojo also offers easy integration with AIR 1.5 and 2.0, Appcelerator Titanium, PhoneGap, and integration hooks with various IDEs and developer tools including Aptana Studio, Komodo, Eclipse, and more.

Demos

We’re in the process of collating the 150 best Dojo 1.5.0 demos. Some of the initial demos were linked to from within this post, and other community plug-in efforts like the MFU project offers an alternative to the dojox.form.FileUploader widget, or the Dojo port of mustache.js as an alternative to dojox.dtl.

Documentation, Support and Assistance

The Dojo web site has a plethora of documentation, and the thriving community offers assistance for everything that’s not already covered. For professional support and assistance, companies like SitePen are available to help you get the most out of Dojo and the open web!

Download Dojo 1.5 today and let us know what you think, and tell us about the great apps you’re building with Dojo!

A Dojo Foundation project

The Dojo Toolkit is part of the Dojo Foundation, the open home for the open web. Beyond the Dojo Toolkit, the foundation has welcomed 3 new projects recently: Zazl, AnimeJ, and wink. cometD-Jetty recently announced a 2.0 release, and Persevere 2.0 beta is due this summer.

Related posts:

1. Platform Optimization Strategies for Ajax Toolkits
2. SitePen Announces Updates to the Dojo Toolkit to Enable Adobe AIR Compatibility
3. End of the Year Dojo News and Updates

Node is a server-side JavaScript platform that is known for being well suited for Comet-style applications that utilize long-lived connections to allow applications to send messages from the server to the browser asynchronously. In fact, the beginning of the front page of nodejs.org starts out with an example of a web application that delays for a couple seconds before sending a response without any type of blocking; the code is asynchronous and efficient.

However, building a real-life real-time application involves more than just a platform that gives you asynchronous communication, there are a number of other important techniques to understand. We will look at these techniques and introduce project Tunguska that provides some helpful tools to assist in building applications. While a number of Comet projects out there attempt to provide a black box solution to Comet, Tunguska recognizes that most real-time applications involve deep integration into the application and its security, messaging, and data structures. Consequently Tunguska is a set of tools for building real-time applications rather than a closed black box that can’t easily be integrated with. Let’s look at some of these tools.

Connections/Message Queues

The most fundamental concept of building real-time applications is providing the ability to asynchronously deliver messages that occur at any time, to the browser. With new technologies like WebSockets, this is relatively straightforward. You can create a connection to the server and then the server is free to send messages to the browser at any time. However, WebSocket capable browsers are still nowhere near widely distributed enough to rely on (and the WebSocket protocol is still in development, a recent revision rendered the first version incompatible, and more changes are likely to occur due to the immature design).

The most widely used technique for Comet is known as long-polling. With long-polling an HTTP request is made to the server which waits until it has a message to send. The message is sent to the browser as the response, and upon receiving the response the client makes another request to the server. However, the complication of this strategy is that the server does not have a continuous, uninterrupted stream to send messages to the client. There is a time gap between each message/response being sent from the server and the next long-polling request from the browser reaching the server.

However, we obviously don’t want to miss the messages that were generated during these time gaps. Therefore we need to provide a virtual connection by creating a queue where messages can be stored during these time gaps between the response and request. These message queues allow us to effectively emulate a continuous connection, whereby messages can be delivered at any time to the virtual connection. If an active long-polling response is waiting the server can immediately deliver the message, otherwise the message can go into a queue to wait for the next request from the browser to send a message.

Tunguska provides this type of virtual connection or message queueing with its queue module, and message queues can be accessed with the getQueue(clientId) function. Queues are identified by a client id. This function will return a queue object for a given client id passed as the argument. The queue object extends JavaScript’s Array (providing array functions for accessing queued events), provides a send() for delivering events, a message event, and a close event (accessible via addListener() or observe()) as the key functionality.

var getQueue = require("tunguska/queue").getQueue;
// get the virtual connection for this request
var queue = getQueue("32a3fa5");
queue.addListener("message", function(message){
   // this will be fired by the send call below
});
// put a message in the queue identified by this request.
queue.send("Hello");
 

The queue module does more than just provide a queue object, it also handles timeouts of non-active connections so as to avoid unbounded memory usage, or memory leakage due to accumulation of unused connection objects. It also handles distributed message queues (across multiple processes/machines) as we will see later.

Tunguska also provides a “jsgi/comet” module for the higher HTTP level interaction. A message queue, which acts as a virtual connection to a client, can be obtained from a request object (JSGI or Node) passed into its getClientConnection(request) function. This function will look for the “Client-Id” header to use as the client id for connection identification. If a connection exists with the given client id, it will be returned, otherwise a new connection object/message queue will be created and returned.

The connection object is based on the WebSocket API. Sending a message to the client is as simple as calling the send(message) function. The connection object is also an array. New messages are pushed onto the array and any listeners are notified.

var comet = require("tunguska/jsgi/comet");
// get the virtual connection for this request
var connection = comet.getClientConnection(request);
// put a message in the queue identified by this request.
connection.send({from:"Kris", body:"hi"});
  Long-polling handler

A long-polling request handler can now listen for messages by initially checking for any messages in the queue for instant delivery and if no messages are ready to deliver to the client, it can listen for message events to trigger delivery. A JSGI middleware appliance is provided with the jsgi/comet module to do just that. We can easily insert this appliance into a middleware stack:

cometApp = comet.Broadcaster(nextApp);
 

The optional nextApp parameter can be used to define another middleware appliance that can handle any needed subscriptions and/or other wiring to event triggers to relay messages to the virtual connection object. For example:

cometApp = comet.Broadcaster(function(request){
  var connection = comet.getClientConnection(request);
  // user function for registering a user to get user events
  listenForMessagesForUser(
    request.remoteUser, // if we have an authenticated user
    function(message){
      // this can be called when a message needs to be sent to the user
      connection.send(JSON.stringify(message));
    });
});
  Message Routing

One of the key aspects of real-time applications is message routing. When an action or event occurs, the message needs to be routed to the appropriate party. In a chat application, when someone sends a chat message, we need to route and send the message to the recipient. A frequently used mechanism for this is a publish/subscribe hub. A hub consists of “channels” or “topics”, and messages can be published to channels on the hub, and subscribers can listen for messages on these channels. This paradigm decouples sending messages and listening for messages. Message publishers do not need to be concerned with who is subscribed and subscribers do not need to be concerned with the details of who and how messages are published.

Pub/Sub Hub

There are certainly alternate approaches to topic-based pub/sub hubs for message routing. Rather than subscribing to a particular named topic, subscribers could provide message filter functions that filter through all messages for appropriate messages of interests. However, this approach does not scale well. As more diverse topics and messages are being sent through a system, a filter often begins to process more non-matching functions and performance suffers. On the other hand, pub/sub hubs scale well. A normal hash-table based hub maintains constant time message routing regardless of the number of topics. This highly scalable approach fits well with the Node philosophy of making it easy to do the right thing (in terms of scalability) and hard to do the wrong thing. By using the named topics, it is very easy to create highly scalable message routing systems.

Creating a very simple pub/sub hub in JavaScript is actually extremely easy. JavaScript’s hash-based objects make a great core of a hub. You can simply create properties on a “hub” object with names that correspond to topics and values that are arrays of listeners. However, more interesting features than simple topic based message routing are often desirable. Tunguska provides a “hub” module with a rich set of publish/subscribe capabilities.

The Tunguska hub supports globbing or wildcard-based subscriptions for listening for a set of topics (rather than only being able to register for a single topic at once).

var hub = require("tunguska/hub");
hub.subscribe("foo/*", function(event){
  // will be fired by the publish below
});
hub.publish("foo/bar", "hi");
 

It allows for “typed” messages to allow for filtering/routing by type as well as topic (essentially allowing a hub to have a two-dimensional namespace). This hub uses “typed” messages for broadcasting subscription events, sending out a “monitored” event when a topic starts to have subscribers or ceases to have any subscribers.

hub.subscribe("foo/*", "monitored", function(event){
  // will be fired by the subscribe action below
});
hub.subscribe("foo/bar", function(){…});
 

Tunguska’s hub also supports echo suppression, allowing subscribers and publishers to indicate a client identifier so that published messages do not echo back to the client. These components are important for various applications and are essential for Tunguska’s distributed messaging system, where hubs can be linked together in a cluster.

var clientHub = hub.fromClient("42a92f3");
clientHub.subscribe("foo/bar", function(event){
  // will be *not* fired by the publish below
});
clientHub.publish("foo/bar", "for everyone else"); Clustering

Building Comet applications on a single Node process (single threaded) can be deceptively easy. But, scaling Comet applications to multiple concurrent processes is where the bulk of complexity usually arises. Inter-process communication, non-deterministic request delegation, message framing, and true parallel concurrency can combine to create a formidable challenge compared to the rather simple single-process application development. Fortunately, Tunguska provides clustering support for linking distributed pub/sub hubs together, allowing applications to maintain a simple hub-based message routing system without having to worry about the inter-process interaction complexity.

The core component for building Tunguska clusters is the “connector” module. The “connector” module provides a means of using connections to other processes to coherently link hubs together. These connectors will listen for subscriptions and send subscription requests to other processes, allowing for collection of all messages for a given topic from all processes. A connector can be created by passing it a connection object (that follows the WebSocket API), and the connector will handle the messaging concerns for propagating subscriptions and message routing.

WebSocket connection objects can easily be utilized with actual WebSocket TCP connections to other machines. For inter-process communication on a single machine, multi-node provides easy access to a WebSocket-based connection to the sibling HTTP serving processes. Connecting a set of processes with Tunguska with multi-node is very simple:

var multiNode = require("multi-node/multi-node"),
    Connector = require("tunguska/connector").Connector;
// start the multiple processes
var nodes = multiNode.listen({port: 80, nodes: 4}, serverObject);

// add a listener for each connection to the other sibling process
nodes.addListener("node", function(stream){
  // create a new connector using the framed WS stream
  Connector("local-workers", multiNode.frameStream(stream));
});
  Jack/JSGI Support on Other Platforms

In addition to Node, Tunguska can run on any other JSGI-compliant server including Jack 0.3, and should be easily adaptable to RingoJS as well. The main complication with different platforms is creating connectors with alternate forms of cross-process or cross-thread communication. Jack 0.3 utilizes threads for concurrency that implement the WebWorker API and includes special events for connecting to other workers for fully connected workers (achieving fully connected workers, where every worker can talk to every other worker is problematic with the standard worker API since it relies on the less ubiquitous SharedWorker API and requires a high level of name coordination). To use the Tunguska connectors with Jack, you utilize the “jack-connector” module, which provides “worker” events that return connection streams that can be used with the connectors:

require("tunguska/jack-connector").addListener("worker", function(connection){
  Connector("local-workers", connection);
});
 

Tunguska provides a solid set of tools for building scalable, non-blocking, real-time applications with the flexibility and modularity needed to integrate with essential application logic including custom serialization, authorization/access-control, and middleware routing to truly leverage the asynchronous I/O capabilities of Node. Next up, we will look at how Tunguska, which is a sub-project of Persevere, fits into the greater Persevere ecosystem for data integration with real-time data views, and RESTful content negotiation.

Related posts:

1. Multi-node: Concurrent NodeJS HTTP Server
2. Which Comet Implementation?
3. JSGI vs Connect for Node Middleware

Modules are an integral architectural piece of robust application development since they allow individual components to be developed with proper dependency management. Modules can specify dependencies and these can be automatically resolved and loaded to bring various pieces together automatically. In application development this is vastly more scalable and easier than having to track all the different dependencies and manually load modules or insert script tags.

The CommonJS module format is increasingly ubiquitous as the de facto module format for JavaScript. However, if CommonJS modules, by themselves, are directly executed, they require synchronous loading of modules. Synchronous loading is known to be very problematic in the browser since it locks the browser user interface, requires eval-based compilation of scripts which confuses debuggers, and is less efficient than using standard script tags.

The CommonJS group has developed a specification for callback based module loading called the module transport format. The CommonJS module transport format is specifically designed to make it easy to wrap CommonJS modules (which also use synchronous require calls) with a callback wrapper to allow for asynchronous loading on the client. For example, if one had a CommonJS module:

// complex-numbers/plus-two.js:
var sum = require("./math").sum;
exports.plusTwo = function(a){
  return sum(a, 2);
};
 

This can be wrapped with the transport format to be loaded asynchronously and used on the client:

// complex-numbers/plus-two.js received by browser:
require.define({"complex-numbers/plus-two": function(require, exports){
// define body in callback
  var sum = require("./complex-number").sum;
  exports.plusTwo = function(a){
  return sum(a, 2);
};
},["complex-numbers/math"]); // indicate the dependencies
 

There are a couple of excellent client-side libraries available for loading and handling modules in the CommonJS modules format including James Burke’s RequireJS and Yabble. RequireJS also provides a number of functions for easing construction of hand-crafted static modules with wrappers (rather than using server-side generation). These are well documented on the RequireJS API page, so I won’t go into those here, but RequireJS makes it very easy to hand-craft modules for asynchronous loading. Yabble is designed to support direct loading of CommonJS modules (with or without wrapping), and supports other CommonJS APIs including require.ensure for explicit dynamic asynchronous loading of modules.

Automated Module Wrapping

A client-side library like RequireJS can then easily load modules and dependencies (like complex-numbers/math.js) asynchronously and execute the module body when they are available. Of course wrapping modules like this requires writing a lot of extra boilerplate code, especially if you are originally writing your code in CommonJS format, so naturally it is helpful to automate this. Fortunately, there are actually several projects that can automate this wrapping, allowing you to write plain CommonJS modules, and automatically wrap the modules so that they can be asynchronously loaded in the browser. FlyScript is a CommonJS module wrapper written in PHP, and modulr is a Ruby-based module wrapper. Each of these come with their own client-side module loader, but with proper CommonJS transport format compliance they could be used with RequireJS as well.

Another module wrapper is my Transporter project that is actually written as a CommonJS module for JSGI servers (like Jack, and Node with an adapter). Transporter is particularly interesting since it is actually designed to run on a CommonJS server, and therefore paves the way for easily reusing the same modules on the server and client effortlessly. In fact, Transporter will default to loading modules using the same lookup path as the server-side CommonJS environment, to make it easy to share modules. Transporter has been primarily tested with RequireJS and Yabble.

Transporter includes the (configurable) ability to automatically resolve all dependencies on the server. This means that when a module is requested, all the dependencies are automatically included, rather than having to wait for the browser to individually request each dependency, which is much slower due to the latency and overhead of multiple HTTP request-response round trips. Transporter includes a number of other useful options including the ability to explicitly include and exclude dependencies from URLs (within script src), automatic running of the modules (in Yabble, modules aren’t auto-executed), specifying module source paths or loaders, and support for alternate converters (so that different formats besides CommonJS can be used). These are described in more detail in the Transporter documentation.

Dojo, RequireJS, CommonJS

These new asynchronous loading capabilities are not just for CommonJS modules. James Burke has been hard at work in integrating RequireJS with Dojo. He has built a tool for a converting Dojo modules to RequireJS-compatible transport format, integrated RequireJS with the Dojo bootstrap process, and created a full Dojo build that incorporates RequireJS with a fully converted copy of Dojo (including DojoX and Dijit). With this build you can use Dojo in fully asynchronous mode, getting the performance, usability, and debugging benefits of RequireJS’s loader with Dojo. After Dojo 1.5 is complete, we will be looking at various possibilities for using this type of technology and integrating asynchronous loading capabilities in core Dojo and possibly supporting CommonJS modules in Dojo as well. However, this is still a work in progress, and there are couple different options we are exploring, so I will save more details on this for a future post.

Related posts:

1. Pintura JSGI Modules
2. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
3. Introducing Pintura

NodeJS has demonstrated impressive performance potential as an HTTP server. By combining highly optimized HTTP parsing algorithms with the speedy V8 JavaScript engine and using an event-based architecture, Node has posted eye-opening request handling numbers. However, Node historically has been limited by its inability to provide true concurrent request handling, greatly limiting its ability to leverage increasingly multiple core servers. The latest release of Node introduced new functionality for socket sharing that can be coupled with child process spawning to achieve concurrent multi-process request handling for a single TCP socket. Multi-node exists to leverage this capability to make it simple to start up a multi-process Node server.

Multi-node is very easy to use. You simply create an HTTP server object as you would normally do, and rather than immediately calling listen(), you pass it to the multi-node listen function:

var server = require("http").createServer(function(request, response){
                … standard node request handler …
});
var nodes = require("multi-node").listen({
        port: 80,
        nodes: 4
}, server);
 

As you can see, we can indicate the number of “nodes” or processes to start up, and the port number to listen on. Multi-node will automatically spawn the appropriate processes and pass the socket to each child process to share. The OS kernel then essentially acts as the load balancer, queuing up incoming TCP requests and handing them off to processes as they can accept them. This can actually be more efficient than a separate load balancer, since socket connections are handled as processes request them, making the load balancing immediately dynamic. If a process is loaded with more difficult processor intense requests, it won’t grab as many requests and the other process will pick up the load.

One decision to make is choosing the number of processes (”nodes”) to run. Perhaps the most common suggestion would be to use the same number of processes as the number of CPU cores on the machine. Indeed this is likely to provide the most efficient utilization of the machine’s CPU resources, with minimal context switching overhead. However there can be reasons for using more or less processes. If you don’t want the Node server to consume all of the machine’s resources, you might specify less processes to keep other CPU cores available for other machine tasks.

Alternately, there can be “fairness” advantages to running more processes. Node won’t start processing another request until it finishes its current queue of events. If it has a large queue of expensive event handlers to process, this may increase the latency handling of a request even if it is small, lightweight request. With few processes, this request processing granularity is large. However, with the addition of more processes, the request processing granularity decreases and smaller requests have better odds of being serviced sooner rather than ending up being queued behind other performance expensive operations in the event queue of a few processes. More processes can slightly increase context switching overhead, but this may be worth the improved fairness and latency reduction for quick requests.

One of the core capabilities that multi-node also provides is setting up fully connected inter-process sockets so that processes can easily communicate with other processes. This is very important functionality for situations where data and messages need to be accessible between different processes. Real-time Comet applications like chat demonstrate this need. Chat messages received on one process may need to be sent to other processes so they can deliver messages to their connected client. Even for simple session management such capability can be useful. Since browsers utilize multiple connections to servers, it is very easy for one web application to be connected to different processes on the server, which may need to share session data.

To use the inter-process communication, simply add a “node” listener to the object returned from the listen() call. The “node” event is fired for each new Node process with a “stream” argument that can be used to communicate to that process:

var nodes = require("multi-node").listen(…);
nodes.addListener("node", function(stream){
        stream.addListener("data", function(data){
                … receiving data from this other node process …
                stream.write(… write data to other process…);
        });
});
 

This event should be fired for every other Node process (for this server) that is created, allowing you to communicate to any other process.

Multi-node allows you to easily leverage Node’s new socket sharing capabilities and build truly scalable, concurrent, Node-based web application servers. While multi-node is intended to be usable on its own, multi-node is a sub-project of Persevere and the Persevere example wiki demonstrates how multi-node can easily be used with Persevere applications. Next we will look at how another Persevere sub-project, Tunguska, can leverage multi-node’s inter-process communication for building scalable multi-process real-time applications.

Related posts:

1. Real-time Comet Applications on Node with Tunguska
2. Deterministic Client/Server Interaction
3. CommonJS/JSGI: The Emerging JavaScript Application Server Platform

One of the most powerful pieces of Dojo is also one of the most underutilized: Charting.  The Dojo Charting library lives within the DojoX (extensions) branch of Dojo, and features numerous chart types, options, and a variety of themes. This post introduce you to the charting library and show you how you can take a boring data collection and make it a beautiful visual chart in any modern web browser.

Requiring Charting Classes

A common misconception with many Dojo libraries is that they require a large number of dependencies — this is not the case.  Dependencies change depending on the chart you’d like to make, but a simple chart can be created with only the following Dojo classes:

/* required modules for the basic chart */
//Main 2d chart class
dojo.require("dojox.charting.widget.Chart2D");
//Class to create a legend for our chart
dojo.require("dojox.charting.widget.Legend");
//A theme for our chart
dojo.require("dojox.charting.themes.PlotKit.green"); Two Ways to Play

Much like most of the widgets within the Dijit and DojoX libraries, the charting library allows you to create charts programmatically and declaratively.  Let’s take a look at an example of each approach.

Sample Data

The following JSON is used for our simple Pie chart:

var pieData = [{ "x": "1", "y": "9420" }, { "x":"2", "y":"10126" }, { "x": "3", "y": "9803" }, { "x": "4", "y": "15965" }, { "x": "5", "y": "17290" }, { "x": "6", "y": "15667" }, { "x": "7", "y": "17762" }]; Declarative Chart Creation

The declarative method of creating charts is done with HTML markup.  All chart data such as type, axis, plot, and other chart information is created within the chart container.  Here’s a simple pie chart:

<!– create the chart –>
<div dojoType="dojox.charting.widget.Chart2D" id="viewsChart" theme="dojox.charting.themes.PlotKit.green" style="width: 550px; height: 550px;">
       
        <!– add the plot –>
        <div class="plot" name="default" type="Pie" radius="200" fontColor="black" labelOffset="-20"></div>
       
        <!– pieData is the data source –>
        <div class="series" name="January" array="pieData"></div>
       
        <!– tooltips! –>
        <div class="action" type="Tooltip"></div>
       
        <!– slice animation! –>
        <div class="action" type="MoveSlice" shift="2"></div>
       
</div>

Programmatic Chart Creation

Programmatic chart creation is also quite easy.  The JavaScript code below reproduces the chart above:

//create the chart
var pieChart = new dojox.charting.Chart2D("pieChart");
//set the theme
pieChart.setTheme(dojox.charting.themes.PlotKit.green);
//add plot
pieChart.addPlot("default", {
        type: "Pie",
        radius: 200,
        fontColor: "black",
        labelOffset: "-20"
});
//add the data series
pieChart.addSeries("January",pieData);
//slice animation!
new dojox.charting.action2d.MoveSlice(pieChart,"default");
//tooltips!
new dojox.charting.action2d.Tooltip(pieChart,"default");
//render the chart
pieChart.render();

Take a moment to compare the two methods and you will quickly notice how similar the structures are. It’s the developer’s choice which method, declarative or programmatic, to use. This tutorial will focus on the programmatic method of chart creation though all examples work with both approaches.

Creating a Simple Chart

Now that the fundamental pieces of the charting puzzle have been explained, let’s create a very basic chart.  This chart will depict website visits over 3 months time.  The chart will be a basic StackedAreas chart with different colors to denote the different months. We’ll take it step by step.

The first step is to place our data in the format with which the chart works. The data for our StackedArea chart must hold each value in one array:

var json = {
        January: [9420,10126,9803,15965,17290, /* … */  13165,12390],
        February: [23990,32975,23477,22513,18069, /* … */ 12956,12815],
        March: [22477,24014,21116,20404,19142, /* … */ 22077,20263]
};

The next step is to create the chart programatically.

//we have a DIV element with an ID of "chart1";  that’s the argument
var chart1 = new dojox.charting.Chart2D(‘chart1′);

Then it’s time to add the plot.

//add the default plot
chart1.addPlot("default", {
        //type of chart
        type: "StackedAreas",
        //show markers at number points?
        markers: true,
        //curve the lines on the plot?
        tension: "S",
        //show lines?
        lines: true,
        //fill in areas?
        areas: true,
        //offset position for label
        labelOffset: -30,
        //add shadows to lines
        shadows: { dx:2, dy:2, dw:2 }
});

Since the numbers in this type of chart are significant, we want to add an X axis (representing days) and a Y axis (representing traffic / views).

chart1.addAxis("x");
chart1.addAxis("y", { vertical:true });
 

Now that the frame of the chart has been created, it is time to add the information for each month. Since we want to fill areas, we’ll need to pass a stroke and fill color within the series:

chart1.addSeries("January Visits",json["January"], { stroke: "red", fill: "pink" });
chart1.addSeries("February Visits",json["February"], { stroke: "green", fill: "lightgreen" });
chart1.addSeries("March Visits",json["March"], { stroke: "blue", fill: "lightblue" });
 

Now that our chart is built and the data is in place, we can render the chart:

chart1.render();
 

The result is a beautiful chart:

Chart Animations, Legends, and Extras

As with every other piece of the Dojo library, the Dojo charting library is flexible enough to incorporate other Dojo classes and widgets.  The basic chart we created above is nice but could use a few useful enhancements.

Legend

Adding a legend to the chart is very simple:

//this legend is created within an element with a "legend1" ID.
var legend1 = new dojox.charting.widget.Legend({chart: chart1}, "legend1"); Chart Animations

As with basic content on a web page, animation adds some flare and shows focus on an element. Each chart type supports different sets of animations. To keep it simple, we can add a “magnify” animation to the markers on our chart so that the marker grows when the mouse enters it:

//add the magnification animation to our chart for the default plot
var magnify = new dojox.charting.action2d.Magnify(chart1, "default"); Tooltips

Our chart features markers at the designated x/y axis numbers but we can enhance those markers with tooltips so that the user can see the exact number the marker represents.

var tip = new dojox.charting.action2d.Tooltip(chart1, "default");

You may also pair Dijit themes with your chart to style the tooltip; I’ve added the “tundra” theme’s CSS file to enhance the style of my tooltip.

Zooming, Scrolling, and Panning

As you would expect from a vector graphic generation tool, Dojo’s charting library provides a method by which you may flawlessly zoom in and out on any chart. You may also accommodate panning and scrolling on your charts. Read Zooming, Scrolling, and Panning in Dojo Charting to learn more about these features or view a great example of panning, scrolling, and zooming!

Chart Events

Dojo’s charting library also has its own event connection method: connectToPlot.  This method allows you to add event listeners to your chart and trigger functionality based on the given event.

chart1.connectToPlot("default",function(e) {
       
        /* do something */
       
});
 

Chart events provide you a wealth of information, including the type of event, coordinates, plot, shape, and more. Visit the Dojo Charting Reference Guide to get detailed information about chart events.

Chart Themes

The Dojo charting library provides over 30 themes to make your charts stand out. You may see a complete list of bundled themes (with preview) in the Dojo Nightly Theme Previewer. SitePen will publish a post in the near future detailing how to create a custom theme!

Chart Families

This tutorial showed you how to create a basic, but beautiful 2D chart; Dojo’s charting library has many advanced charting capabilities though. The different families of Dojo charts includes:

• 2D Charts: The basic two-dimensional chart. (examples)
• 3D Charts: Three-dimensional representations of data. (example)
• Data Charts: Advanced chart, connectable to an external data source. (example | advanced example: Stocker | tutorial)

New Charting Features in Dojo 1.5

The release of Dojo 1.5 introduces new features to the charting library; most notably:

• Linear and radial gradients can now be used with virtually all chart types (and they even work in IE).
• Flexible data sourcing via DataSeries adds the ability to back charts with practically any sort of data store one would like, even supplying secondary information like tooltips.
• Plot and series order manipulations enable one to programmatically shuffle plots and series as needed.
• Indirect events that effectively make it possible for linked plots to share events.
• External events allow a developer to programmatically generate things like mouseovers and apply them to charting objects.
• The default axis is now smarter regarding scaling and sizing.
• A new lightweight invisible axis type allows the user to control plot scale and alignment without displaying an axis.

Great Charting Resources

• Dojox Charting Reference Guide
• Nightly Tests
• Introducing Dojox Data Chart
• A Beginner’s Guide to Dojo Charting, Part 1 of 2
• A Beginner’s Guide to Dojo Charting, Part 2 of 2
• Zooming, Scrolling, and Panning in Dojo Charting

The concept of charting can seem difficult, but Dojo’s charting library is a breeze to dive into!

Related posts:

1. A Beginner’s Guide to Dojo Charting, Part 2 of 2
2. A Beginner’s Guide to Dojo Charting, Part 1 of 2
3. Zooming, Scrolling, and Panning in Dojo Charting

One huge feature that sets the Dojo Toolkit apart from other JavaScript libraries is its UI component system: Dijit. A flexible, comprehensive collection of Dojo classes (complemented by corresponding assets like images, CSS files, etc.), Dijit allows you to create flexible, extensible, stylish widgets. To learn how to install, configure, and use basic Dijits within your web application, keep reading!

Requiring Proper Classes and Assets

Since Dijit includes a collection of UI components, it comes bundled with four supported themes: nihilo, soria, tundra, and (as of Dojo 1.5) claro. Each theme contains images and CSS files to control the overall display of the widgets. CSS files must be explicitly included into each HTML page:

<style type="text/css">
        /* use the tundra theme */
        @import "http://ajax.googleapis.com/ajax/libs/dojo/1.5/dijit/themes/tundra/tundra.css";
        /* Note that if you don’t specify a minor revision, e.g. 1.5.0 or 1.4.3, the CDN will deliver the latest version */
</style>
 

You can view each theme set using the Dijit Theme Tester. You may also define your own theme.

There is another consideration that must be made when Dojo is included in the page: parseOnLoad. Adding parseOnLoad to the djConfig attribute will direct Dojo to scour the page for elements that should become Dijit widgets and make them such:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/dojo/1.5/dojo/dojo.xd.js.uncompressed.js" djConfig="parseOnLoad:true"></script>
 

This parseOnLoad setting must be true if you prefer automatically instantiating Dijits defined declaratively (which we’ll examine below). You will also need to require Dijit classes you plan to use within the page. For example, If you want the dijit.form.Button widget within your pages, you’ll need to require that class:

dojo.require(‘dijit.form.Button’);
 

In the body tag of your page, or in a parent node of your widgets, you need to define the class name based upon the theme you would like to use.

<body class="claro">
 

Now let’s learn the two ways you can create Dijit widgets.

Two Ways to Play

One of the most notable features of the Dijit system is that you may create a Dijit widget in one of two ways: declaratively (using HTML markup and custom attributes) or programmatically (using raw JavaScript, mostly likely within a dojo.ready block). Let’s take a look at how we can make a basic SELECT element an Dojo-enhanced widget.

The Basic SELECT Element <select name="players" id="players">
        <option value="">Select an Arsenal Player</option>
        <option value="Arshavin" selected="selected">Andrey Arshavin</option>
        <option value="Vermaelen">Thomas Vermaelen</option>
        <!– more OPTION elements here –>
</select>
 

This is a static SELECT element containing a series of OPTION elements. We know we’d like for this SELECT element to become a dijit.form.FilteringSelect widget so we must require this class:

dojo.require(‘dijit.form.FilteringSelect’);

Now that the dijit.form.FilteringSelect class is available, we can use the declarative or programmatic method of enhancing our SELECT element.

Declarative Method

The declarative method of enhancing the SELECT element is done within the HTML element itself:

<select name="players" id="players" dojoType="dijit.form.FilteringSelect" autoComplete="true" pageSize="10">
        <option value="">Select an Arsenal Player</option>
        <option value="Arshavin" selected="selected">Andrey Arshavin</option>
        <option value="Vermaelen">Thomas Vermaelen</option>
        <!– more OPTION elements here –>
</select>

This declarative example illustrates the use of the dojoType attribute which identifies which Dijit the given element should become. Since we have declared parseOnLoad “true” in our djConfig, Dojo will find this element immediately upon load (due to the element having a dojoType attribute) and instantiate and initialize the widget.

Options of the dijit.form.FilteringSelect class are also custom attributes. This widget will autocomplete when the user types a value and page every 10 items. Just as with a normal SELECT element, “Arshavin” will be selected initially.

Programmatic Method

The programmatic method of enhancing the SELECT element is done completely with JavaScript:

//require the class
dojo.require(‘dijit.form.FilteringSelect’);

//when the class has been loaded…
dojo.ready(function() {

        //dijitize my SELECT!
        var enhancedSelect = new dijit.Form.FilteringSelect({
                autoComplete: true,
                pageSize: 10
        },‘players’);

        //dijitize any form field individually
        var inputDijit = new dijit.form.Textbox({/*options*/},‘myInput’);
        var textareaDijit = new dijit.form.Textarea({/*options*/},‘myTextarea’);
        var mySelectDijit = new dijit.form.FilteringSelect({/*options*/},‘mySelect’);
        var dateDijit = new dijit.form.DateTextBox({/*options*/},‘myDate’);
        var timeDijit = new dijit.form.TimeTextBox({/*options*/},‘myTime’);
        var checkboxDijit = new dijit.form.CheckBox({/*options*/},‘myCheckbox’);
        var radioDijit1 = new dijit.form.RadioButton({/*options*/},‘myRadio1′);
        var radioDijit2 = new dijit.form.RadioButton({/*options*/},‘myRadio2′);

});
 

Reminder:  the programmatic method of Dijit creation does not require you to enable parseOnLoad within the djConfig settings.

Since you may have a large form and may not have the desire to target individual elements for Dijitization, you may want to use dojo.behavior to create Dijits by selector:

//require the class
dojo.require(‘dojo.behavior’);
//when it has loaded…
dojo.ready(function(){
        //add two behaviors:
        dojo.behavior.add({
                //find SELECT elements, make them a FilteringSelect Dijit
                ’select’: {
                        found: function(item) {
                                new dijit.form.FilteringSelect({/*options*/},item);
                        }
                },
                //find all radio buttons, make them dijit.form.RadioButton Dijits
                ‘input[type="radio"]‘: {
                        found: function(item) {
                                new dijit.form.RadioButton({  },item);
                        }
                }/* ,
                        more selector => Dijit creation….
                */
        });
});

You could also run a few dojo.query searches for elements and loop through them to create new widgets, but the dojo.behavior method provides a cleaner syntax.

A Dijitized form; easily customizable and themeable.

Accessing Dijit Widgets and Widget Properties

Accessing a specific DOM element can be easily accomplished by using Dojo’s byId method. Dijit features its own byId method which retrieves the Dijit widget registered within the widget system with the ID specified. If the element to be made a Dijit has an ID, the widget ID will be that same value. If the source element doesn’t have an ID attribute, a widget ID will be generated. If we wanted to retrieve the widget object for the declaratively created “players” Dijit above, we would code:

//grab the players dijit.form.Filtering widget
var enhancedSelect = dijit.byId(‘players’);

View a listing of all properties and methods of a Dijit using dijit.byId within Firebug.

If we wanted to access the pageSize for which the Dijit widget was created from, we would code:

var pageSize = dijit.byId(‘players’).get(‘pageSize’); // returns 10

If we wanted to change the pageSize for the widget, we would code:

dijit.byId(‘players’).set(‘pageSize’,20); //now pageSize is 20

Note: Dojo 1.5 introduces “get” and “set” methods to handle property values. In Dojo 1.4 and earlier, you would use “attr” in place of both get and set in the previous example.

Listening to Widget Events

Dijit widgets use Dojo’s native connect method to listen to events on the given widget:

dojo.connect(dijit.byId(‘player’),‘onChange’,function() {

        //the user has selected a new item
        //do something

});

Each widget supports a select number of events so be sure to view the documentation for the widget to ensure the event you’d like to listen to is supported.

The Dijit Collection

Dijit is also an incredible UI library that has the potential to enhance your website and save you an immense amount of time in doing so. Dijit includes a large number of solid widgets including:

• Context, popup, and dropdown menus
• Form element replacements like buttons, combo boxes, checkboxes, radio buttons, and text boxes
• Date and time selection widgets
• WYSIWYG Editors
• Horizontal and Vertical Sliders
• Progress Bars
• Tabs and Accordions
• Tree Structures (including Drag and Drop)
• Dialogs and Tooltips
• Layout widgets with slide controls and splitters

And if Dijit doesn’t have a widget you want, there’s a good chance it’s available in DojoX (Dojo Extensions)! To see many of these widgets in action, visit the Dojo Theme Tester.

Dijit Resources

The following resources will help to aid your Dijit education:

• Dijit API Documentation
• Dijit Reference Guide
• Dojo Nightlies (test files)
• Dijit Theme Tester

Stay tuned for more in-depth Dijit posts in the future. This amazing library contains a vast array of useful widgets to enhance your website!

Related posts:

1. Styling Dijit Form Elements
2. Dijit: Prepackaged
3. Usable directory listings with a little Dojo

Creating and Enhancing Dojo Classes

Like all top-notch JavaScript toolkits, Dojo tries to make its classes as flexible as possible, knowing that users of the toolkit may have different ideas about how a given class or class method should work. Luckily, Dojo provides you a number of methods by which you can subclass or modify existing classes. Let’s examine a few ways you can make Dojo classes exactly the way you like.

Creating Dojo Subclasses

The typical method by which you can create a Dojo class (or subclass) is by using the dojo.declare method. The dojo.declare method registers your given class within its designated namespace, while also subclassing any number of classes passed in the method’s second argument.

The following code shows the basic method by which one can create a subclass:

//dojo.declare(’your.name.space’,superClass,{ customProperties });
//or
//dojo.declare(’your.name.space’,[superClass1,superClass2,superClass3],{ customProperties });

dojo.provide(‘davidwalsh.Menu’);
dojo.declare(‘davidwalsh.Menu’,dijit.Menu,{

        /* your custom properties and methods here */
        myCustomProperty: true,
        myCustomMethod: function() {
                /* do stuff */
        }

        //all of dijit.Menu’s methods are also part of this new class
});
 

The new class above, davidwalsh.Menu, is a shiny new custom Dojo class that inherits all methods and properties of the dijit.Menu class. My new class also features a new custom property and a new custom method which can do anything I want. Now that we know how to create a subclass, let’s create a realistic example of a useful subclass: davidwalsh.Menu.

dojo.provide(‘davidwalsh.Menu’);
dojo.declare(‘davidwalsh.Menu’,dijit.Menu,{

        //new option
        allowSubmenuHover: true,

        //another new option
        popupDelay: 500,

        //override the dijit.Menu method
        onItemHover: function(item) {
                if(this.isActive || this.allowSubmenuHover) {
                        this.focusChild(item);
                        //use the new settings to trigger popup
                        if(this.focusedChild.popup &&
                          !this.focusedChild.disabled &&
                          !this.hover_timer){
                                this.hover_timer = setTimeout(
                                  dojo.hitch(this, ‘_openPopup’),
                                  this.popupDelay);
                        }
                }
                if(this.focusedChild){
                        this.focusChild(item);
                }
                this._hoveredChild = item;
        }

});
 

davidwalsh.Menu is an enhancement of the original dijit.Menu class; it features two new options and overrides a dijit.Menu method with an end goal of providing a Menu whose PopupMenuItem opens when hovered over instead of clicked.

You may be wondering how to call methods of a superclass from your new subclass. That’s simple too:

//more methods above
someMethod: function() {
        /* do anything you want here */

        // call the superclass’ "someMethod" method
        // to execute superclass’ original functionality
        var result = this.inherited(arguments);

        /* do anything you want here */
}
//more methods below
 

As you can see, creating subclasses is a breeze! But what can you do if you simply want to modify an existing Dojo class? Monkey patch it!

Prototype Modification or “Monkey Patching”

Sometimes subclassing existing Dojo classes isn’t the best option or an option at all. You may find yourself in the position where you can only patch an existing Dojo install; in that case, monkey patching is the ideal strategy. Monkey patching is the process by which you modify the prototype of an existing object (in our case a Dojo class). Positives of using a monkey patch approach are:

• All existing objects of this type are instantly changed.
• You don’t need access to the core Dojo files.
• Since you aren’t modifying the core Dojo files themselves, upgrading your Dojo builds will be exponentially easier as you wont have to hunt for your past changes.
• Your patches are more portable as they aren’t placed directly in the core Dojo files themselves.

The following code shows the monkey patching format:

(function(){
        //save the old prototype method
        var oldPrototypeSomeMethod =
                dijit.someDijit.prototype.someMethod;

        //modify the prototype
        dijit.someDijit.prototype.someMethod = function(){

        /* some new stuff here */

        // call the old method *only if you
        // want to keep some of the original functionality*
        oldPrototypeSomeMethod.call(this, arguments);
    };
})();
 

Now let’s take a look at a realistic example. I was recently working with the FilteringSelect Dijit when I noticed that if the first option element within the srcNode (select element) has no value (an empty string value attribute), the element’s label will not display. A very odd bug and definitely not a desired result. What I was able to do was patch the Dijit’s postMixInProperties' method to fix the problem.

(function(){
        var dffsp = dijit.form.FilteringSelect.prototype;
        //save the old prototype method
        var oldPMIP = dffsp.postMixInProperties;

        //modify the Dijit’s prototype
        dffsp.postMixInProperties = function(){

          //if this select has no value and the first option is blank:
          //set the displayedValue of this dijit to that label initially
          if(!this.store &amp;&amp; this.srcNodeRef.value == ”){
                var srcNodeRef = this.srcNodeRef,
                nodes = dojo.query("> option[value='']", srcNodeRef);
                if(nodes.length){
                        this.displayedValue =
                                dojo.trim(nodes[0].innerHTML);
                }
          }

        // call the original prototype method;
        // we still want the original functionality to fire
        oldPMIP.call(this, arguments);
    };
})();
 

That’s just one great example of monkey patching. While monkey patching may not look like the most beautiful technique, it’s an essential part of patching your Dojo installs.

Extending Dojo Classes

The dojo.extend method allows us to add new methods to the class’ prototype, thus providing the new methods to every instance of a given class. If a method name is passed to dojo.extend that already exists for a given class, the new method overrides the original method.

The following code illustrates extending the dijit.Menu class, allowing popup menu items to open during its label’s hover event in addition to the click event.

dojo.extend(dijit.Menu,{
        allowSubmenuHover: true, //new setting
        popupDelay: 500, //new setting
        onItemHover: function() { //overriding original method
                if(this.isActive || this.allowSubmenuHover) {
                        this.focusChild(item);
                        //use the new settings to trigger popup
                        if(this.focusedChild.popup &&
                          !this.focusedChild.disabled &&
                          !this.hover_timer){
                            this.hover_timer = setTimeout(dojo.hitch(
                                  this, ‘_openPopup’), this.popupDelay);
                        }
                }
                if(this.focusedChild){
                        this.focusChild(item);
                }
                this._hoveredChild = item;
        }
});

Note that the original onItemHover method isn’t saved to be executed later — the entire prototype has been rewritten. Essentially we’re rubbishing the original functionality of this method. Now we have a dijit.Menu class that conforms to our needs and doesn’t interfere with our Dojo build itself.

To Subclass, Extend, or Monkey Patch?

There are no hard and fast rules for when to extend a class or monkey patch a class. I do have a few recommendations though:

• Do *not* change one of the core Dojo files; monkey patch or extend.
• Monkey patch the class if you need access to the original prototype.
• Subclass with your own namespace when looking to share code between projects.
• If portability is an important goal, extend the class.

Extend Away!

Extending Dojo classes is the perfect way to fix bugs, enhance native Dojo classes, and prevent you from needing to repeat code. The limitations of Dojo are only those that you put on it!

Related posts:

1. Creating Dojo Widgets with Inline Templates
2. Creating Asynchronous Tests with the Dojo Objective Harness
3. Managing Widget Templates in Dojo 1.4

JSGI (JavaScript gateway interface) is the broadly adopted standard for server-side JavaScript web applications and middleware. JSGI is designed specifically for ease of use in asynchronous, evented environments, and consequently JSGI has been a perfect fit building applications on Node. JSGI-Node has existed for sometime as lightweight JSGI server/adapter for running JSGI-based applications on Node. However, “Connect” was recently released as a new alternate middleware system for Node. This spurred me to make sure JSGI-Node was optimized, and compare the Connect middleware design to JSGI.

Performance

To begin with, I did some performance comparisons. The first test is the simplest possible “hello world” application. This was run against a mock server to eliminate any overhead of HTTP parsing and isolate the test to simple request delegation. Here are the results in requests per second:

JSGI-Node: 730K/s
Connect: 204K/s

Here JSGI-Node is shown to be about 257% faster. Next, I created a stack of 9 middleware apps in front of the “hello world” app. These middleware apps do nothing but delegate to the next layer. This was designed to isolate the middleware delegation mechanism. The results are in middleware delegations per second:

JSGI-Node: 80,600K/s
Connect: 2,230K/s

Middleware delegation is over 3000% faster with JSGI than Connect! Why? JSGI is based on a simple, lightweight, pure JavaScript approach to connecting middleware. Middleware apps are simple function closures that accept another app that they can delegate to. Nothing is actually required to wire middleware together. Connect on the other hand has to go through a heavy delegation layer to wire the middleware together. Middleware delegation might not pose a significant bottleneck for simple applications, but as middleware layers accumulate, the speed of delegation becomes increasingly important.

Putting these test together, the full request delegation and middleware delegation through nine middleware layers to a simple “hello world” app results in following requests per second:

JSGI-Node: 675K/s
Connect: 112K/s

With JSGI providing about a 503% better performance with isolated request and middleware delegation.

Here are the test files.

Flexibilty

One of the powerful aspects of JSGI is that middleware is connected by simply passing apps as arguments. This means you can not only create linear stacks, but you can create rich branching tree type request routing structures. Connect on the other hand is really only designed for creating linear middleware stacks. This is useful for demonstrating simple middleware integration, but often real applications require much more sophisticated logic than these simple stacks. While Connect comes with routing middleware, it is not clear if it is possible to create middleware stacks under the different routes (especially with the same ease as JSGI).

Layer Isolation

One of the key concepts of middleware is layering. JSGI provides a simple mechanism for allow each layer to provide a controlled view of the request for the next layer and control the response from that layer. On the other hand, Connect does not provide any mechanism for wrapping and creating new request objects or handling response objects. Altering the view for the next layer means actually altering the request and response objects in place. This fundamentally breaks the layering concept of middleware because when the request object is altered, it is altered across all layers, below and above, not just for the next layer.

Ease of use

Ease of use is in the eye of the beholder, so I will just show so code comparisons. Here is a sample of how you create middleware with JSGI:

function Middleware(nextApp){
  return function(request){
    var response = nextApp(request or our own request object);
    return response or any modified response we want to return;
  };
}
// now wire it up
jsgiNode.start(Middleware(app));
 

And in Connect:

Middleware = {
  handle: function(req, res){
    // must modify req in place
    // modify res handlers in place as well
    next();
    // restore any res handlers that might be used lower down in the stack
  }
};
// now wire it up
var connectServer = new Server([
  {module: Middleware},
  {module: app}
]);
connectServer.listen(…);
 

Pintura has a wealth of examples of other middleware applications.

One complaint that has been made about JSGI is that it was not designed for specifically for Node. However, it was designed for asynchronous event-oriented architecture, which is exactly the point of Node, and so there really is no credibility to discounting JSGI for not being Node specific. It is a perfect fit for Node.

Middleware Availability

Simply take a look at the rapidly growing list of middleware modules for Node to see what is available for JSGI. JSGI is based on a collaborative effort of large number of participants in the CommonJS group. Furthermore, JSGI-Node is licensed under the Dojo foundation, which has the known for rock solid CLA-protected IP-encumberance-free liberally licensed software with a track record of being safe to use.

Now to be fair, Connect is about more than just inventing yet another middleware interface. It has some other extremely useful features including a powerful command line startup mechanism and middleware configuration techniques. In fact, Tim Caswell has actually talked about possibly including support for JSGI in Connect. This would be a great addition, and would allow Connect to actually combine the superior JSGI middleware interface with the other cool features in Connect.

The elegant design of JSGI, based on simple idiomatic JavaScript, allows for simple, intuitive middleware, with flexible connections, incredible performance, and connected to a growing ecosystem of reusable appliances.

Related posts:

1. Pintura JSGI Modules
2. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
3. Real-time Comet Applications on Node with Tunguska

The NoSQL movement continues to gain momentum as developers continue to grow weary of traditional SQL based database management and look for advancements in storage technology. A recent article provided a great roundup of some of the great new technologies in this area, particularly focusing on the different approaches to replication and partitioning. There are excellent new technologies available, but using a NoSQL database is not just a straight substitute for a SQL server. NoSQL changes the rules in many ways, and using a NoSQL database is best accompanied by a corresponding change in application architecture.

The NoSQL database approach is characterized by a move away from the complexity of SQL based servers. The logic of validation, access control, mapping querieable indexed data, correlating related data, conflict resolution, maintaining integrity constraints, and triggered procedures is moved out of the database layer. This enables NoSQL database engines to focus on exceptional performance and scalability. Of course, these fundamental data concerns of an application don’t go away, but rather must move to a programmatic layer. One of the key advantages of the NoSQL-driven architecture is that this logic can now be codified in our own familiar, powerful, flexible turing-complete programming languages, rather than relying on the vast assortment of complex APIs and languages in a SQL server (data column, definitions, queries, stored procedures, etc).

In this article, we’ll explore the different aspects of data management and suggest an architecture that uses a data management tier on top of NoSQL databases, where this tier focuses on the concerns of handling and managing data like validation, relation correlation, and integrity maintenance. Further, I believe this architecture also suggests a more user-interface-focused lightweight version of the model-viewer-controller (MVC) for the next tier. I then want to demonstrate how the Persevere 2.0 framework is well suited to be a data management layer on top of NoSQL databases. Lets look at the different aspects of databases and how NoSQL engines affect our handling of data and architecture.

Architecture with NoSQL

In order to understand how to properly architect applications with NoSQL databases you must understand the separation of concerns between data management and data storage. The past era of SQL based databases attempted to satisfy both concerns with databases. This is very difficult, and inevitably applications would take on part of the task of data management, providing certain validation tasks and adding modeling logic. One of the key concepts of the NoSQL movement is to have DBs focus on the task of high-performance scalable data storage, and provide low-level access to a data management layer in a way that allows data management tasks to be conveniently written in the programming language of choice rather than having data management logic spread across Turing-complete application languages, SQL, and sometimes even DB-specific stored procedure languages.

Complex Data Structures

One important capability that most NoSQL databases provide is hierarchical nested structures in data entities. Hierarchical data and data with list type structures are easily described with JSON and other formats used by NoSQL databases, where multiple tables with relations would be necessary in traditional SQL databases to describe these data structures. Furthermore, JSON (or alternatives) provide a format that much more closely matches the common programming languages data structure, greatly simplifying object mapping. The ability to easily store object-style structures without impedance mismatch is a big attractant of NoSQL.

Nested data structures work elegantly in situations where the children/substructures are always accessed from within a parent document. Object oriented and RDF databases also work well with data structures that are uni-directional, one object is accessed from another, but not vice versa. However, if the data entities may need to be individually accessed and updated or relations are bi-directional, real relations become necessary. For example, if we had a database of employees and employers, we could easily envision scenarios where we would start with an employee and want to find their employer, or start with an employer and find all their employees. It may also be desirable to individually update an employee or employer without having to worry about updating all the related entities.

In some situations, nested structures can eliminate unnecessary bi-directional relations and greatly simplify database design, but there are still critical parts of real applications where relations are essential.

Handling Relational Data

The NoSQL style databases has often been termed non-relational databases. This is an unfortunate term. These databases can certainly be used with data that has relations, which is actually extremely important. In fact, real data almost always has relations. Truly non-relational data management would be virtually worthless. Understanding how to deal with relations has not always been well-addressed by NoSQL discussions and is perhaps one of the most important issues for real application development on top of NoSQL databases.

The handling of relations with traditional RDBMSs is very well understood. Table structures are defined by data normalization, and data is retrieved through SQL queries that often make extensive use of joins to leverage the relations of data to aggregate information from multiple normalized tables. The benefits of normalization are also clear. How then do we model relations and utilize them with NoSQL databases?

There are a couple approaches. First, we can retain normalization strategies and avoid any duplication of data. Alternately, we can choose to de-normalize data which can have benefits for improved query performance.

With normalized data we can preserve key invariants, making it easy to maintain consistent data, without having to worry about keeping duplicated data in sync. However, normalization can often push the burden of effort on to queries to aggregate information from multiple records and can often incur substantial performance costs. Substantial effort has been put into providing high-performance JOINs in RDBMSs to provide optimally efficient access to normalized data. However, in the NoSQL world, most DBs do not provide any ad-hoc JOIN type of query functionality. Consequently, to perform a query that aggregates information across tables often requires application level iteration, or creative use of map-reduce functions. Queries that utilize joining for filtering across different mutable records often cannot be properly addressed with map-reduce functions, and must use application level iteration.

NoSQL advocates might suggest that the lack of JOIN functionality is beneficial; it encourages de-normalization that provides much more efficient query-time data access. All aggregation happens for each (less frequent) write, thus allowing queries to avoid any O(n) aggregation operations. However, de-normalization can have serious consequences. De-normalization means that data is prone to inconsistencies. Generally, this means duplication of data; when that data is mutated, applications must rely on synchronization techniques to avoid having copies become inconsistent. This invariant can easily be violated by application code. While it is typically suitable for multiple applications to access database management servers, with de-normalized data, database access becomes fraught with invariants that must be carefully understood.

These hazards do not negate the value of database de-normalization as an optimization and scalability technique. However, with such an approach, database access should be viewed as an internal aspect of implementation rather than a reusable API. The management of data consistency becomes an integral compliment to the NoSQL storage as part of the whole database system.

The NoSQL approach is headed in the wrong direction if it is attempting to invalidate the historic pillars of data management, established by Edgar Codd. These basic rules for maintaining consistent data are timeless, but with the proper architecture a full NoSQL-based data management system does not need to contradict these ideas. Rather it couples NoSQL data storage engines with database management logic, allowing for these rules to be fulfilled in much more natural ways. In fact, Codd himself, the undisputed father of relational databases, was opposed to SQL. Most likely, he would find a properly architected database management application layer combined with a NoSQL storage engine to fit much closer to his ideals of a relational database then the traditional SQL database.

Network or In-process Programmatic Interaction?

With the vastly different approach of NoSQL servers, it is worth considering if the traditional network-based out-of-process interaction approach of SQL servers is truly optimal for NoSQL servers. Interestingly, both of the approaches to relational data point to the value of more direct in-process programmatic access to indexes rather than the traditional query-request-over-tcp style communication. JOIN style queries over normalized data is very doable with NoSQL databases, but it relies on iterating through data sets with lookups during each loop. These lookups can be very cheap at the index level, but can incur a lot of overhead at the TCP handling and query parsing level. Direct programmatic interaction with the database sidesteps the unnecessary overhead, allowing for reasonably fast ad-hoc relational queries. This does not hinder clustering or replication across multiple machines, the data management layer can be connected to the storage system on each box.

De-normalization approaches also work well with in-process programmatic access. Here the reasons are different. Now, access to the database should be funneled through a programmatic layer that handles all data synchronization needs to preserve invariants so that multiple higher level application modules can safely interact with the database (whether programmatically or a higher level TCP/IP based communication such as HTTP). With programmatic-only access, the data can be more safely protected from access that might violate integrity expectations.

Browser vendors have also come to similar conclusions of programmatic access to indexes rather than query-based access in the W3C process to define the browser-based database API. Earlier efforts to provide browser-based databases spurred by Google Gears and later implemented in Safari were SQL-based. But the obvious growing dissatisfaction with SQL among developers and the impedance mismatches between RDBMS style data structures and JavaScript style data structures, has led the W3C, with a proposal from Oracle (and supported by Mozilla and Microsoft), to orient towards a NoSQL-style indexed key-value document database API modeled after the Berkeley DB API.

Schemas/Validation

Most NoSQL databases could also be called schema-free databases as this is often one of the most highly touted aspects of these type of databases. The key advantage of schema-free design is that it allows applications to quickly upgrade the structure of data without expensive table rewrites. It also allows for greater flexibility in storing heterogeneously structured data. But while applications may benefit greatly from freedom from storage schemas, this certainly does not eliminate the need to enforce data validity and integrity constraints.

Moving the validity/integrity enforcement to the data management layer has significant advantages. SQL databases had very limited stiff schemas, whereas we have much more flexibility enforcing constraints with a programming language. We can enforce complex rules, mix strict type enforcements on certain properties, and leave other properties free to carry various types or be optional. Validation can even employ access to external systems to verify data. By moving validation out of the storage layer, we can centralize validation in our data management layer and have the freedom to create rich data structures and evolve our applications without storage system induced limitations.

ACID/BASE and Relaxing Consistency Constraints

One aspect of the NoSQL movement has been a move away from trying to maintain completely perfect consistency across distributed servers (everyone has the same view of data) due to the burden this places on databases, particularly in distributed systems. The now famous CAP theorem states that of consistency, availability, and network partitioning, only two can be guaranteed at any time. Traditional relational databases have kept strict transactional semantics to preserve consistency, but many NoSQL databases are moving towards a more scalable architecture that relaxes consistency. Relaxing consistency is often called eventual consistency. This permits much more scalable distributed storage systems where writes can occur without using two phase commits or system-wide locks.

However, relaxing consistency does lead to the possibility of conflicting writes. When multiple nodes can accept modifications without expensive lock coordination, concurrent writes can occur in conflict. Databases like CouchDB will put objects into a conflict state when this occurs. However, it is inevitably the responsibility of the application to deal with these conflicts. Again, our suggested data management layer is naturally the place for the conflict resolution logic.

Data management can also be used to customize the consistency level. In general, one can implement more relaxed consistency-based replication systems on top of individual database storage systems based on stricter transactional semantics. Customized replication and consistency enforcements can be very useful for applications where some updates may require higher integrity and some may require the higher scalability of relaxed consistency.

Customizing replication can also be useful for determining exactly what constitutes a conflict. Multi-Version Concurency Control (MVCC) style conflict resolution like that of CouchDB can be very naive. MVCC assumes the precondition for any update is the version number of the previous version of the document. This certainly is not necessarily always the correct precondition, and many times unexpected inconsistent data may be due to updates that were based on other record/document states. Creating the proper update logs and correctly finding conflicts during synchronization can often involve application level design decisions that a storage can’t make on its own.

Persevere

Persevere is a RESTful persistence framework, version 2.0 is designed for NoSQL databases while maintaining the architecture principles of the relational model, providing a solid complementary approach. Persevere’s persistence system, Perstore, uses a data store API that is actually directly modeled after W3C’s No-SQL-inspired indexed database API. Combined with Persevere’s RESTful HTTP handler (called Pintura), data can be efficiently and scalably stored in NoSQL storage engines and accessed through a data modeling logic layer that allows users to access data in familiar RESTful terms with appropriate data views, while preserving data consistency. Persevere provides JSON Schema based validation, data modification, creation, and deletion handlers, notifications, and a faceted-based security system.

Persevere’s evolutionary schema approach leans on the convenience of JSON schema to provide a powerful set of validation tools. In Persevere 2.0, we can define a data model:

var Model = require("model").Model;
// productStore provides the API for accessing the storage DB.
Model("Product", productStore, {
  properties: {
    name: String, // we can easily define type constraints
    price: { // we can create more sophisticated constraints
      type: "number",
      miminum: 0,
    },
    productCode: {
      set: function(value){
        // or even programmatic validation
      }
    }
  },
  // note that we have not restricted additional properties from being added
  // we could restrict additional properties with:
  // additionalProperties: false

  // we can specify different action handlers. These are optional, they will
  // pass through to the store if not defined.
  query: function(query, options){
    // we can specify how queries should be handled and
    //delivered to the storage system
    productStore.query(query, options
  },
  put: function(object, directives){
     // we could specify any access checks, or updates to other objects
     // that need to take place
     productStore.put(object, directives);
  }
});

The Persevere 2.0 series of articles provides more information about creating data models as well as using facets for controlling access to data.

Persevere’s Relational Links

Persevere also provides relation management. This is also based on the JSON Schema specification, and has a RESTful design based on the link relation mechanism that is used in HTML and Atom. JSON Schema link definitions provide a clean technique for defining bi-directional links in a way that gives link visibility to clients. Building on our product example, we could define a Manufacturer model, and link products to their manufacturers:

Model("Product", productStore, {
   properties: {
      …
      manufacturerId: String
   },
   links: [
     {
        rel: "manufacturer",
        href: "Manufacturer/{manufacterId}"
     }
   ],
…
});

Model("Manufacturer", manufacturerStore, {
   properties: {
      name: String,
      id: String,
      …
   },
   links: [
     {
        rel: "products",
        href: "Product/?manufacterId={id}"
     }
   ]
});
 

With this definition we have explicitly defined how one can traverse back and forth (bi-directionally) between a product and a manufacturer, using a standard normalized foreign key (no extra effort in synchronizing bi-direction references).

The New mVC

By implementing a logically complete data management system, we have effectively implemented the “model” of the MVC architecture. This actually allows the MVC layer to stay more focused and lightweight. Rather than having to handle data modeling and management concerns, the MVC can focus on the user interface (the viewer and controller), and simply a minimal model connector that interfaces with the full model implementation, the data management layer. I’d suggest this type of user interface layer be recapitalized as mVC to denote the de-emphasis on data modeling concerns in the user interface logic. This type of architecture facilitates multiple mVC UIs connecting to a single data management system, or vice versa, a single mVC could connect to multiple data management systems, aggregating data for the user. This decoupling improves the opportunity for independent evolution of components.

Conclusion

The main conceptual ideas that I believe are key to the evolution of NoSQL-based architectures:

• Database management needs to move to a two layer architecture, separating the concerns of data modeling and data storage. Persevere demonstrates this data modeling type of framework with a web-oriented RESTful design that complements the new breed of NoSQL servers.
• With this two layered approach, the data storage server should be coupled to a particular data model manager that ensures consistency and integrity. All access must go through this data model manager to protect the invariants enforced by the managerial layer.
• With a coupled management layer, storage servers are most efficiently accessed through a programmatic API, preferably keeping the storage system in-process to minimize communication overhead.
• The W3C Indexed Database API fits this model well, with an API that astutely abstracts the storage (including indexing storage) concerns from the data modeling concerns. This is applicable on the server just as well as the client-side. Kudos to the W3C for an excellent NoSQL-style API.

Related posts:

1. Resource Oriented Programming
2. Introducing Pintura
3. When to Use Persevere: a Comparison with CouchDB and Others

The REST architecture has become increasingly recognized for its value in creating scalable, loosely coupled systems. REST is presented as a network interaction architectural style, not a programming methodology. However, the principles of REST can actually be very meaningfully and beneficially applied in the programming realm. We will look at how the resource oriented approach of REST can be applied as principles for programming language usage and design. The motivation for looking at REST should be clear. Little in history has been as ridiculously successful and scalable as the web, and REST is a retrospective look at the principles that were employed in designing the core technologies of the web, particularly HTTP. Applying such proven principles to our application design will certainly be beneficial.

Roy Fielding’s REST architecture is broken down into seven constraints (and the four sub-constraints of the uniform interface). The individual concepts here are certainly not new, but collectively looking at these concepts as resource oriented programming may provide an interesting new perspective. I will also look at how these principles are exemplified in Persevere 2.0 in its object store framework, Perstore, and its web stack Pintura.

Client-Server

Of all the REST principles the first principle is most targeted at the network paradigm, and is the least compelling for advising programming. However, it does suggest the importance of an asymmetric relationship between modules with separation of concerns. When modules interact, one module should stick to the role of providing a service (acting as the server), and one should act as the consumer of the service. The roles and concerns of the two modules thus stay cleanly separated.

Stateless

If you follow computer science research, you are probably aware that one of the predominant focuses of modern programming languages is seeking to minimize the hazards of stateful imperative programming. Many developers are moving towards functional languages, or at least functional elements of languages, which perform computations on inputs, yielding outputs, rather than being defined as a series of state changes. Concurrency, security, scalability, and many other programming concepts become drastically simpler when mutating state is removed, and applications become much easier to develop. This directly corresponds with the stateless principle of REST. Maintaining transitive state information puts an enormous burden on modules in terms of scalability and complexity. Minimizing or eliminating state and using functional paradigms yields much better code.

When we consider the stateless constraint of REST, there is an important distinction to be made between “state” and “data” or “resources”. Since REST is clearly resource oriented, data/resources are obviously not something that we are trying to eliminate. When we are advising against state, we are advising against the use of transitive state information as the primary programming technique for developing and implementing application flow and logic. The resources and data to which an application provides a user interface are still mutable structures, but this interaction is carefully handled through the uniform interface (coming later).

Cache

Another key concept of a resource oriented approach is that it provides visibility for interaction. REST’s interaction explicitly defines cacheability and idempotence of actions. Certain operations are known to be safe and cacheable. In HTTP, GET requests fit this criteria. In standard object oriented programming conventions, a getter is safe and cacheable as well. If a uniform interface has been defined, one can define one or more retrieval function that are safely cacheable. The uniform interface must explicitly disclose which operations have side-effects. This can have very significant implications for creating well-optimized code. We can re-utilize local copies of data without re-calling retrieval functions if the functions are known to be cacheable. In the realm of purely opaque functions we are afforded no such opportunities (who knows if a function might have side-effects).

Getters should be safe and cacheable (and idempotent), and setters should be idempotent. Getters and setters provide an extremely powerful concept within programming. Distilling everything down to opaque calls hides important semantics from users that can distinctly alter their expectations and use of modules, but getters and setters expose these expectations in a way that benefits users. Developers can expect that object.foo can be evaluated multiple times with the same result unless the property has been altered. Setting object.foo is an idempotent operation (i.e., it can be safely repeated) and can be expected to alter the value returned by object.foo (if successful).

Uniform Interface

The most central concept of REST is the uniform interface, often characterized by HTTP’s standard methods: GET, PUT, POST, DELETE, etc. as the means for navigation and interaction. These methods have clear semantics. GET is safe, PUT, POST, DELETE have side-effects. GET, PUT, and DELETE are idempotent, POST is not. PUT is supposed to update a resource in a way that predictably alters the expected response to a GET.

These methods have analogies in object programming with getters and setters as we have discussed; however a uniform interface can be more than simply getters and setters. The uniform interface does not act solely on the internal state of individual objects, but can affect the collection of some type of objects, providing means for creating and deleting objects, querying for objects, retrieving by unique identifier, and even locking and unlocking the objects.

Perhaps one excellent example of the uniform interface principle in JavaScript is Dojo’s Data API. While it doesn’t utilize the traditional HTTP-style set of verbs, this API does follow the uniform interface principle and provides a single generic uniform interface through which all widgets can access data, and the data source can implement this interface to provide generic access to various different storage systems or file formats.

The W3C is working on a compelling new API called the indexed database API that could provide a more broadly accepted generic uniform interface for JavaScript, and even bears quite a bit of resemblance to the traditional HTTP verbiage for verbs, with get(), put(), and delete() methods. This API is used by Persevere 2.0 in it’s object store framework, Perstore, and further leveraged by its web stack Pintura. With this interface, various different storage systems, especially NoSQL servers, can very easily be modeled and leveraged at different layers within the Pintura/Perstore ecosystem.

The uniform interface is further broken down into four important sub-constraints.

Identification of Resources

A fundamental concept of REST is that resources must have a unique identifier which can be used to retrieve that resource (a URL). The identifier must be unique and carry sufficient information to locate the resource. If the identifier is only unique within a given subsystem, additional information must be added to the identifier if it is referenced outside the subsystem. This is in fact how URLs work. The paths are unique to a server, but the path is combined with the server host name to make them unique across the entire Internet. When applied to programming, resources are objects that are identifiable. Within an application runtime, it is often sufficient to simply use the server unique identifier, or even within the scope of certain modules use an identifier specific to a given table or data store.

Within Perstore, all resource objects must have an identifier property or getId() function that returns the identity of the object. Resource objects can be retrieved using the store.get(id). These mechanisms guarantee a way to discover a resource’s identity and retrieve the resource by the identity.

Manipulation of Resources through Representations

REST defines a distinction between resources and representations. Resources are not necessarily simple files. Resources may represent abstract concepts such people, computers, events, and more, and may be stored and accessed through various means such as database records. In the scope of network communication, representations provide a means for describing these resources using byte-level serialization. REST recognizes that different consumers may need or prefer different types of representations, and supporting multiple representations for a given resource can be highly valuable.

In intra-machine programming, of course modules can interact with higher level constructs than byte-level data, and can use objects, numbers, strings, and language constructs. However, there still may be multiple ways of representing an abstract resource. These can be particularly important if different data consumers expect data to be structured in different ways.

One of the ways that the abstraction between a representation and a resource can be achieved is with facets. I have previously discussed how facets are a powerful tool for securely controlling access to resources in the object capability model. Facets act as a wrapper for an object store, and can not only attenuate access, but can provide alternate “views” of data/resources. For example, facets can easily be used for localization. You could create one facet for a set of resources that provides methods and properties for English speakers, and then another set of methods and properties for French speakers. The facet could include any necessary logic for pulling data from different data sources based on locale. This use of facets is a clear demonstration of separation of representation/view and resource.

Self-descriptive Messages

Self-descriptive messages describe how we view and interact with a resource. There are a couple aspects of this principle. First, we should be able to easily introspect a representation of a resource to comprehend the structure and content, and use this information to intelligently update the object. Objects with properties (or hash/map style name-value pairs, which is the same thing in JavaScript) are well suited for this since the introspection clearly implies how the object can be updated. Self-descriptive objects are easier to interact with because generic tools can interact with them. The API for interacting with the object is evident from the object itself.

Self-descriptive messages also means that in addition to the raw representation of a resource, we should also be able to view additional metadata that describes extra information about the resource such as cacheability, attribution information, etc. Here, Pintura defines a getMetadata() function for resource objects that provides access to metadata information for resources and their representations. This makes metadata readily available for processing conditional requests (based on modification times or etags) and defining caching levels.

Hypermedia as the Engine of Application State

The ability to navigate resources through hyperlinks/hypermedia is the hallmark of REST. This is easy to relate to object oriented programming where property values can be references to other objects or getters can return other objects, making it easy to navigate through data structures. We frequently utilize object mapping to maintain these types of object references from relationships indicated by underlying data sources.

Perstore leverages JSON Schema’s hyperlink description capabilities to define links based on data. Perstore provides a get(relationName) method on all resource objects to navigate to other resources based on the links defined in the schema definition. For example, we can define a model in Perstore:

var Model = require("model").Model;
Task = Model("Task", dataStore, {
  links: [
    {rel: "project", href: "{projectId}"}
  ]
});

And then navigate through the data:

var someTask = Task.get("some task id");
someTask.projectId -> id of a the project it is part of
someTask.get("project") -> retrieves the target project through the defined link Layered

REST advises layering as the technique for composing functionality and scalability from multiple agents. In programming, layering can also be used, and is best described as wrapping or composition. With programmatic layering, one object “wraps” another source object and handles all messages before sending them on to the source object. The wrapper can then add functionality to the source object and still expose the same interface to object users.

The programmatic layering principle is also seen in middleware. The HTTP interface of middleware-centric web applications consists of layers of middleware, where each middleware acts as an HTTP intermediary, adding functionality along the way. Pintura, the HTTP interface for Persevere 2.0, is completely driven by this middleware-centric approach. Pintura is basically a set of middleware providing authorization, CSRF protection, content negotiation, and more.

Perstore makes extensive use of the layering approach as well. Perstore uses a single REST-style uniform interface object store API (with get, put, delete from the W3C Indexed DB API), that is used throughout the framework. Low level data storage adapters (for MongoDB, JSON files, CouchDB, SQL, etc), can implement this API, and then store wrappers can be layered on top to add additional functionality such as caching, subscription/notification, aggregation, replication, and adaptable-indexing. For example, the we can take a store and add notification and caching to it:

var Notifying = require("store/notifying").Notifying,
    Cache = require("store/cache").Cache,
    Memory = require("store/memory").Memory,
var enhancedStore = Notifying( // add notifications
   Cache( // add caching
      dataStore, // our original source store
      Memory())); // cache in memory

Perstore’s models and facets are layers themselves, also implementing the same interface, and provide schema validation and access control. These various layers are effectively an object store form of middleware.

Code on Demand

The Code on Demand principle provides a pathway to extensibility for applications. A traditional basic resource may simply be data, but this principle advises that resources may contain code (or be code entirely), providing language level flexibility for richer content than is possible for simple static data structures. As the LISP mantra goes, “code is data and data is code”. JavaScript can easily be used in homoiconic ways, supporting this paradigm well. The object literal syntax of JavaScript (the proper superset of JSON), is well adapted to include functions holding programmatic logic. By supporting storage of code/functions in databases, one can easily define individual resource-specific logic, and resource consumers can easily interface to resource functions as means for allowing resource-specific extensibility in applications.

Summary

These seven principles (and the four sub-principles of uniform interface) collectively integrate to form the powerful architectural style known as REST. This important network architecture can be applied as a programming style in resource oriented programming, and Persevere 2.0 provides a great framework for building applications with this methodology.

Related posts:

1. Some RPC With Your JsonRestStore
2. Introducing Pintura
3. NoSQL Architecture

Dojo 1.5 introduces a new paradigm for consistent interaction with stateful objects like widgets. Stateful objects now follow a pattern of using get() and set() methods for accessing and setting named properties. Following this paradigm, to get a named property from a widget, we now use:

widget.get("checked");

And to set a named property we do:

widget.set("checked", true);

This essentially supersedes the widget system’s attr() method, providing more readable function names that don’t require branching on variable argument length, and are more easily overridden. The attr() will remain in Dojo 1.5, but now delegates to get() and set(). The use of get() and set() is recommended over attr() now.

Introducing dojo.Stateful

Dojo also introduces a new constructor for creating stateful objects with the dojo.Stateful module. This module provides simple default get() and set() implementations as well as the third complementary method, the watch(). The watch() design is based on the Mozilla’s watch function, and provides an easy mechanism for monitoring state changes. Using dojo.Stateful, we can now create objects that can be mutated, read, as well as monitored:

var obj = new dojo.Stateful({
  price: 9.99
});
 

First, we created a new Stateful object with a property price set to 9.99. Now, we can monitor that object for changes:

obj.watch("price", function(){
  console.log("price changed to " + this.get("price"));
});
 

Later if the property is changed:

obj.set("price", 6.99);
 

This will fire the watch callback function that prints out the new price.

The watch function returns an object that can later be used to discontinue a watch by calling the unwatch() method. For example:

watchHandle = obj.watch("price", callback);
…
// no longer want to watch:
watchHandle.unwatch();
  Possibilities and future directions using Stateful

While widgets in Dojo 1.5 will only provide get() and set() methods (not watch()), we are hoping to have dijit._Widget subclass dojo.Stateful in 1.6 providing a convenient and consistent approach to monitoring widget property changes as well as other objects. We will then be able to use the watch method on any widget.

We are also considering a new Dojo Data design where data items would also follow this interface, allowing even broader consistency in a uniform interface for accessing and monitoring stateful objects, whether they are items from a data store, widgets, or other user objects that extend dojo.Stateful.

One of the key motivations for this design is that it would ultimately enable building functionally reactive “live” templating engines or data-bindings on top of Dojo. The concepts of functional reactive design (or data-bindings) have been demonstrated in other projects like Flex, Flapjax, and Fin. Eventually we will be able to write templates like:

<div>${obj.get("foo")}</div>
 

And the templating engine could determine that it needs to call the watch method to monitor object obj and it could automatically update the rendered view when the object changes. We can also create data bindings between different objects (changes in an object would get automatically synchronized to another object). Projects like Flapjax and Fin that have used this approach are clever but have very limited use because they require an interface that doesn’t match with common existing components. By following a consistent pattern throughout Dojo, we can enable this type of live templating in a way that really is useful. Again, these are future ideas, not yet implemented, but the consistent interface makes this very doable.

For now you can begin to use get() and set() methods on all widgets in Dojo 1.5, and create your own watchable objects with dojo.Stateful that can easily be monitored for state changes.

Related posts:

1. Queued: New Interaction Tricks for the Old Netflix Dog
2. Deterministic Client/Server Interaction
3. HTML Widget Prototyping with the Dojo Toolkit

The dojo.Deferred module has long been a central component of Dojo, providing a powerful interface for asynchronous operations like HTTP requests. Dojo’s Deferreds are a form of promises, providing a separation of concerns between the mechanics of calling a function and the interface for interacting with the eventual asynchronous future result. Passing callbacks directly to functions tightly couple the interface with asynchronous semantics, but using Dojo’s Deferreds (promises) keeps the concerns separate for robust asynchronous code. Furthermore promises can greatly simplify interaction, allowing one to easily get the eventual result of an operation whether it has already finished, or if it is yet to be finished. While the Deferred module has existed in Dojo for some time, version 1.5 introduces some valuable enhancements.

One of the primary new methods on dojo.Deferred is the the new then() method, which can be used in place of the addCallback() or addBoth() methods with some important advantages that we will investigate. First a quick sample:

dojo.xhrGet({url:…}).then(function(response){
   someElement.innerHTML = response;
});
 

Before looking more closely at the new API, the primary new concept in Deferred 1.5 is immutable-once-resolved promises. The advantage of immutable-once-resolved promises can be explained with analogy to synchronous coding. A key characteristic of JavaScript is pass-by-value call semantics. If we execute this function:

function foo(){
  var a = 2;
  var b = bar(a);
  alert(a);
}
 

The call to bar passed the value of a, not a direct memory reference to the variable’s memory slot. Therefore, we can be assured that there is nothing that the bar function can do that will alter the value of a (now obviously if a referred to an object, that object could be mutated, but a’s reference to the object could not be changed, just as its reference to an immutable primitive can’t be changed). The bar function only has control of what is returned to b, and we can safely assume that the alert will be passed a value of 2. This greatly simplifies programming because it is a more functional, safe approach, one does not need to worry as much about the side-effects of called functions.

Likewise, immutable-once-resolved promises act the same way. If one passes a promise to another function, one shouldn’t need to worry about the promise being altered. Two separate functions (such as the caller and callee function) should be able to utilize the same promise without fear of affecting each other inadvertently. We should be capable of using the values generated from functions in various ways. With normal sync functions, a value can be returned, and we can use that value in a variety of ways. We can set it to a variable, use it to make computations, pass it to other functions, etc. By encapsulating the eventual completion and result of an asynchronous function’s execution in a promise, we can do the same thing with promises. We can pass them to other functions, we can register computations to be done on the values, etc. The recipients of promises can safely use the eventual value (registering to receive it) without cause side-effects to each other. Unfortunately, the old dojo.Deferred objects were mutating, the return value of an addCallback changed the value of the deferred. Now, the API provides access to immutable promises that can be safely passed around.

then()

The rewrite of dojo.Deferred provides a few new access points while preserving backwards compatibility with the previous dojo.Deferred versions. The first new method, as mentioned before, is the then() function. This method is based on this CommonJS promise proposal, and discussed in my article on the CommonJS platform. Calling then() is similar to dojo.Deferred’s addCallback or addBoth function. The first argument is the callback, and the second argument is the error handler. Both are optional. The main difference between then() and addCallback or addBoth is that it is side-effect free. The value returned from then() is a new promise, which can be chained but not altered. Consequently, we can write:

var responsePromise = dojo.xhrGet({url:…});
showStatus(responsePromise);
responsePromise.then(function(response){
   return dojo.fromJson(response).content;
}).then(function(content){
   someElement.innerHTML = content;
});
 

One key thing to note is that even though we passed the result of the XHR operation to the function that parses the JSON and sets the content into some HTML, we did not modify responsePromise. It still resolves to the text returned from the XHR request, and can safely be passed to other functions (like showStatus() in this example) without interference.

promise property

Part of ensuring that promises are unaffected by other code is by only exposing a minimal API on the promises. Deferred objects (result of calling new dojo.Deferred()) offer callback and errback functions for resolving, but the objects returned by then() are purely consumer objects and therefore only provide a then() function. However, it is possible to directly use a consumer-only promise object from Deferred with the promise property without having to go through a then() call. For example:

var deferred = new dojo.Deferred();
foo(deferred.promise);
function foo(promise){
   promise.callback(); // this will fail, pure promises can’t be directly resolved
   promise.then(function(){
     // this will succeed, pure promise’s provide listening
   });
}
deferred.callback(3); // this will succeed, the deferred object provides resolution methods
  dojo.when

Perhaps the most useful new function is the addition of dojo.when(). The dojo.when() function can also be used as a static function to register a callback for the eventual completion of a promise or deferred (with chaining support and without side-effect, like then()), but dojo.when() can be used consistently on both normal non-promise values and promises. It therefore provides a powerful form of synchronous/asynchronous normalization. A simple example of using dojo.when, based on our initial example:

dojo.when(dojo.xhrGet({url:…}), function(response){
   someElement.innerHTML = response;
});
 

This doesn’t look much better than any other way of calling dojo.xhrGet, but lets look at another example to see the advantage of using dojo.when(). Suppose we write a function that finds the last item in an array and prints it. A simple synchronous implementation might look like:

function printLast(items){
   console.log(items[items.length – 1]);
}
 

Now suppose we wanted this function to work on promises, we could of course alter the function to do so:

function printLastAsync(items){
   items.then(function(items){
     console.log(items[items.length – 1]);
   });
}
 

However, what if we want our function to work properly with both asynchronous promises as well standard synchronous values? With dojo.when() we can write something like:

function printLast(items){
   dojo.when(items, function(items){
     console.log(items[items.length – 1]);
   });
}
 

Now our findLast can be used uniformly with both normal synchronous value and promise/deferred based values. For example:

printLast([1,2,3])
> 3
printLast(dojo.xhrGet({handleAs:"json",url:…}));
> (last value returned from server)
 

And remember that dojo.when() returns a new promise like then(), so we can still chain promise actions together. For example:

dojo.when(dojo.when(dojo.xhrGet({handleAs:"json",url:…}),
    function(data){
       return process(data);
    }),
  function(processedData){
    return doSomething(processedData);
  });
 

Dojo’s new Deferred module provides backwards compatibility with previous versions, still providing addCallback, addErrback, addBoth, and addCallbacks methods, but these will probably be deprecated in a future release (chaining will still be possible though then() and dojo.when()).

The new Deferred module provides a new set of easy-to-use methods facilitating robust, manageable, asynchronous code. The new module provides promises with minimal API, preserving the object capability model’s principal of least authority, functional side-effect free data flow, and easy synchronous/asynchronous normalization.

Related posts:

1. Creating Asynchronous Tests with the Dojo Objective Harness
2. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
3. Queued: Demystifying Deferreds

There’s really no obvious “winner” in the Apple vs. Adobe spat. While both sides make some good points, they miss the mark on many details as well.

First, Steve, we appreciate your bold Thoughts on Flash, but a few comments:

• Steve, Apple did not invent WebKit… you rewrote Konqueror and KHTML. Give credit where credit is due.
• H.264 is not truly open.
• Your app store review process is anything but open.
• Tools to generate source code that can be deployed to the app store should not be discouraged. Either the app passes or fails in a quality review process that you need to open.
• Flash exists because while the open web is great, it’s not perfect and there are still things that are easier to do with Flash’s development tools (this from a major open web supporter).
• The lack of true APIs for native features on the phone for web developers has been holding us back for three years. Camera, GPS, geolocation, native graphics acceleration, address book, etc. iPhone OS 4.0 helps a bit, but it’s way overdue.
• Give open web developers access to the things they need to make the native app development process less important.

Unfortunately, Adobe’s CEO did himself no favors in his Interview with the Wall Street Journal:

• Developers already have multiple workflows.
• Don’t make excuses and pass the buck on Flash’s performance problems on Mac OSX. Where’s the proof?
• One set of developer tools but support for all platforms is convenient for Adobe or Microsoft. Real developers mix and match tools rather than locking into a single vendor stack.
• Your tool stack isn’t open, so you really can’t complain. Unless you offer a decompile option, it’s hard to judge the quality of your generated application code. Your history of quality generated code leaves me skeptical.
• Flash being an open spec. is a misuse of the term open. Where’s the competing plug-in?
• The Adobe stack should innovate to support the open web as another platform, per your logic, and my advice. See PhoneGap as a great example of working more openly to solve similar problems.
• Overall, your view is that everything you deploy to should be open, except for your tool stack. We disagree.

Related posts:

1. Adobe Engages SitePen to Make Dojo Toolkit Compatible with Adobe AIR
2. SitePen Announces Updates to the Dojo Toolkit to Enable Adobe AIR Compatibility
3. Thoughts on XML, Ajax, and UI Development