The short answer: Yes, if it changes its strategy to one that embraces and augments the open web ecosystem, rather than continuing down the path of trying to compete with or replace it.

With the recent anti-Flash, pro-HTML5 buzz caused by the iPad and sites like YouTube offering HTML5-enabled video alternatives, I thought it would be useful to share my thoughts on the opportunities and struggles Adobe faces with the Flash platform. Given my propensity as a strong open-source advocate, it may seem odd that I bother to discuss this, but it’s an interesting thought experiment for me on where Flash still excels compared to the open web, and how it can leverage that to thrive as part of the world going forward.

Developer Tools

Flash plus Illustrator, Catalyst, Flex Builder and the rest of the Adobe CS collection provide a compelling suite of visual tools for building web sites and applications. While there are a number of attempts at building studio tools using open web technologies, they all tend to be component-driven attempts to clone PowerBuilder or Interface Builder for the web. Flash clearly leads the market in animations and the creation of interfaces that are more movie-like in their experience. For example, while it’s possible to create the famous Sports Illustrated tablet UI demo entirely with HTML5 technologies, today it is much easier to build something this visually rich with Flash. Though ironically, it won’t run on the iPad without Flash compiling to a native iPhone app.

Rendering Engine

The Flash player is really just a replacement rendering engine that works on most platforms that don’t begin with a small i. To date, it’s unfortunately been pretty crash-prone on Mac OS X and has trailed behind on Linux. But it has succeeded because it was a useful hack for improving upon the limitations of the world’s worst browsers. On Windows, it is stable and is a significant improvement to Internet Explorer’s rendering engine. Also, somewhat surprising is that in our performance tests, Flash is not actually any faster at drawing or animating objects than Firefox, Safari, or Chrome, meaning that we do not see them leveraging native hardware 2-D and 3-D acceleration APIs, which are starting to arrive in the next version of the major browsers. Which means that the edge Flash player once enjoyed with its rendering engine is no more.

Revision Cycle

Google Chrome Frame has the advantage now in being able to upgrade the host browser much like Flash can. And really, that’s been Flash’s big advantage historically: it could push updates to the world faster than browser vendors. But in the past few years, versions of Flash have actually been released slower than non-Microsoft browsers, which now include reliable auto-update mechanisms. Being able to patch, hack, and fix the web was Flash’s big advantage, and that edge is gone.

Compilation

It’s fairly interesting that Adobe has made it possible to compile Flash apps into native apps for distribution on the Apple App Store. This idea was likely inspired by PhoneGap and Appcelerator’s Titanium. But it raises an interesting question for me: could Flash compile into native HTML5-compliant applications? That would potentially decouple the benefits of the developer tools from the limitations of the player.

Embracing the Open Web

Adobe to date has dabbled in the world of open source, but Google Chrome Frame is its first real competition in the player/plug-in space, because it has two huge advantages (ignoring its disadvantages for now): it’s open-source, and it builds and extends open web protocols like HTML5, JavaScript, SVG, and more. In my opinion, Flex should have extended HTML4 or XHTML instead of being a complete rewrite from the ground up. This would allow any HTML developer to easily jump into it and then take advantage of Flash’s extension. Imagine Flex apps being unobtrusive upgrades to HTML pages, if the flash player is available! Or, imagine including a Dojo HTML fragment for a grid, using a standardized flash grid component, installed as a native component as part of the player, if that was the best rendering engine available!

Pushing the Open Web into Design Tools

As someone that can’t draw a circle in Photoshop or Illustrator, I would love for everything in a PSD or AI file to be represented by HTML/XML and CSS. Developers loathe having to wait for a designer to change text in a graphics file or modify colors in a background image. As good as the open web gets at moving more and more image functionality to native styling, there will always be concepts that cannot easily be represented tersely enough in the CSS layer. The authoring tools layer is likely the best place to make such modifications. Providing a simple tool or mechanism for developer’s to update or modify files created by designers would be useful and intriguing.

License Pricing

Given the dramatic reduction in costs of computers and software, Adobe is significantly more expensive relative to the cost of other products than it was when it started its CS pricing model. To fix this problem, Adobe has been experimenting with a subscription pricing model in Australia that allows users to pay each month to use the software. That’s more inline with expectations today as it doesn’t require a huge up-front cost. Also, Adobe could start to consider licensing affordable Flash components that are hosted on a CDN, or a number of other fine-grained payment models.

Conclusions

The open web has quickly closed the gap on Adobe’s advantages over the past two years, but there are many new opportunities for Adobe to exist and thrive in the open web ecosystem, and take advantage of the new possibilities this creates, if it can adapt to the completely new world of today’s modern web platforms.

Related posts:

1. Flash, Silverlight and the Open Web
2. Porting Dojo Methods to Flash – Part 1 of 3
3. Lets. Push. Things. Forward.

The object capability model is an approach to security that utilizes object references as the primary means of controlling access and providing authority. Capability-based security follows the principle of using unforgeable capabilities to provide access to resources. Object capability builds on capability-based security by leveraging object references as the primary representation of capabilities, which are naturally unforgeable in memory safe languages. Object capability based security is an elegant approach to security because the goals of object-oriented principles of encapsulation and information hiding are realized in virtually the same exact manner as the principle of least authority that is at the heart of object capability security. This type of security is extremely flexible and customizable since it is based on object-oriented design. Plus, writing good code naturally leads to secure code, security can be designed with object encapsulation hand-in-hand.

In the object capability model, the authority to act on an object is permitted when one attains a reference to that object. Primarily, object references are gained by creating a new object (parenthood) or being passed an object reference (introduction). This means that a separate access control system is not needed, the passing of object references to functions and other objects provides the minimal, but sufficient, means and authority to carry out operations.

Object capability provides a fundamentally superior approach to alternates that have led to a vast number of the security flaws that have plagued operating systems and the web. Operating systems have long been unable to provide any true protection against malicious code. This is because access is defined by the current user’s access levels (ACL approach) instead of through capabilities being appropriately passed to programs. In object capability terminology, providing all executing code (for a given user or other context) with the same level of access is known as “ambient authority”. This has essentially enabled the whole class of exploits known as viruses. On the web, the problem is equally severe. Perhaps the most broad security threat to web applications is CSRF. This is also a direct result of how applications use cookies for ambient authority. Cookies are uniformly attached to all requests to a domain, regardless of who or what triggered the request. This allows malicious sites to trigger requests under the ambient authority of a logged in user.

One of the primary goals in the evolution of JavaScript is improved security. A large amount of the revisions in EcmaScript 5, and proposed ideas in the next version, are the result of object capability model research and how it can be applied to JavaScript to avoid ambient authority exploits. In particular, ES5’s strict mode is specifically designed to enable a new class of sandboxing techniques for executing untrusted code. Caja, FBJS, Jacaranda, dojox.secure, and ADsafe are all technologies that employ the object capability model to safely execute suspicious code.

Challenges: Doing something useful

While the object capability model provides a well-grounded and principled approach to security, it is not necessarily obvious, nor intuitive how to build real applications with this model. Application requirements are not usually defined in terms of object references, but rather in terms of who can do what, and how access is controlled. Since it is easy to just default to ACL-style approaches, how do we translate such requirements into object capabilities instead? We will take a look at how Persevere 2.0’s Pintura and Perstore provide a framework for doing just this.

Object capability model advocates tend to be cautious around user-authentication schemes, since they often translate into ambient authority. However, that does not need to be the case, and the reality is that the vast majority of applications will indeed utilize user authentication to control access to resources. This does invalidate the object capability approach. Pintura makes it very easy to use user authentication. Pintura employs a middleware module that reads user credentials, checks them against a data store for credentials (which can be configured to be any data store), and then calls a function that can be implemented by the application to return a set of “capabilities”, which are a set of references to data interfaces that they can access. These capabilities are passed to the entry REST or RPC handler, but one does not need to rely on ambient authority, code can then explicitly pass appropriate capabilities to any other functions that are executed.

Let’s take a look at the Pintura’s example application to see this in action. The access.js module defines a getAllowedFacets function on the security object that implements the logic of determining which capabilities are designated for each user. In this example, we define administrative capabilities, authenticated user capabilities for editing wiki pages, and the capabilities of an unauthenticated user which include the ability to view pages, authenticate and register/sign-up. The access.js module is loaded by the entry module, app.js, and makes it very easy to define the capabilities of the users.

Facets

Looking at the access.js module you will see that most of the capabilities refer to facets. Facets are an integral part of the Pintura/Perstore (Perstore is the persistence/data modeling framework used by Pintura) security system, and are introduced in the getting started with Pintura article. One of the primary challenges of using the object capability model is that a simple reference to an object often falls far short in terms of access level granularity. If you don’t have a reference to an object, you can’t do anything with it, and if you do have a reference, you can do anything to that object. In reality, applications often need much more fine-grained access levels like read-only access, create-only access, ability to delete from collections and even ability to define these levels at individual property levels instead of just the object level. With the object capability model, we can achieve this level of granularity by creating proxy objects that wrap an object. The proxy object can then be referenced and the proxy can handle attempts to modify and/or read data from the source object and determine which operations to allow, and which to deny.

Building these proxy objects can be onerous, but Perstore makes this easy, providing a complete, robust system for controlling access to store model persistent objects with “facets”. We use the term “facet” in Perstore because proxy is an overloaded term, and facets also indicates the secondary purpose of providing alternate views and interfaces of data. The facets in Perstore automate most of the proxying process, and allow a great deal of flexibility in defining access. Facets actually are very analogous to the concept of a proxy server, and follows the layering principle of REST. Like proxies, facets add functionality (attenuation of access), but presents the same interface to the next level up.

As mentioned in the getting started guide, Perstore provides two facet constructors: Restrictive and Permissive. The Permissive constructor gives full access to the underlying objects by default, and you must explicitly define functions or schema constraints to restrict functionality. The Restrictive constructor restricts access to the underlying objects to read-only by default, and you must override functions to permit access. Perstore facets utilize JSON schemas for further defining the facets. The same familiar structure that is used for model object validation is used for facet definitions. Therefore you can write:

var Permissive = require("facet").Permissive;
var SomeProductFacet = new Permissive(Product, {
  properties:{
    productCode: {readonly: true},
    managerNotes: {blocked: true}
  },
  "delete": function(){
    throw new Error("Can not delete");
  }
}
 

In this example, we have defined a facet for accessing the Product model and its instances. All instances that are retrieved through SomeProductFacet will be writable objects except that productCode will be read-only and managerNotes will be completely blocked (can not read or write to it). Furthermore, this facet defines that any attempt to delete products through SomeProductFacet will be denied.

Facets are the primary mechanism for authorization in Pintura. This fulfills the essential role in a web application of controlling access of clients. However, with the firm basis on object capability model, Perstore’s faceting will be appropriate for the eventual sandboxing of untrusted code. While JavaScript sandboxing is still somewhat of a research and development level topic, and is more of a platform level concern, (Perstore/Pintura do not provide any sandboxing themselves) such mechanisms are in the works, and should work perfectly with facets to provide fine grained access to persisted data for various modules.

Facets are a powerful concept with a multitude of uses. We have seen how to utilize them for access control, but facets can also play an important role in providing different “views” of data. One can utilize facets for applying different query filters, providing different locale-specific data to achieve internationalization, or revealing various levels of object details based on application views. Out of the box, facets are based on authentication information, but facets (and even layers of facets) can also be selected based on locale, user agent, or custom headers.

Conclusion

Persevere 2.0’s new object capability model with facets provides a flexible security framework based on the best principles of security research, avoiding the pitfalls of ambient authority and giving you the tools to quickly and efficiently build solid, secure applications.

Related posts:

1. Using the Persistent Object Model in Persevere
2. Getting Started With Pintura
3. Client/Server Model on the Web

There is so much existing information about the Dojo Toolkit that it can be challenging to know where to begin. The following is a Dojo curriculum (I use this term loosely) highlighting community resources and a logical path for self-learning the foundational parts of Dojo.  If you understand the purpose of a variable and function, or you are new to Dojo, then this is for you.

I also hope that this curriculum will help the Dojo community fill in any gaps for learning the Dojo basics. I’ve started with a focus on Dojo Base and Dojo Core. The other parts of Dojo; Dijit, DojoX, and Util (how Dojo is organized) really deserve their own curriculum and are not the focus of this track.

Curriculum: Prerequisites:

• (book) JavaScript: The Definitive Guide / Chapters 1 to 20
• (book) Object-Oriented JavaScript: Create scalable, reusable high-quality JavaScript applications and libraries
• (book) Professional JavaScript for Web Developers / Chapters 4 to 7
• (6 tutorials/articles) Firebug Tutorials
• (video) The Douglas Crockford JavaScript Master Class Optional! (some of this info can be found at the YUI Theatre)

Dojo 101: The Overview

• (video) Building High Performance Web Applications with the Dojo Toolkit
• (video) Dojo: Patterns for Lovers of JavaScript
• (9 tutorials/articles) SitePen Dojo Quick Start Guide
• (article) Introducing The Dojo Toolkit
• (slide-deck) Various Dojo Presentations
• (slide-deck) Dojo – JavaScript’s Swiss Army Knife
• (article) Dojo Goodness, Part 1

Dojo 201: Beyond the Basics

• (book) Getting StartED with Dojo / Chapters 3 to 7
• (article) Understanding dojo.declare, dojo.require, and dojo.provide
• (article) Inject Dojo
• (article) Dojo Module Packaging and Loading
• (article) Dojo Goodness, Part 3 (Animation Station)
• (article) Dojo Goodness, Part 4 (Easy AJAX)
• (article) Dojo Goodness, Part 5 (Simple Network I/O)
• (article) Exploring URLs client-side
• (resource) 1.4.0 cheatsheet
• (resource) Dojo API reference
• (resource) Dojo Source (Dojo Directory) online
• (article) Debunking Dojo Toolkit Myths
• (article) forEach goodness

Dojo 301: Dojo Base & Core In-depth

• (book) Dojo: The Definitive Guide / Chapter 1 to 10
• (book) Getting StartED with Dojo / Appendix  C
• (article) Dojo Goodness, Part 7 (Injecting Dojo After Page Load)
• (article) Dojo Goodness, Part 8 (JSONified Cookies)
• (article) Dojo Build System
• (book) Mastering Dojo: JavaScript and Ajax Tools for Great Web Experiences / Chapters 3 and 8 to 11
• (article) ShrinkSafe
• (article) A Dojo Plugin Pattern
• (article) More on Dojo “Plugins”
• (article) Creating Your Own $
• (article) Jammastergoat: dojo.hitch
• (article) dojo.hitch and scope: more romance

Dojo 401: Example Dojo Applications & Demos

• (demo application) Stocker
• (demo application) twitterverse
• (demos) Dojo Feature Explorer

It is likely either during or after the process of working through this curriculum that you are going to have questions.  When this occurs consider leveraging the following community resources:

• Dojo Community/Forums
• #dojo chat room on the irc server irc.freenode.net
• Mailing Lists

If you find that a self-guided tour or community support is inadequate, I would suggest an expert-guided Dojo workshop or hands-on assistance through Dojo support from SitePen.

Related posts:

1. The Best Things in Life are Free
2. Introducing the Dojo Toolkit 1.1
3. Now Available: Commercial Support for the Dojo Toolkit, DWR, and Cometd

Pintura is a REST-style web framework that provides a comprehensive solution for Ajax-based thin-server applications. However, Pintura has a very modular design, and many of the modules in Pintura are extremely useful as standalone JavaScript/CommonJS components that can be used with Node, Narwhal and other projects. Let’s look at the JSGI middleware modules. JSGI middleware modules are designed to be used in a JSGI HTTP request handling stack, adding functionality to a web application. An example of using a JSGI middleware:

exports.app = MiddleWare( // wrapped my application with middleware
  function(request){
     … my JSGI application …
  });

All of Pintura’s middleware modules are JSGI 0.3 compliant and fully support promise-based asynchronous request/response handling.

• jsgi/csrf – The csrf middleware module will detect if a request may have been generated by a web page from a different server origin. If it is possible that the request came from a different origin, the crossSiteForgeable property of the request is set to true. Generally, such requests should be treated with suspicion. The request may still include cookies that were used for authentication and authorization, but any authorization module should be aware that the requesting agent may not be authorized to actually utilize these cookies for authorization.
• jsgi/xsite – While the csrf module protects against unauthorized cross-site requests, the xsite helps enable authorized cross-site requests. The xsite module actually consists of three middleware components:
  • JsonP – This adds support for JSONP requests. This will wrap requests with the proper callback in order for the client to receive the response.
  • WindowName – This adds support for window.name requests. This will wrap requests with the proper HTML in order for the client to read the response from an iframe’s name property.
  • CrossSiteXhr – This adds support for native cross-site XMLHttpRequest (now available in modern browsers). This will add the appropriate HTTP headers to allow clients to read the responses.
  • CrossSite – This combines all three of the above middleware to maximize the ability for authorized client’s to interact with the server from other page origins.
• jsgi/http-params – This allows for HTTP headers to be emulated through query parameters. This is useful for cross-site requests and hyperlinks where HTTP headers can not be directly specified, but applications expect to read HTTP headers.
• jsgi/conditional – This module implements handling HTTP conditional requests, checking If-Match and If-None-Match against ETag headers, and checking If-Modified-Since and If-Unmodified-Since against Last-Modified headers, modifying and returning the appropriate HTTP responses when conditions indicate a 304 Not Modified (which improves performance) and 412 Precondition Failed (to avoid conflicting changes).
• errors + jsgi/error – jsgi/error is a JSGI middleware module that catches errors and converts them into appropriate HTTP response codes. There are a variety of error constructors defined in the errors module (which is actually in the Perstore project) that correspond to the different HTTP status codes. For example: var MethodNotAllowedError = require("errors").MethodNotAllowedError,
      ErrorHandler = require("./jsgi/error").ErrorHandler;

  exports.app = ErrorHandler(function(request){
    // my JSGI app,
    switch(request.method){
       case "GET": …
       default: throw new MethodNotAllowedError();
    }
  });
   

  This would allow you to throw an error, break out of the current stack, be caught by the ErrorHandler, and be converted to a 405 response code.

• jsgi/media – This module acts as “cross-over” middleware and performs HTTP content negotiation to choose the best content type (AKA media type). This makes it easy to have a RESTful selection of different representations for each data resource. This middleware wraps an application stack that is allowed to return JavaScript objects, arrays, and any other data values for a response body. Then, this middleware serializes the data to the best possible media type raw format. The media middleware module and the media type handlers also include support for deserializing raw binary/text data from requests into JavaScript data.
  Pintura has a collection of different media type handlers that for serializing and deserializing data to and from different formats. Some of the important media handlers include:
  • json – JSON media handler
  • javascript – Similar to the JSON media handler, but will serialize to additional JavaScript specific types such as dates, NaN, functions, and other types that do not exist in JSON.
  • multipart-form-data and url-encoded – Used for parsing form data.

Pintura includes several more middleware modules that are designed to work downstream of the media module, expecting deserialized input and expecting JavaScript objects and arrays for the body (rather than normal JSGI body values). These are designed to interact with the object store API used by Perstore. These are generally more Pintura specific, but may still be used with proper integration:

• jsgi/faceted – This finds the appropriate facets for the authenticated user for accessing the models/stores
• jsgi/metadata – This sets and retrieves metadata from objects, translating from headers.
• jsgi/auth – This performs authentication against a provided object store, and saves authentication tokens in a cookie for maintaining authentication.
• jsgi/rest-store – This is the core Pintura application for delegating requests to the store facets and models.
• jsgi/transactional (this one lives in Perstore) – This wraps each request in a transaction.

With these modular middleware components, one can easily cherry-pick from Pintura in building custom stacks, or rearrange the middleware stack in Pintura applications. The Pintura framework itself is essentially a middleware stack. The Pintura middleware set is designed for REST-style standards-compliant web applications, and can be useful for any server -side JavaScript REST application regardless of whether or not you are using the entire Pintura framework.

Related posts:

1. Getting Started With Pintura
2. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
3. Introducing Pintura

General Interface (GI) recently joined the Dojo Foundation as part of a transition to a true comprehensive 100-point open source project. TIBCO open sourced GI a number of years ago, but now GI can enjoy the benefits of running under the Dojo Foundation and being integrated with the infrastructure for open SVN, bug tracking, and more. Not only is GI is part of the foundation, but GI now includes significant integration with the Dojo Toolkit. The powerful GI builder, a web-based visual IDE for building client side web applications, now is capable of utilizing Dojo widgets as well as the GI set of widgets.

The GI builder provides support for full drag and drop of Dojo widgets into the application canvas. In addition, the GI builder’s property and event editors can be used with Dojo widgets, allowing for a visual configuration of widgets with full documentation available in tooltips. This requires extensive integration with the builder. This was made possible with a new runtime metadata retrieval module available at dojox.lang.docs. The metadata module provides runtime access to the extensive class description and metadata information that is available through Dojo’s API documentation tool.

Class documentation and metadata information can now be made available at runtime through JSON schema structures on the class. To utilize the metadata module requires access to the API data files. These are included in the source and SVN distributions of Dojo (in the utility directory). Then the metadata module can be started:

dojo.require("dojox.lang.docs");
dojox.lang.docs.init();
 

The API documentation will be loaded, and Dojo classes will have their metadata available. To get the description of a class, you can access the description property. For example:

dijit.ColorPalette.description -> a description of dijit.ColorPalette
 

To access information about the properties that are available on the instances of classes, we can access the properties property. For example, to find the description of the defaultTimeout property of ColorPalette instances, we can access:

dijit.ColorPalette.properties.defaultTimeout.description
 

And likewise, to discover the expected type of the defaultTimeout property:

dijit.ColorPalette.properties.defaultTimeout.type -> "number"
 

We can also retrieve information about the methods that are available for classes. The method definition follows the JSON Schema Interfaces (JSI) structure. For example, to get a list of expected parameters for the onChange method for ColorPalette:

dijit.ColorPalette.methods.onChange.parameters -> [{type:"string", name:"color"}]
 

And the definition of the return value would be available:

dijit.ColorPalette.methods.onChange.returns -> info on the return value
 

We can also look at the class hierarchy, and see what the superclass is for a class:

dijit.ColorPalette["extends"] -> dijit._Widget
 

The GI builder leverages the metadata module to provide access to the type information and descriptions of properties and methods (events) in the visual interface. The comprehensive and canonical information available in the API docs is fully exposed through the builder, and greatly enhances the integration level of the builder with GI for a comprehensive visual development experience.

Related posts:

1. TIBCO Contributes General Interface to the Open Source Community
2. Getting Started with Persevere Using Dojo
3. Code Finished for Dojo Offline’s Default User Interface

CommonJS Utils is a collection of general purpose CommonJS-compliant modules. These modules can be used on Narwhal, Node, and other CommonJS platforms. The modules include:

• json-schema.js – This is a JSON Schema validator module. It can be used to validate data structures using JSON schema definitions. For example: var validate = require("json-schema").validate;
  var data = {name: "Test"};
  var schema = {
    properties: {
       name: {type: "string"},
       age: {type: "number"}
    }
  };
  var validation = validate(data, schema);
  validation.valid -> false
  validation.errors -> indicates that age was not provided
  data.age = 30;
  var validation = validate(data, schema);
  validation.valid -> true
   

  This module also supports using standard native constructors as type definitions. The schema above could be written
  more briefly:

  var schema = {
    properties: {
       name: String,
       age: Number
    }
  };
   
• json-ext.js – This module provides several JavaScript-valid extensions to a JSON for serialization and parsing of richer data structures. The json-ext.js module supports dates, NaN, Infinity, and undefined using standard JavaScript syntax. For example: var JSONExt = require("json-ext");
  var myObj = {now: new Date(), unreals: [NaN, Infinity, -Infinity], notDefined: undefined}
  asString = JSONExt.stringify(myObj);
  // asString -> {now:(new Date(1265260623866)), unreals:[NaN, Infinity, -Infinity], notDefined:undefined}
  myObj = JSONExt.parse(asString);
   
• observe.js – The observe module event provides listening capabilities that normalize subscribing to events for CommonJS observable objects, Node.js EventEmitters, and plain JavaScript property functions. For those familiar with Dojo, the observe function is the Node/CommonJS equivalent of dojo.connect: var observe = require("observe").observe;
  // listen to an "observable" object:
  observe(someObservableObject, "someEvent", function(event){
    … handle event
  });

  // listen to a Node request
  var listener = observe(request, "body", function(data){…});

  // we can later stop listening with:
  listener.dismiss();

  // plain JavaScript objects can also serve as event sources
  myObj = {
    sayHi: function(){
      print("hi");
    }
  };
  // listen for sayHi to be called
  observe(myObj, "sayHi", function(){
    print("Observed sayHi being called");
  });

  myObj.sayHi();
  // this will print out "hi" and then "Observed sayHi being called"
   

• lazy-array.js – One can utilize lazy array-like structures with the lazy-array module. Normally with standard JavaScript arrays, one needs to populate all the slots in the array before it can utilized by a consumer. However, for scalability of large collections of data, it is often beneficial to only evaluate or load data as needed. This can avoid excessive memory consumption (storing everything in collection at once), and avoid unnecessary evaluations or loading operations if only a subset of a collection is actually used. With the lazy-array module, one can easily implement a lazy array-like collection that is an instanceof Array, and provides all of the functions normally available on Arrays. One simply needs to implement the some() function and provide a length property. For example: var LazyArray = require("lazy-array").LazyArray;
  fibonaccis = LazyArray({
    length: 10000000, // effectively Infinity, but that’s not a valid array length in JS
    some: function(callback){
      var a = 0, b = 1;
      do{
        var next = a + b; a = b; b = next;
        var done = callback(next);
      }while(!done);
    }
  });

  // now we can print the square of first 10:
  var i = 0;
  fibonaccis.map(function(value){
    // lazy-array will use lazy evaluation of the map function,
    // so we can safely do this without an infinite loop
    return value*value; // compute the square of each
  }).some(function(value){
    if(i++ == 10){ // after 10, signal we are done
      return true;
    }
    print(value);
  });
   

  Lazy arrays are not true arrays in that we can’t use the [index] operator to access items. However, we can grab items by index using the get export from lazy-array:

  var get = require("lazy-array").get;
  get(fibonaccis, 5) -> 13
   

  Furthermore, lazy arrays also support asynchronously evaluated lazy arrays. This effectively becomes a powerful generic data asynchronous streaming mechanism.

  var LazyArray = require("lazy-array").LazyArray;
  var when = require("promise").defer;
  lazyFileParts = LazyArray({
    length: file.size,
    some: function(callback){
      var deferred = defer();
      fileInput.addListener("data", function(data){
        callback(data);
      });
      fileInput.addListener("complete", deferred.resolve);
    }
  });
  // now we can use it
  var defer = require("promise").when;
  when(lazyFileParts.forEach(function(part){
    // called for each part
  }), function(){
    // all done
  });
   
• jsgi-client.js – The jsgi-client module is an HTTP client module that is based on the CommonJS JSGI standard. The JSGI standard was originally developed for HTTP servers, but this module basically follows the same object structure and design, but in reverse. It also provides some alternate options to make it easier to create HTTP requests. This module relies on the browserjs package and Narwhal’s promise module. For example: var request = require("jsgi-client").request;
  request({
    uri: "http://example.com/path",
    method: "POST",
    body: ‘{"foo":"bar"}’,
    headers: {
      "Content-Type": "application/json"
    }
  }).then(function(response){
    response.body // -> the response body
    response.headers // -> the response headers
    response.status // -> the response status
  });
   
• settings.js – The settings module provides access to local configuration information. This configuration information should be stored in a local.json file in your application. local.json:
  {
    "database": {
      "connection":"jdbc:mysql://localhost/…"
      "type": "mysql"
    },
   …
  }
   

  And then we can access this local information:

  var dbConnectionInfo = require("settings").database.connection;
   
• extend-error.js – The extend-error module provides a means for creating custom error constructors. Creating custom error constructors can be tricky, particularly if you want to preserve the normal error properties with name, message, stack trace information, and standard toString() operations. This module makes it simple: var ErrorConstructor = require("extend-error").ErrorConstructor;
  MyCustomError = ErrorConstructor("MyCustomError");
  // now we can use our new error constructor:
  try{
    throw new MyCustomError("error message");
  }
  catch(e){
    print(e); // prints: "MyCustomError: error message"
  }
   
• smtp.js – This module provides the ability to send email through SMTP. This module requires Rhino. For example: var send = require("smpt").send;
  send({
    to: "recipient@example.com",
    subject: "Example",
    message: "This is an example email"
  });
   

  And then your local.json file would need to have the appropriate SMTP settings:

  {
    "mail": {
      "host":"mail.site.com",
      "defaultFrom": "app@site.com"
    }
  }
   
• xml-rpc.js – The XML RPC module provides basic method invocation through XML RPC. This module is dependent on the browserjs package. For example: var XmlRpc = require("xml-rpc").XmlRpc;
  var endPoint = XmlRpc("http://somesite.com/rpc-target");
  var result = endPoint("someMethod", ["arg1", 2]);

The CommonJS utilities package provides these various general purpose utilities. Some of these modules may eventually warrant their own package/project. Any one of these modules can be downloaded from the CommonJS Utils project repository, or you can download the entire CommonJS Utils package.

Related posts:

1. CommonJS/JSGI: The Emerging JavaScript Application Server Platform
2. Pintura JSGI Modules
3. Dojo JSON-RPC + Java

Dojo 1.4 sports a fantastic tree widget, complete with ARIA compliance, keyboard accessibility, and internationalization (including right-to-left layout for appropriate countries and languages). For large tree data sets, we want to be able to only load the necessary data for the visible nodes of the tree. As a user expands a node, we then want to load the children of that node. Ideally, we only want to make one HTTP request per expansion for optimal performance. Historically, effective lazy loading has been a challenge, but some recent additions will make it much easier to utilize efficient lazy loading mechanisms in the tree.

The Dojo tree widget supports lazy loading, but it typically connects to a Dojo data store for all its data through a model-store adapter. Consequently, the tree itself does not define the actual data loading mechanism, as that is up to the store. The tree merely requests data from the store as needed. However, the JsonRestStore fully supports lazy loading with a well defined mechanism for retrieving deferred data from the server. When the JsonRestStore is used with the tree widget, it makes it possible to lazy load nodes in the tree as nodes are expanded. JsonRestStore supports lazy loading by using JSON referencing, whereby items can be referenced from other properties and arrays and the full item can later be retrieved when needed (with loadItem).

The model-store adapter now in Dojo 1.4 and later supports an option for a loading mechanism that facilitates single requests per node expansion. Previously (in 1.3), when a node was expanded the tree would request all the children and if any of the children had not been loaded, the tree would request that each child’s data item be fully loaded (this is done to ensure that each child has a label and information about whether or not it has children so the child nodes can be properly rendered) resulting in a request for each child of the expanded node, which is clearly inefficient. However, now when the model’s deferItemLoadingUntilExpand property is set to true, the tree will not attempt to load all of the children, but rather will only load the item for the expanded node if needed. This allows us to leverage the JsonRestStore’s support for partially loaded items. When a node is expanded and the item is loaded, the server can provide a JSON representation of the item which includes references to the children, and a partial set of properties for each child, which can include the label and children information for proper rendering. Each of these children can then be fully loaded when they are expanded (with their partial children objects). With this strategy, each node expansion will rely on only a single HTTP request.

Let’s take a look at how to build a lazy loaded tree with the JsonRestStore. We begin by creating a store:

myStore = new dojox.data.JsonRestStore({target:"tree/", labelAttribute:"name"});

Now, we will create the model adapter for the tree to access the store. This is where we use the new deferItemLoadingUntilExpand property:

myModel = new dijit.tree.ForestStoreModel({
        store: myStore,
        deferItemLoadingUntilExpand: true,
        query: "root",
        childrenAttrs: ["children"]
});

Next, we can create a tree:

myTree = new dijit.Tree({model: myModel}, treeNode);
myTree.startup();

Now, we can build our data that is supplied from the server. The first request that the JsonRestStore will make to the server will be a request for the top level nodes (the children of the root). The tree will make the initial request using the query provided to the model, and the JsonRestStore will combine that with the target. In this case, the request will be made to the path “tree/root”. Leveraging the partial loading support in JsonRestStore, we can serialize references to each of the items that will be children of the root node in the response, and only include the properties necessary to render these nodes (label and children info):

request:
GET /tree/root
response:
[
        { $ref: ‘node1′, name:‘node1′, children:true},
        { $ref: ‘node2′, name:‘node2′},
]

This provides sufficient information to render the top level of the tree, as well as the link information for retrieving the full representation of each item. We don’t need to actually include the children, just the presence of a children property will indicate to the Tree that the node is expandable and an expansion icon will be included. Now when a user clicks on one of these nodes, the tree will ask the JsonRestStore to load the item and the JsonRestStore will request the resource for the URI specified by the $ref property. In this case, it will request node1. This URI is interpreted relative to the target URI of the store, so in this case, the JsonRestStore will request “tree/node1″. Your server can then respond:

request:
GET /tree/node1
response:
{ id: ‘node1′, name:‘node1′, someProperty:’somePropertyA’, children:[
                        { $ref: ‘node1.1′, name: ‘node1.1′, children: true},
                        { $ref: ‘node1.2′, name: ‘node1.2′}
]}

Here we see the full representation of node1. This not only includes the name, but may include additional properties that are used in other application logic. We also include an array that lists all the children of this item. Once again we use partial representations of these children to minimize the data transferred over the wire to the client. Whenever any of these children are expanded the process will be repeated and another request will be made to the server for the full representation of that child.

The example code for lazy loaded trees is available in Dojo at /dijit/tests/Tree_with_JRS.html.

If you are updating data in the store, you should be aware of one more tip when using the Tree with the JsonRestStore. By default, the ForestStoreModel adapter will re-query the top nodes on every onNew notification event and every onDelete event that involves a top level item. This can result in queries to the server even though the server has not yet been sent all changes. This makes top level additions essentially disappear when the re-query takes place. You may need to override the _onNewItem and _onDeleteItem to provide your own logic about where new and deleted items should be placed in the hierarchy.

Another powerful feature of the referencing capabilities of JsonRestStore is that individual items can be referenced from multiple parent items. Consequently an item could exist in multiple places in the tree, under different parents.

Together, the Tree and the JsonRestStore provide a powerful combination for lazy loading data that allows for large extensive hierarchical data to be displayed without large upfront data transfers. The JsonRestStore’s partial loading support can be leveraged so that we can perform lazy loading with a single request per expansion.

Related posts:

1. Effective use of JsonRestStore: Referencing, Lazy Loading, and more
2. More Query and Caching Power for Data Stores: ClientFilter, JsonQuery, and JsonQueryRestStore
3. RESTful JSON + Dojo Data

Pintura is a REST-style web framework that utilizes a layered approach to application development that facilitates straightforward, well-designed rich internet applications. Pintura forms the core web framework for Persevere 2.0, and consists of a powerful data modeling and persistence framework called Perstore, in combination with a REST-style web framework. You can read more about the goals and principles behind Pintura, but here we will look at how to get started writing applications.

Pintura-based applications normally consist of server-side data models with three layers: data stores, store models, and model facets. On top of this, different representation handlers (for serializing data to different formats) can be defined, but Pintura comes with a good set of these ( including JSON, JavaScript, multipart, and Atom), so usually that is not necessary. This provides a well-structured separation of concerns, distinguishing storage configuration (data stores), core data logic (models), varying capabilities of access to the data (facets), and data serialization (representations). Perhaps the easiest way to understand this approach to take a look at an example application.

Naturally we need to install Pintura first. Pintura is JavaScript based and can run on various CommonJS JavaScript servers including Node and Narwhal/Jack, and the various engines they can run on. The Pintura installation process is described here.

A basic Pintura application is very easy to create. If we start by making a copy of /template/, we can create an application with a server-side model class by simply adding this to the lib/app.js:

var Model = require("model").Model;
Model("MyData", null, {
});

Once we start up our server for this application, we now have the full Persevere HTTP/REST interface for creating, querying, updating, and deleting objects from the MyTable model. We have everything we need on the server to write a web client application, which can utilize REST methods like GET, PUT, POST, and DELETE, (as well as JSON-RPC) to interact with the server model class to store and query data. We get an enormous amount of functionality for free, including cross-site capabilities (JSONP, window.name, and CORS support), cross-site request forgery protection, authentication, conditional request handling, and more.

However, most applications will require more server-side logic than simply a generic data store. Generally, application code will be written as model and facet definitions using Pintura’s persistence framework (Perstore). With this framework, we can define handlers for different aspects of the object’s interaction and lifecycle, such as get, put, and delete handlers. We can define data integrity contracts by adding JSON schema constraints in basically the same way as in Persevere 1.0 (except our model classes are defined with the Model constructor from the model module rather than a top-level Class constructor, to be more inline with CommonJS and standard web framework terminology). We can define additional methods for the model as well as methods for the persisted object instances (by defining methods on the model prototype), once again using the same structure as the first version of Persevere. For example, we could easily add a new method on object instances:

var Model = require("model").Model;
Model("MyData", null, {
    prototype: {
        addFriend: function(friend){
            this.friends.push(friend);
            this.save();
        }
    }
});

Pintura comes with an example wiki, we will examine how it is constructed to see how we can further understand what it looks like to build an application with Pintura. The example wiki is found in the example folder in the Pintura distribution, and we will break out the various parts and how they fit together.

We will start by looking at the app.js file found in the example/lib directory. This is the entry module which loads the other modules, starting from a user perspective, first defining the application-specific resource representations and user access levels. The example/lib/access.js module defines a set of facets for accessing data, for the different sets of users. For each user, a set of facets is returned and the facet provides the capability through which the user can access the application data. We can then drill down into example/lib/facet/page.js module, which defines how the underlying model class can be accessed through each facet. The example/lib/model/page.js defines the core application logic which in turn accesses the data store that provides the low-level interaction with the underlying storage medium.

Let’s look more closely at each of the levels, starting at the bottom with the store. A data store is the basic entity which allows for RESTful interaction with a storage system. Storage systems can be relational SQL-based databases, document based databases, OS files, remote servers, and so on. The object store API follows the W3C Indexed Database API for an object store. Pintura’s Perstore framework comes with a variety of data store implementations, including a SQL datastore for Rhino, and a JavaScript/JSON file-based data store. Creating your own store is not difficult, it essentially involves implementing get(id), put(object, id), query(query), and delete(id) functions (and optionally create, transaction, and subscribe for additional functionality). In our example application, we don’t define any special stores, we will simply use the default store, which out of the box is the js-file module, an object store which saves records into a JavaScript/JSON file.

The object store API is not only used at the bottom storage level in Pintura, but the model class and facets also follow the same API for consistency; classes and facets are simply wrappers that expose the same store API, adding their own functionality and/or constraining operations. Model classes act as the gateway to stores, providing the central core logic of the application. Any application data logic that should be enforced for all users should be implemented here. Models do not need to implement the entire store API themselves. Any store method that is implemented simply passes through the underlying object store. An empty class is perfectly valid, and would result in all REST actions being passed through to the object store. In example/lib/model/page.js, we can see the put handler is implemented to add a lastModifiedBy property and create a new PageChange instance:

exports.Page = Model("Page", pageStore, {
        put: function(object, id){
                object.lastModifiedBy = auth.currentUser.username;
…

We can also define the prototype for this class, which allows us to create methods that are available on all instances of the class.

At the next layer up, we have the set of facets used to access the Page class. Facets are defined in a similar way to model classes, they follow the store API, wrapping a class, and only methods with specific logic need to be implemented. There are two primary types of facets, restrictive and permissive facets. These differ in how they handle unimplemented methods. A permissive facet will default to passing all actions through to the underlying class. A restrictive facet defaults to only passing retrieval actions (get and query) through, methods have to be explicitly implemented to interact with the class.

In example/lib/facet/page.js we see three facets defined to accommodate different groups of users. The public facet is defined to be restrictive, essentially making the objects read-only for public users. The public facet also defines the query method so that only published pages are accessible. The user facet and admin facet are permissive, allowing all actions on the pages. All of these facets control access to the model class, and in fact, model classes are simply a special permissive facet that applies to data stores.

Continuing back up the stack, the access module can now be understood as the initial logic that determines which facets are available for the authenticated user, so their access is properly controlled.

The initial example/lib/app.js file also defines the custom media module used for generating the HTML for pages. Pintura comes with a good selection of media handlers out of the box; the JavaScript and JSON handlers are the primary use case handlers, allowing objects to easily be accessed through XHR by Ajax applications. Pintura is designed first and foremost for Ajax applications, so normally these will be used most frequently. However, we need a specific HTML handler in this case so that pages can be directly generated as HTML for easy indexing by search engines.

Media handlers are simple to implement. In the example/lib/media/wiki-html.js handler we can see that we just need to define the media type, the quality of the media type (for content negotiation), and the serializer. Bi-direction media handlers can also implement a deserialize method (for handling PUT and POST requests), but this isn’t needed for our Wiki, all updates to pages will be done through JSON-based XHR requests. The Wiki HTML handler simply uses the the wiky package to do the transformation of markup to HTML for us.

By default, Pintura does all the HTTP request handling for you, handling cross-site issues, authentication, transaction management, and converting requests to calls to facets and model classes. However, Pintura is highly modular, actually consisting of about 13 modular middleware appliances, many of which can be used on their own. With these set of modules, one can easily create new HTTP request handling stacks, or reuse individual modules for handling other requests. To see the Pintura stack, you can open up lib/pintura.js, and easily see the different middleware modules that come together in Pintura (this file also lists the set of media type handlers that Pintura registers by default).

For an application running on Jack, the Pintura application stack is defined as the request handler in jackconfig.js (in Node, it is defined in start-node.js). With these startup files we can define our own additional request handling logic, and still defer store related requests to the Pintura handler. This application definition defines static file handlers to simplify development. For the example application, we also created a very simple JSGI middleware module redirect-root that will check for requests to the root path, and redirect to a specific “root” wiki page. This middleware wraps the pintura.app request handler, handling the root path, and delegating all other requests to Pintura.

The example wiki gives us a good starting point for beginning to build with Pintura. Much of the persistence, model and facet APIs are defined at the Perstore project page. In the next article in this series, we will look at some more examples of how to leverage the persistence framework and its various stores.

Pintura provides the complete web client-server REST functionality of Persevere combined with a simple, modular, JavaScript-based, facet-oriented architecture, for easy to build to JavaScript applications, with end-to-end consistency, and flexible control and security.

Related posts:

1. Introducing Pintura
2. Pintura JSGI Modules
3. Object Capability Model and Facets in Perstore/Pintura

Pintura is a CommonJS/JSGI-compliant, server-side JavaScript-based framework for building rich Internet application with the REST architectural style, thin storage-oriented server design, and the consistency of end-to-end JavaScript. The Pintura framework forms the core of Persevere 2.0, and is the first framework built to run on multiple CommonJS platforms like node.js, Narwhal/Jack, and Flusspferd/Zest. It utilizes a layered approach to application development that facilitates straightforward, modular web applications.

Pintura is not a traditional MVC web server framework, which often conflate presentation and interaction concerns across the client and server, but rather follows the REST prescription of maintaining simple storage and serialization oriented server also known as thin server architecture or SOFEA. Pintura is designed to cleanly separate the concerns of presentation and interaction on the client, and storage and model logic concerns on the server. This design fits perfectly with comprehensive JavaScript frameworks like Dojo, General Interface, Cappuccino, YUI, and ExtJS that provide client-side MVC. In particular, Dojo has excellent support for standards-based JSON REST interaction that matches perfectly with this server-side framework.

The core design principles behind Pintura:

• Standards should be leveraged as much as possible to maximize interoperability with other projects, and facilitate freedom in composing web applications with the tools of choice. Pintura is based on RFC 2616 (HTTP/REST), JSON Schema, CommonJS, JSGI, JSON-RPC, and other interoperable standards.
• Presentation and data model logic should be properly separated such that clients can provide the user interface and servers can handle storage and application modeling logic. One should be able to develop multiple web clients for a given application data server.
• Storage and data processing concerns should be separated, with pluggable storage options that can easily be switched and combined (such MongoDB plus a MySQL database).
• The best option choice between configuration and convention is neither. Integration between different entities without dependence upon certain programming patterns (convention or configuration) is optimal.
• End-to-end consistency in data (JSON/JavaScript) handling provides the smoothest communication and interaction from client to server and back.
• The most common cases, like standard CRUD style applications, should be very easy, while still making it possible to build more unusual or sophisticated applications.
• A framework should be simple enough to read the source and easily extend or modify. Applications and frameworks should be organized into layers of functionality.
• Utilize object-capability design for security to allow for flexible security design while protecting against confused deputy style vulnerabilities.
• Full support throughout for promises, providing asynchronous functionality (used extensively by Node), while preserving clean encapsulated interfaces.

Improvements over Persevere 1.0

The server-side JavaScript community has dramatically changed since Persevere development started. There have been numerous exciting new server-side projects that have started like node.js, flusspferd, and Narwhal, utilizing the new generation of advanced high-performance JavaScript engines like V8, TraceMonkey, and JavaScriptCore. Persevere’s main focus is on enabling efficient client server interaction and for well-structured web application development, and matching this functionality with powerful new platforms is a great fit.

In many ways, Pintura represents a simplification of Persevere. With an entirely JavaScript-based modular design, it is very easy to dig in and understand how it works, and the complicated Java-JavaScript bridges have been eliminated. The central request handling mechanism is composed of about 13 middleware modules that can easily be rearranged and used on their own. From the ground up, Pintura has a simple, minimal, direct approach to providing REST functionality.

There are numerous other improvements over Persevere 1.0. Pintura introduces a new security system that is based on the object capability model and is therefore much more flexible and powerful than the previous purely role-based mechanism. Much scientific research has pointed to the superiority of capability based design and how it avoids dangerous confused deputy issues. Pintura utilizes “facets” (also known as proxies) to provide fine-grained access control to different stored objects. Each facet defines what actions are available for objects, with the flexibility of programmatic guard functions. In the end, security is essentially as simple as defining what facets are available for given users, and then defining what actions the facets allow.

Pintura also has moved to implementing the W3C proposed standard Indexed Database APIs object store interface for all data stores. Pintura is based around the Perstore persistence library, which implements this simple API consisting of a REST-style interface. The stores expose get, put, and delete methods and is used at multiple levels in the framework for simplicity and consistency, including for facets, model classes, as well as lower level object stores. Furthermore, additional functionality is easily applied to stores with provided store wrappers that add support for inheritance, caching, replication, event notification and more. Not only that, but with the interface likely to be implemented in browsers soon, this API will provide true end-to-end consistency.

Pintura has also embraced a much more asynchronous oriented design. JavaScript in the browser uses shared-nothing event loop concurrency, and server side projects like Node and Narwhal are following suit. This not only improves client-server consistency, but there is compelling evidence that suggests that event-based design performs and scales much better than traditional thread-based web servers. One of the major challenges of event-based systems is the encapsulation of functionality when dealing with callbacks, but promises are used extensively to greatly simplify working with asynchronous code.

Pintura also leverages the link relation definitions specified by the JSON schema specification that is now a submitted Internet-Draft. Link relations superset the functionality of JSON referencing, and allow for much cleaner, more flexible referencing syntax. This further advances the flexibility of a loosely coupled hyperlink-driven REST architecture.

A core feature of Pintura is now content negotiation, following the REST architectural design of permitting multiple representations of a resource. Resources effectively are persistable JavaScript objects, and representations allows these objects to be serialized and deserialized in multiple formats. Pintura has excellent RFC 2616 conformance to content negotiation and provides a number of out of the box media type handlers, such JSON, JavaScript, CSV, and more, while making it easy to create new custom media type handlers.

Finally, Pintura uses greatly simplified querying syntax. It continues to provide powerful URL-based querying, but the format is simplified to work easily with standard HTML form URL-encoding, aligns with the feed item query language, and has an easy to use and extensible call syntax. Pintura provides a parsing mechanism to help developers implement their own custom query handlers.

Role in Persevere 2.0

Persevere 2.0 is not a single monolithic project, but will be a distribution of several web stack configurations composed of various components including Pintura, Perstore, and the store explorer. Pintura and Perstore are still alpha stage software, but the Persevere 2.0 system is shaping up quickly.

The Next Web Stack

Pintura brings together the cumulative advances of modern JavaScript and web architecture design patterns, including object capability based security, event-driven flow, immutable promises, JSON Schema, REST style, and thin server architecture. Coupled with the powerful new CommonJS platforms and remarkably fast JavaScript engines, this framework fulfills a critical role in the next generation web application stack, and is built to adapt to your JS platform of choice. The next article in this series will explore how to get started with Pintura.

Related posts:

1. Getting Started With Pintura
2. Persevere 1.0
3. CommonJS/JSGI: The Emerging JavaScript Application Server Platform

This article introduces dojo.cache and presents a technique for externalizing your widget templates in swappable configuration files, where they can be referenced by a custom templateKey widget property.

Introducing dojo.cache

Dojo 1.4 adds a new core utility called dojo.cache. To appreciate it we first have to review how Dijit’s templatePath works. When you define a _Templated widget with a templatePath property, the content from that URL is fetched the first time you instantiate the widget, and made available as a string. All subsequent instances get the cached string. Furthermore, when you run a build, your templatePaths get replaced with templateStrings and their content is inlined into the output from the build. This improves performance considerably by removing those synchronous XHR requests, while remaining transparent to the developer.

dojo.cache generalizes this pattern, making the same functionality available from Dojo Core—synchronous, cached content retrieval that gets inlined during the build. dojo.cache usage is much like dojo.moduleUrl:

dojo.require("dojo.cache");
my.stringResource = dojo.cache("module.path", "relative.path.html" );

Runtime configuration

You’ll find many uses for dojo.cache. The ability to define and cache strings was immediately useful for me in the context of runtime configuration files. This is a fairly common technique in which you define and load modules ahead of most of the components to your app, to externalize constants and properties for the application that you don’t want to hard-code into each component. So, you might have something like this:

dojo.provide("myapp.defaultConfig");
// dojo.require("myapp.defaultConfig") populates myapp.config
// with messages with a laid-back informal tone
myapp.config =  {
        loadingText: "We’re loading that for you, won’t be a sec.",
        loadErrorText: "Ugh, that didn’t work out. Sorry."
};

…and a variation:

dojo.provide("myapp.formalConfig");
// dojo.require("myapp.formalConfig") populates myapp.config
// with messages with a more formal tone
myapp.config =  {
        loadingText: "Fetching requested resource",
        loadErrorText: "Error loading requested resource. Please accept our apologies."
};

In this example, switching the “tone” in your app is as simple as switching the one line that requires your config. Your code can use e.g. this.attr("content", myapp.config.loadingText) without having to know or care which variation it is supposed to be using.

Template configuration

Now with dojo.cache, we can manage templates in the same way, using object paths which can be looked up and requested at runtime during development, and inlined by the build for production:

myapp.config = {
  templates: {
    someWidget: dojo.cache("myapp.resources", "someWidget.html"),
    otherWidget: dojo.cache("myapp.alternateResources", "otherWidget.html"),
    collapsiblePanel: dojo.cache("dijit.templates", "TitlePane.html"),
  }
}

The config lets us retrieve template strings by using a look-up like myapp.config.templates.someWidget. With Dojo, we can treat these types of object paths as strings and use dojo.getObject to receive the value they represent. In this sense the object path is like an address—it gives us the ability to reference a variable and potentially a long string like a template with a short, symbolic one. I use these configuration properties by referencing them in a (new and custom) templateKey widget property. The key provides a level of indirection that enables you to pull template management outside of your widget definition entirely. It looks like this:

<div dojoType="my.SomeWidget" templateKey="myapp.config.templates.someWidget"></div> Template Reuse

When all that varies between a set of widget classes is the template markup, you can reuse one widget by simply passing the instance a new templateKey, for example:

<div dojoType="my.SomeWidget" templateKey="myapp.config.templates.someWidgetInversed"></div> The Implementation

To make this happen we need to have all widgets use an extended dijit._Templated to put the templateKey property on the prototype, and handle it in the buildRendering step:

dojo.declare("myapp._Templated", dijit._Templated, {
 
  templateKey: "",

  buildRendering: function() {
    if(this.templateKey) {
      this.templateString = dojo.getObject(this.templateKey);
    }
    this.inherited("buildRendering", arguments);
  }
});

.. which gets used by all our custom templated widgets, e.g.

dojo.declare("myapp.SomeWidget", [dijit._Widget, myapp._Templated], {
  templateKey: "myapp.config.templates.someWidget", // an optional, default value
});

Now, when we run our build, the config file you’ve specified gets the inlined templates, and the templateKey look-up works just as before. We could even monkey-patch this functionality into dijit._Templated and wring yet more goodness from the out-of-the-box Dijit & DojoX widgets.

Conclusion

Dijit’s architecture has always let you override templates, but in practice this has typically necessitated a subclass. The combination of runtime configuration and dojo.cache gives you new, powerful options that keep specifics out of reusable components, while maintaining resources in a manageable way. This is just one new technique that falls out of Dojo 1.4. Would you use it? What have been the highlights of 1.4 for you?

Related posts:

1. Creating Dojo Widgets with Inline Templates
2. HTML Widget Prototyping with the Dojo Toolkit
3. Dojo-Mini: Optimization Tricks with the Dojo Toolkit

CommonJS (formerly known as ServerJS) has become the essential hub around the development of server side JavaScript (SSJS). SSJS for years has suffered from fragmentation, but the CommonJS project has provided the momentum to bring different frameworks together and start building interoperable modules. SSJS is ripe with potential; JavaScript has been soaring in popularity and the ECMAScript 5 specification was recently accepted. JavaScript has proven itself as the language of the web’s client-side (even ActionScript is a derivative of JavaScript). The opportunity to use the same language on both client and server is certainly most realistic and viable with JavaScript as it avoids the need for translation.

CommonJS

CommonJS has focused on developing critical APIs for building reusable modules, particularly for server-side JavaScript environment. The server-side is generally based around database interaction, file I/O, HTTP serving, and the generation of data formats and HTML, whereas the client-side is based around DOM manipulation and the browser object model. There are certainly APIs that can be used on both sides, and JavaScript on the client and server invites the reuse of APIs where possible. The WebWorker, Indexed Database, and XHR APIs are promising to be enormously beneficial on the server side, and with excellent client server consistency. But still the server side requires special attention, and CommonJS is bringing the needed standards and conventions.

There have been a number of different areas that CommonJS has invested in for creating the necessary foundation for SSJS. The primary initial effort was in defining the module system, which provides an easy to use system for loading modules and their dependencies, while ensuring that each module is loaded only once. This module system is now very broadly implemented and extensively used by the other subsequent module APIs defined by CommonJS. The module system provides the essential basis for modular library and application development.

CommonJS is working on building up the low-level I/O components, defining the necessary binary data types, the APIs for file interaction, and encoding. There are proposals in place for unit testing, and a number of modules to provide concurrency support. And finally there is a HTTP interface specification known as JSGI.

Concurrency

One of the most central characteristics of a runtime is the concurrency approach. The inherent dangers of shared memory threading are widely understood, and the browsers wisely avoid exposing developers to the low level complications of shared-memory threads, alternately providing shared nothing event-loop concurrency. With the latter approach, data structures are not exposed to preemptive threads that could cause race conditions. Instead, events are queued up, and deterministically executed. Concurrent operations are handled with WebWorkers, in separate memory spaces, with the ability to communicate to other workers with message passing.

The CommonJS group defines APIs, so it is beyond the scope of CommonJS to dictate the concurrency model used by servers, but there is general consensus recommending that JavaScript-based servers utilize shared-nothing event loop concurrency, and CommonJS APIs are built to support this model. The CommonJS wiki includes proposals for exposing the WebWorker API through a worker module, and for exposing controls for the event loop with the event-queue module.

Promises

One key abstraction that is extremely useful for event-based systems is a promise. A promise is an abstraction that encapsulates the execution of an asynchronous action and its subsequent value that it returns. A promise can be thought of as representing the result of an asynchronous computation. The promise will receive the value returned by the call when it is finished, and acts as a placeholder for the returned value. Promises are critical to developing large scale applications that make heavy use of asynchronicity. Using callback-passing for asynchronous actions does not compose well. It is impossible to properly encapsulate functionality, switching an implementation from synchronous to asynchronous breaks APIs and creates complex flows of passing callbacks around to handle return values. With promises, values can be returned just like synchronous functions, and callers can listen for asynchronous results without requiring additional callback-passing APIs.

The promise API in CommonJS is an asynchronous handling system based on the collective experience of the promises in Dojo, Twisted, and ref_send. The ref_send promises focus on secure immutable promises, but lacks a reasonable API for promise implementors. Dojo provides a promise API (called Deferreds) with convenient chaining support, but has suffered from the use of mutable promises, with side-effect inducing callbacks complicating the data flow. The promise API combines the best of both, providing secure immutable promises with a very simple, easy to implement API for implementors, and full chaining support. The promise API in CommonJS provides a foundation that implementors can easily extend to provide the convenience functions from Dojo’s Deferred API as well as ref_send’s static operators.

An example of using CommonJS promise might look like:

requestSomeData("http://example.com/foo") // returns a promise for the response
    .then(function(response){ // ‘then’ is used to provide a promise handler
        return JSON.parse(response.body); // parse the body
    }) // returns a promise for the parsed body
    .then(function(data){
        return data.price; // get the price
    }) // returns a promise for the price
    .then(function(price){ // print out the price when it is fulfilled
        print("The price is " + price);
    });
  JSGI: HTTP Gateway

The JavaScript gate interface (JSGI) API has been asymptotically reaching consensus in the CommonJS group, as a standard API for serving HTTP requests through JavaScript application servers. The JSGI provides a very easy to use API for responding to HTTP requests, influenced by the design of WSGI and Rack. A basic JSGI application looks like:

app = function(request){
    return {
        status: 200,
        headers: {},
        body: ["Hello World"]
    };
};
 

JSGI has been intentionally designed to make it easy to write “middleware”, modules that exist at different places in the request handling chain to add various forms of functionality. Middleware can include support for cross-domain requests, logging, error handling, special method handling, mapping URLs to different applications, and much more.

Most existing HTTP interfaces were designed around a synchronous model, and have had an extremely difficult time adapting to the asynchronous model, which is essential for efficient Comet support. JSGI on the other hand, has been developed from the ground up with asynchronicity in mind, both to support Comet, and to work properly in an event-based environment. JSGI achieves support for asynchronicity while still maintaining a simple function call and return style by leveraging promises. Here we can see the power of promises, JSGI applications can return a promise to indicate that the response is not yet complete, and no additional callback parameter need to be passed around. With promise support throughout, an asynchronous JSGI app can be as simple as:

app = function(request){
    return doSomethingAsynchronous().then(function(result){
        return {
            status: 200,
            headers: {},
            body: ["Result ", result]
        };
    });
};
 

Since JSGI effectively provides a JSON/JavaScript representation of the HTTP protocol, JSGI has powerful opportunities for use in communication outside of just HTTP server handling. JSGI can be used as an API for making HTTP requests as a client (as a promise based alternative to XHR), and can be used as an inter-process/inter-worker protocol. HTTP is almost certainly the most commonly used communication protocol, and the RESTful power of HTTP can easily be leveraged for worker communication with JSGI’s JSON-based format (that doesn’t require HTTP parsing).

Projects

CommonJS and JSGI are purely API specifications, not implementations. Lets look at some projects that implement these specifications.

Narwhal/Jack

Probably the most advanced and mature CommonJS implementation is Narwhal plus Jack. Basically all of the CommonJS APIs are implemented in Narwhal/Jack and/or their forks. Narwhal provides extensive I/O support, a large collection of useful library modules, and a great package management system called tusk with package resource handling built into its module loader. Narwhal is designed for running on various JavaScript platforms and has an intelligently architected layer of abstractions for dealing with different engines.

Narwhal has the most complete engine support for Rhino. Rhino is almost certainly the most popular JavaScript engine for server side JavaScript due to its seamless integration with Java. With Rhino, the virtually limitless expanse of Java libraries are available and easily accessible. While great efforts are being made to important libraries in other engines, the breadth of libraries that are available through the Rhino’s Java bridge is not likely to be matched anytime soon. It also comes with a good visual debugger. Rhino is also perhaps the most feature-rich JavaScript implementation available, as it implements most of the ECMAScript 5 specification and Mozilla’s JavaScript 1.8 features.

Narwhal also runs on JSCore, the JavaScript engine used by Safari. JSCore is a very high-performance engine (second only to V8, perhaps), and consequently there is great potential for high-performance applications running on Narwhal/JSCore.

Jack is a JSGI implementation that has been developed to run on Narwhal, and is also built to run on various web server technologies. Jack has been somewhat of the de facto reference implementation for JSGI, and runs on Simple, CGI, servlet containers (including Jetty and GAE), and others. Older versions of Jack relied on shared memory threading, but with the latest work on Jack, requests are handled with full event loop based shared-nothing concurrency, with request delegation across multiple workers for efficient multi-core utilization.

Node

Node is an exciting project in the SSJS realm, and has received a lot of publicity recently, for good reason. Node builds on the amazingly fast V8 engine and provides important low-level functionality. V8 is one of the fastest dynamic language engines on the planet. Node uses libev for an event-loop based asynchronous I/O model, providing the same type of event-loop shared-nothing concurrency used in Narwhal. However, Node has had a stronger emphasis on trying to provide asynchronous APIs for all I/O operations. By exposing almost all I/O operations as asynchronous events, Node rarely has to block while waiting for I/O, and is able to execute much more efficiently than spreading workloads across multiple blocking threads. The performance advantages to event-loops over threads are well-documented. This focus, in combination with the blazing fast V8 engine, promises to place Node as the performance king of dynamic language based application servers.

There are certainly gaps that need to be filled with Node. Node implements CommonJS’s module system, but does not implement other CommonJS APIs, and has been slow to add better support for interoperability. Node runs its HTTP server and handling on a single process/thread, which means it can’t utilize multi-cores yet (support for that is coming). Node does not support the JSGI specification; however, it is possible to run JSGI on Node with the JSGI-Node adapter that I built.

There are other important things like package support and module reloading that are definitely needed. From my experience, Node does not feel as complete as Narwhal. However, despite these concerns, Node has a lot of potential, and is an exciting project. Node primarily fulfills a role as a low-level JavaScript platform, on which higher level functionality can be built. The Node community is responsive. There is an effort to run Narwhal on top of Node, which would be a powerful combination.

Ejscript

Ejscript is a JavaScript platform that extends the JavaScript language with many new constructs. Many of these additions are based on the ECMAScript 4 specification that was eventually abandoned because many implementors felt it was too extensive of a revision of the language. However, many of the features included in ECMAScript 4 that Ejscript implements were technically excellent additions, and may be included in ECMAScript 6. Beyond extra language features, Ejscript boasts a very high performance engine with remarkably low memory footprints, claiming performance that rivals Node with less than a tenth of the memory consumption.

Persevere

Persevere was one of the first projects to implement the CommonJS module specification. However, the primary goal of the Persevere project is to provide consistent end-to-end data semantics leveraging RESTful architecture. Persevere 1.0 provided CommonJS module support to provide interoperability with other CommonJS libraries, but being a JavaScript platform is beyond the primary scope of the project. With the new array of CommonJS platforms, future Persevere development will now focus on building on top of these new platforms and providing its data persistence, security, REST architecture, Comet notifications, and advanced web communication capabilities for multiple platforms.

Others

Mozilla’s SpiderMonkey engine is also a capable engine for server side JavaScript. Flusspferd is a project to add low-level system functionality to SpiderMonkey and couples with Zest as a JSGI server and Juice as a web framework on top.

There are number of other important server-side projects including HelmaNG, with high-quality technology that predates CommonJS and now are adapting to work in harmony with the CommonJS ecosystem, but I won’t cover all of them.

Web Frameworks

Some exciting new web frameworks are building on these technologies such as Nitro, Bogart, and JuiceJS. Also some platforms come with their own MVC framework such as HelmaNG, and EJScript. However, for many the move to JavaScript has been motivated by investment in rich client interfaces that already have presentation capabilities (the V and C in MVC), so a more modern REST-style, “thin server architecture” may be a better fit than traditional server-side MVC frameworks for many next generation JavaScript-based web applications. Pintura is the core of the next generation of Persevere, a cross-platform REST-style architecture web framework that is designed specifically for Ajax-style applications that maintain presentation logic on the client separate from the server-side storage concerns. We will cover Pintura in the next article of this series.

Conclusion

The combination of improvements and maturation of ECMAScript standards, with the new line of astonishingly fast JavaScript, the Ajax driven interest in JavaScript, the collaborative efforts of CommonJS standards, and the emerging array of server-side JavaScript projects are all coming together to propel server-side JavaScript to the forefront of web application development. It is not unreasonable to expect that the CommonJS APIs implemented with a web stack of the new line of SSJS projects could be the most important new web platform of the next decade.

Related posts:

1. CommonJS Utilities
2. Introducing Pintura
3. Pintura JSGI Modules

Recently, a number of performance tests have been released that are based on the performance of the top 100 web sites such as SpriteMe savings, the IE8 100 top sites test results, or the JSMeter research. These are in direct contrast with tests such as ACID3 which attempt to test the future of the web rather than just what’s possible today.

These efforts are outstanding and highly useful, especially the JSMeter work and their valiant effort to redefine performance tests that are indicative of today’s web apps.

I completely agree with one of their stated goals:

We hope our results will convince the JavaScript community to develop and adopt benchmarks that are more representative of real web applications.

However, I disagree with their approach: they are testing the performance of today’s already optimized sites! There’s nothing in the middle of testing today’s sites and the more unrealistic “test every feature” ACID tests.

I believe more accurate tests for tomorrow would be useful, testing what’s pushing the limits of the web today, but are not currently top 100 sites. My main objection with comparing performance across the top 100 web sites is this: The top 100 web sites are already relatively highly performant, because they are optimized for what’s possible today. They have and continue to improve thanks in large part to the work of Steve Souders and others in the performance optimization community. Because it costs significant amounts of money in server operations fees and bandwidth, high traffic web sites generally dedicate considerable resources to highly optimizing their sites. High traffic web sites also face significant competition and are highly scrutinized for acceptable page load time. Budget and competition result in popular sites not deploying code that makes pages load slower than their desired performance threshold. Even more importantly, top 100 sites have the budget to make their app work in the future when things change. You can dedicate people on your team to squeezing out performance improvements in all aspects if you have the budget for it. Most web apps cannot afford to do this.

When we’re testing the performance of new browsers or analyzing page load performance, we should also really be looking at what the top 100 sites will look like in terms of features and expectations in five years! So how do we do that today? There’s no simple answer, but here are some ideas:

• Test popular web apps, e.g. mint.com populated with large amounts of data
• Test apps that don’t support IE6, e.g. Google Wave
• Test all sections of popular sites, not just the home page, through an automated performance test harness
• Test ridiculous configurations of popular applications, e.g. enable every feature in modular applications until they slow down
• Test apps over long amounts of time in the browser, not just initial page load time
• Test 50 apps, each in a different tab, all at once, and see how fast you can make a browser like Firefox or IE crash!
• Test throttled networks that emulate the profile of mobile and satellite networks, slow hotel wi-fi networks that often limit the length and duration of connections, corporate proxies, tech conferences, and countries with overloaded pipes (e.g. YouTube in New Zealand)

Only when browsers are pushed to their limits do we see where they break down, and how sites break them. We also need tests and tools (such as instrumented usage of YSlow, PageSpeed, SpeedTracer, etc.) comparing the most complex apps and how they perform across the various browsers, as today’s complex app is potentially tomorrow’s median site.

To be clear, I’m not saying “don’t optimize for today”. I’m saying, stop comparing cutting edge sites to Google search results. Lumping these two together in a common test is like putting apples against oranges because they are both round fruit.

Related posts:

1. String Performance: Getting Good Performance from Internet Explorer
2. Why Apple is Investing in WebKit Performance
3. Web Page Global Variable Performance

Dojo 1.4 is hot off the presses, with more than seven months of significant improvements to performance, stability, and features.

Of particular interest:

• IO Pipeline topics
• dojo.cache
• dojo.contentHandlers
• dojo.hash with native HTML5 onhashchange event support where available
• Traversal and manipulation for NodeLists (the return value for dojo.query)
• dojo.ready (easier to type than dojo.addOnLoad)
• Hundreds of refinements to the Dijit API and collection of Dijits, and a few new widgets in DojoX
• DataChart widget and other improvements to charting
• dojox.drawing lands!
• Editor improvements and new plug-ins in both Dijit and DojoX
• Grid is faster, and the EnhancedGrid lands!
• ForestStoreModel for the TreeGrid
• GFX improvements
• dojox.jq, a very experimental module aimed at trying to match the jQuery API as close as possible, but using Dojo underneath
• Dojo build system optionally supports the Google Closure Tools compiler
• Significant speed improvements, especially in IE

Read the full Dojo 1.4 release notes for more details! And thanks to everyone in the Dojo community that helped make this release great!

Related posts:

1. Dojo 0.9 released
2. Introducing the Dojo Toolkit 1.1
3. Dojo Grid Update

Google recently was asked about something we have suspected: Android and Chrome OS may converge. From our perspective, Android and Chrome OS both offer compelling opportunities for building great web apps, but having two distinct operating systems from Google, each with different approaches to development, just adds complexity and confusion to the overall development landscape. Of course, it still bothers us that iPhone apps and Dashboard widgets aren’t interoperable.

Android has the first mover advantage of being deployed today to many devices, but to really get the most out of it, you really should develop using Java, or employ a toolkit like PhoneGap. Chrome OS offers the promise of using web technologies that are popular today, but is not yet production-ready, or optimized for mobile devices like Palm’s webOS.

The long-term expected convergence from Google makes sense. Convergence of Palm’s webOS with Chrome OS makes sense to us as well as they are both pushing towards having a very similar WebKit-based operating system for delivering web applications. Palm already appears to be moving in the direction of converging with Chrome through their adoption of key technologies in webOS such as Google’s V8 JavaScript engine. There’s nothing confirming that this will happen, other than it just making sense.

While there is a lot of short term interest in apps, you can still get significant results from mobile web apps, and the gap between a native app and a web-based app will quickly shrink, as evidenced by apps like Pie Guy. After all, web apps don’t require app store approval!

Related posts:

1. An Android without Gears?
2. Why We Love Chrome Frame
3. SDK showdown: iPhone vs. Android

It was recently reported that Google Dumps Gears for HTML5. If true, with the investment Google has made in HTML5, Chrome, Chrome OS, and Chrome Frame, this is not surprising, but it does leave a potential short-term gap for offline application development.

In their post, Read-Write Web asks if offline access is even necessary any longer. I guess they don’t spend enough time on airplanes or at hotels and conferences with poor internet connectivity! It’s certainly necessary, and while other browsers have developed significant offline capabilities, older browsers still need a plug-in like Gears.

In the interim, what should you do if you want an offline application? Do you develop for Gears and HTML 5 features? Do you wait for Chrome Frame to integrate offline capabilities? Do you use a toolkit like Dojo which will wrap the various possibilities? Or do you rely on something else like AIR, Titanium, Prism, or Fluid? The answer really depends on your application. If it is live now, you plan for the future but you keep going with Gears and/or Dojo Offline. If you won’t be launching for some time, you may want to talk to us about your offline app options.

At the end of the day, Gears is open source. If there’s a long-term need for its existence, the community can pick it up and run with it! Thankfully end-of-life isn’t the certain death-knell that it is with closed-source software.

Related posts:

1. Google Gears, Dojo Offline, and SitePen
2. Why We Love Chrome Frame
3. An Android without Gears?