One of the most basic tests developers perform in JavaScript is whether or not a particular property exists on an object. Since feature detection is the preferred method of code forking, developers are encouraged to test for the existence of properties before using them. There is a lot of buggy JavaScript code out there as inexperienced developers try to use feature detection. A lot of the problems reside with a lack of understanding as to the nature of object properties in JavaScript.

Where do properties come from?

Before attempting to detect properties, it’s important to understand from where they originate. There are two basic types of properties in JavaScript: those that exist on the object (also known as “own” properties) and those that are inherited through the prototype chain (often called “prototype” properties). Consider the following:

var person = {
    name: "Nicholas"
};

alert(person.name);        //"Nicholas"
alert(person.toString());  //"[object Object]"

In this code, the object person has only one own property, which is name. You can still access other methods on the object such as toString(), but these are inherited through the prototype chain. Object literals inherit from the Object type, so all of the basic methods of Object are accessible on the instance.

The big difference between own properties and prototype properties is the difference between unique and shared values. Own properties belong to that single object instance and can’t be affected by other instances of the same type; prototype properties belong to the prototype of the object, and since the prototype can be shared amongst multiple instances, these properties are also shared amongst multiple instances. Here’s another example:

var person2 = Object.create(person);
var person3 = Object.create(person);

alert(person2.name);   //"Nicholas"
alert(person3.name);   //"Nicholas"

person.name = "Adam";

alert(person2.name);    //"Adam"
alert(person3.name);    //"Adam"

This example uses the Object.create() method from ECMAScript 5 to create two objects, person2 and person3, that inherit from person. The prototype for both person2 and person3 is person, and so name is actually a prototype property that is accessible through person2 and person3. This is why displaying the values of name on both objects results in the same value: they are both sharing the prototype property name. That means when person.name is changed directly, the change is accessible from the instances.

It’s important to understand that name is a prototype property for both person2 and person3, but it’s an own property for person. You can only assign values to own properties, so attempting to assign a value to a prototype property actually causes a new own property of the same name to be created. Example:

alert(person2.name);    //"Nicholas"
alert(person3.name);    //"Nicholas"

person2.name = "Adam";

alert(person2.name);    //"Adam"
alert(person3.name);    //"Nicholas"

Since you can’t assign to prototype properties, assigning a new value to person2.name actually creates a new own property on person2 called name. Own properties always shadow prototype properties, so the next time you access person2.name, you’re accessing the own property instead of the prototype property. That will continue until the own property is removed using delete, such as:

delete person2.name;

alert(person2.name);    //"Nicholas"

You can only call delete on an own property to remove it (calling on a prototype property does nothing). Once the own property name is removed, there is nothing shadowing the prototype property name and so person2.name now refers to the prototype property.

Note: While all native object types (Array, Boolean, Date, all Error variants, Function, Number, RegExp, and String) inherit from Object, non-native object types, such as those that represent the DOM in browsers, don’t necessarily inherit from Object in all browsers.

Detecting properties

Let’s say you want to determine if a given object has a property of name. In experienced developers tend to write code like this:

//doesn't accurately test for existence
if (person.name){
    //yay! property exists!
}

At first glance, this seems okay. However, understanding how JavaScript works reveals some problems with this approach. First, this will only succeed if the value of person.name is truthy, meaning it’s an object, a non-empty string, a non-zero number that’s not NaN, true, and not null or undefined. That means if person.name is the empty string (”"), this check will fail. Failing, in this case, doesn’t mean that the property doesn’t exist. In fact, the property does exist and contains a value, but the value is falsy and so doesn’t pass this test.

Detecting own properties

Keeping in mind that this is about testing for the existence of the property and not for the usability or data type, there are a couple of options. The first option is to detect own properties, and it comes via a method on the Object type called hasOwnProperty(). Since native objects inherit from Object, this property is inherited by these objects and can be used to detect the existence of own properties:

alert(person.hasOwnProperty("name"));   //true
alert(person2.hasOwnProperty("name"));    //false

person2.name = "Adam";

alert(person2.hasOwnProperty("name"));    //true

delete person2.name;

alert(person2.hasOwnProperty("name"));    //false

Initially, person2 has a prototype property name, so hasOwnProperty() returns false. Once an own property is created, calling hasOwnProperty() returns true. And after the property is removed via delete, this method again returns false.

JSON serialization works only for own properties, and non-native JSON serialization utilities used hasOwnProperty() to ensure that only the properties defined on object literals were included in the resulting string.

Detecting all properties

If you only care that the object has a property and don’t care whether it’s an own property or a prototype property, you can use the in operator to determine the existence of the property. Example:

if ("name" in person){
    //property exists
}

The in operator returns true when the named property exists on the object. In many cases, the in operator is all that you’ll need (especially when dealing with DOM objects). In fact, Mark Pilgrim’s All-In-One Almost-Alphabetical No-Bullshit Guide to Detecting Everything for HTML5 makes extensive use of in for detecting the new HTML5 features on DOM objects.

Conclusion

If you just want to check for the existence of properties, and not necessarily what their value might be, then you have two safe options: hasOwnProperty() and the in operator. The hasOwnProperty() property method should be used if you want to detect own properties only. If you want to test property existence and don’t care if it’s an own property or an object property, then the in operator is the one to use.

Update (27 July 2010): Added false and NaN to list of falsy values.

Update (29 July 2010): Fixed description of truthy/falsy values.
Related posts

• Feature detection is not browser detection
• Naked JavaScript objects
• Get the JavaScript global
• Good object-oriented design in JavaScript
• Feature detection != browser detection
• Browser detection versus feature detection

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

Frequent readers will remember my mission to stop browsers from making automatic requests when an empty string URL is reference in script. My mission began with a post entitled, Empty image src can destroy your site, in which I explained just how devastating this browser quirk can be to an enterprise system. After that point, I started contacting browser vendors and discussing the issue on the WHAT-WG mailing list. I also did some research only to find that there were other tags besides <img> that suffered the same effects in some browsers, though in very inconsistent ways. I posted a followup on the progress I’d made in various arenas, including getting HTML5 updated to say that automatic downloads of resources should not happen for empty string URLs.

It’s been eight months since my original post and from when I started contacting people about this issue, so I thought it would be good to update the current state. Keep in mind that Opera is intentionally left off of this list because they never have initiated a request for an empty string URL (hooray Opera!).

Firefox

Firefox 3.5 fixed the issue with <img src=""> so that it did not fire off a request as it did in previous versions. Unfortunately, it still fires a request for <script src=""> and <link href="">. I filed a bug with Mozilla about this (the one that started the WHAT-WG discussion) and while a patch has been submitted on the bug, it hasn’t yet been incorporated into a build. There hasn’t been any movement in over a month, so perhaps it’s time to nudge them again.

Safari and Chrome

When I first went through and looked at the WebKit bug queue, I found an existing bug mentioning this issue. I added my comments as discussions continued with various people. Unfortunately, there wasn’t much movement on the issue, which is particularly dismaying because WebKit made requests for <img src="">, <script src="">, and <link href="">, making it the worst current browser engine for this issue.

Seeing a lack of movement, I filed a bug against Chrome, hoping that they could apply some pressure. It took a little while, but the Chrome team eventually came back and said they would fix the issue. You can follow the conversation from the Chrome bug back onto the WebKit bug, where a patch has now been submitted for this issue. The Chrome bug indicates that this is scheduled to get fixed in Chrome 6 and there is no word yet on when it’s scheduled for Safari.

Internet Explorer

Being the least transparent of all browsers, it was difficult getting contacts to people who could make a difference. My initial attempts to get some movement failed, and I had to go back through my personal network to find another contact before I could make any progress. Finally, I got confirmation that the team believed fixing this issue was the right thing to do and that it would make it into a future build. They couldn’t tell me what version or build this would be fixed in (corporate policy) but did assure me that it would be addressed in the future.

While sitting in the Internet Explorer 9 talk at Velocity, there was a slide featuring a bunch of performance-related fixes made in IE9. Towards the bottom of the slide was a bullet saying that <img src=""> will no longer fire a request. When I got home, I setup the Internet Explorer 9 Platform Preview 3 to find that the issue had been fixed. So Internet Explorer 9 will definitely not fire these requests any more.

Conclusion

We’re not completely out of the woods yet, but with changes to HTML5 and all browsers now on the path towards resolving this issue, we could very well have seen the last of this issue by the end of 2010. I think this is a testament to just how great the web community is: to be able to take an issue in front of the people who can resolve it and see such amazing progress. I’d like to thank each of the browser vendors from responding to this issue. We’re almost there!
Related posts

• History of the user-agent string
• Empty-string URLs in HTML - A followup
• Protect IE from empty img src
• And so the HTML 5 trouble begins
• Cross-domain XHR removed from Firefox 3
• The Web could be heading for another dark age

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

I was sitting in a talk given by Facebook’s Jason Sobel at Velocity this year, when I was a bit surprised by an impassioned plea that he made at the tail end of the talk. To paraphrase, Jason said that CSS sprites require too much work for average web developers and that we should be pressuring the browser vendors to make this process easier. I was perplexed for a moment. First, I don’t think CSS sprites are all that complicated anymore, especially with tools like the online CSS Sprite Generator readily available. Second, CSS sprites only really have to be used for older browsers (looking at you, IE6 and IE7), as this problem is easily solved in other browsers by using data URIs. Jason’s epilogue made it even clearer to me that people still don’t understand the true power of data URIs and how they’ll transform web development going forward.

The purpose of CSS sprites

Before getting into how data URIs make CSS sprites obsolete, it helps to examine the problem that CSS sprites solved. The problem is that HTTP requests are a major performance bottleneck for web pages. The more requests you have, the longer it takes your page to load and the slower it is, so every little image you load onto a page fights against you in your quest for page speed.

CSS sprites solved this problem by combining multiple images into a single file, thus collapsing all of those extra HTTP requests into a single request and vastly speeding up the page. The downside is the overhead of planning for and using CSS sprite images, as the images need to be arranged in a certain order, perhaps with some extra blank space in between. That typically meant that someone had to write down the location of each individual image within the larger sprite image so that CSS could be used to position the image in the correct spot to show the correct image. For more information see Dave Shea’s article, CSS Sprites: Image Slicing’s Kiss of Death.

Basic CSS sprite usage

The pattern I use the most for CSS sprites is relatively straightforward and has the goal of ensuring CSS maintainability. There is a single class that contains a reference to the CSS sprite image and several other classes that simply move the background into different positions. For example:

.mysprite {
    background: url(mysprite.png) no-repeat;
}

.icon1 {
    background-position: 16px 16px;
}

.icon2 {
    background-position: 32px 16px;
}

Suppose you were making a progressively enhanced toolbar with this CSS, so there’s an unordered list with each item representing a button. Imagine that these are styled such that the text is hidden and each list item link simply becomes an image to click on. The HTML for such an example would look like this:

<ul class="toolbar">
    <li class="mysprite icon1"><a href="/save">Save</a></li>
    <li class="mysprite icon2"><a href="/open">Open</a></li>
</ul>

For any element that wants to use the master sprite image, the class of mysprite is applied. Then, a second class is applied to move the sprite into position. Note that there are alternate techniques that have the same result; the reason I like this one is because the URL is only ever referenced once (good for maintainability) and it’s able to be used anywhere on the page.

In terms of performance, the benefit to this technique grows as the number of images in the same file increases. You can end up with one very large image file, but that is still better than making multiple requests for a bunch of small images. You make a single request for the sprite image and after that point it’s cached by the browser, so you no longer have to worry about making a request. Note also that if the CSS is in an external file, it too will be cached.

Using data URIs instead

A little while back, I wrote about what data URIs are and how to use them. In short, data URIs allow you to embed images (and other files) directly into HTML and CSS. Since all of the data is represented locally, there is no extra HTTP request required to access the information.

Remember that the original problem that CSS sprites solved was having too many HTTP requests for images. Data URIs also solve that problem, and solve it in a much more manageable way. Instead of using a single extra request to get the large sprite image, you use zero extra requests to get the images to use. What’s more, there’s no need to combine all of the images - you can keep the images separate and use them as normal background images. The CSS doesn’t really change all that much (full data URIs omitted for space):

.mysprite {
    /* no longer needed */
}

.icon1 {
    background: url(data:image/png;base64,<data>) no-repeat;
}

.icon2 {
    background: url(data:image/png;base64,<data>) no-repeat;
}

Here, the mysprite class actually becomes unnecessary as the image data now resides in each icon class. The HTML doesn’t need to change (though you can remove mysprite if you so desire) in order to create the same effect.

At first glance, this may seem strange to you. The first question that people tend to ask when I describe this approach is that I’m dramatically increasing the size of the CSS by embedding the image data, doesn’t that hurt performance? The answer is no, so long as the CSS lives in an external file and is gzipped and cacheable by the browser. Base64 encoding, which is how the image data is represented, compresses remarkably well when gzipped, ultimately resulting in roughly the same amount of bytes being transferred over the write as compared to downloading the original image file. The added benefit is that you’re making zero extra calls for all of the images. And since these are in your external CSS files, they are also cached, so the next time the user comes to the page the CSS file is pulled from cache with the images already inside.

Automatic transformation

Because I believe in this technique so much, I wrote a tool called CSSEmbed (read the announcement) that makes it trivial to update your image-based CSS into data URI-based CSS. This frees you up to write CSS code like this:

.icon1 {
    background: url(icon1.png) no-repeat;
}

.icon2 {
    background: url(icon2.png) no-repeat;
}

So you write CSS in the old-fashioned, non-performant way with individual images, run it through the tool, and you automatically get a CSS file with data URIs embedded. That’s it, no more arranging images into a single file or keeping track of coordinates.

Note: CSSEmbed also supports an MHTML mode to make IE6 and IE7 compatible stylesheets that use internal images similar to data URIs.

Conclusion

CSS sprites were a solution to the problem of multiple HTTP requests to download multiple images. Data URIs allow you to embed images directly into your CSS files, solving the same problem in a much more elegant and maintainable way. Although we still need CSS sprites for older versions of Internet Explorer, that shouldn’t prevent you from investigating the use of data URIs as a better alternative to CSS sprites. Once IE6 and IE7 go away for good (some day), there really shouldn’t be the need to use CSS sprites so heavily if at all.
Related posts

• Automatic data URI embedding in CSS files
• Data URIs explained
• Internet Explorer 8 document and browser modes
• Writing maintainable code
• New site design
• Wanted: Browser debug mode

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

A couple of years ago, web developers were banging their head against the first wall in Ajax: the same-origin policy. While we marveled at the giant step forward enabled by cross-browser support for the XMLHttpRequest object, we quickly bemoaned the fact that there was no way to make a request to a different domain from JavaScript. Everyone setup proxies on their web sites, which was the onset of a new host of open redirect problems, as a way to get around the restriction. Although developers were working around this limitation using server-side proxies as well as other techniques, the community outcry was around allowing native cross-domain Ajax requests. A lot of people are unaware that almost all browsers (Internet Explorer 8+, Firefox 3.5+, Safari 4+, and Chrome) presently support cross-domain Ajax via a protocol called Cross-Origin Resource Sharing.

Cross-Origin Resource Sharing (CORS)

Cross-Origin Resource Sharing (CORS) is a W3C Working Draft that defines how the browser and server must communicate when accessing sources across origins. The basic idea behind CORS is to use custom HTTP headers to allow both the browser and the server to know enough about each other to determine if the request or response should succeed or fail.

For a simple request, one that uses either GET or POST with no custom headers and whose body is text/plain, the request is sent with an extra header called Origin. The Origin header contains the origin (protocol, domain name, and port) of the requesting page so that the server can easily determine whether or not it should serve a response. An example Origin header might look like this:

Origin: http://www.nczonline.net

If the server decides that the request should be allowed, it sends a Access-Control-Allow-Origin header echoing back the same origin that was sent or “*” if it’s a public resource. For example:

Access-Control-Allow-Origin: http://www.nczonline.net

If this header is missing, or the origins don’t match, then the browser disallows the request. If all is well, then the browser processes the request. Note that neither the requests nor responses include cookie information.

All of the previously mentioned browsers support these simple requests. Firefox 3.5+, Safari 4+, and Chrome all support usage through the XMLHttpRequest object. When attempting to open a resource on a different origin, this behavior automatically gets triggered without any extra code. For example:

var xhr = new XMLHttpRequest();
xhr.open("get", "http://www.nczonline.net/some_resource/", true);
xhr.onload = function(){  //instead of onreadystatechange
    //do something
};
xhr.send(null);

To do the same in Internet Explorer 8, you’ll need to use the XDomainRequest object in the same manner:

var xdr = new XDomainRequest();
xdr.open("get", "http://www.nczonline.net/some_resource/");
xdr.onload = function(){
    //do something
};
xdr.send();

The Mozilla team suggests in their post about CORS that you should check for the existence of the withCredentials property to determine if the browser supports CORS via XHR. You can then couple with the existence of the XDomainRequest object to cover all browsers:

function createCORSRequest(method, url){
    var xhr = new XMLHttpRequest();
    if ("withCredentials" in xhr){
        xhr.open(method, url, true);
    } else if (typeof XDomainRequest != "undefined"){
        xhr = new XDomainRequest();
        xhr.open(method, url);
    } else {
        xhr = null;
    }
    return xhr;
}

var request = createCORSRequest("get", "http://www.nczonline.net/");
if (request){
    request.onload = function(){
        //do something with request.responseText
    };
    request.send();
}

The XMLHttpRequest object in Firefox, Safari, and Chrome has similar enough interfaces to the IE XDomainRequest object that this pattern works fairly well. The common interface properties/methods are:

• abort() - use to stop a request that’s already in progress.
• onerror - use instead of onreadystatechange to detect errors.
• onload - use instead of onreadystatechange to detect successes.
• responseText - use to get contents of response.
• send() - use to send the request.

Preflighted requests

CORS allows the use of custom headers, methods other than GET or POST, and different body content types through a transparent mechanism of server verification called preflighted requests. When you try to make a request with one of the advanced options, a “preflight” request is made to the server. This request uses the OPTIONS method and sends the following headers:

• Origin - same as in simple requests.
• Access-Control-Request-Method - the method that the request wants to use.
• Access-Control-Request-Headers - (Optional) a comma separated list of the custom headers being used.

Example assuming a POST request with a custom header called NCZ:

Origin: http://www.nczonline.net
Access-Control-Request-Method: POST
Access-Control-Request-Headers: NCZ

During this request, the server can determine whether or not it will allow requests of this type. The server communicates this to the browser by sending the following headers in the response:

• Access-Control-Allow-Origin - same as in simple requests.
• Access-Control-Allow-Methods - a comma separated list of allowed methods.
• Access-Control-Allow-Headers - a comma separated list of headers that the server will allow.
• Access-Control-Max-Age - the amount of time in seconds that this preflight request should be cached for.

Example:

Access-Control-Allow-Origin: http://www.nczonline.net
Access-Control-Allow-Methods: POST, GET
Access-Control-Allow-Headers: NCZ
Access-Control-Max-Age: 1728000

Once a preflight request has been made, the result is cached for the period of time specified in the response; you’ll only incur the cost of an extra HTTP request the first time a request of this type is made.

Firefox 3.5+, Safari 4+, and Chrome all support preflighted requests; Internet Explorer 8 does not.

Credentialed requests

By default, cross-origin requests do not provide credentials (cookies, HTTP authentication, and client-side SSL certificates). You can specify that a request should send credentials by setting the withCredentials property to true. If the server allow credentialed requests, then it responds with the following HTTP header:

Access-Control-Allow-Credentials: true

If a credentialed request is sent and this header is not sent as part of the response, then the browser doesn’t pass the response to JavaScript (responseText is an empty string, status is 0, and onerror() is invoked). Note that the server can also send this HTTP header as part of the preflight response to indicate that the origin is allowed to send credentialed requests.

Internet Explorer 8 doesn’t support the withCredentials property; Firefox 3.5, Safari 4, and Chrome all support it.

Conclusion

There is a lot of solid support for cross-domain Ajax in modern web browsers, yet most developers are still unaware of this powerful capability. Usage requires just a little bit of extra JavaScript work and a little extra server-side work to ensure that the correct headers are being sent. IE8’s implementation lags a bit behind the others in terms of allowing advanced requests and credentialed requests, but hopefully support for CORS will continue to improve. If you’d like to learn more, I highly suggest checking out Arun Ranganathan’s examples page.

Update (25 May 2010): Fixed typo in example code.

Update (27 May 2010): Removed trailing slash from Origin headers.
Related posts

• Cross-domain XHR removed from Firefox 3
• Firefox 3.5/Firebug XMLHttpRequest and readystatechange bug
• Mentioned in Microsoft whitepaper
• XMLHttp Requests For Ajax
• Firebug
• Web definitions: DOM, Ajax, and more

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

Not too long ago, I wrote about Internet Explorer 8’s browser mode and document mode settings. The goal of the post was to familiarize everyone with the various modes that IE8 can run your document in. Since that time, I’ve had ample opportunity to play with the browser and figure out a bunch of quirks related to this functionality. I’ve seen IE8 send the wrong user-agent to the server a handful of times, messing up server-side user-agent detection. But the most interesting issue I’ve come across has been the site compatibility view list.

What is it?

When Microsoft announced that IE8 would run in a new “super standards” mode by default, it created a lot of fear that this would “break the web.” The argument was that there were web sites/applications that were created for use specifically with IE7 (or worse, IE5) and that these sites would break as soon as the user upgraded to IE8.

To mitigate this concern, Microsoft took several steps:

1. Introducing browser and document modes so that IE8 was capable of acting like IE7 and IE5.
2. Developers can add the X-UA-Compatible HTTP header or <meta> tag to specify that the page should be rendered in a particular browser mode.
3. The creation of a site compatibility view list, maintained by Microsoft and delivered to IE8, that kept a list of sites known not to work in IE8 browser mode and therefore should be rendered in IE7 browser mode.

Numbers 1 and 2 covered those sites who were fully staffed and able to do work to prepare for IE8 compatibility. Number 3 is designed to help the rest of the web, the sites that aren’t actively maintained and don’t have resources to make any changes.

Where can I get a copy?

Prior to IE8 release candidate, Microsoft created a draft site compatibility view list based on pages that had trouble working in the browser. The list was fed by feedback buttons on pre-release versions of the browser as well as through comments and interactions with the web community. Over time, the list has been updated and changed as feedback was received.

You can download a copy of the site compatibility view list in Excel format from MSDN. The file contains details such as when the site was added to the list, when it was removed (if it was) and why, and more.

The site incompatibility list is downloaded as part of Windows Update. When you download updates from Microsoft, the list comes along with it.

How can I tell what settings my browser uses?

At any point in time, you can view the site compatibility view list that your browser is using by typing the following into your address bar:

res://iecompat.dll/iecompatdata.xml

If you’re ever unsure if IE8 thinks that a site should be run in compatibility view mode, double-check the list to see if it’s included.

What happens in the browser

When you visit a site that’s on the site compatibility view list, the “Compatibility View” button that is normally next to the Refresh button disappears. Since the browser has already determined that the site must use compatibility view and won’t work in IE8 mode, the button is hidden to prevent the user from accidentally putting the browser into a mode in which the page won’t work. This is actually the same behavior as if the X-UA-Compatible header is set.

Learn more

There is a lot of information about the site compatibility list and all things related to browser and document modes. The best place to start is MSDN’s, Understanding the Compatibility View List, which gives an overview of the feature. To understand more about why this is necessary, I’d recommend reading, Defining Document Compatibility.
Related posts

• Internet Explorer 8 document and browser modes
• Empty string URLs - browser update
• How Internet Explorer 8 document mode affects JavaScript
• History of the user-agent string
• Protect IE from empty img src
• XPath in JavaScript, Part 3

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

There is little debate over how important advertising is to the web. Web ads have made Google a powerhouse and everyone else envious. Their Adsense product made ads ubiquitous on even the smallest of web sites and yet people still don’t have a good understanding of ads are tracked, and therefore, how they make money. What follows here is a description of the modern web advertisement tracking paradigm. Keep in mind that this is a bit of a simplification, though the most important aspects are definitely covered.

Example: Amazon Associates links

Before getting fully into a discussion of how ads are tracked, it helps to have an example. Amazon has a great program called Amazon Associates which allows you to refer visitors to Amazon. If a referred user makes a purpose on Amazon, then the associates who sent them along gets a referral fee. The fee is typically pretty low, starting at around 4%, and increases as your referred visitors sell more. All of the book pictures on this site contain an Amazon Associates link, and many other sites do the same thing.

As an associate, you can decide what kind of link to show on your site. The simplest is a text link. Here’s an example text link, pasted from the Amazon Associates site, for High Performance JavaScript:

<a href="http://www.amazon.com/gp/product/059680279X?ie=UTF8&tag=nczonline-20&linkCode=as2&camp=1789
&creative=390957&creativeASIN=059680279X">High Performance JavaScript (Build Faster Web Application Interfaces)</a>
<img src="http://www.assoc-amazon.com/e/ir?t=nczonline-20&l=as2&o=1&a=059680279X" width="1" height="1"
 border="0" alt="" style="border:none !important; margin:0px !important;" />

Note that there are actually two tags in this code: an <a> tag and an <img> tag. You may be wondering why an image is present in what should be a text link. Take a look at the styles applies to the image: it’s a 1×1 image with no border and no margin, which effectively keeps it at 1×1 no matter what styles are on the page. What gives?

Makeup of a web ad

This is actually a very common occurrence in the world of web advertising. Each ad is typically made up of two portions:

1. The payload, which is the markup displayed to the user. This might be plain text, an image, a Flash movie, or other.
2. The tracking beacon, which is used to determine when the payload was displayed. When an image is used, this is frequently called a “tracking pixel.”

In the case of the Amazon Associates link above, the payload is the text link and the image is a tracking beacon. Without the image, Amazon has no idea how frequently the given link is displayed

Most ads use images this way for a number of reasons. First, the image is virtually guaranteed to be downloaded if the payload is displayed due to the proximity of the image markup to the payload markup. The ad server returns an image that cannot be cached (through the use of HTTP headers) and so the image is requested every time the ad markup is parsed. Because it’s simply a 1×1 image, it doesn’t take a long time to download and it doesn’t really affect the layout of the page.

Second, images can be loaded from a different domain than the rest of the page, allowing for different cookies to be and read. Note that the Amazon Associates example uses two different domains: one for the text link and one for the image.

Tracking ads

There are many metrics used to determine both the cost of an ad and how successful the ad really is. Big companies like Google are obsessed with these metrics as they are so tied to revenue.

Impressions

Ad impressions, also called ad views, are the number of times that each ad is displayed to the user. This is precisely the measurement for which the tracking beacon is used. Each time a request is made to the tracking beacon, it means an ad was displayed to a user and so the impression count for that ad is incremented.

Impressions are important because they are typically used to determine the cost of ads. It’s quite common to sell ads based on a thousand impressions (called CPM or cost per thousand - “M” is the Roman numeral for a thousand). Because impressions are so closely tied to cost, it’s imperative that this number be as correct as possible. Tracking beacons are the current state-of-the-art to assure accurate impression count.

Clickthroughs and clickthrough rate

A clickthrough occurs when a user clicks on the given ad. Clickthroughs are tracked by the link in the ad, so in the Amazon Associates example, a click is tracked through the amazon.com domain name based on the extra query string parameters. A clickthrough means that the user arrived on the ad’s destination site and, presumably, is ready to spend money on the product.

Clickthrough rate, often called CTR by businessfolk, is the measure of how many users who saw the link actually clicked on it as a percentage. In order to calculate this number, you divide the total clickthroughs for a given period by the total impressions for the same time period.

Clickthrough rates are typically used to measure the success of an advertising campaign. Most ads clickthrough at a very low rate (0.5% or less, source: 8 ways to improve your clickthrough rate), so an ad that clicks through at a rate of 2% is considered a big success. Advertisers frequently use the known clickthrough rate of a given ad spot combined with CPM to determine whether it would be a good idea to buy an ad in that position or not.

There are pay-per-click (PPC) advertising campaigns, as well, for which the clickthrough count and clickthrough rate are very important.

Page views

Another important metric that is frequently tracked are the page views of the page on which a given ad is displayed. This has to be done by the site that is hosting the ad, since the ad provider has no way of knowing this data. A page view is typically counted when the page begins to output markup from the server. In effect, once the opening <html> tag is sent, a page view is counted.

Page views are important because they are a checkpoint for how many impressions to expect for a given ad. For example, if there are 1,000 page views for a single page during a specific time but 100,000 ad impressions, that could mean something is very wrong. Basically, you’d like to see one ad impression for each page view, though in reality these numbers rarely ad up.

Metrics mixups

Page views and ad impressions are usually tracked together to determine what a “normal” relationship looks like between the two. For some, a normal relationship may be 80% of page views have an ad impression, where for others it may be 90% and still others may be lower or higher. The numbers don’t typically add up to 100% for a few reasons:

• Page abandonment. There are times when the user leaves a page while it’s still loading, so a page view is counted but an ad impression is not. This can happen for any number of reasons: server error results in the page not completely rendering, a network connection is lost so the page isn’t fully served, the user clicks on a link at the top of the page before the rest of the page is rendered, etc. Since the page view is counted right at the beginning but the ad impression isn’t counted until the ad markup is returned, a page view without a corresponding ad impression is considered to be an “abandoned” page.
• Browser cache. It’s possible that the page may be served from the browser’s cache. This happens most frequently when someone clicks a link on the page and then hits the back button. In that case, the page is served from the browser’s cache and so does not generate a page view (remember, page views are measured by the server that is serving the page). Because the tracking beacon is not cached, it will still fire and count as an ad impression, so you have the opposite scenario of page abandonment: an ad impression without a page view.

These two circumstances tend to happen with a predictable rate for any given page on the web, so the ratio of page views to ad impressions tends to remain constant even as traffic increases and decreases due to seasonality (different browsing behaviors based on the time of year). Big traffic increases or decreases may change the ratio. A big change in the ratio without an accompanying big change in traffic means that there’s a serious problem with ads.

Conclusion

Web ads aren’t going away anytime soon, and at some point you may need to understand how they work. What I’ve shared here are the basics that you need to know in order to grasp the concept of web ads and tracking. At some point, you may be asked to help debug some ads tracking and it’s my hope that this post serves as a good jumping off point to help those who end up in that position.
Related posts

• Empty string URLs - browser update
• Empty-string URLs in HTML - A followup
• Protect IE from empty img src
• Empty image src can destroy your site
• The IE6 support problem
• And so the HTML 5 trouble begins

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.

Last year, I started digging into the various client-side data storage alternatives that had popped up as a result of HTML5. These included sessionStorage, localStorage, and client-side databases. Though I was glad to see movement in this direction, I was unhappy with some of the results. I started my exploration around sessionStorage, and even though it is severely limited, I thought it was very useful and nicely wrapped up some of the security issues related to storing data client-side (see the full blog post for more). I was not, and still am not, a fan of SQL on the web as the solution of structured data storage, and I’m glad to see the folks at Microsoft and Mozilla moving in another direction.

That being said, I started looking at localStorage. Truly, this is a grand idea: a persistent storage area shared by all browser windows (or tabs) and tied to a specific domain. I know there’s a lot of dislike amidst browser vendors around this feature due to the complexities of cross-process data management, but my problems with the API have to do with how little control one has over the data.

The problems

There are two major problem the data storage mechanism in localStorage:

1. The data is stored on unencrypted on disk. That means anyone with access to the computer can potentially get access to that data.
2. The data remains on disk until either the site removes it or until the user explicitly tells the browser to remove it. That means the data may remain on disk permanently otherwise.

These are problems because they both increase the likelihood that the data can be examined by those for whom it is not intended.

Suppose I’m running one of the major webmail clients and would like to improve the site’s performance by storing information about the customer emails in localStorage. That way, you can speed up the site’s startup time and only download the new email information. (By the way, this is a really bad idea, please don’t do this.) Now suppose you log off and close the browser. Your email data is still saved on disk because the webmail client didn’t delete it when you left. Not a big deal if it’s your personal laptop; huge deal if it’s a computer in a cybercafe. Imagine if in that cyber cafe, twenty other people end up using the same computer to access the same webmail client and all of their data ends up stored on disk when they leave. Big problem.

You may be asking yourself, “wouldn’t encrypting the data solve that problem?” Yes and no. You could suggest that localStorage always encrypt data when it writes to disk but then it would end up being standard encryption algorithm and standard key. While this would provide a bit of a moat around the data, it would also be easy to figure out the browser’s choice in cipher and key, forcing browser vendors to either be incredibly clever in how they encrypted data to disk or to change the data storage method frequently. (Imagine if someone figured it out and posted the details on the web, there would have to be a mad rush to update the affected browser to ensure secure data.)

Don’t get me wrong, for publicly available data, there’s no reason not to use localStorage. But for anything even remotely personal to the user, you’re placing personal data into an area that is too easily accessed.

The solution

I don’t believe that there’s a clear path forward for localStorage to make it more secure. It’s out there, people are using it, and changing the API now would be a huge problem. When I brought these issues up at the Mozilla Summit on data storage, what I heard most frequently was, “if you can think of some way to solve this, write it up and we’ll talk.” And so I sat down and wrote a proposal for secure key-value storage in browsers called SecureStore.

The proposal is based on a few simple concepts that are shared amongst security-conscious companies:

1. User data should not be stored on disk unencrypted.
2. Even when user data is stored encrypted, the company must control the encryption algorithm and key.
3. User data, even when encrypted, should not persist on disk forever.

These rules have traditionally applied to servers and server-side caches, but seems logical enough to extend to client-side data storage in browsers.

I tried to keep most of the API similar to the already existing client-side data storage APIs so as to not introduce something totally different. One big difference, though, is the way in which you access a storage object. To do so, you must call the openSecureStorage() method and pass in an encryption cipher, a base64-encoded key, and a callback function that will receive the storage object:

window.openSecureStorage("mystorage", window.AES_128, key, function(storage){
   //use storage object
});

This code will do one of two things. If the storage area named “mystorage” doesn’t exist, it will be created and the given cipher and key will be used whenever data is written to it. An empty SecureStorage object is then passed into the callback function. If the storage area does exist, then it is opened, the contents decrypted, and the data is made available on the SecureStorage object. Note that the storage areas are tied to a domain, and there is no limit on the number of storage areas for a particular domain (only a limit on the total amount of space a domain can use).

Once you have a SecureStorage object, you can use the length property to determine how many key-value pairs are available, and all of the standard storage methods are also there:

• getItem(key) - retrieves the value for the given key or null if the key doesn’t exist.
• setItem(key, value) - sets the value for the given key.
• removeItem(key) - removes the key completely.
• key(position) - returns the key for the value in the given numeric position.
• clear() - removes all key-value pairs.

Note that you must use getItem(), setItem(), and removeItem() for manipulating keys; keys don’t automatically become properties on a SecureStorage object. Other than that difference, you use a SecureStorage object the same as you would sessionStorage or localStorage. Also, both the keys and the values are encrypted on disk.

An additional method called setExpiration() is present on the SecureStorage object as well. This method allows you to pass in a Date object indicating when the data should be deleted. For example:

window.openSecureStorage("mystorage", window.AES_128, key, function(storage){

    storage.setItem("username", "Nicholas");
    storage.setItem("super_secret_value", "unicorn");

    //set expiration for a year from now
    var expires = new Date();
    expires.setFullYear(expires.getFullYear() + 1);

    storage.setExpiration(expires);
});

You can set the expiration date any number of times to extend the life of the data.

The API is purposely made a bit generic, so that it’s possible to add additional encryption ciphers easily and to allow the developer to control from where the encryption key is generated. This may be done by the server in some cases, or potentially from some as-yet-undefined API that browser vendors will create in the future. The point is to allow easy extension as web technology continues to evolve.

Why?

One of the most frequent questions I get about this proposal is whether it would be better to create a general JavaScript crypto API that could be used in conjunction with localStorage rather than creating an entirely new data storage solution. First, I’ll say that I think a native JavaScript crypto API would be great and I’m all for it. What I’m looking to avoid, however, is needing to write code like this:

//write name and value so they're both encrypted
localStorage.setItem(AES.encrypt("username", key), AES.encrypt("Nicholas", key));

//retrieve the encrypted username
var username = AES.decrypt(localStorage.getItem(AES.encrypt("username", key)), key);

I’m not sure if this looks as messy to you as it does to me, but it seems like this is a common enough pattern that having a native implementation that prevents us from writing such horrid code is a good idea.

Let’s make this real

There are a lot more details on the full proposal, but I wanted to give some highlights in this post. I’ve received favorable feedback from at least one browser vendor on this proposal, and now I need help to make this real. What I really need is more feedback from people. I’ve already picked the brain of coworkers, and now I’d like to open it up to the public. What I’m interested in:

• Implementers: is there anything about this API that makes it too difficult to implement?
• Web developers: Do you have a use case that this would address?
• Web developers: Is there anything you’d change about the API?
• Everyone: Anything else?

If you’re a contributor to an open source browser, I’m also looking for someone that’s interested in prototyping this API for use in WebKit and/or Gecko. Feel free to contact me if you’re interested or have other feedback that you don’t want to post publicly.
Related posts

• Cookies and security
• Determining if an object property exists
• Cross-domain Ajax with Cross-Origin Resource Sharing
• iPad web development tips
• Custom events in JavaScript
• Maintainable JavaScript: Don’t modify objects you don’t own

P.S. My new book, High Performance JavaScript is now out! If you want more than performance, check out Professional JavaScript, 2nd Edition.