On Monday I finally got around to open sourcing my Lebowski framework that I built to perform automated feature and integration testing of applications built on the SproutCore framework. If you’re interested in trying the framework, please check out http://github.com/FrozenCanuck/Lebowski.

The framework was primarily designed with the intent to easily write test scripts that resembled how an application was built with SproutCore. This compared to testing a SproutCore application using traditional web-based test automation frameworks that focus on interacting with a web application using XPath statements, explicit HTML DOM elements, and low-level JavaScript. That’s not to say those frameworks are wrong, they just happen focus on a different domain of how traditional web applications are built.

and another one

Another great thing from the official SpoutCore blogs ... keep it comming, guys!

looking good :)

Recently I have seen some confusion about DataStore and granularity of its usage in Sproutcore google group. Since I have been playing with this lately and usually to tidy things up in my head, it is best to put thoughts on paper, here is my take on the usage of DataSources.

Before we begin, let's just say, that this article is more trying to answer the why, than how. I assume, you are familiar with at least basic design of SC.DataSource and interaction with SC.Store. If you are not, go, read the docs. This part is covered in great detail on Sproutcore wiki.

The question

The question that sparked my mind to think about this a little bit more was "should I use one data source for record type?" When I first saw that question I sat back and wrote a response to the discussion roughly along these lines "No, use a data source instance per back end type". When I revise that answer again, I think, that more appropriate one could have been given. Sproutcore gives you little constraints about how to write, or not to write, a data source implementation, but as usual in programming, the objective should be elegant code, that will solve the problem at hand. If you write less code, there is less likelihood, that it will break and maybe it is even going to be elegant. So the best answer to the question posed in the beginning I was able to put together is "Write your data source in way, that allows you to abstract common patterns in a certain set of models". This is not implying anything about number of back end servers or number of record types per data source. But to prevent this notion to bring more disarray into your understanding, let's have a look at some examples.

The ideal case

Though it never happened to me, I have heard, that it can actually happen. Imagine, you are writing an application and are free to design both server and client architecture. No legacy apps, no constraints. The application is not that important, but let's say, that it is something involving people, projects and tasks. Each project has several tasks and each person can be assigned to one project. There might be several people working on one project. Let's further assume, that the data coming from the REST API will look like this.

# Person at /people/1
{
    "id":           "/people/1"
    "first_name":   "John",
    "last_name":    "Doe",
    "age":          42,
    "project":      "/projects/1",

}

# Project at /projects/1
{
    "id":           "/projects/1"
    "name":         "Project X",
    "members":      ["/people/1", "/people/2"]
    "tasks":        ["/tasks/1", "/tasks/2"]
}

# Task at /tasks/1
{
    "id":           "/tasks/1"
    "name":         "Make it awesome",
    "project":      "/projects/1"
}

# Also, on uris /projects/, /tasks/ and /people/ there are expected arrays of specific objects

I hope, that these patterns I was talking about just a while ago is already started waving at you from the screen. These three types of objects look very similar. They are already flat, without any unnecessary stuffing, ready to be loaded to the store. Let's try writing a data source, that will tackle all these three resources in one go.

// The method used fof actual xhr request, used by all GET related requests
_getFromUri: function(uri, options) {
  var notifyMethod;
  notifyMethod = options.isQuery ? this._didGetQuery : this._didRetrieveRecords;

  SC.Request.getUrl(uri)
    .set('isJSON', YES)
    .notify(this, notifyMethod, options)
    .send();
  return YES;
},

// Method called by store, when retrieveing query, either local or remote
fetch: function(store, query) {
  options = {
    store:    store,
    query:    query,
    isQuery:  YES
  };
  if (query === Md.PEOPLE_QUERY) {
    return this._getFromUri('/people', {type: Md.Person});
  }
  return NO;
},

// called when query is returned from server
// basically, you just load the records, or their ids and notify store, the you have finished
// there are three ways, you can handle the query, either it is local or remote
// remote can be loaded with fully records or just with keys

_didGetQuery: function(response, params) {
    var store     = params.store,
        query     = params.query, 
        type      = params.type,
        deffered  = params.deffered;

    if (SC.ok(response)) {
      if (query.get('isLocal')) {
          console.log("fetch local");
          var storeKeys = store.loadRecords(type, response.get('body'));
          store.dataSourceDidFetchQuery(query);
      } else if (deffered) {
        console.log("fetch remote deffered");
        var storeKeys = response.get('body').map(function(id) {
          return Md.Person.storeKeyFor(id);
        }, this);
        store.loadQueryResults(query, storeKeys);
      } else {
        console.log("fetch remote");
        var storeKeys = store.loadRecords(type, response.get('body'));
        store.loadQueryResults(query, storeKeys);
      }
    } else store.dataSourceDidErrorQuery(query, response);
},

// Method called by store when requesting individual record
retrieveRecord: function(store, storeKey) {
  this._getFromUri(store.idFor(storeKey), {
    storeKey:       storeKey,
    store:          store,
    type:           store.recordTypeFor(storeKey)

  });
  return YES;
},

// handler called when record returns from server, this method will deal with individual records as well as arrays of records od the same type
_didRetrieveRecords: function(response, params) {
  var store = params.store,
      type = params.type,
      data;
  if (SC.ok(response)) {
    data = response.get('body');
    console.log(data, type)
    store.loadRecords(type, data.isEnumerable ? data : [data]);
  } else store.dataSourceDidError(storeKey, response.get('body'));
},

That is it. Let's go through some interesting points of this particular design.

1. The design is agnostic to server URI space. Since the URI is provided by the server, client doesn't have to care. If this will ever change (which in my experience is happening more often than I would ever guess) (say /people/ to /people_space/employees/), this is one less place to care about.
2. The only thing, that you have to know on the client is an entry point to your application. In my case it is a PEOPLE_QUERY, which causes the server to return all people (yes, this is going to be very app/view specific, but there should be only certain not too big number of those entry points). All subsequent requests are initiated based on relations, thus the expected record type is known as is the URI, which is guid.
3. You can decide if remote queries will be downloaded instantly, or on demand, with one flag change on the client (of course you have to deal with it on the server)
4. Imagine, that new record type is introduced, say Note. Every task can have several notes. Now you have to do this
   1. Add new Note model to your app, alter Task to have a relation
   2. Design new REST API for notes and alter the one for tasks
   3. Crete your controller/view layer

No mention about data source in point 4? Yes, as long as you will stick to the same rules, you used, when you created your other REST resources, there are no needed changes on the data source layer. Everything will just work, so you can clearly see, that we benefit from handling several models in one data source. The Note resource could look for example like this

{
    "id":   "/notes/1",
    "text": "Some note text"
}

Hands on

If you want to play with this live, I have prepared a sample app, which can be cloned here. Though it does not have a UI, it is pretty fine for some tinkering. Part of the example is fake back end server written in Sinatra, so I expect, you have sinatra installed. The server itself is in subdirectory server and you can run it with command

ruby demo.rb -p9393

Don't run it with shotgun or any other tools, that will force it to reload the source code on every request. This way, you would be able to save the stuff back to server and it will be persisted until you restart the server. In the example, the data source is the same as described above, moreover I have implemented one other method, so you can play with updating data to server. Sorry no creation or deletion, you can do that on your own. To get you up to speed, try

l = Md.store.find(Md.PEOPLE_QUERY)
p = l.objectAt(0)
p.get('firstName')
r = p.get('project')
t = r.get('tasks')
t.get('length')
t1 = t.objectAt(0)
t1.get('what')

Play with it, change it, break it.

Multiple responses

Let's show one great advantage of the Sporutcore data store/data source architecture, that cannot be praised enough. I will illustrate it with another example. Let's say, that your hosting provider is a %^&*(%$#, he charges 100$ for each request to the server. Instead of going away to another provider, you decide to leverage great power of Sproutcore and load all your data in one request. You can afford to do this, because in your application there are not so many people/projects/tasks. So how, you would write this in the code? First the server side. Sinatra backand is already at your service and on uri /all awaits a collection of data, that looks like this.

{
    "person": [
        {...}
    ],
    "task": [
        {...}
    ],
    "project": [
        {...}
    ]
}

Notice, that the keys in response are not arbitrary. They are chosen, so that you can infer the record type class out of them. Now go to the client code and change these methods

// Just changing the URI here, all the same otherwise
fetch: function(store, query) {
    options = {
      store:    store,
      query:    query,
      isQuery:  YES
    };
    if (query === Md.PEOPLE_QUERY) {
      options['type'] = Md.Person;
      return this._getFromUri('/all', options);
    }
  return NO;
},

// gets the class of the record based on the key of data
// and then load records
_loadRecordsOfType: function(type, data, store) {
  var recordType = type.capitalize();
  store.loadRecords(Md.getPath(recordType), data);
},

// iterate over the keys in response
_didGetQuery: function(response, params) {
    var store     = params.store,
        query     = params.query, 
        type      = params.type,
        deffered  = params.deffered,
        data      = response.get('body'),
        storeKeys,
        recordType,
        recordData;

    if (SC.ok(response)) {
      for (recordType in data) {
        if (data.hasOwnProperty(recordType)) {
          this._loadRecordsOfType(recordType, data[recordType], store);
        }
      }
      store.dataSourceDidFetchQuery(query);
    } else store.dataSourceDidErrorQuery(query, response);
},

Again, let's break down some key features here

1. You created loader, that will inspect the result and load records of multiple types.
2. Since you used great power of Convention over configuration paradigm, you can add new record types (try with the Note object from above) and it will again work, without any code changes, since the record type is inferred from the result itself
3. You can afford this because the insulation, that is between data store and data source. Your application does not care how and when the data gets into the store, so data source is free to decide. You can evolve these two parts of an application without any dependency on each other. Very powerful.

Wrap up

Well, this was a little stupid example, but you can hopefully see, that you can get smart about your data in many interesting ways, without affecting your other layers of application. As long as you will keep to the rules, you can get done a lot in just couple lines of code and benefit from similarities in your resources by tackling several models in one place. In the instant, you will start breaking those rules, you will need to add tons of conditional logic into your data source implementation, because you will have to deal with discrepancies and little inconsistencies in your data. That is a good sign, that you probably should start thinking about rearranging your data source differently, maybe even break it into pieces and create another one to deal with the odd parts.

In next article, we will do exactly that. We will tackle not so nice situation with legacy app thrown on your back. We will also look into cascading data sources and maybe more.

• Home
• Download
• Wiki
• Screencasts
• Blog

• Archive
• 

  RSS Feed

SproutCore Blog

A Universal Package Manager for JavaScript

April 19

A Universal Package Manager for JavaScript

Thought of as a “toy language” for too long, JavaScript is coming into its own. Today JavaScript is used in major duty applications in both the client and server environments.

The problem is, compared to our power-scripting brethren, our tooling support has never caught up.

Running small JavaScripts in the browser is reasonably easy today, but anything beyond that goes downhill fast. A lot of basic things that other scripting environments take for granted simply do not exist for us.

Don’t Get Me Started

Take for example, packages. Both Ruby and Python make it insanely easy to create a library and share it with other people. For most Rubyists or Pythonistas, building a new product or feature usually begins with a search of their package system for a library that gets them close. Once found, one command will install the library to make it ready to use.

JavaScript? No such luck.

First, search Github. If not found there, search Google. Should you find a library, then you need to figure out how to integrate it into your own JS system. Sometimes it is just a script you can drop in. If it is bigger than that, you may need to get a build tool (which is probably written in Ruby or Python anyway) and use that to generate the file.

Once you have the script, of course, tracking updates is just as hard. Usually it involves visiting a website periodically just to look for updates then going through the integration process all over again.

Even just running a JavaScript from the command-line is a total nightmare these days. For most languages, it’s as simple as:

#!/usr/bin/env ruby

And you’re off to the races. The full power of the standard Ruby or Python libraries are at your disposal. If you don’t like those libraries, no problem! A quick search and install from the package manager and you’ll have something new in no time.

Meanwhile in JavaScript land…first you need to pick a JS runtime. Usually you’ll end up repurposing a JavaScript server like narwhal or node.js. Each has a totally different API, by the way.  Whichever runtime you choose will stick you for the life of the project without major transition pains.

What a mess.

Where Giants Fall

Now, to be fair, some really smart folks have been working on some projects that at least partially solve this problem for some time now.

The CommonJS standard has a great module pattern that is a key building block to solving these problems.  It also has a concept of packages that is going in the right direction but each project seems to interpret it so differently we can’t rely on it globally.

There are, in fact, several implementations of CommonJS available, including node.js and Narwhal among many others.  They even have some package managers designed for their respective platforms.

Easily sharing code isn’t a primary purpose of this project, however.  They come with a bunch of added (different) APIs and subtle incompatibilities as well.  When these things break your code, no one is generally in a hurry to fix it.

Which is totally fine, by the way. That’s how open source works. You scratch your personal itch. If someone wants to do something with your project that isn’t part of your personal charter, you don’t do it.

A Call To Arms

The point is that for JavaScript to have a universal set of tools for sharing and executing code, it needs a project dedicated to that purpose. Specifically it needs to have the following goals:

1. Provide a CommonJS runtime environment that works in both the browser and multiple JS “runtimes”.

2. Be as flexible as necessary to help you adapt older JavaScript code to work with CommonJS while maintaining compatibility.

3. Make it insanely easy to create, share, install, update, and fork JavaScript libraries for both the server and the client.

4. Keep the API as focused on sharing code as possible. All other APIs (file io, browsers, etc.) should be exposed as packages that can be dynamically swapped out.

Basically, we want a tool that is focused on making it easy to adapt and share new and existing code to run in browser, server, and from the command line. It should not take a position on other APIs (such as sync vs async). It should be small, focused, and simple as possible.  Ideally it should run on any JavaScript runtime, not just one or the other.

Seed.js

From my perspective, this project does not yet exist.  I think it is important though, so over the last few months I built one. It’s called seed.js.

Seed.js is a flexible package manager for CommonJS modules. (A CommonJS module is a single JavaScript file, a package is a group of one or more related modules that you install together.) It implements three things:

1. A package-aware CommonJS module runtime that works in both the web browser and most “server-side” runtimes (such as node.js or narwhal). This allows you to write your CommonJS modules to a single standard.

2. A command-line tool (called “seed”) that can install, update, remove, and fork packages.

3. A JavaScript-runner that will execute JavaScript using a chosen runtime (e.g. narwhal, node.js), automatically loading seed on top of it.

Virtually every aspect of both the CommonJS runtime (called “tiki”) and the seed command line tool itself is configurable via simple API’s. The idea is to give you lots of ways to enhance the Seed environment itself when adapting existing code so you can make the code run without having to break backwards compatibility.

The point is that seed’s purpose is to be focused and flexible - to adapt to the needs of the community so that we can have one universal packaging system for use across the board.

Current Status

Right now, seed is usable but not finished. We showed seed.js for the first time publicly yesterday at jsconf 2010. It is available now for you to play with at seedjs.org. To get things rolling we have pre-loaded the seed server with a dozen or so packages.

Currently, seed runs on node.js and stores packages on a JavaScript-based server hosted at seedjs.org. When forking packages, seed uses git. But these were just our starting points. All our platform dependencies are kept in a single place so porting from node.js to another runtime should be straightforward. You can target different server backends just by writing a plugin. Ditto for adding support for new SCMs.

As far a projects go - both the Mozilla Bespin team and SproutCore have been using the Seed CommonJS runtime (aka “tiki”) for several months now and we’re really happy with it. In fact, both SproutCore’s ‘runtime’ and ‘datastore’ frameworks have been converted to CommonJS and will soon be available as packages in the seed repository.

…

If you would like JavaScript to become as easy to use as Ruby or Python, please go give seed a try today. Install some packages, write some code. Create a seed of your own and publish it. Then, join the mailing list or join us on the IRC channel at #seed.js on freenode.net and give us your feedback.

Right now seed is new so just about anything can change. The goal is to end up with a universal set of tools that makes JavaScript a better platform to work with - in the browser, in the server, and direct from the command line. If you want this future too, download seed and let me know what you think.

2 Comments and 0 Reactions - Permalink

blog comments powered by Disqus copyright 2008-2009, Sprout Systems, Inc. All rights reserved. Found a problem with the site? No problem. It's open source!

Greenhouse ... yeah!