How To Build Testable and Framework Agnostic Single Page Applications with Node.js

In a previous blog post, I spoke about the package principles of cohesion:

  • The Release/Reuse Equivalency Principle
  • The Common Closure Principle
  • The Common Reuse Principle

In this blog post, I will use the principles of package cohesion to build cohesive npm packages, and I will use these npm packages to build testable and framework agnostic single page applications.

For this demonstration, we will create a fake analytics dashboard. Here is what the final product will look like.

Screen Shot 2016-01-31 at 10.14.36 PMGathering Requirements

We will describe the application requirements using the “User Story Mapping” approach.

The following snippet describes the feature of our application

We want our “system under test” to be testable; so, we need to place some architectural restrictions to guarantee “testability”.

At a minimum, a testable system must have the qualities of (a) observability, and (b) controllability.

Observability refers to the property that I can arbitrarily observe the states of the “system under test”, and controllability refers to the property that I can put the “system under test” into an arbitrary state.

The two properties are inherently related to each other: you can’t fully observe a “system under test” that you can’t fully control, and there we don’t care to fully control a “system under test” that we can’t fully observe.

I claim (but will not prove) that these architectural restrictions guarantee “testability”.

In order to support the “controllability” requirement of our architecture, we will use the following background block to load fake data into our “system under test” for every new test run.

 

By design, a user story will have various scenarios, and we will use the scenarios to determine what needs to be “observable”.

For this demonstration we will consider the following scenarios:

  • Visit the visualization page
  • Change the selected date on the page
  • Aggregate the items by month
  • Aggregate items by year

This snippet shows the scenario description for “visit visualization page”

This snippet shows the scenario description for “change the selected date on the page”:

This snippet shows the scenario description for “aggregate the items by month”:

This snippet shows the scenario description for “aggregate items by year”:

You can see the complete feature file here.

Testing the User Story

The following code snippets uses cucumber.js and babel (an es6 to es5 transpiler) to test the user stories. I use babel because it lets me use generators and coroutines to simplify the asynchronous nature of selenium code.

This snippet initializes the “system under test” based on the data provided in the background block for the user story. This satisfies the “controllability” property of our architecture.

This snippet executes the action(s) that change the state of the “system under test” for the behavior “When I visit the page at <some_url>”:

This snippet executes the action(s) that change the state of the “system under test” for the behavior “When I select the date <some_date>”:

This snippet executes the actions(s) that change the state of the “system under test” for the behavior “When I select to aggregate by <some_aggregation_type>”:

This snippet executes the post conditions checks of the “system under test” for the block “Then I should see a bar chart for <some_title>”.

Note: I will skip the rest of the blocks for brevity’s sake.

In the acceptance tests, I use a page object to encapsulate the data retrieval algorithms for each page.

For example, the following code snippet shows how the page object encapsulates the algorithm that “observes” when the page has finished loading.

This test looks for a DOM element with the html attribute “qa-chart-type”, and then waits for it to change.

We have to do this because the SPA fetches data using a http request when if first loads, and we want to detect when the SPA receives a response.

However, since we cannot easily detect when the SPA receives an http response, we make the SPA communicate this to our test script by creating and updating special “Quality Assurance” html attributes.

This makes testing with selenium much easier and maintainable.

I also created a page object factory to easily change implementations of the page object. It uses the same principles of a “contract test” that I mentioned in my blog post “Contract Tests: or, How I Learned to Stop Worrying and Love the Liskov Substitution Principle”.  

Let’s suppose that we implemented our application using (a) Backbone.js, and (b) React.js.

In order to test a Backbone implementation, we could use the following code

In order to test a React implementation, we could use the following code

This is what running the cucumber application looks like when I test a React implementation:

screengrab

You can see the complete test suite here

Building Our Components

Suppose that you had the following directory structure for a JAVA application

  • src/
    • use_case001/
      • command/
        • File001.java
        • File002.java
        • File003.java
      • delegate/
        • File001.java
        • File002.java
      • model/
        • File001.java
        • File002.java
    • use_case002/
      • File001.java
      • File002.java
    • use_case003/
      • File001.java
      • File002.java
      • File003.java
    • controller/
      • Controller001.java
      • Controller002.java
    • view/
      • View001.jsp
      • View002.jsp
    • model/
      • Model001.java
      • Model002.java

This is an example of a “monolithic” or “big ball of mud” application architecture.

Architects typically have nothing but contempt for this architecture due to its well known rigidity, fragility, immobility, and viscosity.

These are well known application level architectural properties:

  • rigidity measures how easy an architecture can respond to change.
  • fragility measure how likely something will break when making a change
  • mobility measures how easy an architecture can support moving code
  • viscosity measure how easy an architecture supports maintaining the original design

When faced with a monolithic architecture, most architects I know would refactor the monolith into something that looks like this.

  • src/
    • controller/
      • Controller001.java
      • Controller002.java
    • view/
      • View001.jsp
      • View002.jsp
    • model/
      • Model001.java
      • Model002.java
  • lib/
    • use_case001.jar
    • use_case002.jar
    • use_case003.jar

Each .jar file is a self contained components that is independently testable, releasable, and versioned.

You might think that simply moving code from a directory to a .jar file does not significantly improve the design. However, when applied properly, it has a huge influence on reducing rigidity, fragility, viscosity, and increasing mobility.

We will use this approach, but in this case we will use npm modules as components: a npm module is equivalent to a JAVA .jar file; so, all the same rules apply.

I will use the discipline of Responsibility Driven Design to guide the cohesion strategy for our components. Responsibility Driven Design makes component “responsibility” the most important organizing principle for code.

I claim (but will not prove) that organizing our components by “responsibility” will force our components to obey (a) the Reuse-Release Equivalency Principle, (b) the Common Closure Principle, and (c) the Common Reuse Principle.

From the scenarios in the user story, I can identify the following responsibilities:

  • Bar Chart Visualization
  • User Interface Interaction
  • Data Retrieval
  • QA HTML Attribute Generation

Therefore, I will respectively create the four following packages:

  • analytics-chart
  • analytics-facade
  • analytics-service
  • qa-locator-utility

Create the “Analytics Chart” Component

The actual details of how I created the component are not important; so, I will not talk about it.

However, I feel that I should mention the unit tests I wrote for the visualizations because it isn’t a common thing.

I created unit tests to prove partial correctness of my application using the approach I describe in my previous blog post.

For example, the following code snippet tests the visualization for some items.

This test verifies that certain DOM elements appear in the svg canvas. I consider it a very weak test because there is a good chance that it could pass and the visualization could still be wrong. However, it is also a cheap test to write; so, there is still value in writing it.

You can see the completed npm package here.

Create the “Analytics Facade” Component

This component encapsulates all the user interface “glue logic” (i.e. identifying which screen or page to display), and we use the command pattern and delegation pattern to promote loose coupling between the user interface and this component.

For example, these are the cucumber features that I wrote for this component.

From a client’s perspective, they only need to know how to execute a command from the facade, and how to delegate tasks to the facade. The cucumber tests above provide a very high level description of what types of commands and delegates this facade exposes.

For the sake of completeness, I will also show you the step definitions for this feature.

You can see the completed npm package here.

Create the “Analytics Service” Component

This component simply makes a request to our remote server and return the json results. It isn’t very interesting; so, I will not discuss it.

You can see the completed npm package here.

Create the “QA Locator Utility” Component

Officially, we do not need this component, but we would like to have an easy method to locate elements in our window’s DOM using xpath. This component encapsulates any algorithms that we might need to simplify the process of creating html attributes for the purpose of writing acceptance tests.

I don’t consider it a very interesting component; so, I will not discuss it. 

You can see the completed npm package here.

Composing the Components Into A Single Page Application Framework

To demonstrate how to migrate the application between frameworks, I will compose the components into (a) Backbone 1.2, and (b) React 0.14.

The following component diagram shows the general relationships between the components and framework.

Screen Shot 2016-02-03 at 11.27.24 PM

Backbone 1.2 and React 0.14

We have placed the bulk of the applications functionality inside of framework independent components. Therefore, in order to create a backbone version or a react version of the application, we have to create a special component that coordinates between our “independent” components and the framework.

In the case of React, we created the component backbone_app.

In the case of Backbone, we create the component react_app.

For example, here is the entry point for our Backbone application:

Here is the entry point for our React application:

The parallels in the code are very apparent. I will not go into detail into the code bases because it is not important. I only care about the “shape” of the design.

You can find the Backbone implementation here, and the React implementation here if you really care to dig into the code, however.

Conclusion

This approach might seem very complicated for such a simple application … and it is.

In reality, this approach only makes sense once your application reaches a certain scale. However, simple applications have a tendency of becoming big applications, and it is very expensive to put a test harness on an already existing system with no tests.

Personally, I recommend you always build an application with decoupled testable components from the very beginning because that investment will pay large dividends in the long run.

Further, decoupling your components from the framework has huge benefits even if you have no intention of switching frameworks.

One practical “use case” is with automated testing.

When you decouple your components from the framework then you can test the bulk of your application independent of the framework. This significantly simplifies the overall testing process.

In fact, the package principles were partially created to minimize the work of creating and maintaining tests.

In a future post, I will show you how you can use a Continuous Integration tool to automatically run your tests whenever someone releases a new version of a component.

That should demonstrate the practicality of strictly following package principles.

Advertisements

Scalable Cross-Domain “Cookie” Authentication/Authorization: An Example with macaroon.js

Recently, cookie authentication has fallen out of favor in the web community for it’s inability to port across domains and scale.

However, there exists a method to use cookies in a distributed fashion that is both portable across domains and scalable: macaroons (it’s a better kind of cookie).

Google introduced the world to macaroons in 2014 with a white paper. Unfortunately, there does not exist much documentation or code examples for it; so, I thought I’d help fill that void with this blog post.

(Not) Introducing Macaroons

I will not talk about the theory of macaroons. You can use the following resources to get the theory:

A Basic Example

Suppose that you had a website that needs to authorize a user and authenticate their access before it can do anything. You probably would have the server send the user a cookie. The user would then send the cookie value to the server with each request, and the server would use that value to determine authentication and authorization details.

This has severe limitations because we cannot easily share that cookie across domains, and most attempts to do so usually introduce network bottlenecks. I will not go into the details of why this happens; you can easily google it.

Macaroons overcome these limitations by allowing you to put arbitrary restrictions on a “cookie”. Further, these “cookies” can include information from third parties.

This allows you to easily share them across domains and distribute them arbitrarily.

Let’s consider a simple example.

Suppose we had an application that we logically divided into (a) the browser, (b) the service provider, and (c) the identity provider.

The browser contains the code that powers user interaction, the service provider contains the code that powers data access and manipulation, and the identity provider powers user management of the system.

Suppose that the user wants to see his stock portfolio on a web page.

In order to satisfy this use case, our system needs to complete the following tasks:

  • Verify the user’s identity
  • Verify the user’s rights
  • Find the data
  • Show the data

The following sequence diagram shows how we can satisfy those tasks with the help of macaroons.

macaroon_authentication

I will use node.js and the macaroon.js module to power this example. I will only provide code snippets that are relevant to the most important elements of the use case.

Request Macaroon

For this example, the browser will kick-off the macaroon process by making a request to sp.domain.com for a macaroon. Let’s suppose that sp.domain.com exposed the url “/api/macaroons” for that

Generate Macaroon

The code to generate and return a macaroon could look like the following:

This code creates a first party caveat for the macaroon.

A first party caveat is an enforceable restriction on the macaroon created by the minter of the macaroon. In this case, we retrict this macaroon to a particular ip address.

The code also creates a third party caveat for idp.domain.com. A third party caveat is a restriction that we delegate to another party. In this case, we are requiring the recipient of the macaroon to pass the macaroon to the idp to authorize his requests.

The ability to delegate attenuation to third parties is the secret sauce for macaroons. With normal cookies the service provider would have to manage and coordinate all the services associated with authentication/authorization. However, with macaroons, we can simply delegate authority to a trusted third party.

For security reasons, we generate the caveat key on sp.domain.com and encrypt it using the public key for idp.domain.com. We have to pass the encrypted caveat key to browser along with the macaroon.

The following is the code that could handle the response from /api/macaroons.

The browser is responsible for coordinating the authentication for the macaroon. In this case,

the callback sends the encrypted caveat key and the macaroon to idp.domain.com along with a username and password.

Verify Third Party Caveats

The idp will verify the username and password of the user. If that passes then we can create a discharge macaroon and apply it to our original macaroon. When we create a discharge macaroon, we are satisfying the delegation request from sp.domain.com. In this case, we also assign a timestamp to the macaroon to enable revocation. We are telling sp.domain.com that this macaroon is only good until the time set.

We expect sp.domain.com to obey that restriction.

idp.domain.com will send the original macaroon and the discharge macaroon to the callback. We need them both because macaroon’s hmac chaining algorithm protocol requires them to verify the authenticity of the macaroon.  

The above code makes a request to sp.domain.com for the portfolio information. It also sends the root macaroon and discharge macaroon with the request.

Request Data

We know have enough information to verify the authenticity of the macaroon. I know that we have some handwavy helper functions that verify the authenticity. The particulars are not important, though. The verification details will always changed depending on the application.  

Conclusion

This is not production ready code. There is no error checking or edge case handling. This is just a bare bones examples of how you could use macaroons in a distributed environment.
We can easily scale this process up to an arbitrary number of servers. Hopefully, you’ll be able to use this as a starting point to building your own macaroons.

How to Unit Test Your D3 Applications


TL;DR: you can get the entire code at my repository.


I recently found a way to unit test d3 code, and it has transformed my approach to writing d3 applications.

I want to share this with the d3 community because they typically don’t emphasis unit testing.

Consider the following d3 code.

This code does the following:

  • append a red circle to a canvas
  • change the color of the circle to green when the user clicks on it.

Most of the time, we rely on our eyeballs to verify the correctness of d3 applications.

For example, this is what the canvas looks like when we D3 renders the canvas.

pic1

and D3 will render the following when the user clicks on the circle

pic2

In this case, we can manually verify that the code works.

Unfortunately, we can’t always rely on human eyeball testing.

However, with the jsdom module, we can programmatically verify whether the browser updated the DOM properly: we can (a) set expectations for our DOM, (b) write d3 code to fulfill our expectations, and (c) automatically run the tests with gulp.

First let’s setup our project.

Setup Project

Create a directory for our project, and initialize it as an npm package

Create a directory for our source code and our tests

Install the necessary npm modules

Create gulpfile.js

Create the d3 Application

Put the following code under src/Circle.js

The Circle module takes two arguments. The first argument is the dom element that we will append the svg canvas to. The second argument is the id we will give circle. We will need this id for testing.

Testing the d3 Application

Create the file test/Circle.spec.js with the following contents

So far we have not added any tests to the test suite. This is only boilerplate code that we will need to write our tests.

The function d3UnitTest encapsulates the jsdom environment for us. We will use this function in each test to alter the dom and test the dom afterwords.

Let’s create our first test.

Append the first test to our test suite.

In this test, we simply create check if the id “circleId” exists within the DOM. We assume that if it exists then the circle exists.

Notice the done parameter in our function. We do this to trigger an asynchronous test. jsdom will run asynchronously. We have to provide a callback to it; so, we manually have to call done() to inform jasmine when we are ready for it to test.

Append the second test to our test suite.

Append the third test to our test suite

This code simulates a click on the circle. You have to know a little bit about the internals of d3 to make it work. In this case, we have to pass the circle object itself, and the datum associated with the circle to the click function.

FYI, I couldn’t find a way to get jsdom to simulate user behavior; so, you have to understand the d3 internals if you want to test user behavior. If you know how to use jsdom to simulate user behavior then please let me know how you do it because this method is very error prone. 

Run the Tests

When we run gulp from the command line we should see the following

Conclusion

I understand that I gave a pretty simple and contrived example. However, the principles that I demonstrated apply to more sophisticated situations. I can vouch for this because I have already written tests for very complicated behaviors.

There is a downside to this approach. You have to use node.js and write your d3 code as node modules. Some people might not like to have this dependency. Personally, I don’t find it very inconvenient. In fact, it has forced me to write very module d3 code.

It is also extremely easy to deploy my d3 code using browserify with this method.
Your milage may vary with this approach. I’m sure that there are ways you can achieve the same result with browser based tools. For example, I’m pretty sure that you could do something similar with protractor.

How to Build React Applications with Node.js, Jasmine, Babel, and Gulp


TL;DR: you can find the entire code at my github repository.


Facebook’s React has completely changed the way that I write applications. From my perspective, most javascript frameworks unnecessarily force you into particular programming styles. For example, I often hear people tell me to do things the “angular way” (whatever that means).

This isn’t necessarily a bad thing. Programming standards enable team development. However, I believe that the software architect or the team itself should develop those standards. Framework designers do not know your particular needs and concerns.

In the end, framework designers try to solve their problems. They do not generally care to solve your problems. Many times our problems will overlap with their problems, but it becomes incredibly hard to development when they do not.

We do not always know what designs work at different stages in our software lifecycle; so, we need to have as much control over the design of our applications as possible

In my opinion, React has a much better design simply because it makes much less assumptions.

Consider Angular 1.x. While it is a great framework, it also has an all or nothing philosophy. It is very hard to use it’s data-binding feature without also using its controllers, directives, modules, etc …

React only concerns itself with views.

This makes it possible to create a customized architecture around my particular use cases.

I created simple todo application that demonstrates how you can do this.

I would like to use this blog post to demonstrate in a step-by-step way to build it.

First let’s describe the use case for the application.

Gather the Requirements

Suppose we have the following requirements for our todo application

Use Case 1.0: Add List Items

We expect to give a list item to the application. The application should store this list item, and provide visual feedback that it has stored it.

Use Case 1.1: Detect Invalid List Items

We consider an empty string as invalid input. If we give invalid input we expect the system to detect it and give some visual feedback to the user about it.

Use Case 2.0: Remove List Items

We expect the application to allow us to remove items. The application should provide some visual of all the list items, and it should expose some way to remove individual list items.

When we remove a list item, the application should provide some visual feedback about removing the list item.

Decide on an Tool Set

In order to build the application, we need to make some initial design decisions.

For this tutorial, I decided to build the application in the following tools:

  • React + Babel for views
  • Node.js for the main application
  • Jasmine for testing
  • Gulp for building the application

Setup the Project

I will assume that you are using linux.

Create a directory for the project

change to the directory

Initialize the directory as a node project

Install (a) gulp, (b) jasmine, (c) browserify, (d) eventemitter, (e) run-sequence, and (f) vinyl-source-stream with the following commands:

Create the following folder structure

  • src/common
  • src/model
  • src/view
  • test
  • app

with the following commands

We will put (a) our react code in the “src/view” folder, (b) our node code in the “src/model” and “src/common” folders, (c) our jasmine tests in the “test” folder, and (c) our application into the “app” folder.

Build the Model

Let’s encapsulate the algorithms that fulfill these uses cases into a single node.js module.

Create the following file: src/model/TodoListModel.js.

TodoListModel references another node file called Guid. By design, we force the caller of TodoListModel.addItem to provide a guid as an argument.

I did this to make unit testing easy. We will illustrate unit testing in the next section.

Create the following file: src/common/Guid.js

Create the Unit Tests

We need to validate that TodoListModel actually implements our use cases correctly. We will create separate jasmine specs for each use case.

Create Tests for Use Case 1.0

We want to verify that the model obeys the following constraints when we add list items:

  • The model adds the list item properly when we add one list item
  • The model adds list items properly when we add two list items
  • The model fires events that informs subscribers that the model has added a list item

Create test/AddListItems.spec.js with the following contents to validate that our model obeys the rules for “Use Case 1.0”.

Create Tests for Use Case 1.1

We want to verify that the model obeys the following constraints when we add add list items:

  • Detect when the caller does not provide a valid list item
  • Detect when the caller does not provide a valid guid
  • Verify that the model fires the proper events for an invalid input

Create test/DetectInvalidListItem.spec.js

Create Tests for Use Case 2.0

We want to verify that the model obeys the following constraints when we try to remove items:

  • The model properly removes one item
  • The model properly removes 2 of three items
  • The model fires the proper events when it removes items

Create test/RemoveListItems.spec.js

Execute Unit Tests

We will use gulp to execute our unit tests.

In the main directory, create gulpfile.js with the following contents

Change the NODE_PATH to include ./src. This will ensure that model/TodoListModel.js and common/Guid.js are discoverable.

Our gulp file will now execute our jasmine tests by default. Let’s run it and see the result.

Build the Gulp Workflow

Now that we have a working model let’s build the application. I want to have a very simple deployment. I would like the entire application to run using this html.

I want bundle.js to bootstrap the entire application for index.html.

Let’s modify gulpfile.js to support this.

This will browserify all our react code (which we have not created yet) and package it into bundle.js in the app folder. Further, gulp will use babel to transpile our react code into ES5 javascript. We want to use babel with react because it will provide us with a better syntax than standard javascript.

Build the Views

Create src/Main.jsx with the following contents.

This code shows that we have a UI with three general components:

  • A message component
  • An input component
  • A list component

Further, each component takes a TodoListModel instance as an argument to it’s constructor.

Let’s build out each component one at a time.

Create a MessageArea Component

Create src/view/MessageArea.jsx

Create InputArea Component

Create src/view/InputArea.jsx

Create a TodoList Component

Create src/view/TodoList.jsx

ListBox.jsx creates several ListItems components as children.

Let’s create the component definition.

Create src/view/ListItem.jsx

Build the Application

We can now run gulp.

This gulp build uses babel to transpile the ES6 code into standard javascript and browserfys it into bundle.js in app.

You can open index.html in a browser to execute the application. It should look something like this:

Screenshot - 09162015 - 06:20:49 PM

Conclusion

I know that this project is very artificial. I only want you to take away the principles. You ultimately have to make decisions for you applications. Architecting your projects like this may or may not make sense for you.

How To Automate Jasmine Tests for Typescript with Gulp

I have a gulp workflow that I’d like to share with the javascript community. This workflow automates the compilation of typescript and tests the generated code with Jasmine.

Why Another Workflow?  

I personally like to build my own workflows whenever possible.

I know that advanced editors like Visual Studio have tools that automate a lot. However, I feel that software development is too complicated for a single workflow to automate everything.

IMO, there are simply too many details in software development. At some point, you always need to do something custom. 

Enter Gulp

I feel that gulp has the best approach to creating workflows, currently.

Gulp has a massive collection of plugins that you can reuse for common tasks, and you can stitch various plugins together programmatically.

Creating the Workflow  

I will provide a step-by-step walk-through that shows how to use the linux command line to setup a project. You may not be able to use this process if you use something like Visual Studios to setup your typescript applications.

Create the Folder Structure

Create the following folder structure

  • src/ts
  • src/js
  • spec

with the following commands

We will put our typescript code in src/ts, our generated javascript code into src/js, and our jasmine tests into spec.

Setup the Project

Initialize the directory as a node project.

Install node plugins

Create a custom node module

We will need to take the contents of the generated javascript file that typescript produces and evaluate it in the jasmine files. In order to make this easier, we will create a helper module called __require.js in the root folder.

By design, we will place all generated javascript code under src/js. If we had an alternate location then we would have to reflect that in __require.js.

Alter NODE_PATH

In order to make __require.js discoverable by node, we can change the NODE_PATH environment variable to include the current directory

Create Program and Unit Tests

Create a Set class with Typescript at src/ts/Set.ts

This Set class is a very naive implementation of a basic set for the sake illustration.

Place the following Jasmine code under spec/Set.spec.js

These lines of code in Set.spec.js deserves some explaining:

By design, node modules need to follow the commonjs format. If you “require” a file that does not follow the commonjs format then node will return an empty object.

Since Typescript does not actually generate code that obeys the commonjs format, we need to create a method for us to get the generated code into the scope of a jasmine test.

I created __require.js because it makes it very easy to include the contents of a file under the directory I designated as generated code.

Create Gulp Tasks

We now have everything in place to execute our workflow. Create gulpfile.js under the root directory with the following contents.

This workflow will do the following:

  • compile all typescript code in the src/ts directory
  • place the generated javascript code in the src/js directory
  • run all jasmine tests in the spec directory

runSequence will make sure that your workflow will compile all the Typescript files first before gulp runs the unit tests. If we did not use runSequence then gulp would run the tests concurrently with the compilation.

In order to initiate this workflow we simply have to execute gulp from the directory like so:

Conclusion

In practice, there are many different ways of creating workflows. I don’t claim that this is somehow the best or only way to do it.

I hope that this particular example can help you build a workflow that works best for you.