How To Build Testable and Framework Agnostic Single Page Applications with Node.js

In a previous blog post, I spoke about the package principles of cohesion:

  • The Release/Reuse Equivalency Principle
  • The Common Closure Principle
  • The Common Reuse Principle

In this blog post, I will use the principles of package cohesion to build cohesive npm packages, and I will use these npm packages to build testable and framework agnostic single page applications.

For this demonstration, we will create a fake analytics dashboard. Here is what the final product will look like.

Screen Shot 2016-01-31 at 10.14.36 PMGathering Requirements

We will describe the application requirements using the “User Story Mapping” approach.

The following snippet describes the feature of our application

We want our “system under test” to be testable; so, we need to place some architectural restrictions to guarantee “testability”.

At a minimum, a testable system must have the qualities of (a) observability, and (b) controllability.

Observability refers to the property that I can arbitrarily observe the states of the “system under test”, and controllability refers to the property that I can put the “system under test” into an arbitrary state.

The two properties are inherently related to each other: you can’t fully observe a “system under test” that you can’t fully control, and there we don’t care to fully control a “system under test” that we can’t fully observe.

I claim (but will not prove) that these architectural restrictions guarantee “testability”.

In order to support the “controllability” requirement of our architecture, we will use the following background block to load fake data into our “system under test” for every new test run.

 

By design, a user story will have various scenarios, and we will use the scenarios to determine what needs to be “observable”.

For this demonstration we will consider the following scenarios:

  • Visit the visualization page
  • Change the selected date on the page
  • Aggregate the items by month
  • Aggregate items by year

This snippet shows the scenario description for “visit visualization page”

This snippet shows the scenario description for “change the selected date on the page”:

This snippet shows the scenario description for “aggregate the items by month”:

This snippet shows the scenario description for “aggregate items by year”:

You can see the complete feature file here.

Testing the User Story

The following code snippets uses cucumber.js and babel (an es6 to es5 transpiler) to test the user stories. I use babel because it lets me use generators and coroutines to simplify the asynchronous nature of selenium code.

This snippet initializes the “system under test” based on the data provided in the background block for the user story. This satisfies the “controllability” property of our architecture.

This snippet executes the action(s) that change the state of the “system under test” for the behavior “When I visit the page at <some_url>”:

This snippet executes the action(s) that change the state of the “system under test” for the behavior “When I select the date <some_date>”:

This snippet executes the actions(s) that change the state of the “system under test” for the behavior “When I select to aggregate by <some_aggregation_type>”:

This snippet executes the post conditions checks of the “system under test” for the block “Then I should see a bar chart for <some_title>”.

Note: I will skip the rest of the blocks for brevity’s sake.

In the acceptance tests, I use a page object to encapsulate the data retrieval algorithms for each page.

For example, the following code snippet shows how the page object encapsulates the algorithm that “observes” when the page has finished loading.

This test looks for a DOM element with the html attribute “qa-chart-type”, and then waits for it to change.

We have to do this because the SPA fetches data using a http request when if first loads, and we want to detect when the SPA receives a response.

However, since we cannot easily detect when the SPA receives an http response, we make the SPA communicate this to our test script by creating and updating special “Quality Assurance” html attributes.

This makes testing with selenium much easier and maintainable.

I also created a page object factory to easily change implementations of the page object. It uses the same principles of a “contract test” that I mentioned in my blog post “Contract Tests: or, How I Learned to Stop Worrying and Love the Liskov Substitution Principle”.  

Let’s suppose that we implemented our application using (a) Backbone.js, and (b) React.js.

In order to test a Backbone implementation, we could use the following code

In order to test a React implementation, we could use the following code

This is what running the cucumber application looks like when I test a React implementation:

screengrab

You can see the complete test suite here

Building Our Components

Suppose that you had the following directory structure for a JAVA application

  • src/
    • use_case001/
      • command/
        • File001.java
        • File002.java
        • File003.java
      • delegate/
        • File001.java
        • File002.java
      • model/
        • File001.java
        • File002.java
    • use_case002/
      • File001.java
      • File002.java
    • use_case003/
      • File001.java
      • File002.java
      • File003.java
    • controller/
      • Controller001.java
      • Controller002.java
    • view/
      • View001.jsp
      • View002.jsp
    • model/
      • Model001.java
      • Model002.java

This is an example of a “monolithic” or “big ball of mud” application architecture.

Architects typically have nothing but contempt for this architecture due to its well known rigidity, fragility, immobility, and viscosity.

These are well known application level architectural properties:

  • rigidity measures how easy an architecture can respond to change.
  • fragility measure how likely something will break when making a change
  • mobility measures how easy an architecture can support moving code
  • viscosity measure how easy an architecture supports maintaining the original design

When faced with a monolithic architecture, most architects I know would refactor the monolith into something that looks like this.

  • src/
    • controller/
      • Controller001.java
      • Controller002.java
    • view/
      • View001.jsp
      • View002.jsp
    • model/
      • Model001.java
      • Model002.java
  • lib/
    • use_case001.jar
    • use_case002.jar
    • use_case003.jar

Each .jar file is a self contained components that is independently testable, releasable, and versioned.

You might think that simply moving code from a directory to a .jar file does not significantly improve the design. However, when applied properly, it has a huge influence on reducing rigidity, fragility, viscosity, and increasing mobility.

We will use this approach, but in this case we will use npm modules as components: a npm module is equivalent to a JAVA .jar file; so, all the same rules apply.

I will use the discipline of Responsibility Driven Design to guide the cohesion strategy for our components. Responsibility Driven Design makes component “responsibility” the most important organizing principle for code.

I claim (but will not prove) that organizing our components by “responsibility” will force our components to obey (a) the Reuse-Release Equivalency Principle, (b) the Common Closure Principle, and (c) the Common Reuse Principle.

From the scenarios in the user story, I can identify the following responsibilities:

  • Bar Chart Visualization
  • User Interface Interaction
  • Data Retrieval
  • QA HTML Attribute Generation

Therefore, I will respectively create the four following packages:

  • analytics-chart
  • analytics-facade
  • analytics-service
  • qa-locator-utility

Create the “Analytics Chart” Component

The actual details of how I created the component are not important; so, I will not talk about it.

However, I feel that I should mention the unit tests I wrote for the visualizations because it isn’t a common thing.

I created unit tests to prove partial correctness of my application using the approach I describe in my previous blog post.

For example, the following code snippet tests the visualization for some items.

This test verifies that certain DOM elements appear in the svg canvas. I consider it a very weak test because there is a good chance that it could pass and the visualization could still be wrong. However, it is also a cheap test to write; so, there is still value in writing it.

You can see the completed npm package here.

Create the “Analytics Facade” Component

This component encapsulates all the user interface “glue logic” (i.e. identifying which screen or page to display), and we use the command pattern and delegation pattern to promote loose coupling between the user interface and this component.

For example, these are the cucumber features that I wrote for this component.

From a client’s perspective, they only need to know how to execute a command from the facade, and how to delegate tasks to the facade. The cucumber tests above provide a very high level description of what types of commands and delegates this facade exposes.

For the sake of completeness, I will also show you the step definitions for this feature.

You can see the completed npm package here.

Create the “Analytics Service” Component

This component simply makes a request to our remote server and return the json results. It isn’t very interesting; so, I will not discuss it.

You can see the completed npm package here.

Create the “QA Locator Utility” Component

Officially, we do not need this component, but we would like to have an easy method to locate elements in our window’s DOM using xpath. This component encapsulates any algorithms that we might need to simplify the process of creating html attributes for the purpose of writing acceptance tests.

I don’t consider it a very interesting component; so, I will not discuss it. 

You can see the completed npm package here.

Composing the Components Into A Single Page Application Framework

To demonstrate how to migrate the application between frameworks, I will compose the components into (a) Backbone 1.2, and (b) React 0.14.

The following component diagram shows the general relationships between the components and framework.

Screen Shot 2016-02-03 at 11.27.24 PM

Backbone 1.2 and React 0.14

We have placed the bulk of the applications functionality inside of framework independent components. Therefore, in order to create a backbone version or a react version of the application, we have to create a special component that coordinates between our “independent” components and the framework.

In the case of React, we created the component backbone_app.

In the case of Backbone, we create the component react_app.

For example, here is the entry point for our Backbone application:

Here is the entry point for our React application:

The parallels in the code are very apparent. I will not go into detail into the code bases because it is not important. I only care about the “shape” of the design.

You can find the Backbone implementation here, and the React implementation here if you really care to dig into the code, however.

Conclusion

This approach might seem very complicated for such a simple application … and it is.

In reality, this approach only makes sense once your application reaches a certain scale. However, simple applications have a tendency of becoming big applications, and it is very expensive to put a test harness on an already existing system with no tests.

Personally, I recommend you always build an application with decoupled testable components from the very beginning because that investment will pay large dividends in the long run.

Further, decoupling your components from the framework has huge benefits even if you have no intention of switching frameworks.

One practical “use case” is with automated testing.

When you decouple your components from the framework then you can test the bulk of your application independent of the framework. This significantly simplifies the overall testing process.

In fact, the package principles were partially created to minimize the work of creating and maintaining tests.

In a future post, I will show you how you can use a Continuous Integration tool to automatically run your tests whenever someone releases a new version of a component.

That should demonstrate the practicality of strictly following package principles.

Advertisements

How Frameworks Shackle You, and How to Break Free

I sometimes hate software frameworks. More precisely, I hate their rigidity, and cookie cutter systems of mass production.

Personally, when someone tells me about a cool new feature of a framework, what I really hear is “look at the shine on those handcuffs.”

I’ve defended this position on many occasions. However, I really want to put it down on record so that I can simply point people at this blog post the next time they asks me to explain myself.

Why Use a Framework?

Frameworks are awesome things.

I do not dispute that.

Just from the top of my head, I can list the following reasons why you should use a framework

  1. A framework abstracts low level details.
  2. The community support is invaluable.
  3. The out of the box features enable you not to reinvent the wheel.
  4. A framework usually employs design patterns and “best practices” that enables team development  

I’m sure that you can add even better reasons to this list.

So why would I not be such a big fan?

What Goes Wrong with Frameworks?

VENDOR LOCK-IN.

Just to reinforce the point, let me say that again: VENDOR LOCK-IN.

There is a natural power asymmetry between the designer/creators of a framework and the users of that framework.

Typically, the users of the framework are the slaves to the framework, and the framework designers are the masters. It doesn’t matter if it is designed by some open source community or large corporation.

In fact, this power asymmetry is why vendors and consultants can make tons of money on “free” software: you are really paying them for their specialized knowledge.

Once you enter into a vendor lock-in, the vendor, consulting company, or consultant can charge you any amount of money they want. Further, they can continue to increase their prices at astronomical rates, and you will have no choice but to pay it.

Further, the features of your application become more dependent on the framework as the system grows larger. This has the added effect of increasing the switching cost should you ever want to move to another framework.

I’ll use a simple thought experiment to demonstrate how and why this happens.

Define a module as any kind of organizational grouping of code. It could be a function, class, package, or namespace.

Suppose that you have two modules: module A and module B. Suppose that module B has features that module A needs; so, we make module A use module B.

In this case we can say that module A depends on module B, and we can visualize that dependency with the following picture.

Slide1

Suppose that we introduce another module C, and we make module B use module C. This makes module B depend on module C, and consequently makes module A depend on module C.

Slide2

Suppose that I created some module D that uses module A. That would make module D depend on module A, module B, and module C.

Slide3

This demonstrates that dependencies are transitive, and every new module we add to the system will progressively make the transitive dependencies worse.

“What’s so bad about that?” you might ask.

Suppose that we make a modification to module C. We could inadvertently break module D, module A, and module B since they depend on module C.

Now, you might argue that a good test suite would prevent that … and you would be right.

However, consider the amount of work to maintain your test suite.

If you changed module C then you would likely need to add test cases for it, and since dependencies are transitive you would likely need to alter/add test cases for all the dependent modules.

That is an exponential growth of complexity; so, good luck with that. (See my post “Testing software is (computationally) hard” for an explanation). 

It gets worse, though.

The transitive dependencies make it nearly impossible to reuse your modules.

Suppose you wanted to reuse module A. Well, you would also need to reuse module B and module C since module A depends on them. However, what if module B and module C don’t do anything that you really need or even worse do something that impedes you.

You have forced a situation where you can’t use the valuable parts of the code without also using the worthless parts of the code.

When faced with this situation, it is often easier to create an entirely new module with the same features of module A. 

In my experience, this happens more often than not.

There is a module in this chain of dependencies that does not suffer this problem: module C.

Module C does not depend on anything; so, it is very easy to reuse. Since it is so easy to reuse, everyone has the incentive to reuse it. Eventually you will have a system that looks like this.

Slide4

Guess what that center module is: THE FRAMEWORK.

Slide5

This is a classic example of code immobility.

Code mobility is a software metric that measures the difficulty in moving code. 

Immobile code signals that the architecture of the system doesn’t support decoupling.

You can argue that the framework has so much reusable components that we ought to couple directly to it.

Ultimately, we value code reuse because we want to save time and resource, and reduce redundancy by taking advantage of already existing components.

Isn’t that exactly what the framework gave us?

Well, yes and no.

It depends on what you want to reuse.

Are you trying to reuse infrastructure code? By all means, please use a framework for that.

Are you trying to reuse domain specific business logic? You probably don’t want to couple your business logic directly to a framework.

An example of coupling your business logic directly to a framework is your typical MVC-ORM design. I’ve already explained this in my blog post “MongoDB, MySQL, and ORMS: when and where to use them”; so, I will not elaborate on it, here.

The Best of Both Worlds  

So it seems that we are at an impasse: we want to reuse infrastructure code from a framework, but we also want our business logic to be independent of it.

Can’t we have it both?

Actually, we can.

The direction of the arrow in our dependency graph makes our code immobile.

For example, if module A depends on module B then we have to use module B every time we use module A.

However, if we inverted the dependencies such that module A and module B both depended on a common module — call it module C — then we could reuse module C since it has no dependencies. This makes module C a reusable component; so, this is where you would place your reusable code.

coupling The Dependency Inversion Principle exists to enable this pattern.

The typical repository pattern is the perfect example of how inverting dependencies can decouple your business logic from a framework. I have already talked about it in my post “Contract Tests: or, How I Learned To Stop Worrying and Love the Liskov Substitution Principle”; so, I won’t elaborate on it, here.

So how would you structure your application to support the inverted dependencies?

Well, there are multiple ways you can legitimately do it, but I use the strategy of placing the bulk of the applications code in external libraries.

Under this model, the framework’s views and controllers  only serve as a delivery mechanism for the application. 

Consequently, we can easily move our code to another framework because the components do not depend on the framework in any way.

As an added benefit, we can test the bulk of the application independent of the framework. 

In fact, this is your typical Component Oriented Architecture.

For example, standard Component Oriented Architecture will break large C++ applications into multiple dll files, large JAVA applications into multiple jar files, or large .NET applications into multiple assemblies.

There exists rules about how you structure these packages, however.

You could argue that we would spend too much time with up front design when designing this type of system.

I agree that there is some up-front cost, but that cost is easily off-set by the time savings on maintenance, running tests, adding features, and anything else that we want to do to the system in the future.

We should view writing mobile code as an investment that will pay big dividends over time.

I plan to create a very simple demo application that showcases this design in the near future. Stay tuned.

Scalable Cross-Domain “Cookie” Authentication/Authorization: An Example with macaroon.js

Recently, cookie authentication has fallen out of favor in the web community for it’s inability to port across domains and scale.

However, there exists a method to use cookies in a distributed fashion that is both portable across domains and scalable: macaroons (it’s a better kind of cookie).

Google introduced the world to macaroons in 2014 with a white paper. Unfortunately, there does not exist much documentation or code examples for it; so, I thought I’d help fill that void with this blog post.

(Not) Introducing Macaroons

I will not talk about the theory of macaroons. You can use the following resources to get the theory:

A Basic Example

Suppose that you had a website that needs to authorize a user and authenticate their access before it can do anything. You probably would have the server send the user a cookie. The user would then send the cookie value to the server with each request, and the server would use that value to determine authentication and authorization details.

This has severe limitations because we cannot easily share that cookie across domains, and most attempts to do so usually introduce network bottlenecks. I will not go into the details of why this happens; you can easily google it.

Macaroons overcome these limitations by allowing you to put arbitrary restrictions on a “cookie”. Further, these “cookies” can include information from third parties.

This allows you to easily share them across domains and distribute them arbitrarily.

Let’s consider a simple example.

Suppose we had an application that we logically divided into (a) the browser, (b) the service provider, and (c) the identity provider.

The browser contains the code that powers user interaction, the service provider contains the code that powers data access and manipulation, and the identity provider powers user management of the system.

Suppose that the user wants to see his stock portfolio on a web page.

In order to satisfy this use case, our system needs to complete the following tasks:

  • Verify the user’s identity
  • Verify the user’s rights
  • Find the data
  • Show the data

The following sequence diagram shows how we can satisfy those tasks with the help of macaroons.

macaroon_authentication

I will use node.js and the macaroon.js module to power this example. I will only provide code snippets that are relevant to the most important elements of the use case.

Request Macaroon

For this example, the browser will kick-off the macaroon process by making a request to sp.domain.com for a macaroon. Let’s suppose that sp.domain.com exposed the url “/api/macaroons” for that

Generate Macaroon

The code to generate and return a macaroon could look like the following:

This code creates a first party caveat for the macaroon.

A first party caveat is an enforceable restriction on the macaroon created by the minter of the macaroon. In this case, we retrict this macaroon to a particular ip address.

The code also creates a third party caveat for idp.domain.com. A third party caveat is a restriction that we delegate to another party. In this case, we are requiring the recipient of the macaroon to pass the macaroon to the idp to authorize his requests.

The ability to delegate attenuation to third parties is the secret sauce for macaroons. With normal cookies the service provider would have to manage and coordinate all the services associated with authentication/authorization. However, with macaroons, we can simply delegate authority to a trusted third party.

For security reasons, we generate the caveat key on sp.domain.com and encrypt it using the public key for idp.domain.com. We have to pass the encrypted caveat key to browser along with the macaroon.

The following is the code that could handle the response from /api/macaroons.

The browser is responsible for coordinating the authentication for the macaroon. In this case,

the callback sends the encrypted caveat key and the macaroon to idp.domain.com along with a username and password.

Verify Third Party Caveats

The idp will verify the username and password of the user. If that passes then we can create a discharge macaroon and apply it to our original macaroon. When we create a discharge macaroon, we are satisfying the delegation request from sp.domain.com. In this case, we also assign a timestamp to the macaroon to enable revocation. We are telling sp.domain.com that this macaroon is only good until the time set.

We expect sp.domain.com to obey that restriction.

idp.domain.com will send the original macaroon and the discharge macaroon to the callback. We need them both because macaroon’s hmac chaining algorithm protocol requires them to verify the authenticity of the macaroon.  

The above code makes a request to sp.domain.com for the portfolio information. It also sends the root macaroon and discharge macaroon with the request.

Request Data

We know have enough information to verify the authenticity of the macaroon. I know that we have some handwavy helper functions that verify the authenticity. The particulars are not important, though. The verification details will always changed depending on the application.  

Conclusion

This is not production ready code. There is no error checking or edge case handling. This is just a bare bones examples of how you could use macaroons in a distributed environment.
We can easily scale this process up to an arbitrary number of servers. Hopefully, you’ll be able to use this as a starting point to building your own macaroons.

Cargo Cult MVC

Wikipedia describes “cargo cult programming” as “a style of computer programming characterized by the ritual inclusion of code or program structure that serve no real purpose”.

I believe that MVC web frameworks have essentially forced a cargo cult mentality on programmers.

The Problem

Pretty much all MVC web frameworks use the notion of a front controller.

The front controller has the responsibility of handling every web request, and then gives that request to the proper controller.

For example, in CakePHP the default configuration is to use convention names to choose routes. Suppose I had the following piece of code

The front controller for CakePHP would automatically associate PostController::index() to the url /posts/index

The front controller’s strategies for dispatching work to controllers is pretty arbitrary. However, the most typical strategies are to use xml file configuration, annotations, naming conventions, or fluent interfaces.

Typically, we try to organize our code around use cases because it is a technique that we know promotes loose coupling and high cohesion. A use case in this context is any single useful thing the system must do. For example, “edit an article” would be a use case for a blogging system.

However, there is no guarantee that a single url will always be associated with a single use case. Consider the case of a web search form. Imagine that we have a url of product/search?q=some_search_term. That web request would then be passed to a controller like the following

The problem is that the controller actually has 5 use cases:

  • show zero results
  • show one result
  • show many results
  • show lots of results
  • opps

To understand why, consider how an average person would expect the system to respond.

  • Case Zero: If there were no results for a search then I would want to see a dedicated page telling me that there were no results.
  • Case One: If there was only one result then I would want the system to immediately show me the details of that single item.
  • Case Many: If there were many results that could fit on a single page then I would just like to see a listing of those results so that I can choose to drill down into a specific item.
  • Case Lots: If there were too many results to list on a single page then I would want to see some sort of pagination.
  • Case Opps: If something bad happened which prevented the system from responding to my request then I would want the page to tell me that.

Rather than being MVC, we know have MVVVVVC. This isn’t exactly what we want from an engineering perspective, but it is what we were forced to do in order to satisfy the business requirement.

We can work around that situation pretty easily with the proper use of design patterns. For example, we would simplify the controller logic with the use of the strategy pattern. However, in so doing we are no longer using the MVC paradigm. It would be “Model StrategyContext View Controller”.

Not all cases are that simple to work around, though. Consider the situation where you have different roles for a system, and that each roles is allowed to view different things, and perform different work. Now consider that you wanted to expose that kind of functionality through a single url. How painful would that code be?

The Solution

Do not treat MVC like it is a silver bullet solution. There are plenty of alternative architectures out there. Google the following terms to see

  • Hexagonal Architecture (aka ports and adaptors)
  • Onion Architecture
  • Clean Architecture
  • Lean Architecture
  • DCI Architecture
  • CQRS
  • Naked Objects

Conclusion

Everything has it’s time and place. It is your job as a developer to think. Blind adherence to a framework will not save you.

I should also point out that MVC architectures themselves are not bad. How we use it is bad. MVC architectures definitely had their usefulness, and they probably still do.

For example, MVC architectures are really good at doing very simple data entry system because a single use case can actually fit into a url.

Consider the following mappings between a URL and a Use Case

URL Use Case
/article/create Create
/article/:id: Read
/article/:id:/edit Update
/article/:id:/delete Delete

In this case, each url maps very well to a single use case; so, MVC would probably be a really good choice to power an app like this.

However, in my opinion, most systems will not be this simple. Most systems are now very sophisticated and if they are not sophisticated then they eventually will evolve to something that is. You might as well architect your system with something that you know can help you evolve.