Showing off CSS Critic with example applications

TL;DR I agree that setting up your first test with CSS Critic can be a hurdle that needs overcoming. That’s why I created csscritic-examples.

One benefit of working at ThoughtWorks is that you get to see many different projects. In my last two years I’ve seen three different implementations of a user facing application and more importantly three different ways of delivering an application’s UI.

In the phase of each project I’ve sat down and mulled over how to set up a first user interface tests.

While CSS Critic tries to be as simple as possible to set up, your choice of application framework heavily influences the integration of the tool. Unlike other testing solutions (e.g. Selenium-based) that fire up your application through a web server, CSS Critic wants to be a lightweight solution by picking up small components of your application for testing. This forces the developer to carefully wire it into the development setup.

More clearly it needs you, the developer, to think about components of your UI architecture. Once parts of the UI are split into different files and dependencies are made clear (say for example your HTML template needs an adjacent JavaScript view implementation) you can define small and independent tests. This is the basic formulation of how CSS Critic would like to work.

The latest application I have worked on was an AngularJS-based single-page application. Working with an opinionated framework that AngularJS is I realized how difficult it can be to tap into the inner bits of a framework. While designed to be easy to set-up and use for day-to-day work, accessing inner components of a framework (and for the purpose of UI testing that means the templating system and view binding) can be difficult.

To counter the difficulty of the initial set-up phase I now plan to collect examples of how to bring together CSS Critic with the web framework of your choice. I’ll start with a simple application based on AngularJS and one in plain JavaScript. Have a look at the repository over at GitHub. Cheers go to the TodoMVC project that both examples are based on.

Image

Gathering an extensive set of examples can only be a community-based effort and so I welcome contributions. If you, dear user of CSS Critic, have a new set-up to share, please start a pull-request and have others benefit from your good work.

Advertisements

Testing CSS

For some time now I have been wondering why we test our source code so thoroughly but when it comes to CSS we just plainly stop caring about it. 

Maybe I'm wrong, I'm still relatively new to the TDD business, but looking at my colleagues, everybody is quite eager to have their Java or JavaScript code covered. But speaking of CSS, there isn't much help around for doing tests here.

Looking at the test pyramid, it is mentioned that tests through the UI are brittle, in fact you are testing the whole stack from top to bottom and anything anywhere can go wrong. However that doesn't mean that testing the UI needs to be similarly brittle. In fact you can mock out the underlying functionality that your process rendering the UI depends on.

A broken UI can break the user experience just like a faulty functionally (i.e. source code) does. Especially in a bigger project where several people are involved possibly across teams it is hard to keep the UI consistent and errors out.

In my current project a glitch in the UI can keep the product owner from pushing the next release candidate to production. And there are several teams that together deliver a single page to the user, meaning that bits of the page including the layout come from different sources. In the end we need to make sure that everything comes together just right.

On top of that there is this browser issue. Each browser renders a page quite differently. Consistently checking that changes don't break the layout in browser X can be a very tedious manual task.

I've heard from some people that Selenium is used to do a screenshot comparison, i.e. regression testing on reference images. One example is Needle. There have been undertakings to test actual values of properties on DOM elements, e.g. at Nokia Maps.

Why am I saying all that? Because I'm currently looking into developing yet another css testing tool that I want to share with you.

My take on this problem builds on the image comparison technic similar to the Selenium stuff. However my approach is to keep the stack simple and to make it dead simple to use: everything should be done from inside a browser window.

With the feedback from my colleagues at ThoughtWorks I've set up a small project on Github to implement an experimental solution with the goal of driving out a feasible solution. 

The steps to verify a layout should be pretty straightforward: A new page (either from production or a mock page) that includes the CSS under test is registered with a "regression runner". That is a simple HTML page running the test suite (if you know jasmine and its SpecRunner.html you get the point). On the first run the page under test is rendered and can be saved as a future reference. In subsequent runs this image is used for the page to be compared against. Running the tests is as simple as opening the runner in a browser. If the layout changes, the regression test will fail. If the change was intentional a new reference needs to be created, if not you found your bug. 

Technically this works by rendering the page under test to a HTML5 canvas element inside the browser and using a handy JS library for doing a diff between the canvas and the reference image.

Open points: So far works in Firefox only, and as browsers do not render pages across systems consistently, the solution is local only.

Do watch the screencast to see how it works: 

Continuous Integration for your jQuery plugins

TL;DR If you have tests for Javascript code written in QUnit & Jasmine that depend on the Document Object Model (DOM), here is a way to set up Travis CI using PhantomJS.

My colleagues recently made me aware of a relatively new continuous integration software called Travis CI which, originally built to serve the Ruby community, is a distributed build service able to run code in various languages, including Python & Javascript. As far as I know, it currently only works together with Github, so your code needs to be hosted there.

As Travis’ workers (the ones running the actual build) come with node.js included, I played around a bit getting my QUnit tests to run with jsdom and the QUnit node adaptation. While there are some guides out there on how to test your Javascript with node.js, it gets complicated when depending on the DOM, which most likely is the case when you are developing a plugin for jQuery. However, after reading criticism on testing against something that the original system didn’t target (or are you running jQuery on the server side?) I gave up on that pretty quickly.

Now, in a different context I came upon PhantomJS, a headless browser stack based on Qt’s Webkit. It provides an executable that can work without a graphical system (hence the name headless) and is perfectly suited for testing environments. Ariya, the guy behind PhantomJS, is clearly aware of that and already provides the proper integration for running tests based on QUnit and Jasmine. The test runner is a neat piece of code, that just scrapes the QUnit output from the generate HTML. Installing that locally was easy and running the test suite provides a short output on how many tests were run and how many failed, if any.

The problem was getting PhantomJS running on Travis CI. Travis CI comes with a good set of software (and already includes some of PhantomJS’ dependencies); so far no one has written a cookbook for PhantomJS though. However, this guy came up with an easy solution, after all the worker is just a (virtual) Ubuntu machine and you can install anything on it.

So here is the quick run through: In the .travis.yml which describes the build, we

  • run a small install script setting up the remaining dependency of PhantomJS and PhantomJS itself,
  • start up a virtual framebuffer (xvfb, “headless” is not completely true when on Linux) running on port 99
  • and finally run PhantomJS with the QUnit (alternatively Jasmine) test runner on our test suite.

Here is the full .travis.yml file:

rvm:  - 1.9.3before_script:  - "sudo bash install_phantomjs > /dev/null"  - sh -e /etc/init.d/xvfb startscript:  - DISPLAY=:99.0 phantomjs run-qunit.js test/index.html

The first line indicates that we are wanting Ruby version 1.9.3, even though we don’t need it. I believe we have to chose some target system, so there it goes.

Here is the install_phantomjs script:

#!/bin/bashapt-get install libqtwebkit-dev -ygit clone git://github.com/ariya/phantomjs.gitcd phantomjsqmake-qt4makecp bin/phantomjs /usr/local/bin/

We are ready to test this on Travis. If you haven’t registered there yet, get an account, set up the hook by visiting your profile page, and commit your own .travis.yml together with the PhantomJS install script and the relevant test runner described above. You should pretty quickly find your project in the build queue on travis-ci.org.

Happy testing!

Using Selenium to validate XHTML markup using lettuce

I am currently getting started on providing unit tests for Deniz with Selenium. While deniz is pure Javascript (with HTML & CSS) I am using Python with Lettuce (clone of Ruby’s cucumber) to test the application. Lettuce is a behaviour driven development (BDD) tool and makes testing clean and fun.

In this fashion I wanted to check that the W3C XHTML button is placed correctly, i.e. that deniz actually is valid XHTML. So automatic testing comes into play.

Here’s a small receipt to formulate the validation test. There are some quirks when using Firefox that needed a workaround, so I thought I share:

https://gist.github.com/924043

So I found out that while "static" deniz.html is valid XHTML the components that get embedded once Javascript is run breaks the document’s validity.

Now that brings me to a new question: Should AJAX-style webpages stricly adhere to W3C standards while operating, i.e. going through various states of the application, each changing the underlying HTML code? I guess so.

Hudson/Jenkins multi-configuration

In SuRF we changed a lot of str() calls to unicode() which made code fail under Python 2.6.2. This wasn’t found through unit tests as my tests ran with the newest Python version only.

Luckily Hudson/Jenkins has a multi-configuration (or matrix) mode that runs a test with several configurations. This lets you for example run your tests on different Python versions to check for compatibility.

When creating a new job you just choose “Build multi-configuration project” and receive a configuration page mostly similar to a general project.

Multi-configuration

A bit further down the page there’s a special section called “Configuration matrix”. Here you can set several axes of parameters with multiple values. In the example shown I chose a variable python_version with values 2.5, 2.6 & 2.7. Additional I added a value for the Python distributed with Ubuntu so I make sure to catch newer versions as well as distribution specific issues.

Multi-configuration1

As SuRF tests run against an RDF store tests should run sequentially (checked). Also to keep Jenkins from running all configurations if something fails I defined a “touchstone build”. Only if this build succeeds all the other options will be tried.

I had one issue when setting up a Git repository – the same error I reported before: “FATAL: Could not apply tag jenkins-python_version=DISTRIB”. Setting the appropriate Git options did not work as explained in the simple set-up, and setting a default in Jenkin’s settings did not help either. I could only get the test to start by checking “Skip internal tag” which shows up when you click on “Advanced…” in section “Source code management”.

I now have two axes set-up for SuRF (one axis for different versions of rdflib), totally 16 distinct configurations which still complete pretty fast.

Multi-configuration2

Setting up different Python versions

Here’s a short log on how I installed different Python versions in parallel on an Ubuntu system:

Install all build dependencies and get the Python source code

$ sudo aptitude build-dep python2.7
$ wget http://python.org/ftp/python/2.7.1/Python-2.7.1.tar.bz2
$ tar -xjf Python-2.7.1.tar.bz2

Now configure and install the version in parallel with others to a directory under Jenkins:

$ cd Python-2.7.1
$ ./configure –prefix=/home/jetty/jenkins/python
$ make altinstall

Probably repeat these steps for 2.5, 2.6 (and 3.2?).

Configure

Add axis:

python_version

with values

2.5 2.6 2.7 DISTRIB

And now add a shell script before the actual test code:

rm -rf “.env”
echo “**> creating virtualenv”
if [ “$python_version” = “DISTRIB” ] ; then
virtualenv .env
else
virtualenv -p /home/jetty/jenkins/python/bin/python$python_version .env
fi

PIP_DOWNLOAD_CACHE=/home/jetty/jenkins/pip_download_cache pip install -E .env nose coverage

Now add a second shell script running the actual test. Use the virtualenv as e.g. described in my former post. Voila.

Hudson & Python

Today I’m going to pick some low-hanging fruits as there is a lot of documentation on this topic already. However, as there usually are a lot of gory details I’ll write on how this worked out for me.

“Hudson is a continuous integration tool” (Wikipedia). It is very useful for running Unit testing and can automatically act on new commits made to the project the Hudson server watches. You can for example set-up Hudson to notify the person having committed broken code by E-mail or even have a USB traffic light show the status of the checked source code.

Installing on Ubuntu

While one good thing about Linux distributions is that installing is merely a single command away, this is currently not given for Hudson on Ubuntu or Debian. There is currently an open ticket for packaging Hudson for Debian but so far only an unofficial package from the Hudson team itself exists.

Hudson needs Java and I could only get it to work with the official Java sources. To install Java from Sun Oracle you need to enable the “partner” repository from Ubuntu. Uncomment the following line in “/etc/apt/sources.list” (for Ubuntu 10.10):

deb http://archive.canonical.com/ubuntu maverick partner

 

And then run

$ sudo aptitude update

$ sudo aptitude install sun-java6-jre

 

Now make sure the right Java is used by running:

$ update-java-alternatives -s java-6-sun

 

Once you have Java installed you need to install the “daemon” package that Hudson needs:

$ sudo aptitude install daemon

 

Now download the Hudson “.deb” from http://hudson-ci.org/debian/ and follow the steps described there to install the package.

Hudson should now be running under http://localhost:8080/

Setting up a Python project from Git

As others have already described how to use Python with Hudson before and probably even have done it better than I could do in this place here I urge you to follow the steps described in http://www.protocolostomy.com/2010/12/02/nose-and-coverage-py-reporting-in-hu… . Make sure you have git installed and maybe other hard dependencies your project will need (e.g. python-dev when compiling packages).

There is an issue however with the first git checkout. If you haven’t used and configured Git on the system before you will probably get an error similar to the one below: “FATAL: Could not apply tag hudson-surf.virtuoso_protocol-1“. In this case you need to move to the workspace created by Hudson and do the configuration as described by <a href="http://wiki.hudson-ci.org/display/HUDSON/Git+Plugin:

http://wiki.hudson-ci.org/display/HUDSON/Git+Plugin:</p>

$ cd /var/lib/hudson/jobs/YOURPROJECT/workspace/

$ sudo git config user.name “YOUR NAME”

$ sudo git config user.email “YOUR EMAIL”

 

Managing dependencies via virtualenv

Most projects come with dependencies and while manual installation on the server will work, it is impossible to keep different conflicting versions and in general will need you to install new Python packages once dependencies change. Virtualenv which I’ve mentioned before comes in handy here.

As described in http://www.hudson-labs.org/content/python-love-story-virtualenv-and-hudson there is a quick way to introduce virtualenv into the build process.

By adding a build step “Execute shell” with the following content

if [ -d “.env” ]; then

echo “**> virtualenv exists”

else

echo “**> creating virtualenv”

virtualenv .env

fi

pip install -r requirements.txt

 

a virtual environment will be created in .env and updated on every run of the build process. All we now need is to write the file requirements.txt to include the project’s requirements. This at best is a simple file with one line per package dependency. This actually is a good use case for the method I blogged about before.

Using nose for unit testing

Nose is a very handy tool when doing unit testing. In the post about Python&Hudson mentioned above nose is already used for generating the unit test XML and code coverage results. Here’s an in my opinion nice way to use nose in any project.

Distutils has a command for running tests:

$ python setup.py test

By adding the following two lines to setup.py we can use nose for this handler:

test_suite=”nose.collector”,

setup_requires=[‘nose>=0.11’],

We can then activate xunit results and coverage by generating the file setup.cfg:

[nosetests]

with-xunit=1

with-coverage=1

cover-package=YOUR_PACKAGE

cover-html=1

To have the plug-ins activated we need to use the nose handler though (see <a href="http://somethingaboutorange.com/mrl/projects/nose/0.11.1/api/commands.html):

http://somethingaboutorange.com/mrl/projects/nose/0.11.1/api/commands.html):</p>

$ python setup.py nosetests

With this we have not only created an easy way to run the test from Hudson (using the “Execute shell” build step) but also can we now provide a dead easy way for our users to check the project’s code. I use this for example for https://github.com/nwebs/surf.virtuoso_protocol .