Just about every mature testing framework out there supports the concept of custom matchers, which provide us with the ability to define our very own assertions seamlessly into the tests. Even though this ability is quite old, we don’t see it used too often and I think that’s a shame. I’ve seen this practice heavily used in the mind expanding GOOS book and just now am starting to realize its awesomeness.

Your Testing Toolbox

Matchers 101

Creating your own matcher usually means creating a Matcher class that performs the assertions, supplies human readable error messages and a nice constructor.

Here’s an example from the RSpec documentation:

RSpec::Matchers.define :be_a_multiple_of do |expected|
  match do |actual|
    actual % expected == 0
  end
end

Matchers increase readability and intent

As you should know, one of the most important rules for design is Reveals Intent. Take a quick look here, which way do you think reveals more intent?

# This
response['X-Runtime'].should =~ /[\d\.]+/

# .. or this?
response['X-Runtime'].should be_a_number

Also, which error message do you prefer? “expected false to be true” or something along the lines of “expected comment to be anonymous”?

Matchers create robust tests

The most important advantage of all is how using matchers easily allows you to steer away from fragile tests which are the bane of a lot of testing efforts.
The mark of good tests is that a change in your code doesn’t require you to perform changes in multiple tests that don’t really care for the change.
Take this code for example:

expected_comment = Comment.new(anonymous: true, user: "the dude", reply_to: nil)
commentor.should_receive(:add).with(expected_comment)

This might seem like a standard test, but that’s not really the case. A test should assert for a single piece of knowledge, and this test actually checks several. If the purpose of this test is to check the behavior of anonymous comments, why should it change if we no longer allow replies? Or if we no longer require users for posting comments?

The magic of matchers is exactly here. You create a new matcher to check specifically the aspect your test cares about and *boom*, you’re decoupled!

commentor.should_receive(:add).with(anonymous_comment)

This simple change makes your tests DRY and cool.

Happy testing!

Your should subscribe to my feed or follow me on twitter!

I’ve seen a lot of posts about testing in startups, some saying startups shouldn’t bother to test because they’ll have to change the whole damn thing 5 minutes after they’re done, others claim testing is the only reason they were able to keep working. Here are some of my thoughts looking back.

Our Background

When we started, only two of us had a test-infected background out of the five technical guys, me being big on TDD. Two other developers never wrote tests before. We agreed that tests were important, but that’s about it. I set up a continuous integration server and with that we were off. With time, the habit of writing tests spread out among the team. Some are TDD passionate, some write tests after the fact, but we generally all believe that tests should be written extensively.

None of us, ever

Not everything is worth testing

We’ve seen several quite rapid changes to our UI. Having less tests in this area makes sense. We rely on QA for making sure all buttons are displayed etc. To make this clear: we have no selenium-like tests for UI components but have tests for most logic being done by the UI. I think this is generally a good practice, since having to maintain selenium tests would be hard when you throw things around a lot and change flows. Some basic automated sanity tests pretty much does it.

Everyone learned to love tests

I love seeing other guys in the team delete a line of code to see which test breaks and understand why it’s there. Even more I love the frowning face when no tests break. This addiction to tests shows how much value the team’s getting out of having solid tests, hands down. No need to stress this further I believe.

Tests save our asses repeatedly

Having an extensive suite of tests allows us to make rapid changes to our code base, as is needed in most startups, and rely on the solid tests to tell us whether we’ve screwed something up. All the code that has anything whatsoever to do with sensitive and important information is heavily tested which is a huge bonus and a necessity in our line of business (personal finance protection).

TDD is just magical with complex algorithms

We have quite a few complex algorithms that require multiple entities and ideas to perform. I find that the parts we’re most satisfied with maintainability-wise are the heavily TDD-ed algorithms we’ve got. Being written with rigorous TDD gives us so many advantages:

• This critical code usually has a lot less defects.
• The code is a lot more readable, well decomposed and allows for easy changing once we find out a need for tweaking the algorithms.
• Working in TDD magically forces us to form our problem domain better, making us have a language of our own in talking about the problem. This happens less naturally in other forms of working on algorithms.

Summing our testing experiences

All in all, I think the whole team would agree that dedicating time to writing thorough tests is proving itself valuable and because of that people are writing more and more tests without any of us ever stopping and saying “we should write tests” (well, I swear I didn’t do it too much). It happens naturally when people get the value out of it. It’s fun seeing how today BillGuard has become a company that organically values testing so much I don’t even feel a great need to stress it to new people because they’ll quickly see there’s no real other way. We’re far from being the poster children of Clean Code, but I’ve got my fingers crossed.

If you’re interested in accomplishing the same at your work, you might find this recent post of mine of some help.

You should subscribe to my feed or follow me on twitter!

Today I set out with my pair to add some functionality to a certain class. That class had about 30-40 lines of code and about 10 test cases, which seemed quite decent. We added our changes TDD style and just couldn’t get the thing working. After digging into it for a few more minutes we suddenly realized the class shouldn’t be working at all and checking in the DB showed that indeed the last time that specific feature had any effect was 3 months ago!

Fortunately for us, all the problems that caused this bug are solved problems, we just need to get better at implementing the solutions:

Isolated tests go much better hand in hand with a few integration tests (some might say the right term is acceptance tests) that execute the whole system and make sure the features are working. Had we had those, we would have caught the bug much sooner.

The bug was introduced in a huge commit that changes 35 files and 1500 lines of code. We usually try and go over every commit made, even if it was paired, because we believe in collective code ownership, but it’s impossible to go over such a huge diff and find these intricacies. Working in small baby steps makes it far less likely to break something and more likely that someone else will spot your mistakes. Huge refactorings give me the creeps.

After the change was committed, it was not followed-through: this specific feature is a feature you usually notice over a few days and we missed out on making sure it kept working. We moved on to other tasks and forgot all about it, thinking it was working all this time. Had we taken the time to make sure we were seeing, it would have been squashed by the next deployment.

Any of these would have helped us spot sooner that the isolated tests were actually testing the code against a scenario that never happens. These tiny changes of our workflow would have made several of our users happier over this timeframe.

Hopefully all is well now and the feature is back at 100%, but only time will tell whether we were able to learn from this mishap.

You should subscribe to my feed and follow me on twitter!

A simple example might be handing out jobs to a few workers. If your algorithm for doing that is random, you can usually assert that no one of 3 workers gets all 10 jobs, for example. But, being random, that assert should eventually fail. We’ll assume that with the frequency the team runs the tests, a failure is expected every few days.

Surely no one wants to see the tests fail a couple times a week (especially if you’re keeping score for who broke the build). On the other hand, you’d like to keep the tests. What is a pragmatic coder to do?

If you’re not that meticulous to your suite rarely failing, you might just leave it as it is, which, I think, sucks.

The mega-tester’s approach, which I’ve tried in the past, is usually to stub out the random number generator with values that make sure the failures won’t happen. This is usually cost-effective only for the simplest of cases, and the more complex ones results in brittle tests that are coupled to the implementation and that might need to be changed frequently.

What I rather is to postpone the problem! Say we change our test’s parameters to 10 workers and 3000 jobs. The chances of one worker getting all jobs becomes quite minor. This tweak of parameters in the test is usually simple to do and can guarantee quite a safety net.

And still, sometimes bad stuff happen. 64bit hash collisions are somewhere, out there in the world. If you’re one of those guys that are bugged by that chance, I give you a simple JUnit rule that will retry a specific test in case it fails, making it twice as unlikely to fail. Those 64bit collisions are now more like 128bit! woohoo!

The rule allows you to simply annotate a test to make it retry in case it fails:

public class RetrierTest {
  private static int count = 0;

  @Rule public RetryRule rule = new RetryRule();

  @Test
  @Retry
  public void failsFirst() throws Exception {
    count++;
    assertEquals(2, count);
  }
}

And the implementation is as simple as:

@Retention(RetentionPolicy.RUNTIME)
public @interface Retry {}

public class RetryRule implements MethodRule {
  @Override public Statement apply(final Statement base, final FrameworkMethod method, Object target) {
    return new Statement() {
      @Override public void evaluate() throws Throwable {
        try {
          base.evaluate();
        } catch (Throwable t) {
          Retry retry = method.getAnnotation(Retry.class);
          if (retry != null) {
            base.evaluate();
          } else {
            throw t;
          }
        }
      }
    };
  }
}

With the tests so unlikely to fail, I’d start a lottery at work for whoever breaks them.

Happy testing!

You should subscribe to my feed and follow me on twitter!

The Dilemma

Nowadays, it’s hard to get more than a 100 lines of code before adding an external resource to our code. It might be a web service to manage something, or some convoluted API to receive data from or just about anything. Usually, writing tests for code that directly talks with these resources using the resources themselves is very problematic, for numerous reasons:

• It significantly slows the tests, because it requires network access and processing on the service’s side.
• It might cost you money, send emails, tweet stuff and do things you’d rather not do 300 times a day as you run your tests.
• Making your code handle error conditions with the service is hard or impossible, as you can’t control when those occur.

Basically, all of these factors usually amount up to you having crappy tests that you rarely run. That sucks.

Decouple & Isolate

The best solution I’m aware of is simply isolating the thing. We usually strive to wrap whatever service we’re using with a single-point interface. The decoupling is great since I’ve yet to encounter a service with an API that matched my thinking of the domain problem. Wrapping it up allows us to keep using our own language and logic throughout the system.

A benefit of that is we now have a simple interface or facade we need to stub/mock out during tests. That’s usually relatively easy, and allows us to run our tests blazingly fast and test all those hard to reach to corner cases.

But what if the service changes?

That’s the finishing touch. You should still maintain a suite of tests that run against the real service. Those should be the plain tests that make sure you’re using the API right and that would break if anything you’re relying on changes. These tests won’t be part of your regular suite that gets run constantly. Instead have your CI server run them daily/weekly and let you know when something changes.

This puts us basically in a win-win situation, with us being able to run our tests quickly and yet have the assurance that we won’t miss API changes and the likes.

Happy testing!

You should subscribe to my feed and follow me on twitter!

In the past year I’ve been mainly working on a single project, the longest I’ve worked on a project with TDD. Putting aside how fun it is (TDD saved me quite a few times for me to be sure it’s worthwhile), working on a project for so long I finally got to see some of the main problems people have against TDD.

With the hundreds of test you have, refactoring on the class-interface level (that is, the interfaces of classes, and not inside classes) can be problematic, with you having to update all the tests.

I’m still learning how to handle this efficiently, and would like to share an experience I had today. This is an example of a problem regarding 2 collaborators and an interface change. Such refactorings in a TDD environment weren’t mentioned in the excellent “TDD by Example” book and similar works, so I’m pretty much guessing here.

The example:

The change we’re interested in is making “eject” simply open the lid, without rewinding, and making the rewind operation public. This is in order to allow LazyPerson to take the tape out, without having to wait. Gary Bernhardt wrote about this kind of changes a bit. I agree, the fact I need to make such a change is against the OCP. What can I say, I’m not perfect and made a design mistake. Saying “that’s not OCP” doesn’t help me – I’ve got this code and tests, and I need to change them.

I used to succumb to the temptation and make all the changes in one sweep. That means changing all the tests and the classes, then running the tests and hope they still pass. This, of course, is a crappy way of doing this. Had I been able to actually perform such tasks, I’d write less tests. The secret is baby steps. The pressure Kent Beck puts on baby steps and gradually working towards change made me consider this and force myself to find a safe way of doing this.

I decided to start with the VCR and its test (the AutomaticPresenter doesn’t use the VCR itself but an interface, and the tests use test doubles. This means changing one part won’t break the other’s unit tests). The path to enlightenment lies in finding the baby step that allows starting the refactoring without breaking the rest of the tests. I decided to add a test for the should-now-be-public “rewind” operation, while not breaking the existing tests.

The solution is adding a default value for telling the “eject” function whether it should rewind or not. This means existing users (be them tests or not) will still get the previous behavior, and new tests can start work with the new interface (in Java I’d probably do this with method overloading):

This got me to green pretty fast. Now I can slowly remove every rewind-related assertion from the old tests and also add the “should_rewind=False” flag to them, all with quick-green cycles. And we’re done with the first half.

The next move is to change the AutomaticPresenter to call “rewind” before “eject”, which is now really easy to do in the tests. Once we hit green, we remove the “should_rewind” flag and be done with the refactoring. Baby steps save the day:

Being able to get the refactoring working so easily makes me happy, but I’m still not sure this is the smartest way, and there are harder refactorings to master ahead. Yet, I hope this will help TDD adopters see that it’s possible to handle refactorings even with many tests, because once the right baby-step is found, each test can take practically seconds to convert.

I’d really love getting feedback on this cycle.

You should subscribe to my RSS feed or follow me on twitter!

All you have to do is this (code is based on this blog post, with adaptations to work on slaves that don’t share directories with the master, since the createSummary method runs on the master):

When using nose on Solaris machines, simply running nosetests without specifying the file names will not work if you are the root user. To fix this, you must either not be root, or pass nose the argument --exe. That’s it.

Gory details: by default, nose ignores executable files. Each file it encounters it checks with os.access(test_file, os.X_OK) to see if it’s executable. Problem is that Solaris’ access function always returns success for root, regardless of actual file permissions. This is discouraged by POSIX, but known behavior.

Ensuring that you’re aware of known behaviours is crucial, as it prevents you from looking up issues with software unnecessarily. Of course, you can find these out on the web and it’s not difficult to test run them if you’re unsure what they’ll entail in terms of reversing the process or even repairing a bug or glitch yourself.

Then again, if you’re reading this tutorial you’re probably already fairly IT-savvy, so don’t pay it too much heed. Everything has its own unique digital behavioural patterns, from the latest O2 uk I phone 4 to the newest Alienware MX. It’s just a case of running checks before you start to ensure that you’re prepared for any eventuality.

I hope this saves someone the 3 hours it wasted for me