How to test your multi-threaded code (part 1 of 3)?

CHESS is the answer. At least this is what we have best right now.

In multi-threaded application, bug are hard to almost impossible to find. For the last years the only true way to detect threading problems was to run load test until the system crash. Once it does, every once in thousands of iterations, the tools to reproduce and debug the problem were inexistent.

The RiSE (Research in Software Engineering) team at Microsoft have been working for a long time on a product called CHESS. When run with CHESS, you unit tests will try every possible combination of thread interleave to find a case where you application crash or worst doesn’t give you the result you expect.

Here is a simple demo to show you the power of CHESS. Let’s start with a banking account management system.

public class Account
{
    public double Balance { get; set; }

    private Account(double amount)
    {
        Balance = amount;
    }

    public static Account OpenNew(double amount)
    {
        return new Account(amount);
    }

    public void Deposit(double amount)
    {
        double tempAmount = Balance;
        // potential problem
        lock (this)
        {
            Balance = tempAmount + amount;
        }
    }

    public void Withdraw(double amount)
    {
        double tempAmount = Balance;
        // potential problem
        lock(this)
        {
            Balance = tempAmount - amount;
        }
    }
}

Of course I voluntarily introduce “potential problems” to show how CHESS will get them. With that class you can write this test:

[TestMethod()]
public void BalanceTest()
{
    Account account = Account.OpenNew(10000);

    account.Withdraw(100);
    account.Deposit(100);

    Assert.AreEqual(10000, account.Balance);
}

This will always run fine as it is single threaded. Now what if we change it a little to make it multi-threaded:

[TestMethod()]
public void BalanceMutiThreadTest()
{
    Account account = Account.OpenNew(10000);

    Thread thread = new Thread(a => ((Account) a).Withdraw(100));
    thread.Start(account);
    account.Deposit(100);
    thread.Join();

    Assert.AreEqual(10000, account.Balance);
}

As you can see here we put the withdraw part in a new thread. But even then we can run this test again and again without any problem. You can put it in a loop if you want and never being able to make it return an invalid balance.

Once you have CHESS installed on your system the only thing you have to do is to add a HostType attribute to your method.

[TestMethod()]
[HostType("Chess")]
public void BalanceMutiThreadTest()
{
    Account account = Account.OpenNew(10000);

    Thread thread = new Thread(a => ((Account) a).Withdraw(100));
    thread.Start(account);
    account.Deposit(100);
    thread.Join();

    Assert.AreEqual(10000, account.Balance);
}

Now when you run this test you should get something like:

Assert.AreEqual failed. Expected:<10000>. Actual:<10100>.

You should also have noticed that it took a little longer to run the test. This is because the CHESS host scans your code to build an execution schedule for every possible thread interleave that CHESS can detect. If you double click on the test result you will see how many schedules that were tried before finding the bug, in my case it is 3.

Next time we will see how to reproduce and debug that code.