The Browser/OS Combination Problem

The Problem

You have N number of tests, O number of browsers and P number of operating systems. If you don’t think too much, it doesn’t sound that bad. Besides your using grid or Sauce Labs or something similar; how bad can it be. Then you do the math and start to realize the scale of the problem.

Example

Number of tests

5 Browsers * 4 Operating Systems * 30 Smoke Tests = 600 Test Executions!

Execution time

600 Test * 30 Seconds Per Execution / 5 Nodes = 1 Hour Execution Time or More!
And that's just for smoke tests!

Solutions

There are a couple of solution to this problem. The one that I've seen the most is to reduce the scope. Just test against fewer combinations. Second most common is to get a bigger grid. Both these solutions can work, but they both have drawbacks. The first solution obviously sacrifices your OS/browser coverage. The second is more costly, as it requires more nodes, and is much more likely to result in some serious bottleneck problems.

My preferred solution is to steal one of the strategies leveraged by "High-Volume Combination Testing". Combination testing often results in an impossibly large number of tests. This means doing some kind of paring down. The strategy we will be stealing is random sampling. This strategy is especially useful when you run your test frequently, such as part of CI. Over time you will get practically the same coverage as running all the combinations without nearly the infrastructure cost of greatly extending your grid.

Show Me The Code


        /// <summary>
        /// Get random remote browser
        /// </summary>
        /// <returns>A random remote web driver</returns>
        protected override IWebDriver GetBrowser()
        {
            DesiredCapabilities capabilities = new DesiredCapabilities();

            Random rnd = new Random((unchecked((int)DateTime.Now.Ticks)));

            switch (rnd.Next(0, 4))
            {
                case 0:
                    capabilities.SetCapability("browserName", "Chrome");
                    capabilities.SetCapability("platform", "OS X 10.11");
                    break;
                case 1:
                    capabilities.SetCapability("browserName", "Chrome");
                    capabilities.SetCapability("platform", "Windows 7");
                    break;
                case 2:
                    capabilities.SetCapability("browserName", "Chrome");
                    capabilities.SetCapability("platform", "Windows 10");
                    break;
                case 3:
                    capabilities.SetCapability("browserName", "Firefox");
                    capabilities.SetCapability("platform", "OS X 10.11");
                    break;
                case 4:
                    capabilities.SetCapability("browserName", "Firefox");
                    capabilities.SetCapability("platform", "Windows 7");
                    break;
                default:
                    capabilities.SetCapability("browserName", "Firefox");
                    capabilities.SetCapability("platform", "Windows 10");
                    break;
            }

            this.Log.LogMessage("Capability: " + capabilities.ToString());

            capabilities.SetCapability("username", Config.GetValue("sauceName"));
            capabilities.SetCapability("accessKey", Config.GetValue("accessKey"));

            return new RemoteWebDriver(new Uri(Config.GetValue("HubUrl")), capabilities);
        }

If you are interested in learning more about High Volume Testing, I strongly suggest starting with Cem Kaner’s An Overview of High Volume Automated Testing article.