Automatically detect test coupling in Protractor (randomizing test execution order)

Question

The Problem:

We have a rather large test codebase. From time to time, instead of executing all the tests, we execute them individually or in packs. But, sometimes, we see the unexpected test failures because of the tests being interconnected, coupled. For example, one test assumes there is some data created by a previous test - running this kind of test individually will fail.

The Question:

Is it possible to automatically detect which Protractor tests are coupled in the project?

Our current idea is to somehow randomize the test execution order or randomly pick up a pack of tests from all the available tests and check if there are no failures. Hence, the other question: is it possible to change/randomize the Protractor test discovery and change the order of test execution?

Inspired by Ned Batchelder's "Finding test coupling" blogpost and the Python nose test runner's nose-randomly plugin:

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example, Google’s C++ test runner google test.

Answer 1

  Did you try shuffling "it" blocks like below:

var shuffle = function (items) {

  var item, randomIndex;      

  for(var i = 0; i < items.length; i++){

    randomIndex= (Math.random() * items.length) | 0;

    item = items[i];

    items[i] = items[randomIndex];

    items[randomIndex] = item;

  }

}

describe('Suite', function() {

  it("should a", function () {

      console.log("execute a");

  });

  it("should b", function () {

      console.log("execute b");

  });

  it("should c", function () {

      console.log("execute c");

  });

  shuffle(this.children);    // shuffle the 'it' blocks

 

});