Pareto in Action
principled testing
December 21, 2012As simple as testing seems to those outside the test automation world may seem, a lot of thought actually goes into getting real value quickly out of the tests we write. There are many techniques we use to make testing easier. Some of these techniques are software engineering related, but some of the best techniques are cognitive.
Today we discuss an application of a cognitive tool: the Pareto Principle.
20 percent of focused effort results in 80 percent out come of results!
Planning is important. Start from the feature you're testing. Break it down into individual requirements and develop GWT statements, or any tool that explains the feature concisely.
When a user types an invalid email in the sign-up field, and 10 ms passes, an error is shown, submit is disabled, and the field turns pink.
To me, Pareto equals re-use. the statement above looks like 3 tests, but the first two statements overlap. If we didn't care where a test fails, we could stack our assertions on top of one another, and an earlier failure would prevent further assertions from running:
textField("signup") value = "email#example$com"
wait(10)
find("errorblock") should be('displayed) // fails, subsequent lines don't run
find("submit") should not be ('enabled)
find("signup").get.attribute("class") should be("error")
// and so on...
This test, fails on the first error, and results in a lack of verification for the rest of the cases. The lack of output documentation is also disturbing. We can do better, and catch every failure, like so:
import org.scalatest.{FlatSpec, GivenWhenThen}
import org.scalatest.matchers.ShouldMatchers
import org.scalatest.selenium.Firefox
class EmailFailureMode extends FlatSpec with GivenWhenThen with Firefox with ShouldMatchers {
textField("signup") value = "email#example$com"
wait(10)
"the error block" should "show an error" in {
find("errorblock") should be('displayed)
}
"the submit button" should "be disabled" in {
find("submit") should not be ('enabled)
}
"the email form" should "have class 'error'" in {
find("signup").get.attribute("class") should be("error")
}
close
}
We now have a setup with assertions in quick tests. ScalaTest runs these in order, fails the second test, and still runs the third.
Why is this an application of Pareto? 2/3 of every test above was the same workflow. When we perform that action first, and measure each result independently, we save a majority of our time. We end up performing 5 actions instead of 9. More complicated tests may run dozens of steps in preparation, which would result in greater savings of effort.