It’s almost too easy to dismiss the concept of Exploratory Testing as one of the many dark arts of software testing, in which those that do not do it cannot understand it and those who cannot understand it simply do not do it. This often leads many to wonder what the benefit of it is and sometimes even mistake it for bad testing practice. I ask you to rethink Exploratory Testing as it can be an invaluable tool to any development process but only if it is performed in a meaningful and easy to understand way.
One such idea for this is the concept of Session-Based Testing. This approach aims to marry what is lacking with standard Exploratory Testing with the real meat and bones of what Exploratory Testing provides – accountability and structure with creativity and quick defect discovery.
So how do we go about doing this? The easiest way to consider Session-Based Testing is to think of it as a structured approach to Exploratory Testing – admittedly this may seem counter-intuitive but it’s worth remembering that structure does not mean set in stone. All it simply means is that we have a blueprint for what kind of work we will be carrying out during the session and ultimately how it will be reported.
The Initial Charter
The first thing you will need for this blueprint is an initial charter. Like many of the great explorers of old, they didn’t just set out in the hope of going in any old direction and finding land. They had a plan (admittedly, it didn’t always work out but the results were nonetheless impressive – failure can be a good thing) and so should we. Therefore, your charter should be what it is you wish to achieve. What is it that you want or need to explore – perhaps it’s the interoperability between system components, perhaps it’s a very focused view on one simple feature of your software – either way it’s important to establish up front what the overall goal is and also the scope of the session. People are exceptionally inquisitive beings, especially when told to explore, so a charter with a defined objective and scope will help people reel in their curiosity whilst still remaining attentive.
Another important aspect to consider is in the name itself. Session-Based Testing makes more sense when you restrict the duration. It’s true that testing is never finished, it just needs to stop at some point and the same holds true for Session-Based Testing. Timebox the test effort to something that makes sense both for the charter you have created and for those taking part. If it is a solo effort, you may wish to give yourself more flexibility, such as a dedicated afternoon or day to the effort but if it’s a group activity, it may make more sense to keep it short and snappy; forty-five minutes to an hour or two should suffice. However, it is important to remember that people’s creativity can get burnt out pretty quickly.
Now that we have our objective, scope and time sorted, you will also want to give some consideration to what risks there are with this charter. It’s helpful to call this out so those involved are aware of what may go wrong during the session, for example the environment you’re working with during the session may be “flaky”. Calling these risks out prior to the session will also help provide opportunity to mitigate them before or during the actual session.
Example Test Charter
An example charter that Vasca da Gama may have put together before his journey. Hopefully, you won’t face the potential of pirates or mutiny during your exploration but I guess you can’t rule it out.
Putting Your Crew Together
Now you know what you are testing and how long you are going to test, it’s time for testers to do their thing. If you are testing on your own, you will need to be disciplined enough to wear many hats, you will need to keep an eye on the duration, how you are recording defects, how you are recording tests executed and how what you are testing matches up against the original charter. In my experience, you will get more benefit from these sessions if they are carried out as a group. There will be more ideas put forward, more opinions, more energy and ultimately more fun.
You will need to carefully select people who are open to new ideas, whether they are testers or not. Be careful not to pick people who may bring a disruptive energy to the session, one naysayer can really spoil the mood of a group. These sessions are a chance for people who don’t normally get to interact with each other or don’t normally get a chance to flex their exploratory testing muscles. Also, I’m assuming that people involved in Session-Based Testing are competent in their jobs, these sessions won’t make people instantly better testers but will make your overall testing effort better.
To help you along your way in picking people it can be useful to assign a few roles to attendees and one specific one I’ve found to be essential is the chair; this person will be responsible for ensuring the session keeps to its laid out blueprint whilst also taking notes (or organising for the session to be recorded via video or audio or both) on the tests executed and any defects found. You may also want to include a subject matter expert too to be able to answer any questions without having to stop the flow of the session to get an answer.
There are no passengers on spaceship earth. We are all crew.
– Marshall McLuhan
Keeping a Record
You have your motley crew of explorers, you know what it is you want to explore and you roughly know how long it’s going to take. There’s very little point in exploring something if you’re unable to explain or show others how to find your discoveries, so we need to now think about how we report on our findings. There’s probably a few things that make more sense than others to record – from my experience these are:
- Tests executed - even if they produce a positive result they may be tests worth adding into any regression suite.
- Defects found - Defects should be recorded with enough information for you to write up a bug report after the session.
- Areas covered - Allows you to gauge how successful your session was in comparison to your initial charter.
Of course, you may want to capture more or possibly less than this, it all really depends on your needs. It’s far too easy to fall into a trap of performing exploratory testing and thirty minutes later realise you have no idea what you’ve been doing or how you found that great bug you wanted to tell everyone about but have since forgotten. A surefire way to get an accurate log of events is to physically record the session, ideally both audio and video, however one is better than nothing.
For this purpose I determined to keep an account of the voyage, and to write down punctually every thing we performed or saw from day to day, as will hereafter appear.
– Christopher Columbus
Hopefully now you will be in a position to make some awesome discoveries and to see you on your way, a quote that sums up this concept nicely, from one Bilbo Baggins:
It's a dangerous business, Frodo, going out your door. You step onto the road, and if you don’t keep your feet, there’s no knowing where you might be swept off to.
