Building a testing culture

This article was originally produced as a series of articles for Testing Circus back in 2012 which can be viewed here.  I was extremely proud of the end result, and really thankful for all the effort those who participated gave.

Apart from being published as an article, I also used it with my company at the time, to show the variety of opinions within the testing community and to support the direction I was trying to move testing in.  It's an example of how the community can create a positive voice to encourage testers everywhere ...

Background

Without doubt one of the largest challenges any tester has within their organisation revolves around building up their testing culture and identity, engaging with the rest of the business / project about what testing is about, and being able to have the freedom to apply their skills so they give their company the return they need.

Without these things, we're seen as liabilities on projects, and potentially our hard work not being either understood or appreciated. Worse still it potentially leads us to perform testing tasks which are deemed important to non-testing staff, that we feel professionally are not really adding value.

Throughout August I went around a number of peer testers, asking them the same questions, trying to collect ideas and opinions from testers across the industry, testing disciplines and the world.  The responses I got back really opened my eyes to a lot of common issues and vision across the testing community, and function as the backbone of this series of articles.

Contributors

I would like to thank the following people for giving their time and passion to this project, and helping this series to represent the testing community. 

Adrian Howard  - 'Generalising Specialist' with a strong focus on more Agile ways of doing things.  Twitter id: @adrianh

Ajay Balamurugadas - Senior QA Engineer.  A man I've come to respect for his devotion to learning and teaching testing.  Twitter id: @ajay184f

Bernice Niel Ruhland - Software Testing Manager.  I regularly exchange ideas with Bernice, who is a font of wisdom.  She's also a regular contributor to Testing Circus magazine.  Twitter id: @bruhland2000

Betty Schaar - Sr. consultant and Training Services Lead at BenchmarkQA.  Another teacher of testers! I've come to know her through Twitter where I recogniser her wicked sense of humour.  I was really pleased to get Betty onboard as I'd ended up talking to a lot of Agilists, and her opinions gave a view into the non-Agile testing world, as well as worries about Agile.  Twitter id:  @swQualityQueen 

Elisabeth Hendrickson - "technically founder and president of Agilistry Studio, but you can call me whatever you want. Test Obsessed is a fine title".  I think we all know and love her as the later.  Twitter id: @testobsessed

Geoff Horne -  Prinicpal Test Manager, ISQA.  A man I was truly honoured to meet at the Kiwi Workshop on Software Testing on ethics (KWST2), and I was bowled over by his insight and experience.

Harry Bowen, Jr. - QA Contractor for Userplane.  Another tester who I've met and befriended via Twitter.  Twitter id: @Passionfortest

Janet Gregory - Agile Coach and Trainer for DragonFire Inc.  She's the Canadian based co-author with Lisa Crispin of Agile Testing: A Practical Guide for Testers and Agile Teams.  I need to stop teasing her about the snow in Canada.   Twitter id: @janetgregoryca

Jari Mikael Laakso  - Head of QA.  One of the most passionate testers I have interacted with, who is always on Twitter trying to engage with other testers and question, question, question.  That's a good tester trait.  Twitter id:  @JariLaakso

Simon Knight - Test Lead.  He is also the senior editor on the Testing Planet magazine, and technically my boss there (I'm a co-editor). Twitter id:  @sjpknight

Stephen Janaway - Senior R&D Manager in Nokia.  I have found his insights into mobile testing essential reading.  I am gutted we used to live in the same town as Stephen in the UK (before I moved to New Zealand), and I never got to meet him!  Twitter id:  @stephenjanaway

QUESTION ONE:  If you were asked to provide a testing mission statement for your organisation what would it be? Do you currently have one? How comfortable are you about measuring up to the expectation from one?

To start the ball rolling, Elisabeth provided her elegant and to-the-point catch phrase, “Empirical evidence trumps speculation. Every. Single. Time”.  I like this statement a lot – as often on projects there is a lot of hearsay and promises that a product is ready, but it doesn't mean anything until you've actually started to try and use it, and this is a great starting point for engaging with the rest of the project and managing expectations.

Bernice was rightly cautious about a mission statement that attempts a one-size-fits-all approach to testing “I have thought about writing one but not sure what purpose it would serve. For me I would rather put that time into assessing risk and testing approaches for each release we are working on. Each has its own challenges that need to be addressed. “

However the question got many thinking, Stephen echoed many people's initial reaction, “We don't currently have one.”, but then went on to say, “We should have one - previous groups I have run have had one. I'm comfortable with them - without something like a mission statement then it's not easy to get everyone behind a common goal. I think in testing especially it's important to have a common, shared goal, if only for motivational reasons.”

“If I had to write one now it would be something like 'Taking a pro-active approach to uncovering product risk' - that is, it would not mention the word testing nor the the words quality or assurance. “

Indeed, Betty also ventured her own definition, saying that if she was to write one it would most likely read, “To contribute to the delivery of 'good enough' software (as defined by the software stakeholders) that serves our customer's needs and that is supported by/with 'just enough' useful documentation for operational needs. “

Having considered the idea, Betty went on, “I like the idea of having an agreed upon testing mission statement that can be used as a touchstone for making sure our methods and actions are aligned and support these objectives.  One caveat though is that since testing is cross-functional nowadays, this really ought to be bought-into by all groups that do testing.”  Which is a really good point.

Like Bernice, Jari saw the potential pitfalls of having a generic testing mission statement, “My team is testing 4 different products. Each for the same customer, but the testing is very different. There is no common mission for the team. I guess this is slightly different for outsourcing companies than product companies.”  Indeed, when I was pooling experience within my own organisation, I found very different software delivery lifecycles, lines of supports and consequently impact of risk, which led to slightly different perceptions of the impact of defects (and hence testing).

Simon was another tester without one, but would hazard an opinion that it should be something like "provide rapid feedback to developers and clients by utilising the most effective automated and manual testing techniques intelligently and efficiently."

Ajay also felt that there was no single statement that “everyone is aware of” in his organisation, but again ventured it should be about “delivering value to the customer” by “reducing (the number of) unknown customer bugs.”.  Empowered by the idea he had this to say, “I think this testing mission should be part of every tester's goals. Therefore, I am always working on that direction to accomplish that goal” by communicating this to his team.

Janet put excellently that her personal mission statement with clients was about the need to "be curious, ask questions and collaborate with all team members to work towards having a shared common understanding of what we are building in the effort to prevent defects; Evaluate and test the product to find defects, work with programmers to fix them as quickly as possible so they are not shipped to the customer."  She went on to say there was a bunch of other things she'd probably want to add, but by having too long a mission statement it diluted its goals.

Whilst not a mission statement in itself, Adrian had this acknowledgement of the importance of testing “what we do have is a passion for great products and happy customers. Testing helps us make those happen”.

I have encountered other excellent statements for the mission for testing have come outside of this questionnaire.  I like my colleague Pam Forrest’s response that testing was a foremost about “ensuring a product was fit for purpose … that riskier areas which could impact business and customers have been evaluated within the limits of test system”.  Likewise, I found myself nodding along when James Bach says “the purpose of testing is to cast light on the status of the product and its context, in the service of my clients”.

QUESTION TWO:  Do you feel the attitude towards what testing is and what it should deliver has changed during your career? What do you feel is driving that change? Do you feel this is good or bad for testing?

Stephen reflected back on his journey with his current company, “within Nokia testing has become more accepted and is now seen as a critical aspect. Time is given for testing and resources too; this was not the case when I joined when one manager remarked to me 'can't we just find some housewives to do the testing?'.”

He continued to warn about the potential Cassandra syndrome, where testers become too linked with news the rest of the business is not prepared to believe, “I believe across the industry testing is becoming more accepted although this is mainly due to the push from the community rather than a pull from outside. We need to be careful not to push too much or risk being ignored as the ones who always shout perceived bad news normally are.”

Adrian was originally a developer, and “my early encounters with testing folk were not good. Very bureaucratic. A focus on obeying the letter of the process, rather than the spirit. For example getting new features that were making customers happy marked as 'defects' because they weren't in the original spec. The testing folk 'owned' quality, and that ownership seemed to cause problems.”

He mused that such Testing Police who were “often very focused on checklists. If it passed everything on the checklist it was okay - even when there were glaring problems elsewhere”.

However that was in the past and “I now realise that what I had encountered were bad examples of the testing profession, but without knowledge and experience of 'good' testers I tarred the whole profession with the brush of badness. I know from conversation with peers that I wasn't alone in this view. Developers views were being shaped by their encounters with bad testers and bad testing departments is still a problem I think.”

Adrian during his career had become involved initially in XP programming and later Test Driven Design in Agile.  It led to a real change of understanding, “I started reading more about testing. I was fortunate enough to work a couple of excellent testers, including those with a strong exploratory testing focus. I saw people doing much more inclusive testing with the team as a whole. I saw the value that having people embedded with the team having a strong testing specialisation brought.  I saw testers as people who helped us build quality in, rather than people who (poorly) found problems after we'd built stuff.”

Elisabeth had a similar warning tale from history “when I started in this field there was a trend toward separating testing and programming. That was a dark, dark time in our industry. It led to massive amounts of waste and ironically lower quality at a higher price. Fortunately the rise of Agile has resulted in organizations re-integrating testing into the development cycle, and often integrating the testers and programmers within teams. This leads to faster feedback and a whole lot less duplicated effort, wasted time, and higher quality.”

Janet had this to say, “I definitely think there is a difference, but only for those teams moving to Agile ... What I see more these days is testers trying to be more proactive trying to collaborate with the programmers and customer's more, worried more about delivering the right value than adhering to a given set of requirements.  However, it is a slow and hard change for many because of the radical mindset shift.”

Not everyone was sure that Agile was the way for all software development to go Betty had some concerns, “I've been frustrated a bit by the whole agile movement because of the general lack of focus on documentation (and formal requirements). All is fine so long as the group that started the work stays with it forever but that is not a good expectation in this day and age. I do not advocate for documentation that isn't of value but it's no small task to determine what is just enough/good enough documentation.” 

Given Betty's background in medical scanners, this is understandable – surely in some environments detailed requirements and strict gateways between project phases made sense.  I had some difficulty seeing some of my avionics programs I'd worked on being Agile over V-model.

Harry too had felt a little burned by an Agile project, where going Agile was, in his view, seen as an excuse for 'no documentation'*.  “Testing was made difficult without said documentation as the requirements were never set in stone, but rather evolved with the slide of Executive opinion, which could change at a moment's notice.  We went full circle on one of our products in 13 months.”

[* Of course Agile is about so much more than just “no documentation”, it's closer to Betty's definition of only having documentation which adds value]

Ajay felt that testing has changed within his organisation, and he felt two powerful agents for this change was through, the “reputation and credibility of the testers” and “general awareness (they raised) through testing war-stories”.  Reminding us of our social obligation to build and communicate our values to the rest of the organisation.

QUESTION THREE:  Are there obstacles and expectations around testing which you find challenging? How do you confront them?

A colleague from the UK who wished to remain anonymous came forward with what would be a common theme, “Time. Time. Time. Development slips and gets more time.  The result?  Testing gets squeezed. I don't think there'll ever be an answer to this, you just have to shift your risk analysis parameters”

Geoff had this to say, “testing is often seen as the poor cousin on the SDLC & project teams often consider a project delivered once it is released to test.  Testing in New Zealand is slowly changing however when budget and timeline are the primary considerations, often very difficult to translate testing into benefits over the medium to long term.”

Betty echoed this, “one aspect I find frustrating,  is that the amount of time and the amount of resources needed for testing is so often way underestimated. We're always having to justify our needs and often still not given the adequate time/resources.   I have been known to use the argument that the old rules of thumb for testing resources should be thrown out (n dev/1 tester) unless we're willing to go with double or triple the testers to dev ratio.”

Elisabeth felt herself time and again challenging the prejudice that testers were somehow the bottom of the project ladder, and people who failed to make it as developers, “people expect testing to be easy. 'It's just testing,' they say. But good testing is difficult. It requires an understanding of what we're building and why we're building it, what can go wrong, and how to ferret out the bad things before they bite our customers.”

She put the case for a better valuation of what the abilities testers need “that means that people who test need to be skilled. They have to understand technology, the business, analysis, and test design techniques. Testing yields information. The quality of the information depends very much on the caliber of the person testing.”

“However, because the product of testing is an intangible and ephemeral thing, it can be extraordinarily difficult to tell the difference between a skilled tester and someone who is just going through the motions. I suppose that's why it is so tempting for so many organizations to view testing as something that anyone can do with absolutely no training or skill. I have been fighting this 'anyone can test' attitude my entire career, so I doubt it will get better any time soon.”

Repeating our common theme, “furthermore, organizations tend to baulk if they view the cost of testing as too high. They try to measure the cost of testing separately from development, and minimize it. They fail to understand that testing isn't like insurance where it makes sense to make a risk tradeoff. It's more like scaffolding where the building just isn't going to go any higher up unless you have it. Any attempt to minimize the cost of testing is just as likely to increase the overall cost of quality. It's a systems thing and sometimes difficult to help executives understand.”

Simon echoed the feeling about testing being the poor cousin even on some Agile project and often 'the last to know', “I guess the other thing that tends to annoy me is not being involved in more planning/scoping/requirements gathering type meetings. I always see these as areas where test can add value, but nobody else ever seems to agree”

Bernice echoed the issue of attempting to complete testing against deadline, “I think one challenge is time. To manage time, I need to understand priorities plus perform risk assessment before a testing cycle begins and throughout testing. This helps identifies the scope of testing which may change as we learn more and determine how to pull in different tester's strength into the testing.”

Elsewhere there were several comments about the common misconception that testing is responsible for quality (and breaking the product).  Stephen spoke about challenging the idea “that testing assures quality. It of course does not, it highlights risk, and this is the main challenge - carrying out a role which is seen, when executed correctly, to be raising issues while taking none of the responsibility. Confrontation is best done through education and experience.”

Adrian had much the same experience, remembering that “I still encounter the old testing-department-owns-quality attitude occasionally. Changing that can be hard since it's often embedded into a larger organisation problem of how departments are rewarded/graded.  This is made especially hard since it then colours the rest of the organisation's views on what the value of testing is.”

Harry too talked about a changing role and relationship between tester and developer.  The responsibility for quality was not a 'tester thing', it was something they were trying to challenge, though “the idea that Developers should be responsible for the quality of their code is a bit foreign and partly not fair to them.  They write, we test.  As I pointed out in the presentation, 'Welcome to the 21st century!'.   Writing code and testing it are two different mindsets.”

But just as developers needed to become more test driven, so too testers needed to become more technical, “for me at Userplane, I'm testing mostly web technologies with a sprinkling of mobile.  Becoming adept at Javascript  and Selenium as our automation solution is imperative for success.  My thought processes in testing and managing the process must now include code ability and automation.  The impact is beginning to be felt and they are (so far) very happy with the QA mindset.”

Simon was another tester inspired to get more involved in the technical side, “sometimes I think I'd like to try my hand at writing some production code. I work on a small agile team and really there's no particularly good reason why I couldn't pitch in other than division of responsibility. I don't push it too much though because I've got plenty of other stuff to crack on with usually. Perhaps I just need to get stuck in rather than waiting for an invitation! “

A few anonymous colleagues from the UK and Europe echoed a familiar story “the management and sales teams don't understand testing”.  With the perception that“testers are causing delays on projects.”.  Some of this harked back to a comment that Elisabeth made that the team from developer to manager need to learn to accept that “it's not done until it's tested”.

Ajay touched a little on challenging this, also going back to touch upon on Elisabeth's mission statement of “empirical evidence trumps speculation” by stating that testers need to build “reputation and credibility … You slowly build reputation by showing them the proof to start with, then you show the bug again.”  And that he has started to record and share his testing sessions to emphasise this point.

He is also driven to save time and effort in testing, going on to explain, “the whole company was using test cases and then shifted to user stories but in an excel which took an hr to go through multiple reviews - peer review, development review, product review.  Now, after my demonstration of using mindmaps, the entire company has thrown away user stories in excel, started using mindmaps for review, tracking and saved a lot of time.”

Janet asked about why we expect everything within testing to be written up, even when other phases of the project take a lighter approach, “one of the biggest challenges / obstacles is the need for massive amounts of documentation required - mostly as a 'Cover Your Ass' response. I think this stems from a blame culture of many organizations.  It is hard for testers to let go, and realize that maybe we don't need all those metrics, and documentation.”

QUESTION FOUR: Is there anything your team does to promote and raise the profile of testing within your projects?

Stephan talks about how Nokia encouraged the whole team to be involved with testing, “we are organised as cross disciplined Agile teams and share the responsibility for testing when the deadlines come. Each team has (specialised) testers who will do as much testing as they can in the sprints”, however when push comes to shove near a deadline, everyone from developers to UX design gets involved in testing, with the specialised testers “as test advisers to others  … We use a lot of exploratory testing (ET) and session based testing which help share knowledge and experiences between the whole team.”

The result worked, as “distributing (test team) members to each project, and getting them sitting in the project teams, was the best thing I've seen in the last 12 years for promoting and raising the profile of testing.”

Betty echoed the importance of good relationships with the rest of the project, “fortunately I have been able to generate good rapport with the team and management pretty much wherever I've been and it's by repeated project successes that I've been able to garner the respect for my role. I speak up, point out issues in a way that is non-threatening to individuals, I highlight where actions taken have prevented problems, I work on developing rapport and generating good-will.  I use jocularity liberally as I've found humour to be an invaluable tool for making the work environment fun.”

As a test consultant Elisabeth observed that, “I don't usually need to raise the profile of testing, but sometimes I need to help my stakeholders understand that it cannot be an afterthought. I help them understand that features are not "done" until they are tested, meaning both checked and explored. And I help them understand the connection between fast feedback cycles and lowered risk.”

Geoff echoed much of this sentiment saying it was important to create “continuous impression upon leadership to test early, test often, test properly, test appropriately”.

Harry picks up on a theme from Ajay about building credibility within your organisation “on the previous team, we pushed for other teams to allow for us to do their testing to save them time and resources.  We developed a great reputation for fast and reliable testing.  It allowed for the company to move Mobile to our shop, and  consolidate desktop testing“ within his team.

Ajay said he helped to advance the culture of testing though introducing innovation such as mind maps, as well as developing a reputation for “raising good bugs and good bug advocacy” and engaging with stakeholders and peers about “conducting testing talks” to convey testing values.

Bernice talks about engaging with the rest of the organisation by “providing information on what we are seeing in testing to the developers and business analyst is a value-add. For example we may ask a Business Analyst if what we are seeing is what they expected from the functionality. We try to understand the bigger changes that are in the pipeline to allow us to provide any input and initial risk assessment. We show that we can add value to the overall process and not just test at the end of the process. We offer to test code in the developers sandbox if they would like our testing input earlier. “

Janet really sums up a lot of answers here.  The core thing being “I encourage all the teams I coach is to make testing visible. Create testing tasks for all testing activities. Use big visible charts such a a release test matrix to show progress, as well as extent of testing. Test results - keep them visible on the continuous integration”.

QUESTION FIVE:  What changes have you seen introduced to your test projects which have made significant impacts?  Were these changes met with initial hostility or scepticism?  How was that overcome?

Stephen feels a significant change came within the “team structure as a result of the change to the Agile wow. I've also introduced automation (as well as), using testers to pair test with developers,”.  At Nokia there were “no problems with hostility”.

Stephen then harks back to a common theme of getting testing visible, and getting everyone involved, “as mentioned above, changing to exploratory testing and SBT was a big change. Initial sessions used only testers; we invited everyone but since it was optional and testing related then only one developer every showed up. Some bribery was needed (cakes essentially) to draw people in, and after that we got a steady number of developers coming to the sessions. They would really add value with their different view-points and it was very motivating to the testers who organised the sessions to see them there.”

Betty talks again about the technical challenges of modern systems, “at one point I was having to test service oriented archictecture/middleware.  That was new for me and new for the organization that I worked at. We had to feel our way through to creating a test harness to adequately test the code without a calling application (since there weren't any yet anyway).”  But salvation came in the form of relationships, “fortunately the development team was responsive to helping us create tools to do a good job of testing”.

She also wasn't afraid to question the expectations of what testing should be delivering, “we also found that we had to deviate from standard deliverables mandated by the organization because we were a square peg being forced into a round hole. At one point, when the standard documentation wasn't giving me what I needed to do the testing, they asked me to define what should be contained in the design documents that would be useful.  I have never shied away from digging in on the technical stuff and that has taken some developers by surprise when I start asking technical questions.  I've seen initial resistance to that line of questioning but eventually a new level of respect and rapport with developers when they see how what I am asking for can help overall.”

Elisabeth makes a good point about challenging where testing should sit and 'who does testing', “I don't have "test projects." Testing is part of any project.  That by itself is a change for many organizations, and one that is sometimes met with hostility and scepticism. But when I work with organizations, part of my mission is to help them integrate testing throughout so that they don't end up with a huge inventory of unusable source code that they then have to spend months stabilizing before it's ready for prime time.”

“It takes a whole lot of patience to get people within an organization to shift their thinking from test-last thinking to testing throughout. It typically gets easier when they have experience with the horror show that test-last practices yield, or if they see big improvements from integrating testing throughout their projects.”

Harry like many wanted to introduce new ways to his organisation, “I pushed for exploratory testing to be done as a means of exposing bugs differently.  Skepticism was high so I had to continue writing out test cases and pursuing the exploratory on my own.”  Again the common theme of reputation and credibility kicks in, as “ after a few months of exposing things never before encountered for us, they got the point that we needed to stop doing things 'the way we have always done it'.  We tried for years to get automation brought in to allow for us to get past regression and allow it to be done via automation.  It took until this March to see the beginnings of that implementation (selenium).  As I pointed out in a presentation I gave last fall, test cases are only as good as the last bug written against them.  Exploratory testing will allow for us to do code-path/decision point discovery.”

Jari makes an excellent case for doing the ground work within an organisation and building the relationships by talking and involving others, “I had to make a reorganization of a test team once it grew  from 2  to over 60 people (over a relatively short time). I decided I wanted some team leads/senior assistants in the team as most of the testers were very junior. A few of those people didn't have even a year of testing experience, but they had the kind of attitude and personality I thought would fit the role.”

As many will sympathise not everything was embraced, and “in the beginning the change was challenged quite a lot. I spent a large amount of time talking with individuals, the small teams and the whole team to explain them why I made that change and why I chose those people. I also made it very clear I welcomed comments about the changes.  After some time things started calming down and the team was again productive. Some hiccups here and there, but those are always coming when dealing with people. All in all, I am very happy how we managed to overcome the change as a team.”

Ajay talks about the changes he introduced “some of them were challenged but as soon as the value was demonstrated, others could also highlight the value, people embraced it.”

Bernice makes a good point, reiterating Jari's point about engaging with people, “I find people are open to change if you do not mandate it and if you allow them to participate in pilot programs to understand testing approaches to adopt.”

Janet underlined the importance of starting small and winning people over to create a momentum of change, “my life is about introducing changes, many of them with significant impacts. Most of them are met with initial hostility / scepticism.  I find that the easiest way to overcome them, is to show success. I encourage start with something small and easy, see success. Then move to the next thing. Also, a really important thing to develop is relationships. Sometimes I introduce a practice to one eager person(s), get it to work successfully so that there is a frame of reference, and we can showcase that success to tackle the rest of the team members.”

QUESTION SIX:  Working smarter and not harder, are there any activities you’ve found your team focusing more on?  Consequently are there any thing you find yourself doing less of?

In a common theme involved testers questioning how much value detailed test cases and documentation really delivers, Stephen said that he saw testers “doing more testing and writing less test cases. In the beginning, using waterfall, we used to write hundred of tests, taking weeks then run them. Regression efforts took huge amounts of time as did maintenance of the test cases. Switching to automation, using dedicated developers in test, then swapping most of the manual effort to exploratory testing empowered the testers by recognising their expertise, whilst also removing a large amount of time spent documenting and maintaining test cases.”

Stephen had one final concern, “one area that does need to be checked at the moment is the feeling that you can just automate everything” over understanding the skills that human testers bring to a project.

Simon sees Agile being a big part of the way many products are developed in future, “Agile Test Driven Development tests in a continuous integration environment are VERY helpful and mean I spend much less time regression testing. Mobile (development and testing) is a big area of focus for the future.”

Elisabeth repeat the common themes of more “exploratory testing plus the disciplined engineering practices associated with Agile including TDD, ATDD, automated regression” going on to talk about how we need to do less “bug triage, applying test reduction strategies in an attempt to cut down the regression cycle, manual scripted testing, test documentation (including traceability matrices).”

Harry looks back at how his team has evolved and will continue to evolve “we do less manual (scripted) testing and more exploratory as well as learning/implementing automation.  What passed in the past should continue to pass and it should not fail.  Need to keep moving forward.  Once the bug is fixed, test case passes.  If automation picks up too many bugs, someone goofed”.

Betty examines the ways of working she's finding adding value in order to make testing for her clients simpler and more focused on adding real value, “I use things like test checklists, common tests, decision tables over detailed test cases where I can (and it makes sense to) in order to work smarter.  I use scenario based tests for testing third-party software unless/until issues are discovered that warrant more detailed functional tests. As a contractor who is intended to go away, I have felt strongly that I should leave behind test assets that can be used in the future.  I also test lots of third party software and shouldn't need to be doing intensive functional sorts of testing so the concept of 'just enough' testing documentation is my motto.”  On a technical level, “I have used tools to identify and pull production data, scrub production data or to generate/manufacture test data.  Virtual Machines are a huge time-saver/help when testing multiple client computer configurations.”

 Bernice  sees the value of whole-team approaches “working in teams to approach testing problems allow more testing ideas plus cross-training for the testers. Spending more time on risk assessment throughout the project to understand how risk change and to ensure that initial risks were addressed. Identifying different tools the testers can use based upon what they are testing. This can include Session-based testing, mind-mapping, test matrices, testing ideas - the level of documentation is based upon what is useful and risk level. Plus it should be a living document as it should change as we learn more through testing.

Betty has taught testing classes, and sees in modern software that databases have become more and more the heart of every application, thus “I've been surprised how many testers do not take the step of doing SQL queries to verify the stored data and simply trust what the UI shows them.  Our testing work has definitely gotten more technical over the years and that is one aspect that I've noticed that I have had to grow my skills in order to be better at doing my job - reading database schemas and related documents, design documents, writing SQL queries, etc. have all been skills that I've developed along the way. Long gone are the days of black box testing where we only needed to focus on inputs and outputs.”

Bernice picks up on the importance of engaging with ideas from the world-wide testing community, “we hold a lot of journal clubs and focused discussions with testers who are interested in exploring certain areas. For example, Michael Bolton's News Story approach from STAREast is a popular discussion with testers on how to implement his approach. It is important not to mandate what the testers must do for all testing assignments. Each should be assessed for the proper testing approaches or tools. Whereas a session-base test may work well with one testing assignment, a traditional boundary matrix may work better for another testing problem”.

I will leave the final word with Janet, who sees the future of testing quite simply, “faster feedback - continuous feedback”.

Summary

From all of this there are a number of key take-home points which come across, many of which come as no surprise,

• It's important to engage with stakeholders, and make sure your testing goals and theirs align.  If not, how can this be achieved.  As Ajay points out time and again, try to get them to think like testers through education.
• For testers to add value, testers need to be valued.  Challenge the idea that “anyone can do testing” by making your testing visible, and building your credibility.
• Build relationships.  Testing is more effective if it pools on the ideas and experience of the whole project (developers, market managers etc), not just testers.
• Innovate.  Try to challenge the idea that “we've always done testing that way”, especially if testing has always been that way and delivered little value.
• Pioneer.  If documentation is not adding value, how can it be made to?  Or should it be dropped altogether?
• Time is always going to be a constraint, and thus it's important to always to do risk analysis. Assess the risks and limitations in your testing, and give feedback on them to the project leaders.  Help them to make informed decisions.
• Software is getting more complex, and as testers we need to not shy away from that complexity.
• Challenge the idea that testers are responsible for quality.  The whole team is responsible for quality.