Quality Strategy: Test Automation

What are the key considerations in a software test automation strategy?

I recommend starting on your automation journey with the following considerations:

●       Statement of goals; what do we want to accomplish?

○        This will be a valuable reference point and it can be improved and changed as the organization learns and accumulates experience.

●       Capabilities assessment

○        What skills and automation tools already exist?

○        What skills need to be developed?

○        Inventory of tools in use, analysis, and new tool recommendations

○        Where might external specialists help fill gaps, accelerate progress?

○        Unit testing is a type of test automation; include it in the assessment

●       Engage the enterprise

○        Develop a broad base of knowledge and support for the automation strategy

●       Plan for continuous maintenance and improvement

○        For example, I recommend planning for an improvement phase in the release cycle alongside iteration, feature test, and regression test phases

○        Link to QA activities and phases diagram

●       Set realistic goals to sustainably increase test automation coverage

●       Are there current “hot spots” where the automation strategy may need to focus early?

○        For example, performance testing automation may be a higher priority than GUI functional automation

●       It’s important to have an understanding that automation can save time and increase coverage but may not necessarily cost less than manual testing

●       Not all tests can be automated economically or can be executed faster than manual tests

●       Automation is not easy but with the support of the organization, it can be successful and help achieve goals.  The support is critical.  For example, outreach can help the organization stay the course and reach its goals:

●       Success fuels engagement, creativity, confidence, and skills growth

○        Design the strategy and roadmap to get early wins

○        The early wins will bridge to a sustainable and integrated test automation capability

Metrics: Assessing Quality

Metrics enable better understanding of the business and are essential for decision support.  Here are some considerations when selecting metrics for use in assessing quality.

Enterprise quality metrics such as Net Promoter Score are important to the business to understand overall product and business performance quality.  Composite software test specific metrics are also helpful to understand the state of software quality.  For example, the organization may be interested in knowing the INC (internal non-conformance) rate compared to ENC (external non-conformance or customer identified defects).  Defect metrics get a lot of attention because they’re easy to report on and easy to understand.

Operation performance metrics are useful in assessing the ability to produce and deliver quality software on time.  The organization may want education to accompany the metrics to understand what the metrics mean and why they’re important.  For example, unit test coverage growth metrics can provide insight and understanding of the quality of the development process but some basic insight on what unit tests are would provide context on what the metrics mean.

The Scaled Agile Framework provides an excellent metrics model for the organization to use including:

1. Iteration level metrics (velocity, test cases, story acceptance, etc.)

2. Release level metrics (program velocity, automation rate, etc.)

3. Enterprise level metrics (velocity vs. capacity, feature delivery, escalation rate, etc.)

When assessing software release readiness, a weighted defect trend showing submit rates weighted by severity (impact) and priority (urgency) can be a leading readiness indicator.  If test activity remains at a steady state, scope has been maintained, support training is occurring, SRN (Support Release Note) candidates are being composed, and the defect submit trend is sustained downward, then it may be reasonable to conclude that readiness has been achieved or is approaching.  A defect submit double hump trend is a regular release cycle pattern after feature complete; it occurs as defects and secondary issues from the repairs are wrung out and is also a readiness indicator.

What metrics do you find are the most effective in assessing quality?

Think Small and Make a Big Impact

A colleague of mine emailed me a long, thoughtful message.  She wrote to describe how our discussions on management and quality were helping in her current role.  She listed several things that she thought were helping her make a difference.  None of these small items are necessarily new or original but they work and help drive good results in the business.  Think about, apply, and observe these small things constantly to make a big impact on quality and performance in your organization.

1. Focus on the small things and the big things will take care of themselves.
2. Be strategic, thoughtful, and methodical in the changes you make.
3. Some things require immediate action.  But not everything.
4. Give things a chance to work.
5. Suggest different ways to get things done and let people experiment to find what works.
6. Housekeeping.  Every day.  Use the 5S method.
7. Be persistent.  Be decisive.  Get stuff done.
8. Before you try to fix "broken people", fix broken processes.
9. Know what's important, what's urgent, and what's urgent and important.
10. Seek to understand.

Quality Minute: RACI

Quality and business performance go hand-in-hand.  There are many proven, simple techniques you can quickly put to use to help improve quality and efficiency in your business.  When you do this, more energy and time becomes available to create, innovate, and focus on your customers.

The RACI chart is one of those techniques.  It's a tool that helps teams align by understanding individual roles.  It helps teams go faster and get results.

Definitions
Responsible - The individual who does the activity
Accountable - The individual responsible for the activity or decisions
Consulted - An individual who may need to be consulted prior to a final decision or action
Informed - An individual who may need to be informed after an action or decision is made

Chart template

[Process Name]	Role1	Role2	Role3	Role4
Activity1	R/A/C/I	R/A/C/I	R/A/C/I	R/A/C/I
Activity2	R/A/C/I	R/A/C/I	R/A/C/I	R/A/C/I
Activity3	R/A/C/I	R/A/C/I	R/A/C/I	R/A/C/I
Activity4	R/A/C/I	R/A/C/I	R/A/C/I	R/A/C/I

I have used RACI charts to get fast, turnaround results on several occasions.  It's remarkable how effective it can be when applied.  In each case, the RACI chart quickly cleared role confusion.  Work got done.  People got happy.

You can learn more about RACI charting in this tutorial published by Michael L. Smith and James Erwin:  Role & Responsibility Charting (RACI)

Quality Minute: RCA

Root Cause Analysis (RCA) is a technique that helps you understand how and why problems occur.  Once you develop understanding, you can take action to address the sources of problems instead of just treating symptoms.

What is a Root Cause?

• A specific underlying cause of a problem
• Can be reasonably identified
• The organization has control to fix
• Effective recommendations can be made to fix

The RCA Process

1. Collect the data
2. Chart the causal factors
3. Analysis
4. Identify the root causes
5. Make recommendations
6. Implement recommendations

Primer
I recommend this primer to help you get started with RCA:
Root Cause Analysis for Beginners by James J. Rooney and Lee N. Vanden Heuvel

Resources
I put these resources together to help my teams perform RCA:
RCA policy
RCA procedure
RCA interview template 
RCA map