Today’s Software Reality
Software operates the products and services that we use and rely on in our daily lives. It is often the competitive differentiation for the business. As software increases in size, complexity, and importance to the business, so do the business demands on development teams. Developers are increasingly accountable to deliver more innovation, under shorter development cycles, without negatively impacting quality. Compounding this complexity is today’s norm of geographically distributed teams and code coming in from third-party teams. With so many moving parts, it’s difficult for management to get visibility across their internal and external supply chain. Yet, without early warning into potential quality risks that could impact release schedules or create long term technical debt, there may be little time to actually do something about it before the business or customers are impacted.
This lack of visibility also creates downstream collaboration issues and process inefficiency. The QA and security audit process can take more time and effort, increasing the risk of software defects slipping into the field or into production. The business impact: costly recalls, downtime, field crashes, dissatisfied customers, and brand damage. The development impact: more time spent finding and fixing problems instead of focusing on innovation, which only leads to slower time to market, increased costs, and lost customer satisfaction. It’s a never-ending cycle.
Development Testing is an innovative and more comprehensive approach to Software Testing that augments traditional methods. Development Testing helps organizations deal with the challenges described above, and brings a number of key benefits that help to make Software Testing easier.
Here are five tips to explain how organizations would implement Development Testing to achieve this goal:
- Integrate Testing into Development – as Code is Compiled
Software defects are nothing new, so why is it such a hard problem to solve? First, traditional approaches to testing, while effective testing methods, only get you so far. Traditional testing looks for the ‘expected’ problems—making sure the functionality and performance works as expected. Traditional testing also happens later in the cycle, many times after development. Because of organizational silos and conflicting priorities, it’s hard to get development to fix defects after development is complete. Second, technology adoption in development will be limited unless it’s effortless for the developer. If testing results aren’t presented to the developer in a way that’s actionable, relevant, and presented in their workflow, troubleshooting and fixing defects will slow down the entire process. Given the time to market pressure on development, defects will likely go unresolved, or go undetected altogether.
A new testing approach is needed. It’s time for testing to expand beyond its traditional definition and become integrated into every phase of the lifecycle, starting with the code itself in development. Development testing augments traditional testing to help quickly find and fix the unexpected, hard to spot, yet critical defects in the code itself, when it’s the most cost-effective to fix them.
- Prioritize testing efforts by developers to problematic applications with complex code structures
Rather than rely on traditional QA testing methods, Developers and development managers should also be able to quickly and easily manage the defects in their code – especially where code is complex. This includes prioritizing defects based upon impact and filtering defect information to view only what's relevant to them. Once the defects have been prioritized, developers should be able to automatically find all of the places the defect exists across projects and code branches – thus minimizing duplication of efforts. Then they should be able to collaborate with other developers to share triage information across distributed teams and geographic boundaries.
- Automate test execution and allocation of responsibilities within the SDLC
By helping developers to find hard-to-spot, yet potentially crash-causing defects early in the software development lifecycle, it is much easier to automate the process of test execution. This helps to reduce the cost, time, and risk of software errors.
Many organizations have integrated Development Testing in their nightly build process, which means that Developers – having checked in their code at the end of the business day – are automatically notified of problems the next day when they return to work.
Furthermore, this automation particularly comes into its own when scaling Development Testing to hundreds and thousands of developers working across geographic boundaries. This helps distributed developer teams to find and fix thousands of defects, and millions of lines of code in a single analysis. Through parallel analysis and incremental analysis, code can be analyzed in minutes - enabling developers to scan their code more frequently and efficiently. With Development Testing, analysis can be run on up on multiple cores simultaneously and users can choose to only analyze the files which have been changed or were impacted by a change.
- Educate developers to learn from their mistakes and spot code defect patterns
When faced with thousands of defects, where does a developer start? For every defect discovered, Development Testing provides a clear explanation of the defect, the severity, and location of the defect. This helps developers to quickly improve – in an automated way – their coding practices. With this visibility, developers can also dramatically reduce triage time. Individuals now have actionable information to make better fix/no fix decisions based upon impact to a single project, across all projects, across the product portfolio, and to the business, reducing the risk of schedule slips and quality issues across products. This helps developers to learn very quickly how to resolve defects fast.
Development Testing also means that developers can map every defect to a community-developed defect dictionary, to gather defect information and get a better understanding of defect severity, identify what kind of exploits are found around that defect, and get potential fix guidance. This provides one-click access to a rich knowledge base which takes the guess work out researching unfamiliar defects, and helps you identify the root cause faster.
In addition, certain defects are almost impossible to detect using traditional QA methods. For example – Inter-procedural defects, Development Testing means developers can expand function calls inline and understand the execution path for deeply nested events to get a comprehensive explanation of the defect.
Other defects are much easier to find and fix with Development Testing, such as memory corruption, resource leaks, security best practices violations, and insecure handling of data. This too helps in the process of educating developers on how to drive further quality in their code.
Development Testing helps developers to map the impact of a defect across the entire codebase, alerting them of the presence of a single defect in other projects and products that share code. It also allows developers to visualize all of the code branches together so you can see the defects that matter to them.
The process of defect disposition also becomes precise and manageable, as developers can quickly identify the impact of a defect from one part of the code on the entire product portfolio. And what was before flagged as multiple defects is now considered a single defect, increasing efficiency to fix defects faster and increasing visibility to focus on addressing the high priority defects based upon impact.
- Measure time spent on Testing against Quality Improvements achieved
Viewing and tracking defect history and resolution status at the branch level, the project level, and across projects is critical to make better decisions and measure developer productivity and quality improvement over time. Development Testing allows you to answer three critical questions:
Which defects have been fixed and have all critical defects been fixed?
Have all instances of the defect across shared code been triaged and fixed (or not fixed)?
What does my defect and quality trending look like by product, by release, by checker and defect type, and by user over time?
Development Testing provides better visibility into quality, security and efficiency trends across the organization. It helps developers to see detailed information about the number of defects since the last analysis, the defects that were found in the central build, defects found on their desktop and more. It also helps managers and business executives get a more expansive view of quality and security trends over time, by team or by software component. Managers can quickly drill down into a detailed view to get a deeper understanding of areas of risk in their organization.
About the author:
Chris Adlard is senior manager, Global Communications & Customer Advocacy, at Coverity.