Application development executives told Forrester that their biggest challenge for delivering new and updated in‐house applications is quality (Source: Application Development Executives Need Better Visibility to Become More Reliable Business Partners, Forrester Consulting, May 2007). In the same study, 43% of those executives said that they planned initiatives to reduce application development cost. 35% planned to focus on quality improvements, and 28% planned to work on their time‐to‐market metrics. When it came to quality, they said their biggest problems, aside from the perennial problem of finding enough staff, were time pressures, requirements, complexity, and processes. Poor requirements have been cited as a root problem in software quality (Source: The Root of the Problem: Poor Requirements, Carey Schwaber et. al., September 2006) since time immemorial, most commonly in the infamous "scope creep" that negatively impacts so many software development projects.
You've seen it. We've all seen it. Delivering high‐quality applications is incredibly difficult. Not only are applications commonly perceived by users as buggy, but they're often perceived as simply working incorrectly—meaning they don't meet their requirements. Unfortunately, there's no simple, easy button that you can push to improve software quality—despite some manager's uninformed perception that an add‐on suite of software testing tools will solve the problem overnight. Instead, you'll find that the ultimate answer is a complete overhaul to the way you look at software quality, and a more mature approach to the way you develop software applications, starting at the beginning of the development project. That's what this guide is all about—examining where quality problems begin to enter a project, and defining ways to improve quality throughout the life cycle of the project.
The goal of this chapter is to provide a sort of executive overview for the entire guide. In this chapter, I'll outline the problem and the basic steps toward a solution; subsequent chapters will dive into much greater detail on these steps and help you build a clear picture of what you need to do in order to bring quality applications to your organization.
Few in the IT industry could argue that there's at least a perceived problem with software quality. Few, however, like to acknowledge how difficult quality can be to attain. For example, software such as Microsoft Windows is often perceived by users as frustrating, full of bugs and instabilities, and not working the way people want. Windows Vista has made headlines with the number of users who perceive it as slow, unstable, and not doing what they want. For years, a maxim with Microsoft applications has been "wait until the first service pack" to implement. Now consider: If that's the level of quality perception that the world's largest software company can produce, what hope do you have for doing better?
Perception, of course, is a key word. Application users seem to love to hate the applications they use. But there's a reason for that. Actually, there are several reasons:
One barrier to improving software quality is that we often don't do a very good job of measuring quality in the first place. Many organizations simply don't measure quality at all, instead relying on anecdotal evidence—that is, stories from users, often voiced over the phone to Help desk technicians or managers when the user is frustrated by an immediate problem they're experiencing. Other organizations tend to measure quality in a very onesided manner, either by simply counting the number of bugs encountered by users who are using the product or by measuring the number of Help desk calls the product causes. Neither of these "metrics" actually bears any real relationship to the actual level of quality in the product:
Perhaps the worst problem with these metrics is the myriad of issues that they don't measure or address. Performance is rarely considered except from anecdotal reports of "this application is running slow." So what should you be measuring when you consider the quality of your applications?
Clearly, we need to begin measuring different things in order to accurately assess the quality of our software. The most important of these—performance and the ability to efficiently accomplish tasks—are things that have to be designed into the application long before coding ever begins. This drives right back to the oft‐stated "root of all problems"— requirements. It also drives quality away from a "let's measure defects" attitude and more toward an attitude of quality as a business metric.
It's incredibly easy to lose track of why a particular software application is being created in the first place. Developers get bogged down in the details of their code, testers get focused on creating test patterns, and managers begin to deal with staffing, scheduling, progress reports, and more. Amazingly, few organizations that have produced in‐house software can ever produce written evidence on why they produced the application. Some can perhaps produce a vaguely worded "mission statement" or "vision" for the application, but it's rare to find an organization that can succinctly state the specific business needs that the application was meant to fulfill.
You might think that's an easy task: "We need an application that will allow our sales people to enter sales orders." Simple. But is that really the problem? Are salespeople entirely incapable of entering orders today? Probably not—few applications are created to enable entirely new business functionality; the majority of applications are generally intended to improve business processes. Which means a better statement of the application's purpose might be, "we need an application that will allow our sales people to enter sales orders more quickly and from a variety of platforms including their desktops, laptops, and mobile devices." This begins to state the actual business needs through implication: "Today, salespeople spend too long entering orders, and perhaps can't do so unless they're at their desks."
Ultimately, businesses pay for software applications to be created. They do so because they expect to see tangible benefits—typically stated as a return on investment (ROI)—from the applications. From a business perspective, a "quality" application is one that provides tangible benefits quickly and consistently. That means an application that is, incidentally, free of defects, running at an acceptable performance level, and efficiently accomplishing the task for which it was designed—perhaps the three key broad areas of quality perception that I discussed earlier. But quality from a business perspective is impossible to achieve unless the business' perspective—that is, the things the business expects the application to achieve, and ideally the problems that the application is meant to solve—is clearly stated up front. That brings us right back to the application's requirements, and how poorly written requirements are a root cause of quality problems.
Let's take a moment to get on the same page with some terminology. This guide isn't about teaching you a new software development management framework; you doubtless have one, and you've doubtless worked with many over the years. Instead, let's just define a few common terms to describe the major elements present in any software development management framework so that we have a common basis for communication.
Figure 1.1: Generic software development life cycle.
Broadly speaking, most of us think about software development as occurring in four phases, as Figure 1.1 shows:
I'll use these general terms throughout this guide, and in Chapter 2, we'll focus more specifically on the issues of quality—where it comes from, what quality should actually mean, who cares about quality, and more. We'll also spend just a bit more time further defining this software development life cycle and talking about some of the oftenoverlooked tasks that should be going on during each phase.
Businesses often wonder how much it would cost to build extra quality into software applications. In fact, I've had at least one executive ask me exactly that during a consulting engagement: "Sure, quality. What would it cost to have some more?" It conjured in my mind a sort of retro‐style television ad: "New software applications! Now fortified with extra Quality™!"
Let's start by looking at what having no quality costs, or at least what having less quality costs. It's easy enough to point at bugs—defects is the polite term—and talk about how much they cost in terms of re‐programming, lost productivity, potential damage to the business through lost or incorrect data, and so forth. But I feel there's a bit too much emphasis on defects as a quality measurement already, so I want to look at some of the other, less obvious costs of quality. First and foremost is the cost required to build a software application, with the hopes of getting a tangible ROI, and ultimately not getting that return. Even a small application can cost a business hundreds of thousands of dollars in salaries alone, and if that application doesn't create the tangible return that was hoped for, those dollars are simply lost.
I once worked for a telecommunications company that wanted to develop an in‐house application for their network integration division. The application was supposed to implement a number of ISO‐certified business processes encompassing network design, customer service, and more. A dozen developers worked on the application for most of the 2 years I was at the company, never producing a single release. The project was ultimately shelved at a cost of more than $2.6 million in personnel costs. The primary reason it was shelved? When it neared time for a release, not a single manager in the company could agree that the application accomplished the business tasks they needed, in the way they needed them performed, and at the speed they required. Defects never became an issue because the application's first release had a quality of "zero" in every way that mattered to the business.
So not having quality obviously costs; but having quality costs, too. There's a definite balance that you have to strike between cost and quality. For example, that telecommunications company probably spent about a month gathering requirements— obviously not long enough. But they easily could have spent years doing so, and ultimately wound up with a better piece of software, perhaps 5 years down the line. To turn back to the old standby of software defects, it is always true that more testing will catch more defects. But at a certain point, you start finding fewer and fewer bugs per hour of testing, so there's definitely a point of diminishing return.
So where is quality ultimately lost? The key to finding missing quality is remembering that quality isn't simply a measure of software defects or lack thereof. Quality enters a project from the very beginning or not at all:
The costs of poor quality escalate more quickly in foundation software, such as serviceoriented architectures, where poor quality in terms of function and performance can be magnified as the architecture is built on. You're only as strong as your weakest link, and poor quality in the foundation software means individual applications will always suffer from inherent quality issues.
So why don't we simply put more quality into everything we do? There are a number of barriers to quality; things that keep quality at bay. Learning to recognize them and deal with them is crucial to building an organization, and processes, designed to deliver quality:
Yet, if these barriers can be overcome, both IT and the business stand to reap significant rewards:
Analysts love to measure IT organizations by varying levels of maturity, and software quality is no exception. Fortunately, software quality is an area where measuring your maturity, or "quality level," as I like to call it, can be truly beneficial. No organization can go from low quality to amazing quality overnight; it's a process that you have to follow. You can't skip stages of evolution—you need to experience each one because each one sets you up for the next.
Although everyone in the industry uses different terms, I like to break quality into four main levels:
As I've said, moving through these different levels of quality can be quite difficult because doing so involves the entire organization, not just IT. That's an important lesson that Chapter 4 will really hammer home: By itself, IT is incapable of achieving a very high level of quality. You need commitment and involvement from the entire organization. Think of it this way: If you ask someone to build a house for you, and then leave them to their own devices with no further details or instructions, you probably won't be very happy with the result. But if you remain involved throughout the process, involving an architect, an engineer, and a builder from the very beginning, then you'll have a house that's easier to build, better suited for your needs, and ultimately of higher quality.
In Chapter 5, we'll start from the very beginning—a very good place, as they say, to start. So often identified as the root cause of poor software, we'll spend time looking at what software application requirements should contain, and how your requirements are your highest pinnacle of quality, meaning you'll never achieve any level of quality that isn't present at the beginning, in your requirements.
Your requirements documents need to clearly and unambiguously state the reasons for the application in business terms. What exactly does the business want from the application and, most important, why does the business want those things? If there are existing business problems, state them, and state their impact on the business. It's not enough to simply say, "salespeople spend too long entering orders." Instead, clearly state exactly what the problem is, its impact on the business, and the desired outcome:
Today, salespeople spend an average of 40 minutes entering a single new order, including setting up a new customer in our systems. With an average of 10 new sales entered per day, salespeople spend 6-2/3 hours per day simply on paperwork—leaving little time for actual sales. This means each salesperson is only selling for about 2 hours per day, meaning each salesperson is only about 22% efficient. If each salesperson could be made at least 50% efficient, we could double sales! To accomplish this, a salesperson making 20 sales per day must spend no more than 3 to 4 hours entering those sales, meaning each order must require no more than 5 minutes to enter in an accurate and complete fashion.
Of course, being able to state the problem and the desired outcome in such clear, measurable terms requires a thorough understanding of the problem, and this is where most business' zeal to improve disappears. Business analysis is a tedious, often thankless task that employees often perceive as an investigation of them rather than an investigation of the process and its tools. But by stating the requirement in this fashion, look at all the things that are driven further down the development process:
Further, the value to the business is clearly stated: The intent is to allow a doubling of sales, an obvious business benefit. That benefit is calculable: If sales today are at $1 million, then the goal is to enable $2 million in sales. If the cost of doing so is $1 million, then it may be worth it, as the cost is repaid in a single year. If the cost of doing so is $5 million, some compromise may need to be reached, since the cost so significantly outweighs the benefit. Understanding the real value of the application is useful in making decisions and weighing the balance between risk, complexity, quality, and other factors. For example, if the developers have created a process that requires 7 minutes to enter an order, and estimate that it will take 3 more months to refine the process down to 5 minutes, management may well decide that 7 minutes is good enough. It doesn't meet the original goal, but it does meet a goal that's acceptable and in line with the benefits that can be anticipated.
The requirements must clearly state functionality quality indicators. That is, what things can someone look at, in the function of the application, that will clearly indicate good or poor quality? You may start with a business process flowchart that illustrates the business process and workflow that the application must implement or enable. However, don't leave things at such a high level. Specify other functional indicators, such as:
These types of quality metrics do two important things: they provide clear requirements that designers, developers, and testers can work with; they also reflect and communicate an understanding of current business problems. Reading a list of requirements such as this should remind the reader, "yeah, folks call the Help desk all the time now because they think the application locked up—I can see where displaying a progress bar will help alleviate that." In fact, when those reasons come to mind, they should be documented as part of the requirements so that the purpose and reason for the requirements isn't lost.
Don't fall into the trap that legislators often do and simply specify requirements without specifying the reasons. In Nevada, one county's building code—clearly not updated in recent memory—states that a builder must replace the windows in a home if defects in the glass are "visible from twenty paces at high noon." High noon? Twenty paces? You might be inclined to simplify such a provision and say, "visible from 40 feet during daylight hours," but you'd be stripping the rule of much of its intended functionality simply because the rule doesn't document the reason.
The reason is that, in Nevada, "high noon" is when the sun is typically at its zenith for the day. When the sun is somewhat lower in the sky, say in the morning or afternoon, defects in windows can be more difficult to see because the light will strike them more in parallel and pass through more easily. At noon, light rays are more perpendicular to the plane of the window, so defects show more clearly. It isn't a question of "good light" as it is of the right light.
The twenty paces bit is just a little leftover Western charm.
Here's where traditional methods of capturing requirements most frequently fail the business: Specifying quality metrics that don't relate to what the application does but which are nonetheless things people will complain about later if they're done wrong. The major areas tend to relate to security and performance. For example:
Sometimes the line between functional and non‐functional can be blurry; in an organization burdened with regulatory or industry compliance issues, for example, security requirements may well be seen as "functional." That's fine, and you shouldn't worry much about the distinction between the two. The point is to make sure you're considering these non‐functional things, and stating requirements in a way that can be clearly measured. Any reasonable person on the project should be able to look at the application and objectively determine whether it meets the requirements; any collection of 20 or more people should be able to agree on whether the application meets the requirements. That means not using subjective terms such as "fast" or "small" but rather using absolute measurements.
How will the application be maintained in the long term? Have you given any thought to it? Your requirements should clearly state maintenance requirements because you certainly don't want designers and developers making assumptions about such an important piece of the application. Consider:
Start with an acknowledgment that this application will be a living, evolving thing. That means errors will occur, things will break, and the application will change over time. Decide up front what the requirements are for dealing with these realities.
Finally, your requirements documents should conclude—or even start—with several use cases. Identify everyone who will use the application for any purpose, whether as an end user, a manager, an IT technician, or whatever. Describe what each person will do with the application, what data they'll provide to the application, and what they expect the application to provide them. Tell a short story about how the application will be used, what it will do during that use, and provide examples of the input and output that are expected. These use cases will be the very things that the application designer needs, the developers strive to enable, and the testers verify to ensure the application is doing what it's supposed to.
The idea behind a detailed requirements list is to put management, not programmers, in control of the application. By clearly specifying in objective terms what the application is providing, why it's providing those things, and how it is expected to provide them, you're on the right path to actually getting those things—on the right path, that is, to quality application delivery.
The design process picks up where the requirements definition stage ends. Requirements should be filled with business terms; the application designer's job is to translate those into a technical design capable of meeting the requirements. At a high level, this includes a basic application architecture as well as the specification of key technologies. The design phase is where you get your first look at the real cost of your requirements, and management has the opportunity to re‐state the requirements in order to achieve cost goals, if desired. For example, if the data‐transmission requirements necessitate the purchase of special database drivers, management can decide whether the cost is worth it. It's important that, if requirements are going to be re‐stated, the reasons are also documented so that everyone is clear later on why the quality might be lower than originally envisioned. It's fine to knowingly accept lesser quality as a business decision; it's not fine to acknowledge the decision and make it part of the record. Chapter 6 dives into these design details in much more depth, but let's quickly overview the major components of design with relation to quality.
The designer should translate each of the requirements into unambiguous, technical designs; literally, drawings and diagrams, whenever possible. User experiences should be documented in flowcharts, and critical user interfaces mocked up, leaving little room for imagination and error on the part of the developers. The developers have the difficult task of making it all work; they shouldn't be expected to figure out what everything will look like, too.
Report mockups, indicating the data the reports should contain, should be part of the requirements; the designer can use those mockups to help design the flow of data through the application to ensure the developers have everything they need to produce the required reports.
The real key is to continually ask, "Does this element of the design not only meet the stated requirement but also meet the reason for the stated requirement?" If the requirement merely states that an order must be entered in 5 minutes, that doesn't give the designer much flexibility. However, if the reason for that requirement is to "free up salespeople's time," maybe the designer can realize that some of the data could be entered by another user of the application in a related process. That reason helps to streamline the way the application comes together.
The design must also include quality points. As the designer specifies technologies, user interface elements, and application components, metrics for these should be stated. If a mocked‐up user interface screen contains 50 input fields and is expected to be completed in 10 seconds, that may not be reasonable. Right there, we know that the design and requirements aren't in sync. If the screen is adjusted to contain 5 fields to be completed in 10 seconds, which seems more doable, that metric can be used during the development process to determine whether it's actually possible.
Designers also have to take into account how developers will implement the design, and create a design that lends itself to quality coding practices. Creating clearly defined application modules, or components, with clearly defined input and output expectations, helps place clear boundaries around what each developer must produce.
Designers must identify key points within the application where performance will be measured. It's unreasonable to expect a team of developers to collectively meet high‐level performance requirements: One developer will state that "his" bit of code is as fast as it can be, and that someone else will have to "make up" the time. The designer alleviates this problem by creating more granular performance requirements, each of which must, of course, be reasonable and achievable, and documenting them. This allows fine‐leveled unit testing to determine whether individual components are running properly, and gives each individual developer a performance deliverable.
Maintenance must be designed, as well, starting from the maintenance requirements. Work with the people who will be performing the maintenance to understand any other constraints they may be under and to produce a design that is capable of the level of maintenance required. Push back at the requirements team if the design is becoming overly complex or seems like it will be expensive to implement; allow the business managers to weigh the balance between what they want and what it will cost.
Mock up user processes, task flows, and interfaces and present them to users. Ask them to validate the design before coding even begins, and ask them whether the flows and interfaces shown can be used to accomplish the use cases described in the requirements document. This "field test" of the design can be an invaluable way of discovering "gotchas" that weren't known to the requirements authors. For example, you might find that asking for a customer ID number on a particular screen isn't viable because the customer ID wouldn't be known at that point in the process. A redesign of the interface and even the entire task flow might be required, but doing so in the design phase will be vastly less expensive than a redesign after code has been written.
Design test scenarios based on use cases and performance criteria. Describe what test data should look like; too often, poor test data is used, resulting in a lot of software defects making it through testing. Design a means of testing: You want testing to be happening throughout the life cycle, not near the end, so you might need to design "test harnesses" and other "throwaway" code that can be used to implement testing of individual or incomplete components.
Here's an area where organizations typically have little trouble attributing quality problems! Quality is often seen as being strictly under the purview of developers, but as we've seen so far, developers are really only responsible for code defects and, to a degree, performance, neither of which comprise the whole quality picture.
Most developers are accustomed to basic "unit" tests as they develop, although you always have to be watchful for developers for whom the phrase "successful compile" also means "fully tested." Developers who are working on standalone or independent components of an application should be held to a high standard in terms of unit testing: The designer should specify a complete set of inputs and outputs and the developer can ensure that all the specified inputs result in the specified outputs. The developer is also responsible for ensuring that the various workflows involving his or her component also work properly. Testing should be continuous, throughout the development process, and the results of those tests should be clearly documented for later review.
Developers should also perform performance unit testing, using the performance criteria specified in the design. Whether it's a performance metric for a particular task within the code or a metric for an entire module or component, having each developer code to objective performance metrics is the only way to ensure that the entire finished application meets its performance goals.
Test data is a major portion of the testing process. When developers are left without formal test data, they rely on made‐up data instead—names like "John Doe" and addresses like "xxxxxxx," for example. Providing a full set of valid test data ensures that developers' unit tests are catching more defects earlier in the project's life cycle. If your application will support many languages, be sure to provide multilingual test data. Specify known bad test data, too, to make sure developers' code is handling it properly. Specify data known to be out‐of‐bounds, containing illegal characters, and other problems so that developers have an opportunity to test the full range of data that will eventually be thrown at their code.
It can become difficult in some industries to deal with test data. For example, "real" customer data might be governed by privacy regulations, so creating subsets of production data makes it difficult to use. There are alternatives, however: A jumbling of customer data can be used to mismatch names and addresses, for example, creating realistic test data without compromising individual customer privacy. Test data can be purchased from specialty vendors for a variety of purposes, or specialized test data management products can create test data that doesn't violate any rules and it represents a full gamut of possible inputs. In other situations, test data may need to be sequestered for security, and programmers may need to turn their code over to authorized testers for testing in a specialized, secured test environment.
Basic coding practices—using source control, commenting code, naming conventions, SGD automated builds, and other best practices—must also be managed, typically through peer code reviews, managerial code reviews, and even dedicated code reviewers for larger projects. Poor coding practices are a leading cause of code defects, and eliminating them requires constant vigilance. Specific applications can also be used to help detect code quality problems, and in some cases, to even help fix those problems—useful when you're dealing with legacy code in an application revision project.
The beauty of having a highly detailed design that maps directly to business requirements is that it forces developers to meet those business requirements without necessarily knowing or understanding them. We have to acknowledge that developers aren't hired for their business acumen, and that they don't work full‐time in the actual line of business. Developers' knowledge of specific business needs and individual business problems is going to be necessarily limited; that's why a good design is so important. Developers can code to a well‐made design, and so long as the design maps directly back to those business requirements, then, by definition, everything the developer does will also map to those business requirements.
Of course, that does assume the developers are coding to the design, which is why the design must be absolutely objective and unambiguous, so that anyone with sufficient technical knowledge or skill can examine what the developer is doing, review the design, and determine whether the developer is in fact coding to the design.
I was once hired to work on a documentation project for a piece of software. After the documentation had been written, we all sat in a room and reviewed it. The documentation authors were present, as were the people who had tested all the steps they had documented for using the software. At one point, we'd finished reviewing a rather complicated procedure and everyone seemed happy. One fellow, who was on the team who'd produced the software, looked incredulous. "You can't all be seriously okay with this," he said. "The product doesn't work the way this says."
Everyone's eyes got big, and they started reviewing what they'd written. "No," one of the testers said, "I tested this procedure. The software works exactly as described."
"That's impossible," the product team member said. "I wrote the design spec on this and several of these steps are wrong."
"Well," I said, shrugging, "I guess the guy who actually wrote the code didn't read your spec."
It happens. Provided testing and validation is occurring during the development process, however, you should be able to catch mismatches between the design and the code.
If for any reason the developer does need to legitimately deviate from the design, take that as an opportunity to really review the situation. Deviations from design inherently mean deviations from the requirements, and that's where trouble begins and quality is in trouble. Was the design wrong? If so, change it, but make sure it still meets the requirement. Is the requirement simply not achievable? Then revisit it. Whatever ultimately happens, make sure you're documenting the final result, the decisions that were made that led to that result, and the why behind those decisions.
Finally, we come to testing, traditionally the last stop on the quality train. Of course, testing should be occurring throughout development, and the "field testing" of design elements during the design phase is really a form of quality testing, too. Here, we're talking about functional testing, where the various bits of the application come together for the first time to perform as an integrated whole. This is the focus of Chapter 8, which includes three major areas.
Unit testing is the idea of testing individual modules of software independently to make sure each one is doing what it's supposed to do. This should occur throughout the development process as developers are writing code, but it should also be a discrete step that occurs separately as well. A unit is the smallest testable portion of a program; with a good design, an application will be comprised of a great many smaller units that can be tested independently rather than a single giant, monolithic program that can only be tested as a whole.
The design should specify test cases for each unit, typically specifying test input data and indicating what the respective outputs should be. In most cases, individual units aren't designed to be used independently, so the design should also specify the creation of test harnesses, which allow individual units to be tested by themselves.
Unit testing is a valuable part of the overall testing process because it validates the proper function of smaller working pieces. If a defect is discovered, it's easily narrowed because only one unit is involved. If each unit tests without flaw but later functional testing exhibits problems, the problem comes from interactions between the units—in other words, the scope for potential trouble is smaller. Individual units can also be tested more thoroughly with less effort than an entire application because a unit has a limited range of function compared with the entire application. Unit tests thus become not only a way of verifying quality but also of making more thorough testing more efficient and practical.
The key to good unit testing is solid, high‐quality test data spanning the entire range of data that the unit will ever see in production. Another key is consistent testing, which typically is achieved through the use of test automation suites. The thing developers hate the most is troubleshooting an intermittent bug; by testing each unit the same way every time, defects can be caught more easily. The developer can be given the exact test data that caused the flaw, and once the flaw is fixed, the test can be easily repeated to confirm that the defect was in fact corrected.
Major challenges to unit testing generally come from what I call "sync." It's rare that an application's design will stand unchanged throughout the application's life cycle; feedback and evolution are natural parts of the process. It's critical, though, that all changes derive from the requirements, and then from the design. That way, testing can be included in all changes. Test harnesses will need to be updated, test data modified accordingly, and so forth. Unit testing becomes problematic when these test elements aren't evolving at the same pace as the application itself.
Also called system testing, functional testing is when all the independent components of the application come together and are tested as a whole. Functional testing should absolutely be accomplished using good, realistic test data, automated testing tools, and other elements used in unit testing. In functional testing, however, the primary concern is completing the use cases that were part of both the requirements and the design. This is the time to make sure that the entire application not only works but does what it was designed to do, and that it solves the business problems it was originally intended to solve.
Functional testing breaks down into an almost bewildering array of sub‐categories:
Testing is also the time to dig out the original requirements document and test to it. This is the time to make sure that the originally stated requirements are being met. Ignore the design; if there's some conflict between the design (and thus what the developers created) and the original requirements, now's a good time to find it and make an informed decision about what to do.
And the time to do this testing is always. Don't wait until the end of the project to discover that the code doesn't meet the requirements—that's not delivering quality. Begin rethinking testing and testing to requirements as soon as the first testing is capable of being done. Ideally, functional testing can begin very soon after the requirements phase, concurrent with development.
In Chapter 9, we'll look at an important part of non‐functional testing: performance. Although performance can encompass many things, including scalability, capacity, response times, and more, we'll focus on two major categories: load testing and overall performance testing. These are both foundation topics that directly enable higher‐level goals such as scalability and capacity.
Load testing is the idea of throwing an expected real‐world load at an application to see how it performs. For slightly different reasons, you might also throw higher‐than‐expected loads at an application to see how large a load it can maintain, and to measure its performance at various load levels. The purpose of load testing is twofold: to ensure that the application, as a whole, can perform as required under the anticipated actual load; and to predict how much additional load the application can handle and what drop‐off in performance can be expected at certain load capacities.
Load testing is typically pretty difficult to do accurately, even using automated test suites. Simulating the activity of hundreds or thousands of users can be difficult simply because of computer hardware bottlenecks. A single computer cannot, for example, output the combined network traffic of one hundred individual computers. True high‐end load testing often requires a great deal of computer hardware, and some companies simply can't make that kind of investment.
Load testing at a unit level is typically easier, and it's one reason that load testing can be more usefully applied in multi‐tier application architectures. Simulating thousands of database connections to a database server, or thousands of connections to a middle‐tier component, is much easier than simulating thousands of users using a graphical user interface. Client software typically doesn't require load testing because it will only ever be used by one user at a time, spread across multiple computers; middle‐ and back‐tier components, which are utilized by many user connections at once, are where load testing really becomes important.
The primary importance of load testing comes from the fact that software performance doesn't change evenly as load scales. For example, an application that has a 5 second response time with 1000 users will not necessarily have a 10 second response time with 2000 users. Instead, applications tend to perform well up to a certain point, at which time performance quickly begins to dip and then plummet. Load testing is designed to ensure that the "dip point" in performance doesn't happen while the application is under the anticipated production load.
Figure 1.2: Example of a load testing problem.
Figure 1.2 illustrates this: The horizontal axis represents user connections, while the vertical represents application response time. As the number of connections increases, so does the response time—but not in direct correlation. The blue vertical line represents the anticipated number of production users while the red horizontal line is the maximum response time specified in the application's requirements. As you can see, a problem exists where the response time exceeds this maximum well before the anticipated production load is actually achieved. This is the exact type of problem that load testing is designed to spot.
In addition to overall load testing, general performance testing should be conducted throughout the development process. Individual units should be tested for performance whenever possible, according to the performance metrics detailed in the design. Functional testing of specific use cases should also be conducted, checking specific tasks against the performance metrics specified in the original requirements.
Some developers prefer to work on functionality (for example, getting it working) and then focus on performance later. I don't prefer that approach myself; I like to get the performance testing in nice and early for a very good reason: If the approach I'm using in my code isn't performing well, then I'm only going to get so far by tweaking it. I may need to take an entirely different tack in order to meet my performance metric, and that means I'm going to have to rip apart code. There's little point in "getting everything working" only to find out you have to tear it apart for performance reasons. Also, by testing for performance earlier, I can more easily determine whether the design or requirements performance metrics are achievable. If I can find out early enough that I'm never going to be able to hit my performance numbers, the designer and businesspeople can sit down with the design and requirements and decide whether they want to spend more money and time for better performance or compromise on the performance. At that time, I can also have a more realistic expectation about the performance I can achieve with the code so that the designer and business people know exactly how much performance they might have to compromise.
Finally, in Chapter 10 we'll look at the entire picture of quality application delivery. I'll take all the theory and procedures covered in the previous chapters and put them together into an actionable list that you can use to start improving your quality level. I'll summarize shopping lists for things such as tool sets that can help improve various aspects of quality, provide you with some report cards that you can use to assess your internal quality efforts, and more. We'll come up from the deep‐dive into various aspects of quality and bring it all back together with a high‐level, big‐picture look at how all those pieces fit together and create an interdependent "supply chain" for quality. Take Chapter 10 as your final "to do list" for implementing and improving application quality in your organization, without wasting time and money.
This chapter was intended to give you a broad, high‐level overview of everything that contributes to quality application delivery. In the next chapter, we'll start digging deeper by looking at things like the bad quality life cycle—how bad applications come to be, and how one bad application can set off an ever‐deeper spiral of poor quality inside an organization. We'll look at traditional ways of measuring quality and decide what does and doesn't work, and take a fresh look at how business people outside IT tend to view quality—and how they perhaps should be viewing quality, instead. Finally, we'll spend just a bit of time really outlining the phases of the software development life cycle, and talking about how its iterative nature can create real issues for quality if you're not paying attention.