Test Smarter with Linda Hayes

The Value of Values

Worksoft — Fri, 31 Dec 2010 17:53:00 GMT

As the year and decade end (or not, depending on how you think about the year 0), I have been reflecting on what has come before and what might or should happen next. In my humble opinion, Worksoft has done a masterful job of making test automation feasible...(read more)

Enterprise Operations Assurance

Worksoft — Mon, 29 Nov 2010 21:32:00 GMT

As technology becomes more integral to operations and interconnected across every aspect of the enterprise, the risk and cost of failure is expanding. Yet many companies introduce changes into their production environment on a frequent – even daily -...(read more)

Why Is Most Testing Still Manual?

Worksoft — Mon, 23 Aug 2010 20:58:00 GMT

Despite decades of investment in test automation tools, techniques and time, the majority of enterprise application testing is still performed manually. I have always believed it is because the test tools were too technical and time-consuming to use,...(read more)

In the End, End-to-End is Everything

Worksoft — Mon, 17 May 2010 18:26:00 GMT

Enterprises of any significant size depend on extensive software portfolios to operate. I know of several who manage more than two hundred applications and at least one whose inventory exceeds 500. Granted, some of these are small departmental systems...(read more)

Forget Regression Testing

Worksoft — Tue, 09 Mar 2010 22:45:00 GMT

I must confess: I don't believe in regression testing. Not that I don't believe it can work, just that it doesn't. After spending over 20 years in software testing and working with hundreds of IT shops, I have to report that effective regression...(read more)

Rewriting The Rules

Worksoft — Mon, 16 Nov 2009 20:44:00 GMT

We’ve all heard the rule of software delivery – you can choose any two of fast, good or cheap, but not all three. If you want it fast, it either won’t be good or it won’t be cheap. Or, conversely, if you want it to be good then it either won’t be fast...(read more)

The Logic Against Using Logic

Worksoft — Tue, 01 Sep 2009 18:23:00 GMT

There is no doubt that you can reduce the size and number of test cases by focusing on reusability, the question is whether that should be your priority. My experience says no. Here is the problem: If there is a process whose steps vary based on rules...(read more)

The Secret to Automation Success

Linda Hayes — Thu, 11 Jun 2009 20:34:00 GMT

If you are among the thousands who are struggling with a test automation record/play/script tool, I can tell you the secret to succeeding – or at least giving yourself the best possible chance to succeed – in a single sentence:

What you have is a tool and what you need is an application.

What’s the difference? Let’s use an analogy. Imagine that my accountant is keeping the books in a spreadsheet and, naturally, it is taking too long and he is making too many mistakes. So, I decide that automation is the answer. Would I record the current manual process and try to replay it every month? Would I buy him a compiler and expect him to develop an accounting system on the side, while he continues to keep the books by hand? Would I hire programmers to develop an accounting system for him?

I’m sure you are thinking – you should be, anyway – that all of these options are comical. The obvious thing to do is to buy an accounting application.

Now just substitute “tester” for “accountant” and “testing” for “accounting”, and you get the picture of why test automation has failed. Companies think they are buying a test automation application when what they are buying is a tool that lets them choose one of the above three options. They have no idea what they are getting into.

Recording a manual task does not make it automated. It is simply a stream of events at a given time without structure, logic or documentation. We test software because it changes, and maintenance of these sessions is difficult to impossible. Capture/playback is a marketing technique, not an implementation one.

Expecting manual testers to design and develop their own application while they are also required to meet schedules for the application under test is also absurd. And the best testers usually aren’t programmers anyway, so you need additional resources and skills to even attempt it.

But while buying a test automation application is an obvious answer, testing is not as easy as accounting. Accounting rules are standardized, testing isn’t. There is a never-ending mix of technologies, architectures, languages, and functionality that makes it impossible for one test automation application to handle all possibilities.

Or does it?

What made SAP so successful was the idea that functionality can be represented as a data model instead of as code. Instead of each customer developing their own custom applications because their processes were unique to them, SAP allows them to buy a packaged application and then customize it…without writing any code except perhaps in fringe cases.

And therein lies the core difference between a tool and an application: code versus data.

With a test recording and scripting tool, code is accumulated. Whether it is recorded, generated or developed, it must be managed and maintained. The problem is that application functionality expands over time, but the time and resources dedicated to testing are either flat or declining. This means more and more test code must be created and – especially - maintained with the same or less time and resources.

The math just doesn’t work.

So what’s the answer? A test automation application that uses a data model instead of code.

Now, you should be saying to yourself “Well of course she thinks that’s the answer. She’s a vendor and has a product she wants to sell, so her opinion is suspect.” And you would be partly right.

You would be right in that I am a vendor and do have a product to sell. I founded a company that markets a product that does exactly what I am describing. But I have also taught training classes, tutorials and seminars on how to build one yourself, and have had many successful students who were able to use the tools they had at hand to make it work.

But you would be wrong that my opinion should be suspect as a result. In fact, you should be even more skeptical if I didn’t have such a product, because it would mean I was one of those people who teaches about things I’m not capable of doing. If I hadn’t developed it, how could I be so certain that it works? What value would my opinion have if I had not put it to the test?

The bottom line is not whether or not you buy my product, it’s whether you buy what I’m saying

If you do, then your options are to do it yourself, buy it from Worksoft, or buy it from someone else. And if you decide to do it yourself because you already have an investment in a tool or because your application is not a good fit, I’ll do all I can to help you. Your choice.

Automation Works When You Do

Worksoft — Sun, 24 May 2009 21:46:00 GMT

Why is the majority of testing still performed manually? True, many tools are too hard to use and maintain, but is that the only reason? No. No matter how good the technology is, it is still only a tool. It won’t jump out of the box and start testing...(read more)

The Hidden Costs of Testing

Linda Hayes — Mon, 16 Mar 2009 17:48:00 GMT

Even though testing consumes an average of 50% of IT projects, I'm willing to bet there is not a line item in most corporate budgets called "testing". While those that develop software internally may have a QA department that is budgeted for, those that license packaged applications probably don't because test resources are borrowed from the business or other tasks. As a result, the cost of testing - which is substantial - is effectively hidden.

Even worse, I am certain there are no companies whose financial statements have a line item called "inadequate testing" or "software failures". Yet the evidence is irrefutable that the cost of failures in production can be significant or even catastrophic. There are direct costs such as increased support or helpdesk demands, rollbacks, re-testing and emergency transports, but also indirect costs in the form of lower productivity or even reduced revenue or lost customers. Then there are the ripple effects - resources that are diverted to handle emergencies are not available to complete planned projects, resulting in an overall slowdown.

It's not that these costs are absent, they are just hiding in plain sight as increased operating costs, reduced revenue or market share, and overall decreases in productivity and profitability.

So why does this matter? For the simple reason that if you can't measure it you can't manage it, and without an accurate accounting of costs management can't make informed decisions about the drivers that affect testing and all the related effects of trade-offs.

Maybe it's my inner accountant that bristles at the idea of unacknowledged expenses, or maybe it's the recent hoopla about how transparent the federal budget is or isn't, but no matter what the reason I think it's just plain wrong to essentially ignore such a critical business driver.

The Cost of Widgets

If a company is making widgets, the generally accepted accounting principles (GAAP) that must be satisfied for a clean financial audit demand a detailed cost accounting of all the components that create the widgets: direct variable costs such as materials and labor must be tracked and indirect fixed costs such as supplies and overhead must be allocated. And this is rightfully so: otherwise, how can the company determine if it can make widgets at a profit, or at what production level it must operate to breakeven?

This approach to accounting for costs enables informed decisions. For example, the decision to invest in automation equipment on an assembly line can be objectively evaluated by comparing the costs of performing a particular task manually versus automated, then calculating how many widgets over how much time would be required to recoup the cost. Throw in the cost of capital, estimate a useful life for the equipment, and viola: you have a dispassionate financial analysis of the return on such an investment.

The Cost of Software

When a company makes or buys software, these rules are strangely absent. The only GAAP rules center around whether the costs should be expensed or capitalized, based on the useful life of the application. But the detailed tracking of cost components is missing. At best, companies might track the project timeline by task, but few if any can report with any accuracy how much it costs them to analyze, design, develop, test, implement and maintain their applications.

This lack of accounting for software costs disables informed decisions. For example, I asked the CIO of a global company how much was spent on testing during their last upgrade. He estimated "A lot...at least 20%." The actual cost, according to the project manager, was closer to 70%. This level of misinformation makes it impossible to adequately evaluate decisions such as the automation of testing. It is especially frustrating when executives ask for a ROI analysis for test automation yet can't supply the most basic of metrics about their existing cost structure.

The Cost of Ignorance

In this case, ignorance is not bliss. It leads to poor decisions that result in higher costs, lower revenues and decreased profitability. There are even cases - some of them famous - where inadequate testing literally bankrupted the company. At a minimum, the hidden costs of testing mask forces that affect key business drivers and cripple effective management of scarce human and financial capital.

So how good is your software cost accounting? Take this quick test to find out:

· Of the total cost of your last project, how much was allocated to each task of the application life cycle?
· Do you account for the time spent by business users and other borrowed resources in doing testing? What was the opportunity cost of other tasks they could or should have been doing?
· How many emergency transports did your company perform last year, at what cost? What was the cost of delaying planned projects as a result of these unplanned activities?
· What was the impact on revenue and/or costs of software errors or lack of availability?

If you can't answer these questions, the cost - and value - of testing is hidden from view, and as long as that is true you won't be able to make the best decisions about an activity that consumes as much as half of your IT projects. And by any measure, that's significant.

The Cost of Doing Nothing

You might agree in principle that better information would be helpful, but since you've managed all these years without it, what's the rush? Why do anything different, especially now? Precisely because this is the best time to get smart about costs. When the economy is unpredictable - some might say unprecedented - it is time to change your thinking.

As Yvonne Genovese, vice president distinguished analyst with Gartner Research, points out: "We find that automating testing has a very compelling argument in that it's important during the downturn to help us get more efficient - and we are in dire need of ways to cut costs in our current IT environments," she said, "Manual testing is very inadequate, expensive, and introduces risk in the environment, where automated testing will help us take advantage of uncertainty and position us for the future because it allows us to test -and validate--across the end-to-end of the process, capture the test for reuse, and document the process, all while freeing up valuable resources."

Exposing the hidden costs of testing may reveal opportunities for advantages that will pay benefits far into the future. There has never been a better time.

Test Data: The Hard of the Matter

Linda Hayes — Wed, 11 Feb 2009 21:39:00 GMT

Here's a tough truth: If you can't get control of your test data, you can forget test automation. The core value proposition of automation rests on reusability, and if you can't reuse the data you can't reuse the tests. Even manual testing requires test data, of course, but a manual tester can try to find or create the data they need on the fly.

My experience shows that testers spend 80% or their time or more just trying to locate data they can use or entering the conditions they need. For more on this, see "Automated or Not, It's All About the Data" (http://www.stickyminds.com/s.asp?F=S9033_COL_2).That means having reusable data even for manual testing would improve productivity by orders of magnitude.

So what's the big deal? Can't you just copy production data? Aren't there tools out there that let you extract snapshots from databases, scramble existing data or generate fake values?

The bad news is, these traditional techniques usually don't work. The good news is the solution may be easier than you think.

The Problem with Production

Most companies use production data in some form for their testing. For one thing, it's realistic - after all, it's from production. For another, it's easy - it's data you already have. But there are problems.

The first problem is that it is probably so much data that it's costly to copy and store. Second, all that volume makes it hard to find the exact conditions you need for a particular test scenario, and it may not even exist in the form you need. Third, it's dynamic - a constantly moving picture that is too unpredictable to yield any reuse.

And lately there is a fourth reason: privacy. It's highly likely that some of the data is confidential and cannot be legally disclosed to others, including testers. There are the obvious fields like social security numbers, but others may include personal data like addresses, account numbers, and any transactions related to financial or health matters. See "Keeping Secrets: How Data Privacy Affects Testing", http://www.stickyminds.com/s.asp?F=S8327_COL_2 for more detail on this issue.

And yes, there are tools that can help with these problems. But it's not that easy.

The Trick with Tools

Database tools have been around for a long time and are widely available. Most of them were developed to test databases by generating high volumes of data or extracting selected subsets, but some are specifically targeted at testing and can locate, obfuscate or create required data values.

The trick is that it's tricky to selectively extract or create a meaningful, coherent set of data. By meaningful I mean that it contains the test conditions you are interested in, and by coherent I mean that all of the related data is included. It is not as easy as taking every Nth record or some flat percentage of the data: complex interrelationships must be maintained between data elements.

For example, testing customer orders might require the customer master record, any related contracts on file, all transactions for that customer, plus all of the warehouse locations, product inventories, shipping codes, commission records, and myriad other data elements that touch the customer or the transactions - or anything they touch.

Furthermore, few applications operate on a single source of data. Many have interfaces to other applications that take the form of even more files in a wide array of formats. Some of these interfaces are real-time, some are batch, but all must be coordinated. While database tools may help you trace the relationships between tables and fields, they may break down when external files and formats are in the mix.

And finally there is the question of dates. Dates abound in most applications and are often central to calculations and event triggers. The date of an order may affect its pricing based on contractual terms. Posting a payment in one period versus another may create a late fee or interest charge. The dates of shipments or receipt of goods may trigger automated inventory orders, and so on and on. Database tools may know about table relationships but they don't know about date dependencies.

My experience shows that despite their availability and relatively sophisticated capabilities, few of these tools are actually successfully deployed for testing because the effort and skills required to make them work is too high.

The Advantage of Automation

So what does work? Ironically, test automation presents both the challenge and the solution. It is the challenge because, as pointed out, automation won't work without reusable test data. It is also the solution, though, because automation can create the data it needs.

Think about it. If you need a customer account with particular conditions for testing, you can either try to find it or you can create it. Finding it may take a lot of time and it may not even exist in the form you need. The advantage of creating it is that you know it exists and that has the right conditions because you put it there. Another plus is that the very act of creating the customer is, in itself, a test.

Manually creating all that data is not practical, but with automation it makes sense. Test automation tools can type in data quickly and accurately. By simply planning our your test cycle carefully, you can be sure that all the data you need is there how and when you need it...and expand your test coverage along the way. See "The Test Automation Timetable: Altered States" http://itmanagement.earthweb.com/entdev/article.php/622301 for more on automating data states.

Now this doesn't mean you won't need a starting point that includes basic master data. Believe it or not, trying to start with a blank database is almost impossible: there are far too many pointers, stored procedures, and other arcane contents that you could not understand, let alone create, in your lifetime. So you will still find yourself starting with some production or sandbox data just as a backdrop, but you won't use most of it.

Instead, you will use the data you design. And that's another benefit of automated data: You can design exactly the conditions you need, and because you are in control you can be sure that dates have the right relationship to each other and to the system date.

You will still need a strategy for interfaces that will probably include maintaining copies of files and massaging the values, but since you are in control of the core database contents it will be easier to know what data you need.

If this sounds too simplistic, I can tell you that some of the largest companies in the world, with massive, complex IT landscapes that span the globe, have successfully automated both their testing and their data using this approach.

Do the Math: Automation is No Longer Optional

Linda Hayes — Wed, 07 Jan 2009 19:28:00 GMT

I don't get it. Software development has evolved from a slow, painstaking, manual discipline to a rapid, iterative, highly automated process. Computers have escaped glass-walled rooms with raised floors to land on palm-sized devices. Applications have shifted from back-office productivity aids to front line competitive weapons.

Yet the majority of functional testing is still done manually.

It's not for lack of technology - automation tools have been around for decades. I should know, I co-founded a record/play/script company in the 1980's called AutoTester and followed up with Worksoft to take the coding out of automation. And of course the test tool ecosystem has expanded exponentially with a broad range of companies and products.

Neither it is a lack of need. The graph below is my favorite because it says it all: over time, functionality grows but testing resources don't. Unless you can employ automation to create a cumulative library of tests, you are doomed to steadily decrease coverage and increase risk with each succeeding release.

The risk is also exploding. Whereas before software failure likely meant that the accounting staff had to revert to calculators for a day or two, today errors can instantaneously affect thousands or even millions of customers, tainting the company's brand, reducing revenue and driving up costs. In some infamous cases, inadequately tested software has cost hundreds of millions, even billions, of dollars.

So what gives?

As far as I can tell, it's a combination of four factors:

1. An undefined test process. You can't automate what isn't defined, and manual tests are by their nature informal, usually relying on the knowledge and skill of the individual tester. If you don't have a handle on what you need to test, you can't predict the time and effort for automation. See "The Importance of Testing Software Requirements" http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1275907,00.html for a deeper dive into this topic.

2. Lack of software testability. Many modern application interfaces are dynamically generated on the fly, resulting in window and objects that have no useful or persistent names. A lack of testability standards may also cripple automation by presenting components that tools can't interact with. These combine to make automation either impossible or so time-consuming that the payback isn't there. See "Automated Testability Tips" http://www.stickyminds.com/s.asp?F=S6013_COL_2 for more.

3. Inadequate resources. Too many companies think that all they need to do is write a check to the tool vendor and automation is on the way. But buying a tool is like joining a health club: you still have to invest the effort. And you can't expect existing staff to keep up with manual testing while automating in their spare time. See "Join the Club" http://www.stickyminds.com/s.asp?F=S10695_COL_2

4. Unrealistic expectations. Record and play demos paint a false picture of what automation is all about, so management believes that it requires nothing more than capturing the manual process and therefore expects immediate results. This leads to a failure to instill development disciplines and allocate the proper time, attention and resources to getting the process defined and automated. See "The Demise of Record Play Script" http://www.worksoft.com/pdf/TheDemiseofRecordScriptPlay.pdf for a better understanding of the true costs of record and play.

So what to do?

First of all, face up to the fact that manual testing is no longer a viable option. Don't get me wrong - you will never automate 100% of your testing, simply because some tests are not worth automating and others, like usability tests, depend on actual user interaction. But the vast majority of functional testing can, should - even must - be automated in order to achieve anything close to comprehensive coverage on time and within budget.

Second, develop and present a reasonable plan for automation that takes into account the current state of your test process and sets realistic expectations. Accept a strategy for gradual progress, and reject any plan that fails to account for the additional time and effort that will be required to get established. Granted, automation saves time and resources, but that's only after you get it into place.

Third, don't expect to exclude your experts. Unless you have clear, current and comprehensive test case documentation (and who does?), you will need access to your best and brightest to help you identify the most critical aspects of the application and the business rules that govern them.

And fourth, attack the test data and environment issue head-on. Simply put, automation requires repeatable, stable data, and this is one of the most challenging components of any automation strategy. In fact, it's such an important topic that it deserves its own separate treatment.

So stay tuned.

Forget Quality, Focus on Value

Linda Hayes — Wed, 05 Nov 2008 20:15:00 GMT

In my last blog entry I said that it is not a tester's job to find bugs. Now I'll take it one step further - testing isn't "quality assurance" either. I have never liked this term to describe testing, because testing can't assure quality. You can't test quality into a product; it's either there or it isn't.

But there is another reason, too. My observation is that companies aren't looking for quality in their software products, they want value. So what's the difference?

Let's use an analogy. You might want a high quality car, but that doesn't mean you are willing to pay $250,000 for a Rolls Royce. There is a limit on what your transportation is worth to you, especially if you could buy a house for the same price. Neither would you be willing to walk or take the bus while you saved up the money to buy a Rolls.

Instead, you would buy a car that you could afford that still met your needs for transportation and safety. For some people that might be a used Honda, for others a new Lexus, and for some a Bentley might make sense. It all depends on your perception of the value of transportation and the trade-off in time and money you have to make to get it.

Software is no different. Companies buy or build software because it provides functionality they need. While they might say they want zero defects, they are rarely willing to pay a lot more or wait a long time to get it. Instead, they will balance the utility of having the software against the inconvenience of any defects it might have. They will reject the software only if those defects prevent their enjoyment of the functionality in some material way.

The proof of this is everywhere. Companies ship or install software with known defects all the time. Most software companies have backlogs of reported defects that number in the hundreds or even thousands, and some of them have been there for a long time. So why don't they get fixed? Because to fix a defect you have to trade-off against adding a new feature or delaying a release. Quite simply, it's a value judgment.

So why does this distinction matter?

It matters because it causes a disconnect between the testers and the users. Testers report their results as a list of defects, usually categorized by severity measured by system impact. For example, a severity 1 might cause a crash or loss of data, severity 2 means a feature doesn't work, and so forth. I have been in too many release meetings where the status is reported as "X number of sev 1, sev 2 and sev 3" issues while the users' eyes glaze over. What ensues is a negotiation over downgrading the severity of an issue so that some arbitrary policy - "No sev 1 issues!" - can be mollified.

At the end, the testers come away resentful that their hard work in uncovering these issues has been devalued or ignored. Worse, they fear - sometimes rightly - that they will be blamed for the poor quality once it goes into production. Users, on the other hand, become frustrated that testers seem to find never-ending reasons to delay or deny their enjoyment of the software.

What to do?

The easiest place to start is by changing the conversation to be value-oriented. Instead of reporting "We found a high severity defect" or even "The product drop-down list is corrupt", say "You will not be able to enter an order". This shifts the perspective from quality to value. You are no longer talking about the impact on the system, you are talking about the impact on the user. Then let the users evaluate the risk and the priority.

Then, try not to prejudge what is and isn't critical to the users. You might think that not being able to enter an order would be unthinkable, but the users might decide that the majority of orders are transmitted electronically anyway, and waiting to fix this will delay the availability of a new product or service that generates more revenue than the handful of manually entered orders.

Next, check your virgin sense of quality at the door. Don't imagine that you are there to find bugs or "assure quality", realize that you are there to deliver value, and value is whatever the users say it is. Your job is to help them understand the trade-offs between enjoying the functionality and dealing with the defects so they can make an informed decision.

Finally, make sure the users understand where you stand. Ask them what is most important to them about the software and test that first. Don't let them get away with hazy or missing business requirements; insist that you know the critical processes the software must support and make sure they work. Treat users like both a customer and a partner: a customer because you work for them, and a partner because you must work together to get anything done.

And let's not forget to lose "QA". How about something that creates value...like Business Process Assurance? See "A New Twist on Test Processes" at http://www.stickyminds.com/s.asp?F=S7069_COL_2. Also, "To Win, Change the Game" at http://www.developer.com/java/article.php/614401

Test Smarter: The Top 3 Testing Myths

Linda Hayes — Tue, 26 Aug 2008 16:21:00 GMT

Welcome to my blog. My hope is to share 20+ years of testing experience gained the hard way to make it easier for you. The best way to start is to dispel the basic myths that have been around the industry ever since I got started. From there, we can move on to more positive subjects, such as how to really succeed as a tester by testing smarter. Along the way I'll also share links to other columns and articles by myself and others to expand and expound on the topic at hand.

So, here are the top 3 myths we need to lose so we can move on to what testing is really all about:

Myth #1: Your job as a tester is to find bugs. This premise is fundamentally flawed because it requires you to prove a negative, which is that there are no bugs remaining when you are finished. Sorry, but that is neither possible nor practical.

It is not possible because the potential combinations of user behavior, system environment and code conditions are essentially infinite, so you will never be finished. It is not practical because it would take too long and cost too much to even try. By the time you completed the exhaustive testing necessary to even approach full coverage the software would be obsolete and your users would have found another solution. Perfect tax software is worthless on April 16th.

Further, this myth leads to the belief that the more bugs you find, the better. This in turn encourages testers to engage in bizarre behavior and to focus on the fringes of functionality in the hopes of finding more bugs. Anyone can get software to fail if they try hard enough, but frankly users are more interested in what works than what doesn't. They developed or bought the software because they need to use it, and finding random reasons why they can't will not make you popular.

Reporting weird bugs invites developer derision and user frustration because it diverts time and resources from the real goal: making sure the software does what the users need. If you don't believe users will sacrifice quality for functionality, look no further than the 50,000+ known bugs shipped with Windows itself.

This myth also discourages testers from exercising the core functionality - the so-called "happy path" - that is most widely used because it is less likely to yield bugs. But just because a feature has worked every previous time doesn't mean it can't fail this time, and if it is vital to the user then a single failure is one too many. Waving the list of bugs you found in the weeds will do nothing to placate users if mainstream functionality fails. For more on this, see "When Bugs Don't Bug Me" at http://www.itmanagement.earthweb.com/entdev/article.php/621801.

So what is your job as a tester? It's actually much more fun and interesting than just finding problems...but you'll have to stay tuned to get the rest of the story.

Myth #2: Test new functionality first. This one has a certain surface appeal, since of course new functionality is unproven. The issue is that if it is new then users aren't using it yet, but if it is old then the odds are they are. A broken new feature may be embarrassing, but breaking functionality the enterprise already relies on could be catastrophic.

This myth leads to the dangerous practice of focusing on changes first and only doing regression testing if there is time left over. The fact is that testing must be risk-based, and change is only one risk factor. Others include volume, financial impact, and exposure. Clearly a new feature may be high risk for many reasons, but the fact that it is new is only one of them.

And, since the overwhelming amount of testing is still performed manually (more on that next), this inevitably means that regression testing is slighted. There just isn't time to test everything that is changing plus everything that isn't supposed to change.

The irony is that over its lifetime, software tends to become less reliable, not more. This is caused by a variety of factors: developer turnover, repeated patches, constant enhancement, and changing technology landscapes. The volume of code and the functionality it supports continues to expand while developer knowledge is lost over time and turnover, and as a result the software becomes increasingly more complex and less stable. For more on this, see "Anatomy of Ugly: How Good Code Goes Bad" at http://www.computerworld.com/printthis/2005/0,4814,106105,00.html.

Thus, each successive change increases the probability of unintended consequences. Coupled with decreasing regression test coverage, this is a formula for disaster. Happily, changing your strategy can make the most of your efforts and win you popularity with developers and users alike.

Myth #3: You can only automate regression tests.The overwhelming majority of testing is still performed manually, even though test automation tools have been around for decades. You have to wonder why. Part of this is because most testing is centered on new functionality, as previously discussed, and the rest is because of code-based test automation tools.

Originally, test automation tools were based on an approach called record and play, or capture/playback, which required that testers perform the test manually as it was recorded into a script file. This meant, of course, that you could not create the test until you could perform it, and you could not perform it until the software was available and working as expected.

Later the scripting languages became more sophisticated and theoretically you could code tests in advance. Unfortunately the effort to develop and maintain the code was so onerous that it didn't make sense to do it for any functionality that was in a state of flux. Thus, testers still needed to wait for the application to stabilize, and even then they only automated regression tests for features that weren't likely to change. When they did change, it was usually easier just to start over than to modify the code

What makes this whole idea silly is that we test software because of changes. Even so-called regression tests can be impacted by new capabilities. By adopting an automation approach that is so sensitive to change that affected tests are easier to throw away than fix, you lose the primary benefit of automation, which is reusability that enables cumulative coverage. And, if you lose this benefit, then the whole value proposition of test automation is also lost: you spend so much developing and re-developing tests, or attempting to maintain them, that you don't save enough time and money to pay for it. Thus, shelfware is rampant for test tools. For more on this see "The Demise of Record/Play/Script" at http://www.stickyminds.com/s.asp?F=S10939_MAGAZINE_2.

The good news is that test automation has evolved away from code to a data model that not only allows tests to be easily and quickly developed before the code is available, but also supports automated maintenance that makes reusability a reality.

So now we have cleared away the misconceptions and set the stage for an exploration of how to test smarter, giving you the opportunity to be successful and enjoy yourself along the way.

See you next time.