Talk:Regression testing

Older discussions

Does anyone know how to document regression testing or any other kind of testion for that matter?Stanko 19:10, 27 September 2006 (UTC)Stanko

I removed this line, as it clearly belongs in talk and not in the article itself: Remaining questions - who invented the term? Who was the first to use it systematically?

And I'm not sure about this quote (I left it in, but added quotations marks to make clear that it's a quote.)

"As a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix one must run the entire batch of test cases previously run against the system, to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly." -- Fred Brooks, Mythical Man Month (p 122)

-starwed

References

The article would really need some references/external links. Anyone having some? Cheers - David Björklund 11:21, 13 Mar 2005 (UTC)

Relation with extreme programming

The last paragraph states that "regression testing is an integral part of the extreme programming software development methodology". This is IMO abusive since it is part of many, if not all, of those methodologies. Furthermore, the article about extreme programming does not mention regression testing, which tends to prove my point. Regression testing is merely a software engineering trick that many of us use (whereas all should). - 81.56.146.158 15:58, 25 October 2005 (UTC)

I concur. Extreme or Agile programming has a different view with regards to testing in that it preaches to write tests before implementing features, something which is less common and seldom explicitly preached in other development methods. In other methodologies it's not necessarily discouraged or not practiced; it's just not explict as one of the most import principles. Wouter Lievens 08:06, 7 April 2006 (UTC)

I think that eXtreme Programming preaches to write tests before implementing features for a few reasons. One of those reasons is so that you have a full suite of unit tests to run as a regression test at any time. And I think that the reason for that is because in extreme situations, there can be no tolerance for slippage, or regression. Every update to the code must be a step forward.

Like 81.56.146.158 mentioned, regression testing is a trick that many of us use and most of us should. But I think that eXtreme Programming demands it. Maybe it has more to do with Test Driven Development. And maybe someone should update the eXtreme Programming article to mention regression testing. DRogers 12:28, 19 March 2007 (UTC)

Why does this apply only to software?

I hoped to see a definition of regression testing that applied to systems in general, not just software. Is there some reason this definition shouldn't be applied to entire systems, subsystems, and integrated hardware/software units? —The preceding unsigned comment was added by 192.45.72.26 (talk • contribs) 23:53, 25 September 2006 (UTC).

Well 192.45.72.26, the best answer I can give is that software people started this article and software people maintain it. Regression testing as applied to software is all I know. If you think that it applies to something else, I encourage you to start an article. DRogers 12:19, 19 March 2007 (UTC)

Regression testing is one of the key elements when designing integrated circuits, ASICs and FPGA based designs. In this field there are quite a lot of specialiced tools on the marked to help designers with it. On this top 10 asic design and verfication list, regression testing is on the second place. the problem is a bit, that this article belongs to the software testing portal (which is ok) and that there is no article about asic design methods and verification where it would also belong to. --84.20.36.138 (talk) 07:13, 20 April 2011 (UTC)

It belongs to Wikipedia. If you want to add a section on other areas, feel free to. You could also create a different article. It just sounds like there are different methods involved though and the concept remains the same. --Walter Görlitz (talk) 13:52, 20 April 2011 (UTC)

Important questions

The article does not answer 2 important questions:

1. What does the word "regression" in this context mean?
2. And why are the tests called "regression tests"?

I recommend you to read the much better article "Regression Testing" at http://www.wrox.com/WileyCDA/Section/id-291252.html by Adam Kolawa

—The preceding unsigned comment was added by 195.72.132.1 (talk • contribs) 09:58, 28 March 2007 (UTC1)

regression as opposed to progression - a step back not forward.

From Latin 'gressus' - Perfect active participle of gradior (“‘step, go, walk’”).

--87.194.174.252 (talk) 11:00, 17 September 2009 (UTC)

(NJgeezer (talk) 15:51, 16 November 2010 (UTC)) The Wiley article has this definition: "Regression testing identifies when code modifications cause previously-working functionality to regress, or fail..." The important point here is that the existing def in this wiki article does not explain the added nuance of "regression" , just "testing". The function of regression testing is not to test the change itself (commonly known as unit testing), but to ensure through broader testing that the change has not caused a different bug. This will almost always involve programs or modules outside the scope of the actual code changes. e.g. a change to a DB extract may have resolved a missing country's records, but those "new" records may contain data that causes a completely separate, downstream system to crash since their code does not allow for that new country's currency code. So we are testing not that the change works, but that it has not caused any application regression (loss of functionality) i.e. a failure of previously working, unchanged code.

Types of regression

61.95.203.186, I removed your edits. First, I don't think that those are types of regression. Maybe they would belong under the risk mitigation section. And in that case, I think "Complete test suite repetition" and "Partial test repetition based on traceability and analysis of technical and business risks" cover your points. Does anyone disagree? DRogers 12:48, 23 May 2007 (UTC)

Added citations and a testing resource

Since this topic is lacking cited content, I added some from Albert Savoia and Kent Beck. Kent Beck is the father of XP and JUnit (although I deliberately avoided making the new content Java-specific and stuck with what would transcend multiple languages). Alberto Savoia has written extensively on unit testing, regression testing, and developer testing.

In order to avoid tying content to any language, I dropped "JUnit Factory" down to the bottom as an external link instead of making it a reference.

Cheers. MickeyWiki (talk) 16:25, 29 November 2007 (UTC)

first paragraph a bit mistakable?

I dont know what regression testing is - but as i understand the explanation here, it is a test applied just after changes to software have been made. These changes may be "normal" changes or bugfixes. If this definition is correct, i find the the second sentence "The intend of regression testing is to assure that a bug fix has been successfully corrected.." mistakable; this sounds to me like "it is a test, that is applied after trying to fix a bugfix". So there was a mistake _fixing a bug_? I think this is too special..

In the case that my interpretation is correct, i suggest following two sentences for the beginning:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes) have been made, by partially retesting the program. The intent of regression testing is to assure that a bugfix itself is correct, or more general, that a change did not introduce new bugs.

and maybe the next sentence should be shorted as to not become repetitive.

len48 —Preceding unsigned comment added by 95.119.140.35 (talk) 19:07, 30 December 2009 (UTC)

I second the motion but don't feel comfortable making the change myself. If you need more evidence: "Regression testing is performed after making a functional improvement or repair to the program." [The Art of Software Testing, by G. Myers]

75.87.186.225 (talk) 22:15, 22 October 2010 (UTC)

I updated it taking into account what has been noted here. AliveFreeHappy (talk) 23:10, 22 October 2010 (UTC)

Is this claim too bold?

In the corporate world, regression testing has traditionally been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix.

This latter statement in the background section seems to be a bit odd and does not have any citation or reference. I would have thought defects found after QA would be even more expensive, but does agree that QA tests are more costly then dev team tests. — Preceding unsigned comment added by 95.45.199.134 (talk) 10:20, 10 June 2011 (UTC)

Both are correct. There have been several studies that show increasing cost for bug fixes at each stage of development. It is 100 time more expensive to fix the bug if found by the client than if found at the time of creating requirements. However, the phrase above just needs a little tweak: "defects found at this stage are the more costly to fix than if found earlier." --Walter Görlitz (talk) 13:50, 10 June 2011 (UTC)

Regression Testing vs Non-regression testing

Should this article be merged with Non-regression testing? What is the difference between these two terms (I do not see any). Is one of the terms more correct than the other (I was always saying Regression Testing, but recently met people that always say Non-regression Testing)? — Preceding unsigned comment added by Loomchild (talk • contribs) 14:19, 10 January 2012 (UTC)

No. The concepts are different, and "NRT" is unsupported in the literature. --Walter Görlitz (talk) 15:32, 10 January 2012 (UTC)

Thanks for explanation and for raising this topic on NRT Talk page. --Loomchild (talk) 14:27, 11 January 2012 (UTC)

I propose to remove the link and comparison to the non-regression testing page. I think the non-regression testing article is quite problematical with its lack of relevant citations and references (I couldn't find one that was relevant in the article.) I didn't find any published references and literature on non-regression testing that described it in any other way than testing, so I think the article needs a lot of work to be useful.Yorkyabroad (talk) 14:46, 31 December 2017 (UTC)

Please remove it. Should we request that the article be deleted? Walter Görlitz (talk) 01:33, 2 January 2018 (UTC)

Yes, I think article deletion is appropriate - it is potentially confusing and misleading. I guess the policy/guideline it breaches would be WP:OR as it seems to lack reliable published support. Yorkyabroad (talk) 10:27, 2 January 2018 (UTC)

We have a few choices.

1. We can request a WP:PROD. If no one contests it, the article will be deleted without a trace. Anyone can request it be undeleted, and an admin can do that.
2. We can request an WP:AFD. We discuss it and if there is a good, policy-based reason for deletion, it will be deleted. Again, it can be undeleted upon request, but that's more difficult. On the other hand, if a policy-based argument is made for keeping it (or redirecting its content after merging it), it could be closed as no consensus.
3. We (and by we I mean someone other than me) can salvage what's salvageable, and we redirect the topic here.

My preference is 3., and if we do that, it would be appropriate to have one sentence explaining that the term has been used, but it's not supported in the literature, or something to that effect. In short, non-regression testing is all testing that isn't regression testing. Walter Görlitz (talk) 05:41, 3 January 2018 (UTC)

I'm Ok with #3 - I guess that means stripping out piece-by-piece from the article - to avoid some sort of bulk-edit-undo ping-pong. I can make a start. — Preceding unsigned comment added by Yorkyabroad (talk • contribs) 12:15, 3 January 2018 (UTC)

I think we have a winner now. It came to after I saw your recent edits to the article. All that was left was the referenced testing of engine control units, and it hit me: this is an article about a term used in that field. Please check what I have done to see if that makes sense. Walter Görlitz (talk) 15:46, 3 January 2018 (UTC)

Yes, it now looks much better. I was trying to find the content of the remaining first reference to see if NRT was used there, but didn't get to the source. I'm happy to leave it as it is now. Thanks! Yorkyabroad (talk) 16:43, 3 January 2018 (UTC)

I replaced that page with a redirect to this article after finding nine books using "non-regression testing" in this meaning (6 with preview available and 3 with a sufficient snippet shown) and no book with the purported meaning. I also know customers (e.g. Siemens) requiring "non-regression testing" in this meaning. Is that sufficiently frequent to mention the term on this page as a synonyme, e.g. "Regression testing (rarely non-regression testing) ..."? --Rainald62 (talk) 07:15, 29 July 2018 (UTC)

The redirect now removes the referenced description for "non-regression testing" in engine management systems. If literature references have been found for "non-regression testing" in a software testing context it would seem that there are a couple of possible actions:

1. Insert a section on the regression testing page with a description on its equivalence to "non-regression testing" in referenced literature, with references of course,
2. Insert a section on the non-regression testing page with a description (and references) of the software testing context Yorkyabroad (talk) 14:12, 29 July 2018 (UTC)

@Walter Görlitz:, @Rainald62: - any comments on the reversion of the redirect pending inclusion of a literature-referenced comment as per #1 or #2? Yorkyabroad (talk) 14:38, 29 July 2018 (UTC)

It looks good to me. Walter Görlitz (talk) 16:11, 29 July 2018 (UTC)

1. is what I suggested above. Done. Not because the term is frequently used, but to counter the harm to the outside world caused by the hoax (purported other meaning of "non-regression testing").
2. would be a no-go (more than one article for synonyms). It's not even another context since regression testing is very common for embedded software. I added to this intro hardware changes as triggers for (non-)regression testing, with a reference from avionics (which happens to be close to my professional field). In the main text, that point could be detailed using the case of "non-regression testing in engine management systems" as example (I estimate that just one sentence remains after removal of what is already here or covered by a link to HIL testing). --Rainald62 (talk) 17:35, 29 July 2018 (UTC)

edits by anon Woodbridge, Connecticut

These edits are not correct. You want to find errors, not defects. The errors are created by developers and they are not defects until someone actually triggers them. Also, defect is a euphemism for a software bug which is described as "A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways." This is why we should not be using defect. As for "or other software properties", what sort of weasel words are those? Functionality is all that is important in regression testing. There are no other relevant "software properties" (whatever that means). --Walter Görlitz (talk) 22:59, 28 February 2012 (UTC)

I have no hope of convincing Walter but here are some definitions that should illuminate the two issues. Emphasis added.

error – “a human ACTION that produces an incorrect result, such as software containing a fault" (ISO 24765:2010, Systems & Software Engineering Vocabulary)

defect – “a problem which, IF NOT CORRECTED, could CAUSE an application to either fail or to produce incorrect results” (Id.)

regression testing – “testing required to determine that a change to a system component has not adversely affected functionality, RELIABILITY OR PERFORMANCE, and has not introduced additional defects” (Id. Also: ISO 90003:2004, Software Engineering Guidelines for the Application of ISO 9001:2000 to Computer Software)

The other option is to argue with international standards. — Preceding unsigned comment added by 69.183.41.121 (talk) 02:44, 29 February 2012 (UTC)

It's not about convincing me, it's about not being vague and not using terms in incorrect ways. --Walter Görlitz (talk) 03:09, 29 February 2012 (UTC)

I gave my sources of authority. What are yours? — Preceding unsigned comment added by 69.183.41.121 (talk) 03:30, 29 February 2012 (UTC)

A library of a dozen books, fifteen years of Better Software Magazine and twenty years in the industry. Many of the testing articles on Wikipedia were either started or heavily edited early on by me.

The fact that you have sources don't mean that your additions are any more clear than what was there. Your addition of "or other software properties" is vague and your sources (which are not listed in the article per WP:V and WP:RS) don't help make it less vague. The fact that your ISO terminology differs from some of the rest of the industry doesn't mean that it's right, its one opinion. Check the software bug article to compare with consensus. --Walter Görlitz (talk) 03:35, 29 February 2012 (UTC)

ISO standards come into existence through world-wide consensus among experts in the subject domain. May I ask one more time for the precise sources for your def’ns of error, defect and regression testing, with quotation of these definitions, so everybody can compare authority and actual words. 69.183.41.121 (talk) 04:02, 29 February 2012 (UTC)

Check the software bug article. --Walter Görlitz (talk) 04:16, 29 February 2012 (UTC)

That's what I did and that's why I made the changes - just in the first sentence, for a start, the article needs it. If the present version is representative, nobody should be surprised by the sorry outcomes of much of the software engineering industry. Cheers, 69.183.41.121 (talk) 04:47, 29 February 2012 (UTC)

The only sorry thing is your addition of ambiguous statements. Please expand the "or other software properties" phrase or remove it. --Walter Görlitz (talk) 04:56, 29 February 2012 (UTC)

Expanded phrase, done. Thanks. 69.183.41.121 (talk) 05:05, 29 February 2012 (UTC)

Now that I have time to read the whole I understand. You're confusing functionality and functional testing, which is why you're comparing it to non-functional requirements. Functionality is not the same. Fixing. --Walter Görlitz (talk) 06:13, 29 February 2012 (UTC)

No, you don't understand the whole - but that's ok. Your edit sounds as if it comes from a German or Dutch speaker. No verbs at the end of a sentence. Try again. 69.183.41.121 (talk) 07:45, 29 February 2012 (UTC)

Fixed the incorrect change to the relative clause. --Walter Görlitz (talk) 14:15, 29 February 2012 (UTC)

Good to see “error” removed and “non-functional” introduced. The non-functional aspects are the “number one risk” in software engineering. (CrossTalk, April2007, p.12) Being such a risk the non-functionals need full attention by regression testing. 69.183.41.121 (talk) 15:55, 29 February 2012 (UTC)

But regression testing doesn't usually include non-functional testing despite it being the number one risk. We're not creating policy but should be reflecting actual use. --Walter Görlitz (talk) 15:57, 29 February 2012 (UTC)

You have to start moving to different circles. 69.183.41.121 (talk) 16:12, 29 February 2012 (UTC)

Vague definition

A few comments on the heading of this article, if I may:

"Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them."

Here, "regressions" are equated with "new software bugs". 'Regressing', or 'taking a step back', clearly points to the past, whereas 'new' does not. Regression testing seeks to verify that a system has not regressed to a prior, less functional or less operational state after the introduction of changes.

"The purpose of regression testing is to ensure that changes such as those mentioned above have not introduced new faults."

Regression tests are pre-existing tests (made during earlier development) that either test functionality or bug fixes, and as such can only test for known conditions, not new ones. The kinds of tests suggested here regarding enhancements etc. might be called progression tests.

To illustrate: in the latter case (testing bug fixes), when a bug is found during use of a system, a test case is written that reproduces the occurrence of the bug, testing for the expected outcome. When the bug is fixed, the test will pass. These fixes and tests are committed to the system's source repository to ensure the bug does not re-appear unnoticed. While fixing such a bug, the test is usually classified as a unit test or functional test, depending on the scope. In future software revisions, re-running those earlier tests is what constitutes regression testing.

"One of the main reasons for regression testing is to determine whether a change in one part of the software affects other parts of the software."

This is, IMHO, besides the point; such determinations are the purpose of Integration testing.

"Common methods of regression testing include rerunning previously completed tests and checking whether program behavior has changed and whether previously fixed faults have re-emerged."

This is more to the point; however, regression testing only consists of rerunning previous tests. Any new tests added will not be regression tests until they are re-run in the future; while they are new, they are unit/functional/integration/what-have-you tests. Also, "checking whether program behavior has changed" is too vague of a purpose; perhaps "verifying program behaviour has not changed" is more precise.

"Regression testing can be performed to test a system efficiently by systematically selecting the appropriate minimum set of tests needed to adequately cover a particular change."

I would expect the heading to present a clear, concise definition and not go into details about the possibility of optimizing for efficiency. Further, it suggests that one of the purposes of regression testing is to be able to only select those tests that test the impact of a change, which it is not. Typically, regression tests are "blind": they test system functionality to ensure a change does not have unintended side-effects, which goes counter to being able to adequately determine all the effects of any change.

"Contrast with non-regression testing (usually validation-test for a new issue), which aims to verify whether, after introducing or updating a given software application, the change has had the intended effect."

Now this is good; and it is easily seen that when future changes are made, re-running such a test is considered a regression test as it will assert that the previously added functionality is still in effect.

Kenneyw (talk) 13:17, 27 June 2015 (UTC)

Regression means to revert to a former or less developed state. https://en.oxforddictionaries.com/definition/regress. Regression testing is when you ensure that the bug you fixed yesterday isn't reintroduced today. Any new bugs might be discovered in functional testing, integration testing, or even in unit testing. The idea that old and new bugs are found in regression testing would imply that the only testing there is, or needs to be, is regression testing since all bugs, old and new, are found there. If you go through the functional testing and it breaks - you have a bug. But the expectation is that you shouldn't have yesterday's bugs and since you (should) have already created a test for yesterday's bug when you fixed it, you run your regression test, or suite of tests related to bugs you fixed, to ensure that no regression exists before moving on. You may run your regression tests and your functional tests in the same click of the mouse or you may separate them, but the purposes are distinct. New bugs are not regression. They cannot be, by the definition of regression. - Dale — Preceding unsigned comment added by 138.32.32.166 (talk) 16:11, 20 February 2018 (UTC)

Regression Testing

Hey, I had edited the content of regression testing in Wikipedia page and was left with your comment

"Revert as WP:OR the content is focused on web testing and was almost entirely unreferenced. The writing style was inappropriate with a lot of bullet lists. Refs do not go into headings"

1) I had the same feeling that there were a lot of bullet points and am working on correcting them.

2) I had posted a few references from published journal papers. Are journal papers not a good source for content?

3) And about the web testing, some of the tools were mentioned and not all. NarenThanikesh (talk) 15:43, 14 September 2016 (UTC)— Preceding unsigned comment added by NarenThanikesh (talk • contribs) 15:30, 14 September 2016 (UTC)

1. I appreciate the removal of bullet lists
2. I did not see reference. Yes journals are fine, but books would be better. See WP:RS
3. Regression testing does not require tools at all, and the tools you listed, Selenium, WATIR, etc. are web tools. What about xunit or other automated functional GUI testing tools? There are also drawbacks to using tools, most famously addressed in James Bach's "Test Automation Snake Oil". Walter Görlitz (talk) 15:58, 14 September 2016 (UTC)

Thank you for pointing out the changes to be made.

1. Will look into the tools section, What about the drawbacks section? Was the content at fault , because again i read about those drawbacks in journals and referenced them as well. NarenThanikesh (talk) 16:41, 14 September 2016 (UTC)

Use of Blockchain in Regression Testing

The use of regression testing in blockchain technology and cryptocurrencies is important to mention in this article due to following reason:

1) Regression test (Regtest) mode provide developer to run in local private network, without knowing the use of regression test mode, developer will affect public blockchain information which is undesirable and have big risk from loss.

2) Blockchain technology, unlike traditional technology which can delete testing data, data in blockchain are immutable and cannot be delete.

Therefore, regression test mode in blockchain can be more important than traditional software technology. Joeccho (talk) 02:59, 17 April 2023 (UTC)

There is absolutely nothing distinctive about this usage as compared to regression testing of any other type of software. The only source provided is a primary source. Per WP:UNDUE it does not merit a mention. MrOllie (talk) 03:08, 17 April 2023 (UTC)

Traditional software technology, including mission critical software, have a strict control on deployment, it is not possible for developer to deploy the update without going strict testing.

Blockchain, however, ofter adopt loose control, developer may not know about regtest and will suffer loss if they directly launch to live without regtest.

So I think it is worth to mention. Joeccho (talk) 03:14, 17 April 2023 (UTC)

That's not true, in either case. Non-blockchain (I guess that is what you mean by traditional) has many incidents of untested code being accidentally deployed to production or polluting production databases. And if it were possible for one developer to accidentally damage the main blockchain, it would have happened long ago. Prevention of things like that is the reason that the entire infrastructure of miners/validator nodes exists. MrOllie (talk) 03:23, 17 April 2023 (UTC)

I think the risk are different. The risk of non-blockchain software is often bear by the company and the public. However, the risk of blockchain is often the developer himeself. The developer risk to loss his money if he do not have sufficient test.

For the source, the use of secondary source are more reliable than primary source are often true in medical and natural science discipline, but not in software technology discipline. Say a meta analysis and review on medicine are often more reliable than the primary research of the medical producer. However, you cannot say the secondary source about Windows is often more reliable than Microsoft itself about Windows (except the criticism of Windows). Joeccho (talk) 03:33, 17 April 2023 (UTC)

There is no exception for software topics in Wikipedia's sourcing policies and guidelines. Secondary sources are always preferred. The reason we require them is to avoid original research. That's what you are doing here - I think the risk are different. We cannot have a section in this article based on your personal opinion that there is some sort of difference between testing blockchain software vs anything else - there is no such difference, which is why there are no secondary sources to support your claims. MrOllie (talk) 03:48, 17 April 2023 (UTC)

There are no one-size-fits-all guideline for reliability. For example, If you talk about software end-of-life policy or software support policy of a software, primary source (official website) must be only reliable source and other secondary source (university, academic paper, newspaper, etc) are consider less reliable. However, if you talk about benefit of a medicine, secondary source meta-analysis are usually reliable than primary. If you talk about a user procedure or user guideline or user manual of a product, primary source are often the most reliable sources, but if you talk about the benefit and the criticism of the product, then secondary sources are generally more reliable due to potential conflict of interest for primary source. In my case it is a user manual so it is more reliable than secondary source. There are many secondary source to support, mainly from blockchain and cryptocurrencies communities, and some books, for example, Blockchain by exmaple. p. 39. ISBN 9781788473910.. However, official website consider most reliable. It seems this is the limitation of the guideline and need to found a way to improve it. Joeccho (talk) 04:19, 17 April 2023 (UTC)

What you're saying here is plainly contradicted by Wikipedia's policies at WP:V, WP:NOR, and WP:RS. You might hold such beliefs personally, but you're not going to be able to put that into practice on Wikipedia. And once again, you have not shown that regression testing on the blockchain is any different from anywhere else, and per WP:UNDUE your section doesn't belong in the article. MrOllie (talk) 12:45, 17 April 2023 (UTC)

For the reliable source, there are another guideline on primary source, see WP:PRIMARYNOTBAD. Joeccho (talk) 11:10, 18 April 2023 (UTC)