ML p(r)ior | Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

Maintenance of Automated Test Suites in Industry: An Empirical study on Visual GUI Testing

2016-02-03
Context: Verification and validation (V&V) activities make up 20 to 50 percent of the total development costs of a software system in practice. Test automation is proposed to lower these V&V costs but available research only provides limited empirical data from industrial practice about the maintenance costs of automated tests and what factors affect these costs. In particular, these costs and factors are unknown for automated GUI-based testing. Objective: This paper addresses this lack of knowledge through analysis of the costs and factors associated with the maintenance of automated GUI-based tests in industrial practice. Method: An empirical study at two companies, Siemens and Saab, is reported where interviews about, and empirical work with, Visual GUI Testing is performed to acquire data about the technique's maintenance costs and feasibility. Results: 13 factors are observed that affect maintenance, e.g. tester knowledge/experience and test case complexity. Further, statistical analysis shows that developing new test scripts is costlier than maintenance but also that frequent maintenance is less costly than infrequent, big bang maintenance. In addition a cost model, based on previous work, is presented that estimates the time to positive return on investment (ROI) of test automation compared to manual testing. Conclusions: It is concluded that test automation can lower overall software development costs of a project whilst also having positive effects on software quality. However, maintenance costs can still be considerable and the less time a company currently spends on manual testing, the more time is required before positive, economic, ROI is reached after automation.
PDF

Highlights - Most important sentences from the article

Login to like/save this paper, take notes and configure your recommendations

Related Articles

2018-06-02

To reduce manual effort of extracting test cases from natural-language requirements, many approaches… show more
PDF

Highlights - Most important sentences from the article

2019-05-16

Automated UI testing is an important component of the continuous integration process of software dev… show more
PDF

Highlights - Most important sentences from the article

2019-05-06

Regression testing is an important part of quality control in both software and embedded products, w… show more
PDF

Highlights - Most important sentences from the article

2019-04-05
1904.04748 | cs.SE

Unit testing has been considered as having a key role in building high quality software, and therefo… show more
PDF

Highlights - Most important sentences from the article

2019-03-25

Context: Software has become an innovative solution nowadays for many applications and methods in sc… show more
PDF

Highlights - Most important sentences from the article

2019-02-17

Background. The migration from monolithic systems to microservices involves deep refactoring of the … show more
PDF

Highlights - Most important sentences from the article

2019-02-07
1902.02610 | cs.SE

Software testing is an important phase in the software development life-cycle because it helps in id… show more
PDF

Highlights - Most important sentences from the article

2018-08-28

Coincidental correctness (CC) arises when a defective program produces the correct output despite th… show more
PDF

Highlights - Most important sentences from the article

2019-03-13
1903.05432 | cs.SE

Mutation testing is a means to assess the effectiveness of a test suite and its outcome is considere… show more
PDF

Highlights - Most important sentences from the article

2019-03-13

Combinatorial interaction testing (CIT) is a well-known technique, but the industrial experience is … show more
PDF

Highlights - Most important sentences from the article

2019-04-11

Test automation is important in software industry but self-assessment instruments for assessing its … show more
PDF

Highlights - Most important sentences from the article

2019-03-02

Deep neural network (DNN) based systems have been deployed to assist various tasks, including many s… show more
PDF

Highlights - Most important sentences from the article

2019-01-12
1901.03865 | cs.SE

What is the impact of software engineering research on current practices in industry? In this paper,… show more
PDF

Highlights - Most important sentences from the article

2018-08-29

The ever-increasing architectural complexity in contemporary ASIC projects turns Design Verification… show more
PDF

Highlights - Most important sentences from the article

2018-07-23

Test Case Prioritization (TCP) is an important component of regression testing, allowing for earlier… show more
PDF

Highlights - Most important sentences from the article

2019-02-09
1903.01222 | cs.SE

Software testing process consists of activities that implemented after it is planned and including t… show more