1

Effort Estimation Studies in Agile: A systematic Review

Wanderley E. G.1, 2, Ávila, B. T.2, Vasconcelos, A. M. L. 2
1
Instituto Federal de Pernambuco IFPE, Campus Garanhuns, Garanhuns, PE, Brasil
2
Universidade Federal de Pernambuco UFPE, Recife, PE, Brasil

This paper presents the results of systematically reviewing the current research literature on effort estimation in agile projects
(APs). The primary purpose is to show evidences about common trends, gaps and recommendations for future studies. The results
were limited to peer-reviewed conference papers/journal articles, written in English and Portuguese and published before 2014. The
synthesis was made through classifying the papers based on different properties. The analysis indicates the need for future research
on: 1) estimating effort for the remaining APs, 2) analyzing the impact of properties of historical/current project data, 3) using
benchmark data, 4) using composite models and 5) further empirical validation of the estimation models.

Index Terms—systematic review; effort estimation; cost estimation; size estimation; agile; scrum; XP.

I. INTRODUCTION B. Data Source and Search Strategy

Agile methods have been adopted in many software The well-known electronic databases were searched
organizations [2][3][4]. These are approaches to software including IEEE Xplore, Scopus, ACM Digital Library,
development based on iterative and incremental development, SpringerLink, Science Direct, IEEE Xplore, ACM Digital
direct involvement of the customer in the process, fast Library, Springer Link, Science Direct. To help the search was
delivery of features for greater business value and answers fast used a academic tool called ReviewER[10].
to changes [5]. However, the effort estimation in the APs is The search strings was built using a combination of RQ1
dealing with new challenges [6]. Recently, a number of keywords and its synonyms or derived words, concatenated
estimation approaches have been proposed for the APs. In this through the Boolean operators "OR" and "AND". The search
review, we seek to evaluate, synthesize, and present the string derived from the combination is presented below:
empirical findings concerning estimation approaches for the
APs. The objective is to identify common trends and gaps in  (("software metrics" OR "metric sized software" OR
current researches that could be focused on by future "metric software" OR "metrics size software" or
researches. This study is an extension of the mapping driven "estimating software" OR "measurement
by Cao and Cong [7], removing the part of the mapping where software") AND ("agile" OR "agility") AND
they do not evaluate agile development methods. ("scrum" OR "extreme programming" OR "xp" OR
The rest of the paper is organized as follows. Section II "dynamic system development" OR "DSDM" OR
describes the review process. Section III reports the results. "crystal methodologies" OR "crystal clear" OR
Section IV presents conclusions and future research directions. "crystal orange" OR "red crystal" OR "crystal
blue" OR "feature driven development" OR "dd"
OR "lean software development" OR "adaptive
II. REVEW PROCESS software development" OR "test driven
development" OR "tdd")).
We developed a review process by following guidelines of
In addition to the selection of articles, they were also
B. Kitchenham [1]. The process specified research questions,
selected manual way of articles using three approaches:
search strategy, data source, inclusion/exclusion criteria,
• Specific conferences and journals of the areas, for the last
quality criteria, data extraction and synthesis.
five years;
• The references cited in the selected studies;
A. Research Question • Studies of researchers who are references in the study
area.
Regarding the need for conducting a systematic review, the The conferences consulted:
research questions for this study set as follows: • Agile Development Conference;
• Conference XP.
RQ1: What metrics for measuring software effort are used in The journals consulted:
projects that adopt agile methods? • ACM Transactions on Software Engineering and
Methodology;
RQ2: Under what circumstances metrics for measuring • Empirical Software Engineering;
software effort are successfully applied? • IEEE Software;
 2

• IEEE Transactions on Software Engineering; identify potential primary studies. Duplicated studies will be
• Journal of the ACM; discarded. All studies identified at this stage should be named
These searches resulted in 97 papers kept for inclusion and by a unique ID and stored in a web repository shared with all
exclusion process. researchers.

Stage 2: Each selected study in prior stage should be analyzed

C. Inclusion and Exclusion Process by a pair of researchers, based on analysis of the title and
abstract, discarding studies that are clearly irrelevant for the
The papers were considered independently by the authors. search. When the researchers were in doubt whether to include
Each paper was checked against the following criteria: (1) a study or not, it should be included for consideration at a later
Inclusion criteria: studies dealing on measurement software stage.
effort in software projects using agile methods, industry Stage 3: Once the studies are selected in the prior stage, they
studies or academia, qualitative or quantitative research, will be reviewed by a pair of researchers, applying the
primary studies and (2) Exclusion criteria: studies written in a inclusion and exclusion criteria. In case of disagreement over
language other than English or Portuguese, duplicate or the inclusion / exclusion of a certain paper, we stress again to
repeated studies, studies that do not treat software metrics, discuss if the study should be included or not. However, if
they do not come to an agreement, the opinion of a third
incomplete studies, drafts, presentations or summaries slides,
researcher should be considered.
tertiary studies and meta-analyzes, academic studies that
Stage 4: Thus, a list of primary papers was obtained for
address the educational agile methods, studies that do not
critical appraise according to criteria defined in Section 2.1.8.
address at least one agile method, articles that are not available Stage 5: The data obtained from the studies were extracted to
for free download in institutional environments CIn / UFPE. answer the RQ1 and were synthesized.
Finally, we keep 40 papers as primary studies, as shown in the
Appendix.
F. Data Extraction and Synthesis

D. Quality Assessment Data extraction was performed by a single researcher and

reviewed by one of the advisors‟ research. The strategy used
The quality analysis was defined based on an adaptation of in extracting data consists in using spreadsheets of Microsoft
the questionnaire defined by Dyba and Dingsøyr [8]. The Excel™, where all relevant information about each study
studies were divided in half for each of the teams who applied which will be useful at the moment of extraction and synthesis
the questionnaire to each item, requiring the complete reading was recorded. The classification scheme identifies ten data
of these studies. These criteria are listed below: properties: (1) publication year, (2) country, (3) software
1. Is there a clear statement of the aims of the research? development method, (4) size metric, (5) validation method,
2. Is there an adequate description of the context in which (6) study environment, (7) data source.
the research was carried out?
3. Was the research design appropriate to address the aims
of the research? III. RESULTS
4. Was the recruitment strategy appropriate to the aims of
the research?
The results are presented to answer the research questions
5. Was there a control group with which to compare
treatments? A. What metrics for measuring software effort are used in
6. Was the data collected in a way that addressed the projects that adopt agile methods?
research issue?
7. Was the data analysis sufficiently rigorous? 1) Studies over the years: As shown in Figure 1, the
8. Has the relationship between researcher and participants distribution of papers over the last decade is quite stable. Most
been considered to an adequate degree? of estimation studies started after 2001 when there has been a
9. Is there a clear statement of findings? steady rise of the APs. This result may indicate that estimation
10. Is the study of value for research or practice? problems in the in the APs have been still being studied with a
small number of studies.

E. Study Selection Procedures

The selection of studies was conducted according to five

stages, always conducted by a pair of researchers to identify
potential primary studies. Each stage was explained as
following
Stage 1: Each pair of researchers will be responsible for
conducting the manual and automatic search, in order to
 3

Figure 1: Studies over the years

Figure 4: Size Metrics
2) Country: As shown in Figure 2, the majority of studies was
in USA with India and Brazil second and third place.
B. Under what circumstances metrics for measuring software
effort are successfully applied?

5) Validation Method: Estimation models need to be

empirically validated for acceptance. The Figure 5 show that
most of effort estimation models were validated via case
studies. However, many other effort estimation models still
have not yet empirically validated. Some estimation models
have been proposed but their validations were still in
experimental stages. They may need further empirical
validation in a varied set of situations or in large-scale
industry trials.
Figure 2: Country

3) Software Development Methods: The Figure 3 shows that

the majority of studies are interested in general agile, with XP
and Scrum in the second and third place. Each development
method has it own characteristics, thus it may need studies for
other APs such as DSDM[12], FDD [13], etc

Figure 5: Validation Method

6) Study environment: The Figure 6 conducted shows that the

majority of studies was made in study contexts of
Figure 3: Software Development Methods professional/industrial projects, however many studies used
student projects.
4) Size Metrics: As shown in Figure 4, story points was the
most used many metrics. There is also the large number of
studies that use somehow FP as the metric of effort. It is worth
noting that traditional models such as COCOMO II [9] have
not been identified in either study.
 4

However, many studies still have not yet completely

empirically validated. The Use Case was the most validation
method used. We also found that the industries was the major
data source in the studies. Unfortunately, many studies did not
use relevant date or did not use any data.

B. Implication for Future Research

Future studies could do more on: (1) analyzing the impact

of properties of historical/current project data, (2) using
benchmark data, (3) using composite models and (a
combination of several estimation approaches), (4) compare
Figure 6: Study environment the variety of approaches in the same data, and (5) validating
the effort estimation models under a varied set of situations or
7) Data source: It is show in the Figure 7 conducted shows in large-scale industry trials.
that the majority of studies was made using data of
professional/industrial projects, however many studies does
not have any kind of experimental data. The data properties REFERENCES
have a strong impact on estimation results but there is very
few papers paid attention to them [11]
[1] B. Kitchenham, S. Charters, “Guidelines for Performing Systematic
Literature Reviews in Software Engineering”, Technical Report EBSE-
2007-01, School of Computer Science and Mathematics, Keele
University, 2007.

[2] K. Beck, “Extreme Programming Explained: Embrace Chage”, Addison-

Wesley, 2004.

[3] K. Schwaber, “Agile Project Management with Scrum”, Microsoft

Press, 2004.

[4] M. Cohn, “Agile Estimating and Planning”, Prentice Hall, 2006.

[5] Sommerville, I.,2011; Engenharia de Software. Editora Addison-

Figure 7: Data Source Wesley.

[6] Kardile Vilas Vasantrao, “Understanding software effort estimation at

the early software development of the life cycle - a literature view”,
IV. CONCLUSION IOSR Journal of Engineering (IOSRJEN), Vol. 1, Issue 1, pp. 041-057,
2011.

[7] Cong D. N.; Cao D. T., 2013; A Review of Effort Estimation Studies in
A. Findings Agile, Iterative and Incremental Software Development. Can Tho
University, Vietnam. 2013 IEEE RIVF Intt. Conf. on Computing &
Communication Technologies - Research, Innovation, and Vision for the
To respond RQ1. We found some commonalities in the Future (RIVF)
current studies. The number of studies in the decade is
relatively small. The USA is the country that have the most [8] Dyba T; Dingsoyr T., 2008; Empirical studies of agile software
study in the area. The majority of studies focused on general development: A systematic review. Information and Software
Technology, Butterworth-Heinemann Newton, MA, USA.
agile, XP and scrum. The story points are the most size
metrics. About a half of studies used them. Functions points [9] COCOMO 2004; Cocomo II Model Definition Manual.
also was a number of study about. Estimations are usually http://sunset.usc.edu/research/ COCOMOII , 2004.
made to support for planning and tracking project. The model- [10] REVIEWER 2013; https://sites.google.com/site/eseportal/tools/reviewer
based, monitoring based and expert-based estimation
approaches are popular. There is lack of studies that focus on [11] M. Jorgensen, M. Shepperd, “A Systematic Review of Software
the impact of the properties of historical/current project data Development Cost Estimation Studies”, IEEE Transactions on Software
Engineering, Vol. 33, No. 1, Jan. 2007, pp. 33-53.
on estimation results, on using benchmark data and composite
models. The estimation models regularly use the current [12] J. Stapleton, ”DSDM: Business Focused Development”, Pearson
project data rather than the historical project data to improve Education, 2003.
the accuracy of estimations over iterations/releases. In
[13] S.R. Palmer, J.M. Felsing, “A Practical Guide to Feature-driven
response to RQ2, we found that there is about 65% of studies Development”, Prentice Hall, 2002.
using empirical data to validate their estimation models.
 5

[P17] G.B. Alleman, M. Henderson, R. Seggelke, “Making agile development

work in a government contracting environment-measuring velocity with
APPENDIX: PRIMARY STUDIES earned value”, Proceedings of the Agile Development Conference
(ADC‟03), 2003, pp. 114 – 119.

[P18] Y. Yong, B. Zhou, “Extreme Programming Effect through System

[P01] Alshayeb, M. et al.: An empirical validation of object-oriented metrics
Dynamics Modeling”, International Conference on Computational
in two different iterative software processes. IEEE Transactions on
Intelligence and Software Engineering, 2009, pp. 1 – 4.
Software Eng., Vol. 29, Issue 11, 2003, pp. 1043 – 1049.
[P19] L. Cao, “Estimating Agile Software Project Effort: An Empirical Study,
[P02] S. Kuppuswami, K. Vivekanandan, P. Rodrigues, “A System Dynamics
y. In Proceedings of Americas Conference on Information Systems
Simulation Model to Find the Effects of XP on Cost of Change Curve”,
(AMCIS), 2008.
In the Lecture Notes in Computer Science, Vol. 2675/2003, 2003.
[P20] E. Berardi, L. Santillo, “COSMIC-based Project Management in Agile
[P03] S. Roock, H. Wolf, “Agile Project Controlling”, In the Lecture Notes in
Software Development and Mapping onto related CMMI-DEV Process
Computer Science, Vol. 3092/2004, 2004, pp. 202-209.
Areas”, International Workshop on Software Measurement – IWSM
2010, Stuttgart, Germany, Nov. 2010.
[P04] A. Cabri, M. Griffiths, “Earned Value and Agile Reporting”, AGILE'06,
2006, pp. 17-22.
[P21] N.C. Haugen, “An empirical study of using planning poker for user
story estimation”, In: Agile Conference, 2006.
[P05] T. Sulaiman, B. Barton, and T. Blackburn, “AgileEVM – Earned Value
Management in Scrum Projects”, Proceedings of AGILE 2006
[P22] K. Petersen, C. Wohlin, “Measuring the flow in lean software
Conference, 2006, pp. 7-16.
development”, Softw: Pract. Exper., 41: 975–996. doi: 10.1002/spe.975,
2011.
[P06] R. Madachy, B. Boehm, J. Lane, “Spiral lifecycle increment modeling
for new hybrid processes”, Proceedings of the International Software
[P23] M. J. Rees, ”A Feasible User Story Tool for Agile Software
Process Workshop and International Workshop on Software Process
Development?”, Software Engineering Conference, 2002. Ninth Asia-
Simulation and Modeling (SPW/ProSim 2006), China, LNCS, vol. 3966,
Pacific, pp. 22 – 30.
Springer-Verlag, 2006, pp. 167-177.
[P24] H. Sharp, H. Robinson, “Collaboration and co-ordination in mature
[P07] P. Abrahamsson, R. Moser, W. Pedrycz, A. Sillitti, G. Succi, “Effort
eXtreme programming teams”, International Journal of Human-
Prediction in Iterative Software Development Processes – Incremental
Computer Studies, v. 66, n. 7, p. 506-518, 2008
Versus Global Prediction Models”, In Proceedings of the First
International Symposium on Empirical Software Engineering and
[P25] I. Hussain, L. Kosseim, O. Ormandjieva,“Approximation of COSMIC
Measurement (ESEM '07). IEEE Computer Society, Washington, DC,
functional size to support early effort estimation in Agile”, Data &
USA, 2007, pp. 344-353.
Knowledge Engineering, v. 85, p. 2-14, 2013.
[P08] R. Lavanya, S. Chandrasekaran, V. Kanchana, „Multi-criteria approach
[P26] R. Popli, N. Chauhan, “A Sprint-Point Based Estimation Technique In
for agile software cost estimation model”, In: Proceedings of the
Scrum”, In: Information Systems and Computer Networks (ISCON),
International Conference on Global Manufacturing and Innovation in
2013 International Conference on. IEEE, 2013. p. 98-103.
Engineering, GMICIT, 2006.
[P27] R. Turner, D. Ingold, J. A. Lane, R. Madachy, D. Anderson
[P09] P. Hearty, N. Fenton, D. Marquez, M. Neil, “Predicting Project Velocity
“Effectiveness of kanban approaches in systems engineering within
in XP Using a Learning Dynamic Bayesian Network Model”, IEEE
rapid response environments”, In: Procedia Computer Science, v. 8, p.
Transactions on Software Eng., Vol. 35, Issue 1, pp. 124 – 137, Jan.-
309-314, 2012.
Feb. 2009.
[P28] Zhong, S., Liping, C., Tian-en, C. Agile planning and development
[P10] S. Bhalerao, M. Ingle, “Agile Estimation using CAEA: A Comparative
methods. In Computer Research and Development (ICCRD), 2011 3rd
Study of Agile Projects”, IPCSIT Vol.2, IACSIT Press, Singapore, 2011.
International Conference on (Vol. 1, pp. 488-491). IEEE.
[P11] E. Miranda, P. Bourque , A. Abran, “Sizing User Stories Using Paired
[P29] C. Santana, C. Gusmão, “Uso de Análise de Pontos de Funções em
Comparisons”, Information and Software Technology, Vol. 51, Issue 9,
Ambientes Ágeis”, In Engenharia de Software Magazine, p. 33 – 40, 20
Sept. 2009, pp. 1327-1337.
dez. 2009.
[P12] S. Kang, O. Choi, J. Baik, “Model-Based Dynamic Cost Estimation and
[P30] A. M. Fuqua, “Using Function Points in XP Considerations”, In:
Tracking Method for Agile Software Development”, IEEE/ACIS 9th
Extreme Programming and Agile Processes in Software Engineering.
International Conference on Computer and Information Science, pp.
Springer Berlin Heidelberg, 2003. p. 340-342.
743-748, 2010.
[P31] C. A. Santana, F. S. Leoneo, C. M. G. Gusmão, A. M. L. Vasconcelos,
[P13] I. Hussain, L. Kosseim, O. Ormandjieva, “Towards Approximating
Using Function Points in Agile Projects, In: Agile Processes in Software
COSMIC Functional Size from User Requirements in Agile
Engineering and Extreme Programming. Springer Berlin Heidelberg,
Development Processes Using Text Mining”, In the Lecture Notes in
2011. p. 176-191.
Computer Science, Vol. 6177/2010, 2010, pp. 80-91.
[P32] A. U. Banerjee, B. K. Narayanan, and C. Mahadevan P, “Estimating
[P14] E. Miranda, P. Bourque, “Agile monitoring using the line of balance”,
Agile Iterations by Extending Function Point Analysis”, In:
Journal of Systems and Software, Vol. 83, Issue 7, 2010, pp. 1205-1215.
WORLDCOMP‟12
[P15] M. Abouelela, L. Benedicenti, “Bayesian Network Based XP Process
[P33] M. R. R. Braga, C. I. M. Bezerra, J. M. Monteiro, R. Andrade, “Uma
Modelling”, International Journal of Software Engineering &
Linguagem de Padrões para Estimativa de Software em Métodos Ágeis”,
Applications (IJSEA), Vol.1, No.3, July 2010.
In: MiniPLoP Brasil 2011
[P16] L. Cao, B. Ramesh, T. Abdel-Hamid, “Modeling dynamics in agile
[P34] M. Kunz, R. R. Dumke, N. Zenker, “Software Metrics for Agile
software development”, ACM Transactions on Management,
Software Development”, In: In: Software Engineering, 2008. ASWEC
Information Systems,Vol. 1, No. 1, Article 5, Dec. 2010.
2008. 19th Australian Conference on. IEEE, 2008. p. 673-678.
 6

[P35] D. Horvat, “Function Point Analysis and Agile Methodology”,

Available in: http://www.qpmg.com/

[P36] V. Mahnic, I. Vrana, “Using stakeholder-driven process performance

measurement for monitoring the performance of a Scrum-based software
development process”, In: Elektrotehniski vestnik, v. 74, n. 5, p. 241-
247, 2007.

[P37] L. Buglione, A. Abran, “Improving Estimations in Agile Projects:

Issues and Avenues”, In: Proceedings of the 4th Software Measurement
European Forum (SMEF 2007), Rome (Italy). 2007. p. 265-274.

[P38] T. Cagley, “Agile Estimation Using Functional Metrics” The IFPUG

Guide to IT and Software Measurement IFPUG CRC Press

[P39] A. J. Alexander, “Case Study: Function Point Analysis and Cost

Estimation in An Agile Development Environment” Available in
http://www.devdaily.com/fpa/fpa-louisville-agile/

[P40] A. M. Fuqua: "Using Function Points in Extreme Programming".

Available in: http://groups.yahoo.com/group/extremeprogramming
/¯les/XPandFPA.pdf.