A systematic literature review on crowdsourcing in software engineering

Background: Crowdsourcing outsources a task to large groups of people by open call format, and it recently plays significant role for software practitioners. Aim: The purpose of this study is to conduct a comprehensive overview on crowdsourcing in software engineering (CSE), concerning business models, tools, platforms, software development processes, and software economics. Method: We conducted a systematic literature review on CSE. We identified 158 relevant studies and 6 secondary studies. We further reviewed 67 primary studies that passed our quality assessment criteria. We defined 10 research questions and synthesized different approaches used in primary studies regarding each question. Results: Majority of studies report the application of crowdsourcing for coding and testing tasks. Crowdsourcing follows a unique methodology in which project planning, task specification and deployment have more emphasis. There is not enough literature on effort estimation approaches in CSE and associated cost factors. Complexity of the task and its expected duration play significant role in estimation. Conclusions: Future studies should focus more on economic models, experience reports, specific software development methodologies, and strategic pricing mechanism for CSE.