Evaluation in Pilot-Based Research Projects: A Hierarchical Approach

Thomas Zefferer*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review


The evaluation of project results is a crucial part of most research projects in the information-technology domain. In particular, this applies to projects that develop solutions for the public sector and test them by means of pilot applications. In these projects, involved stakeholders require meaningful evaluation results to derive lessons learned and to steer future research activities in the right directions. Today, most projects define and apply their own evaluation schemes. This yields evaluation results that are difficult to compare between projects. Sometimes, inconsistent evaluation schemes are even applied within a project. In these cases, even evaluation results of a single project lack comparability. In absence of coherent evaluation schemes, lessons learned from conducted evaluation processes cannot be aggregated to a coherent holistic picture and the overall gain of research projects remains limited. To address this issue, we propose and introduce an evaluation scheme for arbitrary pilot-based research projects targeting the public sector. The proposed scheme follows a hierarchical approach. Concretely, it organizes evaluation criteria on different layers of abstraction. Furthermore, the proposed scheme describes in detail necessary process steps to carry out evaluations using the defined criteria. This way, the proposed scheme enables in-depth evaluations of research projects and their pilot applications. At the same time, it assures that obtained evaluation results remain comparable anytime. The hierarchical evaluation scheme introduced in this article has been successfully applied to the international research project FutureTrust, demonstrating its practical applicability, and showing its potential also for future research projects.

Original languageEnglish
Title of host publicationWeb Information Systems and Technologies - 14th International Conference, WEBIST 2018, Revised Selected Papers
EditorsMaría José Escalona, Francisco Domínguez Mayo, Tim A. Majchrzak, Valérie Monfort
Number of pages22
ISBN (Print)9783030353292
Publication statusPublished - 2020
Externally publishedYes
Event14th International Conference on Web Information Systems and Technologies, WEBIST 2018: WEBIST 2018 - Sevilla, Spain
Duration: 18 Sept 201820 Sept 2018

Publication series

NameLecture Notes in Business Information Processing
Volume372 LNBIP
ISSN (Print)1865-1348
ISSN (Electronic)1865-1356


Conference14th International Conference on Web Information Systems and Technologies, WEBIST 2018
Abbreviated titleWEBIST


  • Evaluation scheme
  • Hierarchical evaluation model
  • Project evaluation

ASJC Scopus subject areas

  • Management Information Systems
  • Control and Systems Engineering
  • Business and International Management
  • Information Systems
  • Modelling and Simulation
  • Information Systems and Management

Cite this