D-Lib MagazineSeptember/October 2015 Success Criteria for the Development and Sustainable Operation of Virtual Research Environments
Stefan Buddenbohm AbstractIn many areas of research, virtual research environments (VREs) have become an essential part of modern research processes. The providers of VREs need to respond to this growing importance with functioning and efficient processes for the development, operation and quality assurance of VREs. We have developed a life-cycle model for VREs, which focuses in particular on the success-critical points for the transition to a VRE's sustainable operation. Furthermore, we discuss a set of success criteria that enables all involved in the VRE (operators, funding bodies, users) to identify which aspects will be relevant to their specific needs prior to the creation of a new VRE. In light of the heterogeneity of VREs, this set of criteria is supplemented in individual cases by discipline-specific criteria. 1 Introduction1Virtual research environments (VREs) have become increasingly important in recent years in a variety of research disciplines and belong to the standard repertoire of digital tools and infrastructures being used. However, up to now very few VREs have made the leap from the prototype phase to everyday use [3]. Furthermore, the increased use of VREs in research environments has not resulted in any universally accepted attributes or guidelines for the development and operation of these infrastructures [4]. On the contrary: the different stakeholders particularly from research funding bodies, academia and information infrastructure providers are searching for criteria that could help with the development, operation and quality assurance of VREs and with the optimisation of processes. For a variety of reasons, these findings pose a challenge for the development of universally applicable success criteria and guidelines for the development of VREs. First, there are significant variations in the research processes, not only between different specialist areas of research but also within the research disciplines themselves. This applies to methodological approaches as well as to the tools that are used. Considering this heterogeneity, a universally applicable set of success criteria is therefore not feasible. On the other hand, it would be time-consuming and costly to plan every VRE individually and from scratch. Therefore it makes sense to identify similarities in the features and processes of VREs. These similarities can then serve as a basis for the development of guidance and support instruments as well as for evaluating the success of VREs. There are very few classifications and systematic evaluations of VREs in German-speaking countries [2]. The German Research Foundation-funded project "Success criteria for the development and sustainable operation of virtual research environments (DFG-VRE, DFG=German Research Foundation)" has set itself the task of bundling these approaches and turning them into useful recommendations. The project is particularly focused on big, cross-institutional VREs, as these involve specific organisational and legal challenges that can be addressed with pertinent recommendations. Smaller projects, on the other hand, have more straightforward planning and management needs and are therefore given lower priority in the framework of the DFG-VRE project. 2 MethodologyFor the purposes of our project, we took the definition of the Allianz AG of "Virtual Research Environments" as a basis for all further considerations: "A Virtual Research Environment, or VRE, is a working platform that enables a cooperative research task to be carried out by numerous researchers in different locations at the same time and without restrictions. As a professional tool, it potentially supports the entire research process from collecting, discussing and processing data, through to the publication of the results while technologically, it is based above all on software services and communication networks. Virtual research environments are important components of modern research infrastructures and play a decisive role in the productivity and competitiveness of the research". [1] Our project did not encompass a comprehensive empirical collection of data, for example resulting from surveys or standardised interviews. Based on the preliminary studies of project partners, our approach was focused on developing general approaches and identifying framework conditions that could set the course for the successful and sustainable operation of VREs right from the start. Based on the relevant experiences and contacts of the project partners, we decided to focus on the developers and operators of VREs. An important tool was to talk to experts, in the form of open, non-standardised interviews. During these discussions, one or two subject areas were discussed on the basis of material gathered earlier in the project. The different subject areas and the dialogue partners are presented in Figure 1. Figure 1: Overview of the expert interviews conducted between October and December 2013 and the topics discussed. Apart from institutions that have developed or are operating a VRE, we also conducted interviews with representatives from the European Grid Infrastructure (EGI), the Deutsche Forschungsnetz e.V. (DFN) and the Hochschul-Informationssystem GmbH (HIS). The expert interviews were conducted over a period of three months in the autumn of 2013 and were individually tailored to the respective interviewees. The results of the individual expert interviews influenced the subsequent interviews and led to a modification of the questions as well as of the presented preliminary project results (for example life-cycle-phase model or success criteria). Clearly the subject is connected with a number of requirements for all stakeholders, whether from a user perspective or the perspective of infrastructure developers and operators, or funders of research infrastructures. Focal points of the discussion were defining success criteria for VREs, weighing up individual criteria, and the important constellation of cost structures and financial sustainability. 3 Life-cycle phases of a virtual research environmentThe life-cycle of a VRE can be divided into different phases. It should be noted that this kind of schematic illustration always presents an idealised situation. That is why the diagram shown in Figure 2 cannot be directly applied to existing VREs. It should also be noted that the focus of our observations was primarily on the operator's perspective. Throughout the entire lifespan of a VRE there are five phases. During the prototype phase, the technical and organisational structures of the VRE, as a rule within one project of limited duration, are established and made available to the first users in the form of a prototype. This enables the viability of the system to be put to the test and for it to be evaluated under normal operating conditions. The acceptance of the VRE within its user community can also be assessed during the prototype phase. Building upon this prototype, a series of aspects are then addressed during the development phase that are essential for the sustainable operation of the VRE, but which could not be adequately addressed during the development of the prototype. These aspects include, in particular, an operational concept, in which the services that can be provided by the VRE and the necessary resources to achieve these are presented, as well as a cost and financing plan. Once this work has been completed, the VRE is a fully developed product ready for the operational phase. In this phase, the finished version of the VRE is available to researchers for their academic work. From the perspective of the operators, the focus is now on assisting users and on the ongoing technical and organisational development of the VRE. If at some point the decision is made to terminate the VRE, the purpose of the transfer phase is to transfer components worth keeping such as research data or programme codes to other (infra)structures. The remaining technical and organisational structures are then dissolved in the dismantling phase. In terms of the life-cycle phases it is necessary to differentiate between two situations. The described, linearly structured schema applies initially only to VREs that are operated as independent infrastructures. However, if the VRE is one of several modules integrated in a comprehensive infrastructural context (e.g. in a data centre), then each module has its own prototype, development, transfer and dismantling phases, while the operation of the VRE applies across all modules (see Figure 2). In this case, individual components are not necessarily transferred to an external structure during the transfer phase, but possibly also to other phases of a new VRE module. Figure 2: Life-cycle phases of a modular VRE. A first module A is established analogously to the life-cycle in Figure 2 and moves through the respective phases. However, the operating/product phase is shared with other modules of the same infrastructure. At the end of the lifespan of module A, components can be transferred back into the overall operation or into the prototype and development phase of a new module B. Naturally, the transition between the different phases is of particular importance, especially in terms of entering the next phase. In this paper, we describe these conditions as milestones. These phase transitions, each with their respective milestones, are included among a VREs success criteria and are therefore discussed below. 3.1 The prototype phaseThe prototype phase begins with preparatory work, in which the prototype project itself (e.g. the creation of the internal organisational structures and resource planning) is planned, and the environment of the planned VRE is assessed in the context of a requirements and market analysis. A requirements analysis includes the following aspects:
The market analysis, on the other hand, comprises:
Figure 3: The prototype phase of a VRE. During the technical specification, preceding analyses of the academic workflows are used to define the services of the VRE, and a development plan is made for their implementation with the help of an established software development model, e.g. the waterfall model [6]. Also a decision is made about the standards and formats (for data as well as metadata) the VRE will use and which hard- and software will be used for the services to be provided. A number of organisational measures also need to be carried out, including establishing access rights and policies, and developing strategies for community work, training sessions and the necessary support facilities. The VRE is also promoted within the relevant academic disciplines with the help of initial community-forming measures. Initial training sessions for users (and later "power users") can already be carried out. The described developments finally lead to a prototype which is made available to a limited number of users. This serves not only to test the system under operating conditions but also provides an opportunity to adapt the VRE to the requirements of the community with small improvements based on the feedback of the pilot users. In order to gather reliable information about the requirements of a VRE, particularly with a view to the subsequent operational phase, all described phases are accompanied by intensive monitoring, which is part of the project management. 3.2 The development phaseThe pilot phase is important for gaining first impressions of the VRE in a working environment and for evaluating user acceptance in particular. This information is used to make a decision about whether to continue funding the VRE. If a positive decision is reached, further components necessary for the sustainable operation of the VRE are developed in the development phase (see Figure 4). Figure 4: Components of the development of a VRE into a product during the development phase. As a rule, the prototype phase of a VRE is financed as a research or infrastructure project. However, long-term operation requires an individually tailored operational concept that specifies the services provided by the VRE and the necessary resources needed to implement them, as well as a corresponding cost and financing plan. Furthermore, it is important to formulate a concept for subsequent usage, which clarifies how and where the components worth keeping (e.g. research data, software) will be transferred after the VRE reaches the end of its regular operational life. Ideally this work would be carried out at the earliest possible stage, but usually this does not happen because of the specific project character of the prototype phase. In particular, if the pilot phase is judged to be a failure due to lack of user acceptance or for other reasons, the work would have been in vain. These steps are therefore carried out during a second financing stage, after which the VRE can be turned into a product and the operational phase can begin. 3.3 The operational phaseDuring the operational phase, the services of the VRE are available to the community for research purposes. Besides the technical operation of the VRE, the focus during this phase is also on assisting the user community and on the continued technical and organisational development of the VRE. Involving users in the further configuration of the VRE and acceptance on the part of the targeted specialist communities are particularly important. The operational phase encompasses a series of key aspects which are illustrated in Figure 5. Figure 5: Central aspects of the operational phase A help desk, appropriate support as well as record keeping for working with the VRE is at the disposal of users. The community that has formed around the VRE is supported and extended through appropriate events (e.g. workshops, trainings). Publicity work helps to promote the VRE within the academic community and, if applicable, in the public realm. During the lifespan of the VRE, a variety of development work has to be carried out, which, however, should not impact the stable operation of the VRE. Enhancements to the VREs functionality become necessary particularly based on user feedback and new requirements that inevitably result from the growing circle of users. The coordination of these tasks rests with the requirements management. Major add-ons are carried out in separate projects. Bug fixes and small add-ons are carried out within the existing structure. Besides the production system, a VRE should also operate a development and test system, so that the enhancements or changes to the VRE can be seamlessly transferred to the live environment. 3.4 The transfer phaseIf a VRE or part of a modular VRE is to be dismantled, the purpose of the transfer phase is to transfer components of the VRE that are worth keeping to suitable existing or yet to be created structures. This comprises:
Figure 6: Transfer phase of a VRE The transfer may be necessary for legal reasons or due to other specifications (e.g. rules regarding correct academic practice as stipulated by the funder. But there might also be other reasons, for example an existing user base that wants to keep part of the dissolving VRE. 3.5 The dismantling phaseDuring the dismantling phase, the organisational structures of the VRE are dissolved. In some cases it might be useful to establish an appropriate follow-on organisation. Migration support helps the still-active users to integrate the transfer of components to other infrastructures, described above, into their workflows. Figure 7: Dismantling phase of a VRE 3.6 Complete life-cycle phases of a VREFigure 8: Interviewed VREs mapped on the life-cycle phases Figure 9: Complete life-cycle phases of a VRE 4 Criteria for the success of a virtual research environmentIn this section we will first outline how the success of a VRE can be defined. We will then present the criteria set resulting from our project research. This will lead into a discussion about milestones, which are aligned to the individual life-cycle phases and which are important for the success of the VRE. The chapter ends with an overview of the methodology used for evaluating success. 4.1 Definition of successBefore establishing a set of success criteria, it is necessary to consider what in fact makes a VRE successful. This involves defining the success of a VRE in general and in relation to stakeholders and establishing a coherent definition based on different criteria. For this purpose we will first define a generic set of success criteria which establishes various aspects of success by means of selected indicators. Important aspects of this set are based on the VRE definition by the Allianz AG [1]. The set is not a fixed catalogue that can be equally applied to every VRE. It is necessary to select the relevant criteria on a case by case basis from this grouping similar to selecting something from a menu and to evaluate it critically at the appropriate time. Ideally, the goals for a VRE should be defined at the start which is as early as the initial application procedure. The user community strongly influence the service spectrum of the later VRE and who are later catered to with appropriate services by the operator. An optimal alignment of user requirements with the choice of services offered by the VRE lays the foundation for a successful VRE. On the other hand it is necessary to take into account possibly conflicting objectives, for example the desire for a comprehensive choice of functions by the user compared to limited resources on the part of the developer/operator. All of this results in an individual success definition for the VRE. 4.2 Criteria setThe success criteria illustrated in Figure 10 provide a template to be adapted to specific cases. "Case-specific" means that the choice of individual criteria can be varied ("menu"), and can be supplemented with subject-specific criteria, e.g. depending on the disciplinary focus of the VRE. The success criteria are aligned in particular to the use and benefits of the VRE for academic users. They were discussed in the above mentioned expert interviews and to a large part can be traced back to statements from the specific VRE operators. The diagram groups the criteria around the three most important stakeholder groups. This clearly illustrates how each group judges success by different criteria (weighting of the individual criteria). In addition as also illustrated above there are case-specific criteria. Figure 10: Success factors and measurable success criteria, weighted by relevance for the different stakeholder categories. The individual criteria:
The success criteria can be differently weighted on a case-by-case basis. It is not about achieving results that can be quantitatively measured, but rather about defining objectives. The achievement of these objectives enables the success of the VRE to be determined at a later date. An important instrument in the implementation of these success criteria are milestones, which are described in the following chapter. Apart from this the expert interviews revealed the usefulness of the success criteria although they may not be suitable for quantification or direct comparison with other VREs. 4.3 MilestonesIn the course of discussing the success criteria described above with the different VRE stakeholders it quickly became clear that these abstractly formulated criteria are closely connected with concrete milestones. In this sense, the milestones place the success criteria of the individual life-cycle phases in chronological order and connect them with specific tasks for the operator of the VRE. The lack of such milestones, or milestones that are not achieved, can be classified as "failure criteria" (negative success criteria). Assessing these milestones in terms of their structure and content is a further step in the qualitative evaluation. Some of these milestones are outlined in Figure 11. This list is just a suggestion, as depending on a given case there may be different milestones or the individual milestones may be more strongly structured. As already touched in the discussion about life phases, above, many more milestones are needed in the prototype and development phase than in the subsequent phases. Figure 11: Milestones relating to the life-cycle phases of a VRE (selection). The success of a VRE depends to a great extent on integrating the user group right from the start. This takes the form of a user requirements and market analysis, accompanied by project monitoring and is designed to ensure the biggest possible overlap between the requirements of the user group and what the VRE will later offer. To broker any resulting conflicts of interest, we have created a cost and financing plan, a development plan and a concept for the organisational form. Above all, the integration of all stakeholders in the organisational form enables a continuous exchange of ideas. At the same time, the organisational form and the cost and financing plan set a possible course for the future sustainability of the infrastructure. During the interviews conducted as part of the DFG-VRE project, realistic cost estimation for the establishment of a VRE was consistently described as problematic. It is difficult to make general assumptions, but it is certainly possible to come up with relatively realistic weighted cost structures at the very beginning of planning for the VRE, based on the specific tasks and the size of the user group. These cost structures provide a firm basis for financial planning. 4.4 Methodology for evaluating successThe perceived success of a VRE is determined by the interests of the stakeholders. An academic user community generally has different criteria from a funder or infrastructure provider. These differing perspectives are expressed as described above in a conflict of objectives that must be conceptually addressed. The process for evaluating success is illustrated in Figure 12: Figure 12: Evaluation of success in relation to the development phase of the VRE. For all stakeholders, it begins with the selection of criteria from the overall set of success criteria described above. These are supplemented by subject- or case-specific criteria, which together contribute to a definition of success. The objective of defining success is carried out before the VRE is developed, usually in the context of the application for the prototype phase. The further design and specific development is then carried out in line with these objectives. In this case it is important that the objectives are defined by the stakeholders themselves and that there is process of consensus. 5 ConclusionThis paper examines different relevant factors for successful development and sustainable operation of a VRE. Structural, functional, organisational and financial aspects are combined. To an extent the different stakeholder perspectives are taken into consideration, although we focus particularly on the developer and operator perspective. The work is based on experiences gathered in various collaborative projects and was verified in expert interviews and workshop discussions with developers, operators, users and funders. We also took into consideration insights from the development of larger software systems. In summary, the feedback from the community suggests that:
The findings made in the course of the DFG-VRE project are just a first step towards an approved instrument for evaluating the success and usability of such infrastructures. The systematic and advanced approach of the pilot study examines the conditions required for the sustainable development of a VRE in relation to today's research infrastructures and funding structures. It would be useful to do an even more in depth study in the form of a bigger survey of the issues examined and developed in this paper. However, the results of the pilot study already allow for current and future projects to reflect on their own criteria for sustainability and success and to orientate themselves using the life-cycle-phase models. Notes1 This article is a translated and heavily revised edition of a former German language publication. The German edition has been published as DARIAH-DE Working Paper in 2014. References[1] Arbeitsgruppe Virtuelle Forschungsumgebungen der Allianz-Initiative Digitale Information (2011): Virtuelle Forschungsumgebungen Ein Leitfaden. [2] Arbeitsgruppe Virtuelle Forschungsumgebungen der Allianz-Initiative Digitale Information (2012): Momentaufnahme: VREs in 2011/2012. [3] Carusi, Annamaria, Reimer, Torsten (2010): JISC Collaborative Landscape Study. [4] Fritzsch, B. (2013): Erfolgskriterien für VREs: Naturwissenschaftliche Sicht Erfahrungen aus C3Grid, Workshop: "Erfolgskriterien für Virtuelle Forschungsumgebungen", Leibniz Institut für Astrophysik Potsdam, 19. November 2013. http://hdl.handle.net/10013/epic.42856 [5] Ludwig, Jens, Enke, Harry (2013): Digital Curation of Research Data. Experiences of a Baseline Study in Germany. VWH-Verlag. [6] Winston Royce, Proceedings, IEEE WESCON (Ed.) (1970): Managing the Development of Large Software Systems. Institute of Electrical and Electronics Engineers. About the Authors
|
||||||||||||||||||
|