EOSCpilot Open Science Monitor specifications: Towards a dynamic and open European Open Science Monitor

The EOSCpilot is the first project towards the realisation of the EC’s vision for a common and trusted environment for practicing Open Science. Part of the project’s activities is a Policy Work Package aiming to develop an EOSC tailored Policy Framework complemented by a number of services to support this venture. Such Policy Framework will facilitate operational and strategic priorities set by the EOSC in the areas of Open Science and Open Scholarship, Procurement, Ethics and Data Protection for infrastructures, services, data and skills, i.e. the primary components that the EOSC will be comprised of. Besides adoption guidelines and implementation check-lists preparation, the Policy Work Package assists monitoring of Open Science in the European area both with policy recommendations concerning, among others, the use of metrics and data collection in the EOSC, as well as with delivering the specifications for a monitoring mechanism intended for use by funding bodies, research performing organisations, research infrastructures as well as researchers themselves.

Moreover, the EOSCpilot OS Monitoring framework will support the aforementioned stakeholders in measuring:

• levels of compliance with European Union’s laws, regulations and policies regarding research and research results dissemination;
• Open Science Resources’ (i.e. research artefacts, educational resources, research collaboration and citizen science activities) levels of openness, trustworthiness and FAIRness that cover each stage of the research lifecycle;
• impact of science on society and economy.

The design of the framework focused on the identification and definitions of:

• A concrete methodology for deriving clear and measurable targets (called high-level monitoring targets in this context) which drive the monitoring process.
• Key resources and trends that constitute the backbone of Open science, including open access to publications, FAIR research data, open source software, citizen science, etc. The monitoring targets and their associated indicators aim at measuring aspects of these OS resources, e.g., a monitoring target may refer to and measure the degree of accessibility of (open) research data across data repositories in EU.
• Relevant stakeholders in the EOSC and most importantly their interest and requirements for monitoring targets and OS resources. EOSC stakeholders, mainly Research Performing Organisations (RPOs), Research Funding Organisations (RFOs) and Government Bodies as well as researchers and citizens pose different requirements on the monitoring goals and the OS resources to be measured.
• Appropriate metrics and indicators which will quantify each monitoring target. These indicators can be quantitative or qualitative, scalar or binary (e.g, a certain practice adheres to a policy or not), and can be combined together to provide an overall score for a quality indicator. Moreover, each indicator will be associated with a data source from which monitoring data are collected, as well specific processes for further processing, validation and aggregation.

In addition, the first phase of the study lead to the definition of technical specifications and compilation of a pool of indicators and open sources for data collection. In achieving this, work concentrated primarily in understanding the Open Science environment and its dimensions focusing on the more mature efforts in measuring Openness but also at the recent “trend” and communities’ demand in measuring FAIRness. Taking into consideration that a digital object is possible to be FAIR but not open and the other way round, an exercise to find commonalities and differences between those two aspects eventually formed the outline of the Open Science Monitoring Targets Framework. An extensive landscape review of existing monitors and recommendations for measuring elements of Open Science followed, filling the Framework with indicators.

 
[caption id="attachment_3782" align="aligncenter" width="431"] fig.1 - EOSCpilot OS Monitoring Targets Framework


 
Technical specifications to measure those targets revolve around a dynamic schema and depend on a two-way communication between the EOSCpilot OS Monitor and the EOSC stakeholders. For an in depth examination of this work, you may read the full report here.

[caption id="attachment_3783" align="aligncenter" width="554"] fig. 2 - EOSCpilot OS Monitor Architecture


 
Updates on the presented Framework will follow, as the Open Science Monitor is currently in the second phase of clearing the Monitoring Targets’ and indicators’ information so as to provide levels of Openness and FAIRness to its targeted resources.