News

Published

May 3, 2021

Author

Androniki Pavlidou

Evaluating research outcomes and the impact of open science

May 3, 2021

The demand for effective evaluation tools to inform decisions on research funding and impacts will continue to increase in line with public investment in R&D and innovation

OECD report on the enhancement of research performance through evaluation, impact, assessment, and priority setting

The background: There is an increasing need for end-to-end (raw data to platform visualizations) systems for evidence-based, timely and accurate monitoring and evaluation of research activities and their impact. Such systems are also meant to inform policy-making decisions for the future allocations of funds and, in general, to be used for the reporting and analysis needs of different stakeholders. This also leads to a clear understanding of the contribution of research to a country’s economic and social development, thus increasing the interest in evaluating entire research systems and research portfolios.

What's at stake: The value of the research impact assessment is very high on the agenda of many nations that have even defined their own national models to measure it (UK, Finland, Norway, The Netherlands, China, Israel, Japan, Australia, Canada, Sweden, etc.). On a higher level, the European Commission, and more specifically the Horizon 2020 research and innovation framework, “sets some expected impacts at individual, institutional and systemic levels”.

The response: OpenAIRE comprehends the need for the research assessment of open science and as a pioneer, responds with a new enhanced service: the OpenAIRE MONITOR is a tailor-made, on-demand service for tracking research activities and their impact, meant to be used for data-driven, evidence-based evaluation and policymaking, with a focus on open science. A one-stop shop for multiple monitoring needs:

incorporates highly customizable, user-friendly, well-documented, timely and reliable indicators (based on the OpenAIRE Research Graph);
offers downloadable visualizations of indicators;
provides collaborating and sharing capabilities for team members in private (internal) or public (external) mode;
notifies members on changes on data or new indicators; and
allows users to browse through individual research results and projects for furhter analysis.

The impact story: As an example, we present one scenario (a profile) to unfold the research activities input-to-impact story and target the monitoring needs of various research funding stakeholders. We employ a dedicated team of data scientists working on covering different dimensions within each category to provide a well-rounded perspective on research investment outcomes.

Main Category	Subcategories
Input	Project participants \| funding
Output	Publications \| datasets \| software \| other research products
Open Science	OA routes \| FAIR aspects \| Linked results
Networks & Collaborations	Between organizations \| countries \| subject areas
Impact	Academic \| economic \| societal (e.g. policy documents)
Advanced	Correlations \| composite indicators

How it works: All information is based on transparent and trusted data from the OpenAIRE Research Graph, a collection of scholarly works, project outcomes, research data, research products, full texts, software and funder information. Starting from the (1) collection of data , (2) its categorization and classification, (3) inter-connection of data through identifiers and common entities, (4) we perform a statistical analysis to produce the open science indicators and graphs, (5) enrich via network analysis to produce collaboration indicators, and (6) work with 3rd parties to do analysis to produce innovation and social indicators.

Test it out: We invite all to explore the public profiles and get familiar with the information and visualizations. As an example we use the European Commission public profile where you can view the input to MONITOR (in terms of #of projects per country, over time, funded countries and organizations, etc.) and the output of the resources and funds allocated (in terms of #publications) and more indicators.

As a MONITOR Dashboard administrator that designs and manages your organizational profile, you have full control of the setup of information you select to see (or show publicly). You are able to select which indicators and information you want to combine and view.

Recapping:

Who uses the MONITOR Dashboard?

Essentially anyone who wants to measure research outcomes, inspect open science, and investigate different aspects of impact. Users are global, national, international, local, research funders, policy makers, research infrastructures and institutions.

The application domain of the MONITOR

Currently the application domain of OpenAIRE MONITOR is the evaluation or research activities for funders. We have also produced application profiles for domain discipline research infrastructures, and for institutions.

Ask us!

How much does it cost?

OpenAIRE MONITOR business model, is fee based. For European organizations, the EC subsidizes the costs through the OpenAIRE-Nexus H2020 project for the next 3 years. In any case, if you need monitoring service, we urge you to contact us and after consultation create a fully customizable profile.

As of today (March 2021), OpenAIRE MONITOR hosts 12 funders and 2 Research Initiatives.

Get an early start: Here is a list of things you should have in mind when setting it up:

Contact an OpenAIRE NOAD (National Open Access Desk ) that is appointed by OpenAIRE on their country and request an appointment.
Discuss the needs of the organization, nation they represent.
Delineate on the domain and topic(s) of interest.
List the available resources that are important and available. This will ensure that OpenAIRE aggregates content (metadata and relative full-texts) from providers relevant for the stakeholder (e.g. repositories of their country), if not already aggregated.
Provide some simple (meta)data related to the stakeholder’s organization function: e.g., funding database, related projects, registry of people… All this done under confidential agreements. For funders in particular, detailed information can be found here: https://www.openaire.eu/funders-how-to-join-guide.
We then get the ball rolling, i.e., ingest your data, identify and extract related information, infer links, and connect with the OpenAIRE research graph. Results are refined based on your feedback, until final validation and approval.
Meanwhile, your personalized monitoring dashboard is designed by the MONITOR team. You can then Manage your portal and decide what you want to publish, what to keep private, what to share with colleagues.

It is worth noting that OpenAIRE NOADs play an instrumental role in the above process by being the connecting link between stakeholders and the OpenAIRE technical team. With their knowledge and awareness at the local, national level, they help ensure that the data shown in the MONITOR portals is relevant and accurate.

Related OpenAIRE Products/Services

OpenAIRE Research Graph: https://graph.openaire.eu/

OpenAIRE Research Graph is an open resource that aggregates a collection of research data properties (metadata, links) available within the OpenAIRE Open Science infrastructure for funders, organizations, researchers, research communities and publishers to interlink information by using a semantic graph database approach.