Authors: Leonidas Pispiringas, Ioanna Grypari, Harry Dimitropoulos, Alessia Bardi

The assessment of research activities is a key step in the decision-making process for every organisation investing in such activities; whether it is a funder evaluating the proper allocation of grants to maximise societal impact, an institution looking to find hidden potential and room-for-improvement areas, or a research initiative considering expanding its network, to name a few examples. The reliable and timely monitoring and evaluation of research activities are indispensable for the efficient allocation of resources and the overall decision-making process. 

The OpenAIRE MONITOR is an on-demand service built upon the OpenAIRE Research Graph with the mission to fulfill these needs. It offers tailor-made data and visualisation monitoring dashboards for funders, institutions and research initiatives, populated with well-documented metrics and indicators of research activities. To meet the requirements of each individual organisation, a default dashboard is improved upon in one-on-one co-designing sessions, that include the validation of data shown and the creation of new indicators, if needed.

After the launch of the Institutional Dashboard of OpenAIRE MONITOR, in May 2022, and working closely with the community, we focused our efforts on upgrading the service in three areas: (i) updating the user interface, (ii) improving the documentation of methodology and indicators (the new Resources tab), and (iii) redesigning all MONITOR dashboards with new indicators and visualisations. 

User Interface (UI)

MONITOR has an updated interface, focused on offering an intuitive user journey and a smooth and clear user experience. We have updated the look-and-feel, flow, content, and section structure to help and guide both potential users looking to learn and understand the service (home, browse dashboards, get started pages), and also active users that interact regularly with their dashboard (manager and team member settings, visualisation options).

Documentation

We regularly re-evaluate our indicators by examining the trends in research assessment globally, receiving feedback from our users and experts and closely following the improvements on the OpenAIRE Research Graph.

In the most recent update of the dashboards, we redefined the grouping of indicators, improved visualisations and added requested indicators.

We introduced an “Overview” category where the users can have a quick glance at the research activities of their organisation by seeing a collection of some key performance indicators (Figures 1, 2); for a detailed analysis, someone can enter the dedicated categories of the dashboard.

 
Figure 1: Overview - publications trends (All compared to Peer-reviewed and Grant-supported)

Figure 2: Overview - publications openness trends (All compared to Open Access and Gold Open Access)

We updated the Funding, Research Output and Collaboration categories by revising the current visualisations and adding new indicators, e.g., indicators for the peer-reviewed publications of organisations (Figure 3). 

Figure 3: Peer-Reviewed publications indicators

We significantly enriched the Open Science category, with metrics on the organisations’ publications trends by Open Access Route (green, gold, hybrid, bronze), by journal business model (Figures 4, 5) and across a set of FAIR aspects, to name a few examples.

Figure 4: Publications trends by Open Access Route (green, gold, hybrid, bronze)

Figure 5: Publications trends by journal business model

What’s Next?

Currently, our team is working on expanding the indicators’ coverage of relevant monitoring aspects focused on further enhancing the content quality of the dashboards.

Focusing on the Academic Impact, we introduced indicators on the downloads of publications by users in different data sources (journals, repositories), as harvested by the OpenAIRE’s Usage Counts Service (Figure 6). The latter are at a trial stage and the dashboard managers of the currently registered organisations are reviewing them. The next step in expanding the Academic Impact indicators is to generate citation metrics from Open Citations.

Figure 6: Publication download trends by year of publication and by year of download

We will introduce new indicators distributing the research outcomes by Fields of Science (FoS) and by the classification scheme of UN Sustainable Development Goals. The latter will assist in viewing contributions of research towards complex challenges for humanity such as climate change, biodiversity loss, pollution and poverty reduction. 

We will enrich the Open Science indicators with the introduction of Article Process Charges from OpenAPC, Plan S indicators and composite indicators measuring the institution’s average share of:

We will introduce Collaboration indicators based on the co-authorship/co-creation of research outcomes. We are planning to integrate data and produce metrics on the Horizon Europe Funding Programme from the European Commission and on other Funding Streams for other Funders further enhancing the content quality of the dashboards.

Finally, taking advantage of the links in the OpenAIRE Research Graph between different types of research products (e.g., a publication linked to a dataset and a software) we will introduce new metrics that will help us understand their role in Open Science and other aspects.

We are constantly evaluating and improving the MONITOR service. Our goal is to have a community-driven service with a strong focus on the feedback we get from the dashboard managers. After evaluating the latter, we respectively incorporate the suggestions for providing better monitoring services and assisting the organisations in measuring the impact of research activities, by constructing indicators that suit their needs and improving data quality.