The KB ‘Dynamic Window’ into Bibliometric Data: how to plan and build a visualization
Introduction
For the website of the Kompetenznetzwerk Bibliometrie, we built a dynamic visualization of bibliometric data with the aim of providing a visually engaging insight into how bibliometric data is actually generated and how dynamic the publication system is. The processes of indexing and enrichment of publication data are generally rather opaque to (potential) users and non-specialists. This visualization is meant to offer a glimpse into the “engine room” of the publication system.
It provides a vivid insight into the dynamics of the publication system, whilst also touching on the internal mechanisms of scientific and publication systems, such as global balances or imbalances in publication output. The first part of the visualization focuses on a dynamic display of new publications, shown as single bubbles on a world map. The second part moves the recipients’ attention to what happens when these single events are aggregated; it displays publication counts of German publications, again on a map, thus resembling more typical bibliometric maps and representations.
This post documents the conceptual considerations, lessons learnt from a pilot project and the technical solutions, not least to document for those interested how such an application of bibliometric data might take shape. The technical and graphic design were carried out by the agency 3pm.
You can find the live application on KB’s website and below.
World map and ticker:
German Map:
Features
- A world map where publications newly indexed in CrossRef and OpenAlex are displayed as quickly as possible.
- Publications are shown as bubbles on a world map where the geocode is usually derived from the address affiliation of the first author. Relatively older publication bubbles fade out. Bubbles have pop-up windows displaying core bibliographic data when clicked upon as well as links to their journal pages.
- A ticker containing a reduced set of metadata (date and time, title with DOI, first author’s affiliation) is shown in a vertical order. Publications without a geocoded affiliation are also displayed to some extent in the ticker.
- A scalable German map with aggregate publication numbers for institutions, to be zoomed in and out.
- Both maps and the ticker are visually adapted to the overall KB website layout.
Challenges
- Interesting metadata in terms of time and geographical data are not covered uniformly by databases; in our case, we would like to use geocodes provided by OpenAlex and the minute-level accuracy of the create-metadatum in CrossRef.
- After a publication is first indexed in CrossRef, it takes a day for most of the geocodes to be added in OpenAlex, a real on-the-fly mechanism is thus not feasible.
- On some days, there is very little traffic (weekends, possibly public holidays, the start of the year, etc.). During the pilot implementation stage, no content was visible on the map and the ticker stood still following such days. The same happened as soon as all the publications loaded for that day had been displayed.
- It has been observed recently that OpenAlex has had coverage deficiencies in affiliation data of certain publishers (and continues to some degree).
Conceptual solutions
- For accurate timestamps and geographical data, the data are first loaded from CrossRef and then matched with OpenAlex based on the exact DOI match.
- The most recent 150 geocoded publication records plus the 10 most recent non-geocoded ones are cached. At the same time, a limit of 100 publications is set as the maximum number for the world map (some testing showed that displaying more would make the map too cluttered). ‘Older’ publications should only fade out when newer ones emerge. The real-time mode is implemented as follows: a new publication emerges on the map and the ticker exactly at the same hour, minute and second it has been indexed in CrossRef two days before.
- This implementation should generally ensure that the map and ticker are never completely empty, but there may little dynamic visible (if there was little traffic on the date the data was loaded). Based on the duration period of bubbles on the world map, different temporary geographical patterns emerge constantly on the world map. While minute patterns are driven randomly (i.e., by the precise order in which they have been indexed), the density of bubbles is quite often relatively higher in North America, Europe and East Asia compared to other world regions. Of course, the emergence of interpretable patterns through the aggregation of large quantities data is a guiding principle of bibliometrics (as well as other quantitative sciences), which leads to the map of Germany. The German map displays aggregated data for institutions (with a planned quarterly update).
- For the German map, we select 264 institutions based on the KB institutional coding and a threshold of 100 publications in OpenAlex in 2024.
- We can’t do much about missing affiliation data in OpenAlex but will continue to monitor this issue and (and may consider switching the German map to a different data source in future).
Technical solutions
The backend handles data ingestion and delivery using a PHP web application with a MySQL database. This application handles the automated data import functions from CrossRef, OpenAlex and ROR. For the frontend, there is a separate API endpoint for both the world map and the map of Germany.
World map and ticker
Following a number of tests and attempts to provide a database that is as dynamic as possible, yet still stable, for “feeding” the world map, the following solutions have been implemented at present:
Data import via a sequence of three cron jobs, once a day between 04:00 and approximately 05:05.
Import of articles via the CrossRef API
- The import routine works with the following filters:
- journal articles only (
journal-article) - at least one affiliation (
has-affiliation:1) - created on (date): -2 days, between 00:00 and 06:00 on the following day
- journal articles only (
- Only the details required for display are imported, extracted and stored:
DOI(Digital Object Identifier),created(timestamp),title,author(authors, affiliation),published(date). In a few cases, a ROR identifier is already linked to the affiliation for further processing. This is then stored as well. - The time filter is the result of extensive testing and observation. Ensuring that OpenAlex supplements the CrossRef data with GEO components as comprehensively as possible (2nd import step) is only guaranteed with sufficient lead time (therefore, data are loaded 2 days before the day they will be displayed). The inclusion of data up to the 6th hour of the following day is intended to bridge the gap between import operations:
- At the start of the import process, the “old” data are copied to a second, empty publication table. The “regular” table is emptied before the new import starts. However, the most current “old” data (150 records with geocoding and 10 records without) from the previous day is retrieved again from the second table to ensure that there is always a sufficient baseline of data available for the world map and the ticker.
- Whenever the application is launched, the last 150 geocoded data and 10 non-geocoded data are reloaded, as well as any data whose creation time (hour, minute, second) falls within the next 30 minutes of the current time (but from two days ago).
- The number of records imported daily usually fluctuates between 4,000 and 12,000, although a configurable upper limit of 15,000 is currently in place. Above this level, any further increase in activity is barely noticeable.
- The import routine works with the following filters:
Importing GEO data via the OpenAlex API
- To supplement the database created in the first import step with geo-relevant information, each publication is queried via the OpenAlex API using the DOI. This process can take up to 50–60 minutes (whereas steps 1 (CrossRef) and 3 (ROR) are each completed in less than 1 minute).
- We loop through the affiliations (in the order of the authors) and stop upon the first occurrence of an ROR ID.
Importing affiliation data via the ROR API
- Whilst only the publication data required and suitable for display until the next import is retained, the GEO data for the affiliations that has already been collected is preserved. With each new import, the internal comparison of the ROR identifier ensures that only the affiliation data not yet stored at that time is retrieved via the ROR API, thereby limiting the number of daily retrievals to 100–200 (for up to 10,000 affiliations with linked ROR identifiers per day). Currently, just under 42,000 ROR/affiliation records are stored. The following are stored:
lat,lng,country_code(abbreviation),country_nameandname(city).
- Whilst only the publication data required and suitable for display until the next import is retained, the GEO data for the affiliations that has already been collected is preserved. With each new import, the internal comparison of the ROR identifier ensures that only the affiliation data not yet stored at that time is retrieved via the ROR API, thereby limiting the number of daily retrievals to 100–200 (for up to 10,000 affiliations with linked ROR identifiers per day). Currently, just under 42,000 ROR/affiliation records are stored. The following are stored:
Once all import steps have been successfully completed, the copy table for “old” data is emptied and the system reverts to the newly populated import table.
German Map
- The data for the map of Germany is updated at longer intervals via manual CSV imports directly through the database’s admin interface. The front-end can make this call via the dynamic API endpoint; however, given that the data remains unchanged over a longer, predefined period, a static JSON file can also be generated and integrated. This eliminates the need for database queries for the map of Germany during live operation.
- For the German map, ROR-IDs are selected from the KB OpenAlex in-house database, the ROR-IDs are then used to fetch Wiki-ID and the URL of the institution webpage from ROR (using the ror4r library). Geo coordinates are then fetched from Wikidata (using the tidywikidatar).
- The German Map is based on the toolbox Mapbox which provides configurable cross-platform maps.
Conclusion
The visualization illustrates both our ideas and the practical challenges of using bibliometric metadata for dynamic and interactive representations of the scientific publication system. We hope that these applications will help, in a playful way, to foster a better understanding – beyond expert communities – of how bibliometric information is generated and how aggregation leads to interpretability.
Citation
@online{schmidt2026,
author = {Schmidt, Marion and Tobolski, André and Strick, Eva and
Witteler, Arne and Albinus, Janis and Frölich, Arne and
Stahlschmidt, Stephan},
title = {The {KB} “{Dynamic} {Window}” into {Bibliometric} {Data:} How
to Plan and Build a Visualization},
date = {2026-08-04},
url = {http://www.open-bibliometrics.de/posts/2060804-DigitalWindow/},
langid = {en}
}