Methods, calculations & glossary

How this dashboard was built, how scores are computed, and what the metrics mean.

What we did

This dashboard presents journal-level rigor and transparency metrics derived from the same large-scale analysis pipeline described in Menke et al. (2022) (J Med Internet Res; DOI 10.2196/37324). We applied the established methodology to assess all papers in the PubMed Central (PMC) Open Access subset and scored each manuscript with SciScore version 3 (dataset: 2026_sciscore_v3).

SciScore is an automated natural-language processing tool that detects transparency criteria aligned with reproducibility guidelines (NIH rigor standards, MDAR, ARRIVE, and related checklists) and identifies key biological research resources (antibodies, cell lines, organisms, software tools). It produces a numerical score for each paper and aggregates those scores into the Rigor & Transparency Index (RTI) at journal, publisher, and benchmark levels.

Corpus. Full-text articles from the PMC Open Archives Initiative (OAI) open-access subset — the same machine-readable corpus used in the 2022 institutional RTI study. Abstract-only articles, manuscripts without a detectable methods section, and articles available only as non-parseable PDF supplements are excluded from scoring.

Publisher labels. Primary publisher names come from the official Scopus journal list (Elsevier source record). Where a society publishes under a commercial publisher (for example ASCO journals under Wolters Kluwer Health), the dashboard may also list the society as an additional publisher filter so client portfolios match editorial ownership. SciScore did not invent these mappings independently.

Time range in this dashboard. Journal-level data spans 1960–2025 where papers exist in the embedded corpus (not every calendar year has coverage). Corpus-wide benchmark series may extend slightly earlier. Year tabs show only years with data for the selected journal or publisher scope.

SciScore version. Version 3 (released August 2024) — see SciCrunch release notes for details on classifier and scoring refinements. The core framework (10-point scale, conditional rigor logic, resource findability) follows Menke et al.; this dashboard applies v3 across an expanded PMC corpus through 2025.

How individual papers are scored

Each manuscript receives a SciScore from 0 to 10. The score has two components, each contributing up to 5 points:

1. Rigor adherence (up to 5 points)

SciScore detects whether a paper addresses reproducibility-related reporting criteria — for example IRB/IACUC approval, randomization, blinding, power analysis, sex as a biological variable, inclusion/exclusion criteria, and related items. The rigor score compares the number of found criteria against the number expected given what the manuscript contains, using conditional scoring logic: criteria are only expected when contextually relevant (e.g. cell-line authentication is expected only when a cell line is detected; ethics-related criteria trigger expectations for study-design reporting).

2. Resource findability (up to 5 points)

SciScore identifies key biological resources mentioned in the manuscript — antibodies, organisms, cell lines, and software tools — and checks whether each is uniquely identifiable (via a Research Resource Identifier, RRID, or an RRID suggestion). The resource score compares uniquely identifiable resources to the total resources detected.

Not applicable (score = 0)

Papers that contain no scorable criteria or resources (e.g. editorials, highly theoretical work, or manuscripts with no methods section) receive a SciScore of 0 and are marked not applicable. These papers are excluded from RTI aggregation — only papers with SciScore > 0 contribute to journal and portfolio averages.

Open science & identifiers (journal aggregates)

SciScore detects data availability, code availability, protocol identifiers, and data/code identifiers at the individual-paper level. This dashboard now includes journal-year aggregates for these criteria (rates and, where applicable, detection counts) alongside RTI, study-design rates, and resource-findability rates. Rates are the share of scored papers in that journal-year where the criterion was detected; they are shown in the Open Science & Identifiers panel and benchmark tables.

Higher RTI values indicate more methodological detail and better resource identification. They are associated with greater replicability potential but do not guarantee that a study can be replicated without additional information. In the Reproducibility Project: Cancer Biology, replication studies scored significantly higher (mean RTI 7.6) than the original papers they attempted to reproduce (mean RTI 3.4).

How dashboard metrics are calculated

Journal RTI

For a given journal and publication year, RTI is the arithmetic mean SciScore across all scored papers (SciScore > 0) published in that journal that year. The paper count (n) shown in the dashboard is the number of scored papers included.

Publisher / group portfolio RTI

When viewing a publisher or publisher-group overview, metrics are paper-weighted aggregates across all journals in scope. For each rate metric (RTI, sex reporting, randomization, etc.), the portfolio value is:

Σ (journal rate × journal paper count) / Σ (journal paper counts)

This ensures journals with more papers contribute proportionally more to the portfolio average, rather than treating each journal equally regardless of volume.

Sub-metric rates (study design & resources)

Beyond RTI, the dashboard shows the proportion of papers addressing each criterion within a journal or portfolio — for example, the percentage of papers reporting randomization, blinding, or power analysis, and the percentage of detected antibodies, organisms, cell lines, or tools that carry an RRID. These are stored as rates between 0 and 1 and displayed as percentages in the UI.

Benchmark comparisons

The Compare against selector lets you benchmark a journal or portfolio against:

All journals — corpus-wide averages from the by_year sheet in the SciScore v3 dataset.
Discipline groups — averages across journals tagged with a given discipline category (from the extended journal list).
SciScore client publishers — aggregated metrics for journals published by SciScore client organizations.

Delta badges show the difference between the selected subject and the chosen benchmark for each metric.

Europe PMC RRID sampling (journal view)

The live RRID panel samples up to 24 open-access PMC articles per journal per year via the Europe PMC API, fetches RRID annotations, and compares observed RRID usage against SciScore benchmarks. This is a spot-check validation layer independent of the bulk SciScore scoring pipeline.

Glossary

Term	Definition
RTI	Rigor & Transparency Index — the mean SciScore over a set of papers. Used to compare journals, publishers, institutions, or countries on reporting quality.
SciScore	Per-paper score (0–10) from automated NLP analysis of a manuscript's methods and materials. Combines rigor adherence (0–5) and resource findability (0–5).
PMC	PubMed Central — NIH's free full-text archive of biomedical and life-sciences literature. The open-access subset (PMC-OAI) is the machine-readable corpus used for bulk scoring.
RRID	Research Resource Identifier — a persistent ID for antibodies, cell lines, organisms, software tools, and other research resources, enabling unambiguous identification across publications.
ROR	Research Organization Registry — a persistent identifier for research institutions worldwide (ror.org). The institution dashboard matches CSV names to ROR via the public affiliation API and shows parent/child hierarchy from ROR relationships where available.
MDAR	Materials Design, Analysis, and Reporting — a publisher-driven multidisciplinary reporting framework whose checklist criteria inform SciScore's rigor checks.
ARRIVE	Animal Research: Reporting of In Vivo Experiments — guidelines for reporting animal research; criteria incorporated into SciScore.
NIH rigor	U.S. National Institutes of Health guidelines on rigor and reproducibility in grant applications, covering study design, biological variables, authentication, and more.
IRB	Institutional Review Board — ethics approval for human subjects research. Dashboard rate = proportion of papers with a detected IRB/ethics statement.
IACUC	Institutional Animal Care and Use Committee — ethics approval for animal research. Dashboard rate = proportion of papers with a detected IACUC statement.
Sex (sex)	Proportion of papers reporting sex as a biological variable or sex of study subjects.
Power (pwr)	Proportion of papers reporting a power analysis or sample-size justification.
Randomization (rand)	Proportion of papers reporting random assignment of subjects or samples to groups.
Blinding (blind)	Proportion of papers reporting blinding of investigators, assessors, or analysts.
Antibodies w/ RRID (ab)	Among detected antibodies, the proportion with an RRID or RRID suggestion.
Organisms w/ RRID (org)	Among detected organisms/model systems, the proportion with an RRID.
Cell lines w/ RRID (cl)	Among detected cell lines, the proportion with an RRID.
Software tools (tool)	Among detected software tools, the proportion with an RRID.
Data availability (data)	Proportion of papers with a detected data-availability statement or repository link.
Code availability (code)	Proportion of papers with a detected code-availability statement or repository link.
Protocol identifiers (prot)	Proportion of papers with a detected protocol identifier (e.g. protocols.io).
Data identifiers (data_id)	Average number of detected persistent data identifiers per scored paper (can exceed 1.0 when papers cite multiple dataset IDs).
Code identifiers (code_id)	Average number of detected persistent code identifiers per scored paper (can exceed 1.0 when papers cite multiple code/archive IDs).
Paper-weighted	Aggregation method where each journal's metric is weighted by its paper count, so high-volume journals influence portfolio averages more than low-volume ones.

Limitations

Dataset coverage. This SciScore v3 dataset covers scored, machine-readable PMC open-access papers. Full-year completeness has not been independently verified. Paper counts mean scored PMC open-access papers, not total journal output. Changes in scored counts or RTI may reflect coverage or article mix; aggregates alone cannot establish a change in reporting practice. Missing years and blank metrics are unavailable, not measured zero.

Open-access subset bias. The PMC open-access corpus represents only a fraction of all biomedical literature. Journals with restrictive access policies are underrepresented.
Supplementary PDFs. Methods or resource details provided only in supplementary PDF files are not machine-readable and may be missed by automated scoring.
Field differences. RTI is most applicable to experimental biomedical research. Theoretical, computational, or non-life-science papers may score low or be marked not applicable despite being rigorous in their own context.
Automated detection. NLP classifiers can produce false positives and false negatives. Entity-level agreement rates are generally >80% (many >90%) based on manual curation in the 2022 validation study, but edge cases remain.
Not a replicability guarantee. RTI measures reporting transparency, not experimental correctness. A high score means a paper contains more methodological detail — a necessary but not sufficient condition for replication.

References

Menke J, Eckmann P, Ozyurt IB, Roelandse M, Anderson N, Grethe J, Gamst A, Bandrowski A. Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality. J Med Internet Res. 2022;24(6):e37324. doi:10.2196/37324
JMIR Publications. Rigor and Transparency Index: Large Scale Analysis of Scientific Reporting Quality [announcement]. jmir.org/announcements/339
Menke J, Bandrowski A. Rigor and Transparency Index (RTI) — version 1.0 methodology. J Med Internet Res. 2021;23(11):e31282. doi:10.2196/31282
SciCrunch Inc. SciScore Version 3 beta released. scicrunch.com/version-3-released (August 2024).