One-Click DOI Lookup: Using Crossref Content Negotiation Instead of Scraping Pages

Publisher pages are presentation layers. They change layouts, gate content behind consent dialogs, and render metadata after hydration. A DOI is the opposite: a stable identifier with a machine-readable record behind it. Any citation workflow that reads the page instead of the identifier is choosing the less reliable source. Browser extensions can resolve a DOI in one click and return CSL-JSON, which is exactly what Zotero, Pandoc, and most reference tools consume.

The content negotiation endpoint

Crossref supports content negotiation on https://doi.org/{doi}. Send an Accept header for the format you want:

curl -LH "Accept: application/vnd.citationstyles.csl+json" https://doi.org/10.1038/s41586-023-06004-9

The response is CSL-JSON: title, author[] with family and given, container-title, volume, issue, page, issued, and DOI. For BibTeX, request application/x-bibtex; for RIS, application/x-research-info-systems. The same DOI resolves through DataCite for datasets and preprints, so a single lookup path covers most of the literature a student will touch.

In an extension, fetch with the header, then post the JSON to your citation parser. The DOI's host may redirect (publishers do this), and CORS is usually permissive on doi.org and api.crossref.org, but you should still request host permissions for https://doi.org/* and handle opaque redirects. If a redirect drops the Accept header, fall back to https://api.crossref.org/works/{doi}.

Validating the record against the page

The lookup tells you what the DOI says. It does not tell you that the DOI belongs to the page in front of you. That check is the extension's real value.

Compare three fields: the first author's family name, the issued year, and the container title. A mismatch usually means one of three things — the page advertises a related DOI (a correction or a reply), the article is an online-first version with a different record, or the DOI was copied from a reference list rather than the article header.

For corrections, Crossref exposes relation fields such as is-correction-of. If the page's DOI has a relation, surface it before the user cites the paper as primary evidence.

Handling ambiguous and missing identifiers

Not every page has a DOI. Preprints use arXiv IDs or SSRN handles; books use ISBNs; reports use agency URLs. A good extension detects the identifier type from the URL and meta tags, then routes to the right resolver: Crossref for DOIs, DataCite for datasets, arXiv's Atom API for preprints, OpenLibrary for ISBNs.

When no identifier exists, do not fabricate a CSL-JSON record from the page's visible text and present it as verified. Mark the item as "unverified" and let the user edit it. This distinction matters in systematic reviews, where a record with a sourced identifier can be reproduced and a scraped record cannot.

Caching and rate limits

Crossref asks clients to identify themselves with a mailto parameter in the User-Agent or query string. Anonymous traffic is throttled, and a student clicking through fifty references will hit limits. Cache resolved records in chrome.storage.local keyed by DOI with a timestamp; metadata changes rarely, and a 30-day cache is reasonable.

Do not cache the publisher HTML. It is large, changes constantly, and creates stale-citation bugs. Cache only the identifier response.

Where AI fits, and designing the lookup for failure

An AI layer is useful for the parts that require judgment: deciding whether two records with slightly different titles are the same work, normalizing non-English titles, or summarizing what changed in a correction notice. It is not useful for extracting the DOI itself. A regex over the URL and citation_doi meta tag is more reliable and cheaper than a model, and it fails in ways you can log.

The workflow that holds up: detect the identifier, resolve it through content negotiation, validate the three critical fields against the page, and store the CSL-JSON with its source noted. Everything downstream — bibliographies, in-text citations, reference-manager imports — inherits that accuracy.

Most DOI lookups succeed, which is why the failures are badly handled. Three failure classes need distinct UI.

Transient network failure. Retry twice with backoff, then show a retry button. Do not cache the failure.

Not found (404). The DOI may be malformed or registered with a different agency. Fall back to DataCite before reporting "not found", and check the DOI's prefix.

Match mismatch. The record resolves but the title does not match the page. Show both and ask the user to confirm rather than silently substituting the resolved record.

An extension that shows a spinner and then nothing has taught the user that DOI lookup is unreliable, which is worse than a clear error.

Batch resolution without abusing the API

A reference list of 60 DOIs should not produce 60 simultaneous requests. Concurrency of four to six with a small delay keeps within polite usage, and Crossref's works endpoint accepts filters that can resolve several DOIs per call in some cases.

Cache by DOI with a 30-day TTL, and store negative results for a day to avoid hammering a bad identifier. Include a mailto in the user agent or query string so Crossref can contact you about traffic; anonymous heavy use is the first thing throttled.

Normalizing the response for the bibliography

CSL-JSON from different registration agencies is not uniform. issued may have a single year or full date-parts; author may include organizational names as literal; page may be a range or an article number.

Normalize at the boundary: trim whitespace, collapse doubled spaces in titles, convert HTML entities, and mark organizational authors explicitly. Store the raw response too, so a later normalization fix can be reapplied without re-querying.

Every downstream consumer — a reference manager, a Pandoc build, a Word plugin — trusts the identifiers you store. If a DOI is wrong, the error propagates silently and looks like a citation-style problem. Validate the DOI checksum locally before the request; a single mistyped character is far more common than a registry outage and can be caught without a network call.

What the extension should never do

It should not invent a DOI from a title search and present it as authoritative. Title matching produces false positives on similarly named papers, and a wrong DOI is harder to detect than a missing one because it resolves to a real article with plausible metadata.