pkgxray calibration — methodology

What was scanned

1,301 published packages in one static pass, split across three target lists resolved on 2026-07-19:

top-1000 (the calibration list, 1,000 packages) — the anvaka npmrank.json pagerank pool re-ranked by real last-week download count (api.npmjs.org/downloads/point/last-week), top 1,000 pinned to their resolved latest version. This is the list the false-block number is measured on.
MCP (300 packages) — an npm keyword search over mcp-server, modelcontextprotocol, mcp, capped by relevance then re-ranked by last-week downloads. A hunting list, not part of the false-block denominator.
known-malware corpus — reconstructed from OpenSSF malicious-packages and published advisories (see below).

Exact tool and command

pkgxray 1.0.3 (build 1fed962, Node v26.0.0), one invocation per package:

pkgxray guard npm:<name>@<version> --format json

Static only — pkgxray reads the tarball's bytes and queries OSV; it never executes package code and never connected to any hosted endpoint. Exit codes: 0 safe, 2 block, 3 review. A scan that failed to produce a parseable verdict is recorded as a scan error, never counted as safe.

How false blocks were adjudicated

A block has two very different causes, and they are split before anything is called a false positive:

OSV block — the only high finding is a known CVE. Blocking here is by design and is not a false positive.
Heuristic block — a malware-signal finding (install hook, exec, exfil, obfuscation, credential/agent access…). These are the reputation-staking calls, and every one was read by hand.

Every top-1000 block was a real CVE (OSV, by design); zero were heuristic false positives. The one pre-retune heuristic false positive — a ~30-line utility that reads .npmrc solely to return the configured registry URL, never touching the auth token, with no network egress at all — was minimized into a benign corpus fixture and the over-firing heuristic retuned, so on this engine it resolves to review. Re-running the full 1,000-package list on the fixed engine confirms 0.0% = 0 / 1,000, with no new false block.

Reconciliation note (read this)

This run is the re-scan on the retuned engine. An earlier provisional run (2026-07-19, kept unedited in run history) published the as-measured pre-retune figure of one top-1000 false block, flagged as pending a re-run. It now has one: re-running the full 1,301-package scan on the fixed engine yields zero top-1000 false blocks, and the earlier run's number was left in place rather than silently edited. This is also the scoping correction that reconciled the README's earlier "0 false blocks" line: that claim held only on a stale 2019 dependents list; the figure here is corpus-gated and scoped to "the top-1000 most-downloaded."

Catch rate and the corpus

npm removes confirmed malware, so live-registry recall is untestable — of 22 curated known-malicious versions, only one is still downloadable. Catch rate is therefore measured against a committed, reconstructed corpus run through the real static engine (node benchmark/run.js): 19 of 20 malicious samples block outright (95.0%), and 0 passed as safe — the remainder are routed to review by policy (download-then-exec and geo/locale-gated payloads), not missed. CI hard-fails on any false block, any full miss, and any recall regression.

Reproduce it

The resolved target lists (names, versions, and download counts — inputs only, no verdicts) are committed here:

https://github.com/adamsjack711-ux/pkgxray/tree/main/validation/calibration-2026-07-19

Re-run pkgxray guard npm:<name>@<version> --format json over the top-1000 list and you will re-derive the verdict distribution and the false-block count. The corpus and benchmark harness live in the repository under benchmark/.