pkgxray calibration — methodology

What was scanned

1,000 published packages in one static pass, across download-ranked target lists resolved around 2026-08-04:

1,000 — top-1000 PyPI download-ranked list: the false-block denominator; committed list of the 1,000 most-downloaded PyPI project names. inputs

Counted separately and not part of the 1,000-package denominator above:

5 — reconstructed PyPI known-malware corpus: catch-rate corpus — NOT part of the 1,000 denominator. inputs

The top-1000 PyPI list is committed as ranked project names (validation/top-pypi-1000.txt), regenerable from the hugovk/top-pypi-packages download ranking via scripts/build-pypi-list.js. Guard resolves each name to its latest sdist at run time, so the resolved version set is reproducible in method; project-name membership is byte-identical.

Exact tool and command

pkgxray 1.0.6 (build 1067b74, Node v26.0.0), one invocation per package:

pkgxray guard pypi:<name>@<version> --format json

Engine 1.0.6 was a pre-release build, not published to npm (targets 1.1.0) — npx pkgxray@1.0.6 will not resolve. The public npm release at run time was 1.0.5. To reproduce on the exact engine, check out the commit:

git checkout 1067b74 && node scripts/validate-at-scale.js --ecosystem pypi --cohort pypi

Static only — the scanner reads the tarball's bytes and queries OSV; the static scan never executes package code and never connected to any hosted endpoint. Exit codes: 0 safe, 2 block, 3 review. A scan that failed to produce a parseable verdict is recorded as a scan error, never counted as safe.

How false blocks were adjudicated

A block has two very different causes, and they are split before anything is called a false positive:

OSV block — the only high finding is a known CVE. Blocking here is by design and is not a false positive.
Heuristic block — a malware-signal finding (install hook, exec, exfil, obfuscation, credential/agent access…). These are the reputation-staking calls, and every one was read by hand.

Every top-1000 PyPI block was a known CVE or a documented defensible true positive; zero were heuristic false positives. An earlier build of this engine false-blocked 79 of the top-1000 (7.9%): the behavioral engine — first calibrated for JavaScript — mis-read ordinary Python idioms as attacks (a bulk os.environ read as a token-harvest, importlib / __import__ as a computed require, and a syntax lexer's .bashrc filename string as an rc-file write). The fix scopes that JS-primitive detector suite to the languages it models: a Python sdist is now audited by its setup.py / pyproject install-hooks, OSV, and the language-neutral checks (prompt-injection, hidden-unicode), while the JS-shaped behavioral detectors no longer run on Python source. Re-running the full 1,000-package list on the fixed engine confirms 0.0% = 0 / 1,000. Of the four blocks in the top-1000, three carry a known-CVE finding (OSV, by design) and one is a defensible true positive — a package that ships a live .claude/settings.json hooks config, an install-time-equivalent agent auto-exec surface.

Reconciliation note (read this)

This is the first published PyPI-ecosystem run, so there is no prior PyPI figure to reconcile. The 79 → 0 reduction described above happened before publication, on the pre-release engine; the number on this page is the post-fix re-measurement on the committed 1,000-package list, not a provisional figure awaiting a re-run. The PyPI cohort is reported at its own stable URL and is not merged into the npm calibration denominator.

Catch rate and the corpus

Catch rate is measured against a committed, reconstructed PyPI sdist-dropper corpus run through the real static engine (node benchmark/run.js --cohort pypi): 5 of 5 malicious samples block outright (100.0%), and 0 passed as safe. This is a deliberately small, honest denominator — five setup.py install-time droppers that decode (base64 / marshal / zlib) or network-fetch a payload and exec it during pip install — reported as-is, not extrapolated to a population claim. CI hard-fails on any false block, any full miss, and any recall regression.

Reproduce it

The committed top-1000 target list (names, versions, and download counts — inputs only, no verdicts) is here: https://github.com/adamsjack711-ux/pkgxray/blob/main/validation/top-pypi-1000.txt.

Check out the exact engine and run one invocation per package over the top-1000 list to re-derive the verdict distribution and the false-block count:

git checkout 1067b74 && node scripts/validate-at-scale.js --ecosystem pypi --cohort pypi

The corpus and benchmark harness live in the repository under benchmark/ (node benchmark/run.js).