Method

Where the data comes from, how the models were trained, and what the output is currently worth.

Where the patches come from

lore.kernel.org's web pages and search endpoint sit behind bot protection, so patches are fetched from the public-inbox git mirrors instead — each mail is a commit, and a shallow clone of the recent few thousand is enough. A message qualifies if the subject contains [PATCH, the body carries Reported-by: syzbot+…, and there is a real diff. Crash reports come from the syzkaller dashboard.

Only patches still under review are kept — threads with no v2 yet. That is the situation this tool is meant for: the patch has just been posted and review has not arrived.

Deciding which message is v1

Two traps, both now handled in the fetcher:

• [PATCH 5.10] is a stable backport and [PATCH 08/19] is a series position. Neither is a version number, and treating them as one manufactures threads that appear to have fifteen revisions.
• The more damaging one: an untagged [PATCH] posted after a v8 is usually a maintainer's applied copy. Taking it as v1 hands the model the final code and asks what is wrong with it. The fetcher now requires version order to agree with posting order, and discards the thread otherwise.

Not showing what the models memorised

Every candidate is checked against a list of 928 bug ids the models were trained or evaluated on, and any match is dropped. The list ships as plain text with the code — it used to be derived from the training corpus, which is git-ignored, so after a clone it silently shrank to 50 entries and one memorised bug made it back into the demo. A short list is now a hard error rather than a quiet degradation.

The models

Five bases — Gemma-4 12B, Gemma-3 12B, Qwen3.5 9B, Qwen3 14B, Llama-3.2 11B — each fine-tuned with LoRA (r=32, α=64, bf16, 4 epochs, max_len=4096) on 869 syzbot bugs, mapping a v1 patch to the revision points review actually demanded. At inference every model reads the same prompt: crash report plus v1 patch, and returns four to six points.

One detail that matters: some commit messages embed an entire syzbot log, so truncating a patch from the top can remove every hunk and leave the model reviewing a diff it never saw. The message header gets a small budget and the rest goes to the diff.

How far to trust the output

In an offline benchmark with known answers, the advisors proposed under 10 % of what reviewers actually asked for. Two systematic weaknesses are worth knowing about:

• Rejecting is easier than proposing. Saying "v1 does this wrong" is markedly more accurate than saying what to do instead.
• Process requests are almost absent. Adding a Fixes: tag, rewriting a changelog, retargeting a tree, adding a selftest — roughly a quarter of real review comments — barely appear, because they are under 2 % of the training targets.

The task framing is also being revised: the training instruction asks about "the final accepted fix", which does not exist while a patch is still under review.

Treat this as a lead generator. Every claim needs checking against the kernel source and the real thread.

How it updates

The site is static. One run of the refresh script fetches a new batch of patches, runs the models on a local GPU, regenerates data/*.json and redeploys. There is no backend and no request at page load beyond the JSON files themselves — and the per-patch detail is only fetched when you open it.

./script/refresh.sh --push-site

Code and full evaluation data: github.com/sysec-uic/Linux-Patch-Advisor