LLMs for security reasoning

CWE-Trace

Calibration without comprehension — diagnosing the limits of fine-tuning LLMs for vulnerability detection in systems software.

Zibaeirad, A. & Vieira, M. UNC Charlotte ISSRE 2026 · Main Research Track

Back to projects Paper (PDF) arXiv:2606.20502 ISSRE 2026

// 01 Abstract

Overview

834curated Linux kernel samples

74distinct CWE types

8 + 15vanilla / LoRA fine-tuned models

52.1%best detection score (vs. 50% chance)

Whether LLMs scoring well on vulnerability benchmarks genuinely reason about security or merely pattern-match on contaminated data remains unresolved. We present CWE-Trace, a framework for LLM vulnerability detection built from 834 manually curated Linux kernel samples spanning 74 CWEs. The framework enforces a strict temporal split (pre-2025 historical set / post-cutoff leakage-free set), preserves context-aware vulnerable–patched pairs, and introduces two diagnostic metrics: the Directional Failure Index (DFI) and Hierarchical Distance and Direction (HDD). We evaluate eight vanilla LLMs and 15 LoRA fine-tuned variants across non-targeted detection, targeted detection, and CWE classification.

Our analysis yields two key results. First, data contamination provides no measurable advantage — function-level analysis shows that 84% of nominally contaminated samples carry no usable memorization signal: vulnerable functions are absent or cross-mapped across datasets, and ~31% of contaminated samples carry CWE misclassification. Second, backbone directional priors dominate fine-tuning — models exhibit stable, systematic failure modes (DFI ranging from −85.5 to +94.8 pp) that persist from historical to post-cutoff data and resist correction. Fine-tuning shifts the output threshold without changing the decision policy — this is calibration without comprehension. The weakest backbone at binary detection (DeepSeek-R1) gains the most in coarse CWE classification, revealing that detection and understanding are decoupled capabilities. The best detection score reaches only 52.1% (+2.1 pp above chance); exact CWE ranking remains below 1.3% Top-1 accuracy.

// 02 Framework

How CWE-Trace works

CWE-Trace framework overview — The pipeline extracts and manually pairs 834 Linux kernel samples (417 pairs), strictly split into historical (PBD, ≤2024) and leakage-free (LFD, 2025) sets. Vanilla and fine-tuned LLMs are evaluated across three tasks — Non-targeted Detection, Targeted Detection, and CWE Classification — via a suite of metrics covering detection accuracy, pair-wise consistency, and classification ranking.

CWE-Trace pairs vulnerable and patched code extracted from Linux kernel commits into context-aware blocks that preserve cross-file dependencies (macros, type definitions, interacting functions) often discarded by function-isolated baselines. A strict temporal split separates genuine generalization from memorization. DFI quantifies whether a model's errors skew "paranoid" (over-flagging) or "skeptical" (under-flagging); HDD measures how far off a wrong CWE prediction lands in the CWE-1000 taxonomy graph, and in which direction.

CVE-2024-41010 motivating example — CVE-2024-41010: the Use-After-Free (CWE-416) is invisible in `sch_ingress.c` unless the model also verifies the type definition in `tcx.h` — multi-file context is necessary for realistic vulnerability detection.

// 03 RQ1

Vulnerability detection & directional failure

Do vanilla and fine-tuned LLMs actually detect vulnerabilities, or do near-50% accuracy scores hide systematic directional bias?

RQ1 mechanism analysis — Fine-tuning deltas inside vs. outside the FT label space, and on contaminated vs. clean samples.

Fine-tuned vs vanilla shift map — Fine-tuned vs. vanilla shift map across backbones and datasets.

1 Base models perform near-random with severe directional bias

Overall accuracy clusters at 49–53% (chance = 50%). CodeLlama, the best vanilla model, reaches only 52.1%. DeepSeek-R1 is strongly paranoid (DFI +94.8pp), GPT-4.1-mini is strongly skeptical (DFI −85.5pp). These patterns persist from PBD to LFD.

2 Fine-tuning effect tracks the base model's initial competence

Qwen3-4B gains +27.5 to +41.5pp mainly by overcoming abstention, not by learning transferable semantics. DeepSeek-R1 fine-tuning consistently degrades detection while amplifying its paranoid prior.

3 Label-space transfer is flat — fine-tuning shifts policy, not knowledge

Gains on CWEs covered by the fine-tuning label space are nearly identical to gains on uncovered CWEs — a global response-policy shift, not CWE-specific learning.

4 Contamination has no detectable effect

No consistent advantage of contaminated "seen-CVE" samples over clean ones across any backbone family.

5 Context depth is necessary but not sufficient

Single-function (L1) vs. multi-file (L2) samples show almost no split-level difference — near-chance performance persists at both context depths.

// 04 RQ2

CWE-1000 classification before & after fine-tuning

Can models at least place a vulnerability into the correct coarse CWE-1000 root family?

RQ2 taxonomy fine-tuning delta heatmap — Fine-tuning Δ Root-Micro@1 for coarse CWE-1000 classification relative to the vanilla backbone. Red = fine-tuning hurts; green = fine-tuning helps.

RQ2 bias persistence — Directional bias persists from PBD to LFD.

1 Base models reach moderate coarse accuracy but with macro disparity

StarCoder2 leads on LFD at 61.0% Micro@1, but Macro@1 stays as low as 9.7–19.3% on PBD — models concentrate on a narrow set of frequent root families.

2 Fine-tuning response is backbone-dependent — and inverse to RQ1

DeepSeek-R1-32B improves on every fine-tuning dataset (+9.2 to +25.9pp); Llama3.1-8B degrades on every one (down to −49.0pp). The backbone hardest to improve in detection can gain the most on coarse taxonomy placement.

3 Binary-label fine-tuning causes catastrophic forgetting of CWE knowledge

Llama3.1-8B Devign-FT drops −49.0pp on LFD Root-Micro@1 — the worst single result in the study. Binary-only fine-tuning can erase coarse taxonomy knowledge rather than improve it.

// 05 RQ3

Semantic depth & hierarchical error

Can a model that recovers the broad weakness family also recover the exact CWE and place it correctly in the taxonomy?

RQ3 semantic depth analysis — Per-model MRR ranking (a); MRR vs. HDD mean distance for fine-tuned models (b); HDD error profile — Exact / Shallow / Deep / Lateral / Invalid (c); VDISC-FT specialization on its five labeled CWEs vs. all remaining classes (d).

CWE ranking comprehensive analysis — Comprehensive CWE ranking analysis across all models and splits.

CWE knowledge depth analysis — CWE knowledge-depth analysis.

Top 5 LFD CWEs analysis — Per-class error breakdown for the five most frequent LFD CWEs.

1 CWE exact identification is unsolved

Best vanilla Top-1 on LFD is 14.71% (GPT-4.1-mini); best MRR is 27.55%. DeepSeek-R1 and StarCoder2 score 0% Top-1.

2 Lateral errors are the universal dominant failure mode (86.5% mean)

Across all eight vanilla models, 86.5% of wrong predictions land at the correct hierarchy depth but the wrong sibling branch — only 4.2% are exact matches.

3 MRR and HDD are only weakly coupled, validating HDD as a distinct diagnostic

Qwen3-Coder reaches MRR = 14.61% on LFD but HDD mean distance = 6.22; Llama3.1 has lower MRR (5.52%) yet tighter hierarchical proximity (HDD = 4.95). Ranking correctness and hierarchical proximity are not the same signal.

4 VDISC specialization is sharp (+33.3pp) but trades plausibility for exact match

On samples within VDISC's five labeled classes, VDISC-FT improves Top-1 by up to +33.3pp — but the same models are flat-to-worse outside those classes. Narrow supervision produces brittle specialization.

// 06 Discussion

Why this matters

The backbone's directional prior dominates fine-tuning

Near-random accuracy (~50%) masks extreme, stable, backbone-determined bias. Selecting a backbone is selecting a failure mode — fine-tuning mostly shifts the output threshold, not the underlying decision policy.

Fine-tuning decouples detection from understanding

VDISC fine-tuning produces large exact-Top-1 gains on its five labeled classes but not on uncovered ones — organizing a narrow label set without transferable root-cause semantics.

CVE-level contamination yields no detectable advantage

CVE-level overlap does not imply function-level memorization. Combined with ~31% CWE label noise, an estimated 84% of nominally contaminated samples carry no usable memorization signal.

Implications for reliability

The bottleneck is supervision fidelity, not dataset size. A more promising path: training data pairing each CVE with its root-cause function, plus objectives that explicitly counter directional collapse.

// 07 Supplementary

Beyond the paper

Complete per-model radar charts (appendix) — Appendix — complete per-model behavioral radar charts for all 8 vanilla and 15 fine-tuned models (the main paper shows only representative archetypes from this set).

Beyond the contamination summary reported in the paper (281 samples audited across PrimeVul, MegaVul, and LineVul, 68.7% combined CWE-label accuracy), the full audit pipeline ran nine function-level contamination experiments:

69.3%PrimeVul CWE-label accuracy (88 samples)

67.1%MegaVul CWE-label accuracy (144 samples)

77.6%LineVul CWE-label accuracy (49 samples)

Exp 1 — SVD/HPS by source. Per-model semantic-detection and hierarchical-prediction-score breakdown on PrimeVul/MegaVul-contaminated samples.
Exp 2 — Code similarity. Best-match similarity between PBD CVEs and PrimeVul/MegaVul candidates.
Exp 3 — HPS/HDD breakdown. Exact/parent/child/sibling/same-root/unrelated decomposition per model.
Exp 4 — CWE consistency. Cross-checks PBD ground-truth labels against PrimeVul/MegaVul for the same CVEs.
Exp 5 — Label bias. Vulnerable vs. safe label ratios per contaminated CVE.
Exp 6 — Class imbalance. Training vs. predicted vulnerable-class percentage per model.
Exp 7 — VDISC CWE coverage. Performance on VDISC's five labeled CWEs vs. all others.
Exp 8 — Exact-match case study. Qualitative case studies of exact-match predictions.
Exp 9 — Cross-architecture. Training vulnerable-class percentage and balancing strategy vs. accuracy, across backbones.

// 08 Cite

Citation

@misc{zibaeirad2026calibrationcomprehensiondiagnosinglimits, title = {Calibration Without Comprehension: Diagnosing the Limits of Fine-Tuning LLMs for Vulnerability Detection in Systems Software}, author = {Arastoo Zibaeirad and Marco Vieira}, year = {2026}, eprint = {2606.20502}, archivePrefix = {arXiv}, primaryClass = {cs.CR}, url = {https://arxiv.org/abs/2606.20502}, }