all agent alignment api update architecture benchmark business cuda dataset deployment embedding fine tuning game dev hype inference infrastructure library ner new model open source optimization plugin probe targeted prompt engineering quantization rag research security testing tool training tutorial workflow
Making LLMs tell you how confident they really are through probe-targeted fine tuning.[R]
r/MachineLearning · 9h ago · 8 · fine tuning research probe targeted open source

Research demonstrating that instruct-tuned LLMs internally distinguish correct from incorrect answers (0.76-0.88 AUROC) despite displaying uniform 99% confidence externally. The authors use LoRA fine-tuning on probe-extracted hidden state targets to align the model's expressed confidence with its internal knowledge, validated through activation patching experiments showing causal relationships (ρ=0.976) across 8 models (7B-70B). Code and pre-registration are publicly available.