paper / ananyaku / May 12
Physics-informed neural networks (PINNs) suffer from directional gradient conflicts between competing loss terms (e.g., PDE residuals vs. boundary conditions), which fundamentally limit first-order optimizers. This paper proves that second-order optimization — specifically the quasi-Newton method SOAP — implicitly achieves gradient alignment via Hessian preconditioning, bypassing the need for manual loss balancing heuristics. The approach yields state-of-the-art results across 10 PDE benchmarks, including the first demonstrated PINN solution for turbulent flows at Reynolds numbers up to 10,000, with 2–10× accuracy gains over prior methods.
physics-informed-neural-networkssecond-order-optimizationgradient-alignmentmulti-task-learningdeep-learningscientific-computingpde-solvers
“First-order optimizers are theoretically limited in PINNs due to directional conflicts between gradient components of competing loss terms.”
paper / ananyaku / May 12
Language models produce unreliable confidence scores when asked to self-assess in absolute terms (e.g., "rate your confidence 0–1"), yielding coarse-grained outputs that poorly predict correctness. This paper proposes framing confidence estimation as a pairwise preference task — asking models which of two questions they're more confident answering — then aggregating match outcomes via Elo or Bradley-Terry ranking into calibrated confidence scores. Evaluated across 5 frontier LLMs and 14 STEM, social science, and commonsense QA benchmarks, relative confidence estimation yields consistent gains over both absolute and self-consistency baselines in selective classification AUC.
llm-confidenceuncertainty-estimationai-evalslanguage-modelscalibrationpreference-learningbenchmarking
“Relative confidence estimation achieves an average 3.5% gain in selective classification AUC over direct absolute confidence estimation across all tested models and datasets.”
paper / ananyaku / May 12
Recursive automatic differentiation in Physics-informed Machine Learning (PhiML) is fundamentally ill-suited for high-order PDE constraints—it produces inaccurate derivatives, blows up memory, and degrades under noise. Mollifier Layers address this by replacing autodiff with convolution operations using analytically defined mollifiers, reframing derivative computation as smoothing integration. The module is architecture-agnostic, attaches at the output layer with no structural changes, and demonstrates improved accuracy, memory efficiency, and training speed across first-, second-, and fourth-order PDE benchmarks. The approach was validated on a real biomedical inverse problem: inferring spatially varying epigenetic reaction rates from super-resolution chromatin imaging data.
physics-informed-mlinverse-problemspartial-differential-equationsautomatic-differentiationdeep-learningscientific-mlparameter-estimation
“Recursive autodiff produces inaccurate high-order derivatives, inflates memory usage, and underperforms in noisy settings for PDE-constrained learning.”
paper / ananyaku / May 12
Discriminative classifiers tend to latch onto spuriously correlated features, causing failure under distribution shift. This paper demonstrates that generative classifiers — built on class-conditional generative models (diffusion-based and autoregressive) — sidestep this by modeling the full joint distribution of features rather than just the most predictive (often spurious) ones. The approach requires no specialized augmentations, regularization, or prior knowledge of spurious correlations, yet achieves state-of-the-art results on five image and text distribution shift benchmarks and improves robustness in medical and satellite imaging tasks. A Gaussian toy analysis is used to characterize the inductive biases and identify data regimes where generative classifiers have an edge.
generative-classifiersdistribution-shiftspurious-correlationsdiffusion-modelsdiscriminative-modelsrobustnessmachine-learning
“Generative classifiers achieve state-of-the-art performance on five standard image and text distribution shift benchmarks.”