Chronological feed of everything captured from Matt Levine.
RubricRefine addresses a structural weakness in LLM tool-use agents: inter-tool contract violations (wrong output shapes, incorrect routing, broken argument provenance) that execute silently without raising errors, making runtime feedback fundamentally insufficient for catching them. The method generates task- and registry-specific rubrics, scores candidate code against explicit semantic contract checks, and iteratively repairs failures before any execution occurs — entirely training-free. On M3ToolEval, it achieves 0.86 averaged across seven models with zero execution attempts, versus 0.65 baseline and only 0.75 with real execution feedback, while delivering up to 2.6× lower latency. The approach is explicitly scoped to multi-step tool chains; it shows no benefit on single-step benchmarks like API-Bank, confirming its dependence on inter-tool contract structure.
Masked language models (MLMs) define local conditional distributions that are generically inconsistent with any valid joint distribution over sequences — a property the authors formalize and empirically confirm via a "rectangle test." Modeling iterative masked-token resampling as a Glauber dynamics Markov chain, the authors derive that mixing time is O(n log n) under bounded cross-token influence (high temperature), but becomes exponentially slow under a uniform local margin condition (low temperature), implying the chain can become trapped in semantic basins. Empirical results confirm a temperature- and sequence-length-dependent phase transition in mixing behavior, with measurable consequences such as long-lived traps and recurrent semantic attractors.
Levine & Weiss demonstrate that the apparent creativity of diffusion models — their ability to generate novel, realistic images beyond training data — is not an emergent generalization property but a direct consequence of the inductive biases baked into the denoiser architecture. A Bayes-optimal denoiser would simply reproduce training samples; real-world architectures like UNet deviate from this optimum in structured ways that produce novel outputs. The authors provide closed-form theoretical characterizations of generated sample distributions for three denoiser families (linear, polynomial, bottleneck) and show empirically that small UNet modifications cause dramatic, often harmful shifts in output quality. The core takeaway: diffusion model success is contingent on tight alignment between the denoiser's inductive bias and the true data distribution.