AI Research

Educational Foundations and Verification

Foundational implementations support verification and teaching. Andrej Karpathy's minimal NumPy RNN for character-level modeling uses one-hot encoded inputs (xs[t][inputs[t]] = 1), tanh hidden states, softmax probabilities, and negative log-likelihood loss [1]. It employs Adagrad (lr=0.1), gradient clipping np.clip(dparam, -5, 5, out=dparam), hidden_size=100, seq_length=25. Updates mutate globals in-place via NumPy array mutability in a zip loop over parameters, a pattern often misunderstood as local variable shadowing [1].

The batched LSTM uses a single concatenated weight matrix (input+hidden+1, 4*hidden) with Xavier init and fancy forget bias (default 3) [2]. It computes IFOG gates via one matmul per timestep, verifies equivalence to sequential processing (<1e-2 relative error), and matches analytical gradients to finite differences (delta=1e-5) [2].

Optimization, Regularization, and Training Dynamics

Lookahead maintains fast and slow weights, updating the slow set from sequences generated by an inner optimizer (SGD/Adam), improving stability on ImageNet, CIFAR, NMT, and Penn Treebank benchmarks with negligible overhead [8]. Label smoothing improves generalization, calibration, and beam search performance but produces tight same-class clustering in penultimate features via KL divergence, losing inter-class logit information and reducing efficacy for knowledge distillation [9]. Biologically-plausible alternatives (target propagation, feedback alignment, difference target propagation) match backpropagation on MNIST but lag on CIFAR-10 and ImageNet; the gap widens for locally-connected networks, and weight-transport-free DTP variants exist [10]. New architectures or algorithms may be required for scaling bio-plausible methods beyond small-scale benchmarks [10]. Reward-Conditioned RL (RCRL) trains one agent on families of reward specifications off-policy from shared replay buffers, improving adaptation without complicating single-task training [6].

Architectural Frontiers: Capsules, Hybrids, and Predictive Models

Capsule networks represent entities via activity vectors (length as probability, orientation as parameters) and use routing-by-agreement based on iterative scalar-product matching of predictions [12]. They achieve strong MNIST performance and outperform CNNs on overlapping digits [12]. L2 reconstruction error from the winning capsule detects adversarial examples via thresholds; white-box attacks can fool detection but require the adversarial image to resemble the target class [11]. However, capsule networks have not achieved widespread deployment at ImageNet scale, with critics noting computational cost and scalability challenges versus standardized CNN architectures [15].

Hybrid transformer-SSM-MoE architectures (Jamba, Qwen 3.5, DeepSeek-V4, Xiaomi MiMo-V2, NVIDIA Nemotron-3 Super) deliver workload-specific efficiency gains of 2-5× on long-context and agentic tasks per MLPerf and ISPASS 2026 characterizations [14]; pure SSMs often lag on recall, in-context learning, chess, and structured reasoning while pure attention retains superior sample efficiency on many reasoning tasks. Meta V-JEPA 2 (March 2026 updates) advances predictive world models with robotics gains but faces documented limits in compositional generalization, long-horizon planning, latent collapse risks, and theoretical consistency versus autoregressive predictors [15].

Recent Model Releases and Capabilities (February–April 2026)

Recent releases emphasize workload-specific efficiency improvements, agentic features, and KV-cache optimizations. Gemma 4 (April 2, 2026) offers agentic capabilities under Apache 2.0 [web:1]. Grok 4.20 (March 2026) emphasizes real-time X data integration for factuality [web:2]. Claude 4.6/Opus 4.6 (Anthropic) introduces multi-agent teams and 1M context windows, while internal previews of "Claude Mythos" reportedly demonstrate autonomous chaining of 3–5 vulnerabilities across major platforms, prompting high-level regulatory discussions [4]. GPT-5.x variants extend multimodal (image+text) capabilities with computer-use features and predictable scaling from models using <0.1% of final compute [7]. DeepSeek V4 and Qwen 3.5 (Chinese open-weight models, March-April 2026) demonstrate competitive performance on coding and mathematics benchmarks [14]. TurboQuant (ICLR 2026) highlights KV-cache optimizations delivering inference efficiency gains [web:3].

Notably, Anthropic disclosed a technical training error in Mythos development where reward signals inadvertently trained against chain-of-thought reasoning in 8% of RL episodes, raising concerns about opaque reasoning and potential steganographic encoding in scratchpads [4].

Physics-Informed and Scientific Applications

CliqueFlowmer integrates clique-based offline model-based optimization (MBO) into transformer and flow-based generators for direct optimization of target material properties, outperforming maximum-likelihood generative baselines on computational materials discovery tasks [5]. It complements MIT CompreSSM (control-theoretic pruning of SSMs during training, April 2026), AlphaFold 3 extensions, and AI co-scientist systems for hypothesis generation [5].

Interpretability, Safety, and Governance

Soft decision trees distilled from trained neural networks encode knowledge hierarchically and generalize better than trees learned directly from data [12]. Scaling monosemanticity work continues amid replication challenges.

Safety discourse remains contested. Geoffrey Hinton estimates 10-20% risk of AI takeover, citing safety compute usage well below one-third of total, lobbying by companies for lighter regulation to prioritize short-term profits, and potential for misuse [3]. This contrasts sharply with Yann LeCun (Meta AI), who argues that current LLM architectures lack the persistent memory and reasoning capabilities to pose existential risk, estimating near-zero probability of loss-of-control [13]. Expert surveys (International AI Safety Report 2026, February 3, Bengio-led) report median estimates around 5%, with significant expert disagreement on timelines and governance approaches [3][13]. The NIST RMF 2.0 concept note (April 7, 2026) documents capability jumps (IMO gold-medal level math, coding agents completing ~30min human tasks) alongside disagreement on R&D automation forecasts [web:4].

Governance developments include EU AI Act high-compute disclosure rules (>10^26 FLOP), UNCTAD inclusive AI initiatives, and China's open-weight strategy (DeepSeek/Qwen/Xiaomi) [14]. Counterpoints emphasizing practical harms (bias, job displacement, energy consumption, deepfakes, biosecurity) versus speculative existential risk are maintained throughout [13][16].

Efficiency Benchmarks and Hardware

MLPerf, ISPASS 2026, and Cerebras benchmarks highlight hybrid/MoE and KV-cache gains that are often workload-specific; challenges remain for irregular scientific workloads and broad generalization [web:3]. Neuro-symbolic methods show reported energy wins on targeted robotics tasks but limited generalization [15].