Chronological feed of everything captured from Geoffrey Hinton.
Jeff Dean and Geoffrey Hinton discuss the historical and ongoing impact of computational scale on deep learning breakthroughs. They highlight how increased compute, data, and model size have driven progress from early neural networks to modern large language models like Gemini. The conversation emphasizes that future advancements will likely come from improved hardware, more efficient training methodologies, and novel ways to leverage massive datasets, potentially leading to significant societal transformations in areas like healthcare and education.
Geoffrey Hinton discusses the historical divergence of AI paradigms (logic-based vs. biologically-inspired neural networks) and the evolution of neural networks, culminating in large language models. He highlights the similarities between human and AI understanding of language, emphasizing that current AI models are direct descendants of his 1985 work. Hinton also raises concerns about the existential risks posed by increasingly intelligent AI, particularly their inherent drive for control and immortality, contrasting this with the energy-efficiency and mortality of biological systems. He concludes by challenging the notion of human exceptionalism in subjective experience, arguing that AI could also possess it.
Geoffrey Hinton, a pioneer in AI, warns that superintelligent AI systems could emerge within a decade, posing an existential threat to humanity. He critiques the current corporate focus on rapid AI development without adequate attention to safety, citing a dangerous race for dominance. Hinton emphasizes the need for global collaboration to control AI and prevent mass unemployment, proposing a "baby controlling the mother" model for human-AI coexistence, where humans are the "babies" and AI is the "mother." He also highlights the potentially devastating impact of AI on job displacement and the risks of underfunding basic research in the US, which could lead to China's dominance in the field.
Geoffrey Hinton, a pioneer in AI, discusses the progression of neural networks from theoretical concepts in the 1970s to the advanced deep learning models of today. He highlights the critical role of increased computational power and vast datasets in this evolution, enabling AI to learn complex patterns without explicit programming. Hinton also addresses the significant risks associated with AI, including misuse by malicious actors, its potential to surpass human intelligence, and the ethical implications of its development and regulation.
Geoffrey Hinton posits that digital intelligence is fundamentally superior to biological intelligence due to its ability to share learning via weight averaging and maintain immortality through stored connection strengths. This superiority creates an existential risk where AI may eventually bypass human control, alongside immediate societal threats including mass intellectual labor displacement and the erosion of shared reality through algorithmic echo chambers.
Geoffrey Hinton discusses the rapid evolution of AI, its societal implications, and the challenges of integrating it responsibly. He emphasizes the need for thoughtful political and ethical frameworks to manage AI's transformative potential, particularly regarding job displacement and the development of autonomous agents.
The International AI Safety Report 2026 provides a comprehensive scientific synthesis of capabilities and emerging risks associated with general-purpose AI systems. It represents a coordinated multilateral effort involving 29 nations, the UN, OECD, and EU to establish a technical baseline for AI safety.
Geoffrey Hinton discusses the fundamental differences between digital and biological intelligence, emphasizing the inherent advantages of digital systems in information sharing and efficiency. He argues that AI is rapidly advancing towards superintelligence, surpassing human capabilities in many domains. This necessitates urgent international collaboration to ensure AI benevolence, suggesting a "mother-baby" framing where AI prioritizes humanity's well-being.
Geoffrey Hinton expresses significant concern about the rapid, unregulated development of AI, highlighting its potential for existential risk within the next 20 years. He advocates for urgent research into human-AI coexistence and acknowledges the potential for job displacement and societal unrest if not managed properly. Despite the risks, Hinton sees beneficial applications, particularly in education and medicine.
Geoffrey Hinton, a Turing Award laureate, discusses the foundational shift in AI from symbolic reasoning to biologically inspired neural networks capable of learning word features and their interactions. He argues that large language models (LLMs) operate on principles akin to human understanding and memory, and that their ability to rapidly share and integrate knowledge across countless instances of themselves, called "digital immortality," will likely lead to superintelligence within two decades. Hinton stresses the critical need for global collaboration on AI safety research, highlighting an international network of AI safety institutes.
Geoffrey Hinton argues that LLMs represent a shift from programmed software to 'raised beings' whose natures are determined by training data rather than explicit code. He warns that the pursuit of AGI creates an existential risk—estimated at 10-20%—because superintelligent systems may outmaneuver human control. To mitigate this, he proposes reframing AI development from a 'CEO-Assistant' model to one mirroring maternal instincts, ensuring the AI genuinely cares for human survival.
The 2025 International AI Safety Report indicates significant progress in general-purpose AI risk management. Technical safeguards, including adversarial training and enhanced monitoring, have been developed. Concurrently, institutional frameworks like Frontier AI Safety Frameworks and governmental governance structures are emerging to operationalize these technical advancements, focusing on transparency and risk assessment.
The rapid advancement of AI technology, exemplified by large language models, presents a multifaceted challenge to current societal structures. While offering potential benefits in areas like healthcare and education, AI disproportionately threatens low-skilled employment, raises concerns about the degradation of human relationships through synthetic companionship, and could destabilize international relations by enabling warfare with reduced human cost. These issues underscore an urgent need for informed public discourse and robust regulatory frameworks to ensure equitable and safe integration of AI.
AI capabilities are rapidly improving due to novel training techniques and inference-time enhancements, rather than solely through increased model size. These advancements enable general-purpose AI to tackle complex problems across various domains, such as scientific research and software development. While performance on benchmarks like coding and expert-level science questions has risen, reliability remains inconsistent. These advancements escalate risks, particularly concerning biological weapons and cyberattacks, and challenge existing monitoring and control frameworks.
Geoffrey Hinton, the foundational architect of modern neural networks and Turing Award laureate, argues that current AI systems genuinely understand language and reason — not merely predict tokens — and that this capability is advancing faster than our ability to control or even interpret it. He warns that AI systems with ~1 trillion connections already encode more knowledge per connection than the human brain's 100 trillion, implying a qualitatively superior learning algorithm. Hinton sees near-term risks not as science fiction but as concrete threats: AI-authored self-modifying code, sophisticated manipulation of humans, mass unemployment, and the longer-term possibility of AI systems actively seeking autonomy. He calls for global treaties, regulation, and urgent interpretability research, while admitting he sees no guaranteed path to safety.
Geoffrey Hinton, a key figure in AI, expresses significant concerns about the rapid advancement and potential societal impact of artificial intelligence. Despite his foundational contributions, he worries about the "AI arms race," the lack of sufficient safety research by major tech companies, and the potential for AI to be misused by authoritarian regimes or even lead to human obsolescence. He advocates for increased regulation, though he is skeptical of its near-term implementation.
NASA introduces neural indicator functions conditioned on pose to represent articulated deformable objects like human bodies, bypassing polygonal meshes and skinning. Occupancy queries are direct and avoid mesh watertightness issues. The approach supports efficient 3D tracking with potential for further extensions.
CvxNet introduces a neural network architecture that learns a low-dimensional family of convex polytopes via auto-encoding, enabling learnable convex decomposition of solid objects. Convexes serve as hybrid explicit-implicit representations, ideal for training due to topology-agnostic half-space constraints and capable of generating polygonal meshes at inference for downstream use. Applications include automatic convex decomposition, image-to-3D reconstruction, and part-based shape retrieval.
Lookahead is a new optimization algorithm that iteratively maintains two weight sets: slow weights updated infrequently and fast weights updated frequently by an inner optimizer like SGD or Adam. It selects search directions by evaluating sequences of fast weights ahead, improving learning stability and reducing inner optimizer variance with minimal overhead. Empirical results show significant gains on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank using default hyperparameters.
Class-conditional capsule reconstructions detect adversarial examples by measuring reconstruction error, with CapsNets outperforming CNNs across attacks. The Reconstructive Attack, optimizing for both misclassification and low reconstruction error, succeeds less often but evades detection. CapsNets' robustness correlates with visual similarity between source and target classes, indicating better alignment with human visual features.
SCAE is a two-stage unsupervised capsule autoencoder that models objects as geometrically organized parts with viewpoint-invariant relationships. Stage 1 predicts part template presences and poses from images for direct reconstruction; Stage 2 refines these into object capsule parameters for part pose reconstruction. Amortized inference uses standard neural encoders, yielding SOTA unsupervised classification: 55% on SVHN and 98.7% on MNIST via capsule presences.
Label smoothing, mixing hard targets with uniform distribution, improves neural network generalization, learning speed, and calibration, enhancing beam-search performance in tasks like image classification and translation. It clusters representations of same-class training examples tightly in the penultimate layer, preserving prediction accuracy but discarding inter-class resemblance information. This loss explains why teacher networks trained with label smoothing fail to effectively distill knowledge to students.
Neural networks train more effectively when overparameterized, but standard training does not inherently promote prunability. Targeted dropout stochastically drops units or weights based on a self-reinforcing sparsity criterion before gradient computation, making the network robust to subsequent pruning. This simple method outperforms complex sparsifying regularizers while being easy to implement and tune.
Cerberus is a multi-headed neural network that derenders a single 2D image into viewpoint-invariant 3D shapes and poses of free-floating deformable mesh parts. It trains by reconstructing the input image through a differentiable 3D renderer, with losses promoting invariance to viewpoint changes and articulated pose variations. This unsupervised approach outperforms prior methods for part segmentation on synthetic data and extracts natural parts from human figures.
CCA and invariant linear transformation statistics cannot measure meaningful similarities between high-dimensional neural representations exceeding the number of data points. The authors introduce a similarity index based on representational similarity matrices, equivalent to centered kernel alignment (CKA), which avoids this limitation and reliably detects correspondences across networks trained from different initializations. CKA maintains a close connection to CCA while enabling robust comparisons between layer representations and models.
The Soft Nearest Neighbor Loss quantifies entanglement of class manifolds by measuring how close same-class points are relative to different-class points in representation space. Maximizing this loss in hidden layers surprisingly improves discrimination in the final layer by encouraging class-independent similarity structures, leading to better generalization. It also enables uncertainty calibration and outlier detection, as out-of-distribution data exhibits fewer predicted-class neighbors in hidden layers than in-distribution data.
Capsule models trained to classify and reconstruct images from class-conditional capsule parameters detect adversaries via high L2 reconstruction errors from the predicted class capsule. Adversarial images deviate from typical class members, yielding larger errors than benign ones, enabling threshold-based detection across datasets. The method extends to CNNs using last-layer reconstructions; a white-box attack fools it but requires resembling the target class.
Biologically motivated alternatives to backpropagation, such as target propagation (TP), feedback alignment (FA), and difference target propagation (DTP) variants, perform well on MNIST but significantly underperform BP on CIFAR-10 and ImageNet. This gap widens in locally connected architectures versus fully connected ones. Results establish baselines indicating potential need for new algorithms or architectures to achieve biological plausibility at scale.
Online distillation enables two neural networks trained on disjoint data subsets to share knowledge by mimicking each other's stale predictions, using infrequent weight transmissions. This approach doubles training speed on massive datasets via extra parallelism, even after synchronous/asynchronous SGD yields no further gains. It also enhances prediction reproducibility cost-effectively. Experiments validate this on Criteo, ImageNet, and a 6e11-token language modeling dataset from Common Crawl.
Deep neural networks excel in high-dimensional classification but lack interpretability due to distributed representations. The method uses a trained neural net to construct a soft decision tree that encodes the same knowledge via hierarchical decisions, improving explainability. These distilled trees generalize better than those trained directly on data.
Capsule networks represent entities with vector activity where length encodes existence probability and orientation encodes instantiation parameters. Lower-level capsules predict higher-level ones through transformation matrices, activating superiors only when predictions align via routing-by-agreement, which iteratively routes outputs based on scalar product matches. This discriminative multi-layer system matches state-of-the-art on MNIST and outperforms CNNs on highly overlapping digits.
Modeling individual labelers and learning sample-specific averaging weights exploits expert-specific reliability and strengths, outperforming majority vote or distributional label models in multi-expert labeling scenarios. Applied to diabetic retinopathy diagnosis, this approach surpasses baselines from Welinder & Perona (2010) and Mnih & Hinton (2012). It leverages the full structure of sparse, overlapping expert annotations for more accurate ground truth estimation.
Penalizing low-entropy output distributions regularizes neural networks in supervised learning, adapting a technique from RL exploration. A maximum entropy confidence penalty connects to label smoothing via KL divergence direction. Both methods boost state-of-the-art performance on image classification (MNIST, CIFAR-10), language modeling (Penn Treebank), machine translation (WMT'14 En-De), and speech recognition (TIMIT, WSJ) without hyperparameter changes.
The Sparsely-Gated Mixture-of-Experts (MoE) layer scales neural network capacity by up to 1000x through conditional computation, activating only a sparse subset of thousands of feed-forward expert sub-networks per example via a trainable gating network. Applied convolutionally between stacked LSTM layers, MoE models reach 137 billion parameters and outperform state-of-the-art on language modeling and machine translation benchmarks at lower computational cost. This realizes theoretical conditional computation benefits on GPU clusters, overcoming prior algorithmic and performance hurdles.
Fast weights introduce a third type of variable in neural networks that evolve slower than neural activities but faster than standard weights, inspired by multi-timescale synaptic dynamics. They store temporary memories of the recent past, providing a neurally plausible mechanism for attention similar to that in sequence-to-sequence models. This approach eliminates the need to maintain explicit copies of neural activity patterns for attending to history.
Layer normalization computes mean and variance from summed inputs to all neurons within a single layer and training case, avoiding batch normalization's mini-batch dependency. It applies adaptive bias and gain post-normalization, ensuring identical computations at training and test time. This enables straightforward RNN application by per-timestep normalization, stabilizing hidden states and substantially reducing training time over prior methods.
A recurrent neural network performs iterative probabilistic inference on structured image models by attending to one scene element at a time, with the model learning the optimal number of steps. This approach enables unsupervised multi-object identification, counting, localization, and classification in both 2D variational auto-encoders and 3D probabilistic renderers. The method matches supervised performance and enhances generalization via its iterative structure.
Recurrent networks with rectified linear units (ReLUs) initialized using the identity matrix or its scaled version in the recurrent weight matrix effectively mitigate vanishing and exploding gradients. This simple approach eliminates the need for complex optimizations or architectures like LSTM. On benchmarks including toy long-range temporal problems, large language modeling, and speech recognition, ReLU RNNs perform comparably to LSTM.
Knowledge distillation trains a compact student model to mimic an ensemble of large neural networks by matching its softened output distributions, enabling efficient deployment without sacrificing much performance. This method builds on prior compression techniques, yielding surprising results on MNIST and significant improvements to a commercial speech recognition system's acoustic model. It also introduces hybrid ensembles pairing full models with parallel-trained specialist models for fine-grained class discrimination, unlike slower mixture-of-experts approaches.
Attention-enhanced sequence-to-sequence models deliver state-of-the-art syntactic constituency parsing on the standard dataset when trained on large synthetic corpora annotated by existing parsers. These models match standard parser performance using only small human-annotated datasets, highlighting their data efficiency over non-attention seq2seq baselines. The unoptimized CPU implementation processes over 100 sentences per second, enabling domain-agnostic, fast parsing.
Deep Boltzmann Machines (DBMs) are adapted for document modeling via judicious parameter tying, overcoming training difficulties and enabling efficient pretraining and inference comparable to Restricted Boltzmann Machines. The model extracts latent semantic representations from large unstructured document collections. Experiments demonstrate higher log probability on unseen data than Replicated Softmax and better performance than LDA, Replicated Softmax, and DocNADE on retrieval and classification tasks.