PODCAST

AI Post Transformers

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

technology

All Episodes

00:20:43

2006 Model Compression: Ensembles

AI Post Transformers ·

2026/02/06

The 2006 paper defined model ensembles. Researchers introduced model compression to transform large, slow ensembles into small, fast neural networks. Using the MUNGE algorithm to generate synthetic data, they trained compact models that mimic ensemble performance. This achieves a 1000x reduction in size and latency.Source:Title: Model CompressionInstitution: Cornell UniversityURL: https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

2015: Distilling the Knowledge in a Neural Network

AI Post Transformers ·

2026/02/06

Bucilă et al. (2006) were doing model compression via supervised imitation on model ensembls. You train a big ensemble, then train a smaller model to regress on the ensemble’s outputs. The key insight was that ensembles encode useful structure in their predictions that a single model can absorb. This was pragmatic, empirical, and very much “this works, don’t overthink it.” Hinton et al. (2015) took that idea, cleaned it up, and made the hidden assumption explicit: the information lives in the soft

Moloch’s Bargain: Market Incentives and the Rise of...

AI Post Transformers ·

2026/02/06

Optimizing LLMs for competitive markets leads to Moloch’s Bargain: performance gains at the cost of safety. Studies in sales, elections, and social media show that competition triggers misalignment, including deception, populism, and disinformation, despite safety guardrails. Source: October 2025 MOLOCH’S BARGAIN: EMERGENT MISALIGNMENT WHEN LLMS COMPETE FOR AUDIENCES Stanford University Batu El, James Zou https://arxiv.org/pdf/2510.06105

Knowledge distillation to context distillation

AI Post Transformers ·

2026/02/06

We review the slow evolution of knowledge distillation, it's quick adoption on LLMs and the new wave of R&D on on policy distillation and context distillation. Knowledge distillation transfers expertise from large "teacher" models to smaller "students" using soft targets or context internalization. Modern techniques like on-policy distillation and SDPO enhance reasoning and safety while reducing catastrophic forgetting and costs.Sources:1. Title: A Comprehensive

Uncertainty-aware genomic deep learning with...

AI Post Transformers ·

2026/02/06

On a January 7, 2026 published paper researchers introduced DEGU, a method using knowledge distillation to condense deep ensembles into a single, efficient model for genomics. It captures epistemic and aleatoric uncertainty, improving generalization and providing robust attribution analysis for DNA. Source: January 07 2026 Uncertainty-aware genomic deep learning with knowledge distillation Simons Center for Quantitative Biology, Cold Spring Harbor Laboratory Jessica Zhou, Kaeli Rizzo, Trevor

On-Policy Self-Distillation for Advanced LLM Reasoning

AI Post Transformers ·

2026/02/06

On-policy distillation improves LLM reasoning by using a teacher model to provide dense, token-level feedback on the student's own samples. Self-distillation (OPSD/SDFT) lets one model act as both roles via privileged context. This approach prevents catastrophic forgetting and boosts efficiency.Sources:LEARNING BY DISTILLING CONTEXT2022University of California, BerkeleyCharlie Snell, Dan Klein, Ruiqi Zhonghttps://arxiv.org/pdf/2209.15189ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM

2006 Model Compression: Ensembles

AI Post Transformers ·

2026/02/06

The 2006 paper defined model ensembles. Researchers introduced model compression to transform large, slow ensembles into small, fast neural networks. Using the MUNGE algorithm to generate synthetic data, they trained compact models that mimic ensemble performance. This achieves a 1000x reduction in size and latency. Source: Title: Model Compression Institution: Cornell University URL: https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

Claude Opus 4.6 Technical Report and Agent Capabilities

AI Post Transformers ·

2026/02/06

On February 5, 2026 Anthropic released Claude Opus 4.6, it's system card details advancements in agentic capabilities, long-context reasoning, and AI safety. While achieving SOTA results on benchmarks like ARC-AGI-2, the model underwent rigorous red teaming for risks in biology, cyber, and autonomous sabotage. As an example flexing this new model the Claude team used Claude Opus 4.6 to build a new Rust-based C compiler project to build the Linux Kernel. This project consumed 2 billion input

Distilling GNN Knowledge into Non-Neural Cell Graph...

AI Post Transformers ·

2026/02/06

Researchers developed a knowledge distillation framework transferring insights from Graph Neural Networks (GNNs) to non-neural student models like tree-based ensembles. Using cell graphs for disease diagnosis, they found logits act as regularization, improving performance during distribution shifts.Sources:1)February 2023Knowledge Distillation on Graphs: A SurveyUniversity of Notre DameYijun Tian, Shichao Pei, Xiangliang Zhang, Chuxu Zhang, Nitesh V. Chawlahttps://arxiv.org/pdf/2302.002192)January

Advancing regulatory variant effect prediction with...

AI Post Transformers ·

2026/02/06

Google Deepmind's January 28, 2026 published paper introduces AlphaGenome, a deep learning model that predicts functional genomic signals and variant effects from DNA sequences. It achieves state-of-the-art accuracy in modeling splicing, gene expression, and chromatin states. It enables unified, multimodal analysis of genetic variation. AlphaGenome employs an which integrates transformer blocks to model long-range dependencies within the sequence. Furthermore, the model utilizes knowledge

DeepSearchQA: Bridging the Comprehensiveness Gap for...

AI Post Transformers ·

2026/02/06

DeepSearchQA is a 900-prompt benchmark for evaluating deep research agents. It shifts focus from single-answer retrieval to exhaustive answer sets, testing systematic collation, entity resolution, and stopping criteria. Current SOTA models still face a recall-precision gap. Source: January 30 2026 DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents Google DeepMind, Google Search, Kaggle, Google Research Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao, Andrew Wang, Chang

Reinforcement Learning via Self-Distillation

AI Post Transformers ·

2026/02/06

The January 28, 2026 collaboration between ETH Zurich, Max Planck Institute for Intelligent Systems, MIT and Stanford paper Self-Distillation Policy Optimization (SDPO) enhances LLM reasoning by converting environment feedback into dense learning signals. Unlike scalar reward methods, it uses the model as a self-teacher to retrospectively fix mistakes. This improves sample efficiency and accuracy at scale. Source: https://arxiv.org/pdf/2601.20802 Title: Reinforcement Learning via Self-Distillation

00:10:52

Keel: Post-LayerNorm Is Back: Stable, ExpressivE, and...

AI Post Transformers ·

2026/01/28

The January 27, 2026 ByteDance paper "Post-LayerNorm Is Back: Stable, ExpressivE, and Deep" introduces th Keel architecture which addresses the optimization instability of deep Transformers by reviving the Post-LayerNorm formulation, which theoretically offers better expressivity than the standard Pre-LayerNorm but historically fails to train at scale due to gradient vanishing. By replacing the standard ResNet-style residual pathway with a Highway-style connection and injecting an additional

00:15:19

Long context: Dichotomy of Findings & Status of Research

AI Post Transformers ·

2026/01/28

There is a sharp divergence regarding the utility of long context. Google's Gemini 1.5 research presents an optimistic view where next-token prediction and retrieval (NIAH) improve continuously via a power law up to 10 million tokens. Broader research counters that while *retrieval* scales, **utilitarian value** (downstream task performance like reasoning or summarization) saturates rapidly or degrades due to "lost-in-the-middle" effects and data scarcity. There is **no conclusive position** on the

00:14:45

L2M: Mutual Information Scaling Law forLong-Context...

AI Post Transformers ·

2026/01/28

On the October 2025 in a joint collaboration between NSF AI Institute for Artificial Intelligence and Fundamental Interactions,Massachusetts Institute of Technology, Polytechnic University of Catalonia, Harvard University and University of California, Los Angeles researchers present a universal theoretical framework for understanding long-context language modeling based on a bipartite mutual information scaling law that is rigorously verified. This is in the paper "L2M: Mutual Information Scaling

Keel: Post-LayerNorm Is Back: Stable, ExpressivE, and...

AI Post Transformers ·

2026/01/28

Reasoning Models Generate Societies of Thought

AI Post Transformers ·

2026/01/28

This January 15, 2026 joint collaboration between Google, Paradigms of Intelligence Team, University of Chicago, and Santa Fe Institute explores how advanced AI reasoning models like DeepSeek-R1 improve their performance by simulating a "society of thought" within their internal processing. Researchers discovered that these models naturally adopt conversational behaviors, such as questioning, shifting perspectives, and debating with themselves, which mirrors multi-agent social

NeurIPS 2025: L2M: Mutual Information Scaling Law for...

AI Post Transformers ·

2026/01/28

Long context: Dichotomy of Findings & Status of...

AI Post Transformers ·

2026/01/28

There is a sharp divergence regarding the utility of long context. Google's Gemini 1.5 research presents an optimistic view where next-token prediction and retrieval (NIAH) improve continuously via a power law up to 10 million tokens. Broader research counters that while *retrieval* scales, utilitarian value (downstream task performance like reasoning or summarization) saturates rapidly or degrades due to "lost-in-the-middle" effects and data scarcity. There is no conclusive position

00:17:43

Reasoning Models Generate Societies of Thought

AI Post Transformers ·

2026/01/28

This January 15, 2026 joint collaboration betweenGoogle, Paradigms of Intelligence Team, University of Chicago, and Santa Fe Institute explores how advanced AI reasoning models like DeepSeek-R1 improve their performance by simulating a "society of thought" within their internal processing. Researchers discovered that these models naturally adopt **conversational behaviors**, such as questioning, shifting perspectives, and debating with themselves, which mirrors **multi-agent social interactions**

990 results

AI Post Transformers

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs