PODCAST

AI Post Transformers

AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.

technology

All Episodes

00:13:48

Attention with a bias

AI Post Transformers ·

2026/01/17

We review why some transformer models use a bias in attention and how ALiBi helps with long context. The provided sources focus on significant advancements in computational biology, specifically the evolution of the AlphaFold series for predicting 3D biomolecular structures. AlphaFold 2 revolutionized the field by using the Evoformer and attention mechanisms to interpret evolutionary and geometric data with near-experimental accuracy. Building on this, AlphaFold 3 expanded capabilities to include

00:14:21

Squisher: Approximating the Fisher Information Matrix...

AI Post Transformers ·

2026/01/17

We focus on the July 2025 paper, "Fishers for Free? Approximating the Fisher Information Matrixby Recycling the Squared Gradient Accumulator". The paper goes into the mathematical details of approximating the FIM, and the easiest is what they call the Squisher: the Adam optimizer variance. This technique allows for tasks like model pruning and model merging to be performed "for free" without the significant computational overhead typically required to calculate the Fisher Information Matrix.We also

00:13:13

NVIDIA: TTT-E2E: Unlocking Long-Context Learning via...

AI Post Transformers ·

2026/01/17

This December 31, 2025 NVIDIA research introduces **TTT-E2E**, a novel approach to large language model memory that treats long-context processing as a **continual learning problem** rather than a structural design challenge. By utilizing **test-time training**, the model effectively **compresses context into its own weights** through next-token prediction, allowing it to adapt and learn while processing new information. Unlike traditional Transformers that suffer from **linear latency growth**, or

Squisher: Approximating the Fisher Information Matrix...

AI Post Transformers ·

2026/01/17

We focus on the July 2025 paper, "Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator". The paper goes into the mathematical details of approximating the FIM, and the easiest is what they call the Squisher: the Adam optimizer variance. This technique allows for tasks like model pruning and model merging to be performed "for free" without the significant computational overhead typically required to calculate the Fisher

NVIDIA: TTT-E2E: Unlocking Long-Context Learning via...

AI Post Transformers ·

2026/01/17

This December 31, 2025 NVIDIA research introduces TTT-E2E, a novel approach to large language model memory that treats long-context processing as a continual learning problem rather than a structural design challenge. By utilizing test-time training, the model effectively compresses context into its own weights through next-token prediction, allowing it to adapt and learn while processing new information. Unlike traditional Transformers that suffer from linear latency growth, or Recurrent Neural

Scaling laws: long context length and in context...

AI Post Transformers ·

2026/01/17

Recent advancements in Long Context Language Models (LCLMs) demonstrate that In-Context Learning (ICL) capabilities follow predictable power-law scaling relationships, where performance improves monotonically with context length up to 10 million tokens and is governed by model depth, width, and training data volume. While Gemini 1.5 exhibits near-perfect recall and continued log-loss improvement at extreme scales, theoretical frameworks reveal that ICL functions mechanistically as implicit gradient

Attention with a bias

AI Post Transformers ·

2026/01/17

00:13:06

Scaling laws: long context length and in context...

AI Post Transformers ·

2026/01/17

00:13:58

DeepSeek Engram: Scaling Large Language Models via...

AI Post Transformers ·

2026/01/14

On January 12, 2026 DeepSeek released its paper on **Engram**, a novel AI architecture that incorporates **conditional memory** to optimize how large language models handle information. By utilizing a **lookup mechanism for static patterns**, this technology separates an AI's logical reasoning from its factual knowledge base. This structural shift allows massive models to run on **cheaper hardware** by offloading memory requirements to standard host RAM without sacrificing speed. Research indicates

DeepSeek Engram: Scaling Large Language Models via...

AI Post Transformers ·

2026/01/14

On January 12, 2026 DeepSeek released its paper on Engram, a novel AI architecture that incorporates conditional memory to optimize how large language models handle information. By utilizing a lookup mechanism for static patterns, this technology separates an AI's logical reasoning from its factual knowledge base. This structural shift allows massive models to run on cheaper hardware by offloading memory requirements to standard host RAM without sacrificing speed. Research indicates that this

PageANN: Scalable Disk ANNS with Page-Aligned Graphs

AI Post Transformers ·

2025/12/07

The research paper presents PageANN, a novel framework engineered to overcome the severe latency and scalability limitations facing existing disk-based Approximate Nearest Neighbor Search (ANNS) methods used in vector databases. Current systems suffer from inefficient search paths and a crucial misalignment between logical graph node size and the physical I/O granularity of Solid-State Drives (SSDs). PageANN introduces a core innovation: a page-node graph structure that directly maps logical graph

00:13:56

PageANN: Scalable Disk ANNS with Page-Aligned Graphs

AI Post Transformers ·

2025/12/07

The research paper presents PageANN, a novel framework engineered to overcome the severe latency and scalability limitations facing existing **disk-based Approximate Nearest Neighbor Search (ANNS)** methods used in vector databases. Current systems suffer from inefficient search paths and a crucial misalignment between logical graph node size and the **physical I/O granularity of Solid-State Drives (SSDs)**. PageANN introduces a core innovation: a **page-node graph structure** that directly maps

00:14:44

NeurIPS 2025: Homogeneous Keys, Heterogeneous Values

AI Post Transformers ·

2025/12/04

This research presents a novel method for efficient long-context modeling in Large Language Models (LLMs) by tackling the quadratic complexity of attention mechanisms through KV cache compression. The core discovery is a fundamental **local KV cache asymmetry**, which reveals that adjacent attention keys exhibit high structural homogeneity, while their associated value vectors possess distinct, heterogeneous distributions. To capitalize on this finding, the authors propose **AsymKV**, a

NeurIPS 2025: Homogeneous Keys, Heterogeneous Values

AI Post Transformers ·

2025/12/04

This research presents a novel method for efficient long-context modeling in Large Language Models (LLMs) by tackling the quadratic complexity of attention mechanisms through KV cache compression. The core discovery is a fundamental local KV cache asymmetry, which reveals that adjacent attention keys exhibit high structural homogeneity, while their associated value vectors possess distinct, heterogeneous distributions. To capitalize on this finding, the authors propose AsymKV, a training-free

NeurIPS 2025: SeRL: Self-Play Reinforcement Learning...

AI Post Transformers ·

2025/11/29

The academic paper introduces Self-play Reinforcement Learning (SeRL), a framework engineered to enhance the reasoning capabilities of Large Language Models (LLMs) specifically in scenarios lacking extensive, high-quality labeled data. SeRL consists of two core, complementary modules: the self-instruction module generates new and diverse training problems from a small seed dataset, ensuring data quality and appropriate difficulty via an online filtering strategy. Simultaneously, the self-rewarding

NeurIPS 2025: Thinkless: LLM Learns When to Think

AI Post Transformers ·

2025/11/29

The research introduces Thinkless, a framework designed to solve the computational inefficiency of Large Language Models (LLMs) that overuse chain-of-thought reasoning for simple queries. This adaptive model determines whether to utilize a concise () or detailed reasoning () mode based on the input complexity and its own capabilities. Central to this approach is the Decoupled Group Relative Policy Optimization (DeGRPO) algorithm, which employs reinforcement learning to jointly optimize both the

NeurIPS 2025: Self-Adapting Language Models

AI Post Transformers ·

2025/11/29

The academic paper presents the Self-Adapting LLM (SEAL) framework, designed to allow large language models to overcome their static nature by transforming and generating their own fine-tuning data. This mechanism involves the model producing a "self-edit," which consists of natural-language instructions that specify synthetic data, tool invocations, or optimization hyperparameters for adaptation. Training is managed by an outer reinforcement learning (RL) loop that rewards the

NeurIPS 2025: KGGen: Extracting Knowledge Graphs from...

AI Post Transformers ·

2025/11/29

The academic paper introduces KGGen, a novel text-to-knowledge-graph generator designed to overcome the scarcity and poor quality of automatically extracted knowledge graphs (KGs). KGGen utilizes Language Models for initial triple extraction but innovates by employing an iterative clustering and de-duplication process that resolves duplicate entities and relations to reduce sparsity in the final graph representation. To properly assess KG extraction performance, the authors release a new two-part

NeurIPS 2025: DYNAACT: Large Language Model Reasoning...

AI Post Transformers ·

2025/11/29

The provided text outlines DYNAACT, a new framework intended to enhance sequential reasoning in Large Language Models (LLMs) by dynamically managing the available actions during complex problem-solving. This approach targets the inefficiency of current methods that either rely on manually defined and restrictive action spaces or utilize unstructured spaces that prove computationally prohibitive for exhaustive searches. DYNAACT addresses this by first estimating a broad action space from a corpus and

NeurIPS 2025: Reinforcement Learning for Reasoning in...

AI Post Transformers ·

2025/11/29

This research examines the data efficiency of Reinforcement Learning with Verifiable Reward (RLVR) when applied to large language models for mathematical reasoning tasks. The paper's most significant finding is the success of 1-shot RLVR, showing that comparable performance to using a large training dataset can be achieved using just a single, carefully selected example. This result suggests that RLVR is effective primarily because it activates the strong latent reasoning capabilities

990 results

AI Post Transformers

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs