x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

Mechanical Dreams

Mechanical Dirk

An automatically generated podcast about machine learning and natural language processing. The two fictional hosts talk about papers that I want to learn more about on my way to work. It"s not good, but it"s useful.

science natural sciences

All Episodes

0:21:09

Lost in Backpropagation- The LM Head is a Gradient...

Mechanical Dreams ·

2026/03/28

In this episode:• Chapter 1: Introduction to the Bottleneck: Linda introduces the paper and the general concept of the LM head. Professor Norris expresses initial skepticism about revisiting the softmax bottleneck.• Chapter 2: Expressivity vs. Optimization: The hosts discuss how the paper shifts the focus from the classical expressivity limitation to a fundamental optimization problem.• Chapter 3: The Math of Gradient Destruction: Linda breaks down the matrix math, explaining how backpropagating a

en-us

0:23:56

Let's (not) just put things in Context- Test-Time...

Mechanical Dreams ·

2026/03/27

In this episode:• The Context Window Illusion: Norris and Linda introduce the episode and the paper, discussing why million-token context windows don't automatically solve reasoning tasks.• The Math of Score Dilution: Linda dives into the theoretical bottleneck of static self-attention, explaining why the target-distractor margin must scale logarithmically.• Query-Only Test-Time Training: Linda reveals the paper's solution: updating only the query projection matrices at inference time to avoid

en-us

0:20:34

Learning State-Tracking from Code Using Linear RNNs

Mechanical Dreams ·

2026/03/26

In this episode:• Introduction to State-Tracking: Linda and Professor Norris introduce the paper and discuss the historical context of state-tracking in sequence models.• The Next-Token Prediction Testbed: The hosts discuss how the authors used Python REPL traces with print statements to evaluate models using next-token prediction instead of sequence-to-sequence.• DeltaNet Triumphs Over Transformers: Linda explains how DeltaNet with extended eigenvalues perfectly extrapolated the tracking task,

en-us

0:24:36

GLM-5

Mechanical Dreams ·

2026/03/24

In this episode:• Welcome & The End of Vibe Coding?: Linda introduces GLM-5 and the paradigm shift from passive vibe coding to autonomous agentic engineering.• Architecture & DeepSeek Sparse Attention: Professor Norris and Linda examine the 744B parameter model and how transitioning from dense to sparse attention drastically cuts compute costs.• Asynchronous RL and the Slime Framework: A deep dive into decoupled training engines, addressing off-policy drift with TITO and token-level clipping.•

en-us

0:21:19

Cautious Optimizers

Mechanical Dreams ·

2026/03/23

In this episode:• Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism.• The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function.• The One-Line Fix and Scaling: Linda breaks down the PyTorch implementation of the cautious mask, explaining how it zeros out

en-us

0:22:10

Backward Gradient Normalization in Deep Neural Networks

Mechanical Dreams ·

2026/03/22

In this episode:• Welcome and Introduction: Professor Norris and Linda introduce the episode and the paper of the week: 'Backward Gradient Normalization in Deep Neural Networks'.• The Ghost of Gradients Past: A discussion on the classic vanishing and exploding gradient problems, and why existing solutions like Batch Normalization and ResNets still leave room for improvement.• Unpacking Backward Gradient Normalization: Linda explains the core mechanics of the BGN layer, detailing how it leaves the

en-us

0:21:00

Attention Residuals

Mechanical Dreams ·

2026/03/21

In this episode:• The PreNorm Dilution Problem: Professor Norris and Linda introduce the episode and discuss the fundamental limitations of standard residual connections, focusing on the unbounded magnitude growth caused by PreNorm.• Attention Residuals and the Time-Depth Duality: Linda introduces the core concept of Full Attention Residuals, treating network depth like sequence length. Professor Norris raises concerns about the memory and communication overhead.• Block Attention Residuals: The

en-us

0:20:28

Decoupling the "What" and "Where" With Polar...

Mechanical Dreams ·

2026/03/20

In this episode:• Introduction: Linda and Professor Norris introduce the podcast and the focus of the episode: the PoPE paper.• The Problem with RoPE: A discussion on Rotary Position Embedding and how it entangles content and positional information.• Introducing PoPE: Linda explains the mathematical shift to polar coordinates to decouple the what and the where.• Empirical Triumphs: Reviewing the massive performance jump on the Indirect Indexing task, plus music, genomics, and language modeling.•

en-us

0:20:33

Scaling Laws for Precision

Mechanical Dreams ·

2026/03/19

In this episode:• Introduction to Precision in Scaling Laws: Linda introduces the new paper which adds precision as a third variable to the Chinchilla scaling laws. Professor Norris reflects on how precision is usually treated as an afterthought.• The Post-Training Quantization Paradox: The hosts discuss the surprising finding that overtraining models on too much data actually makes them degrade worse when applying post-training quantization.• Effective Parameters and Low-Precision Training: Linda

en-us

16:22

Midtraining Bridges Pretraining and Posttraining...

Mechanical Dreams ·

2026/03/14

In this episode: • Introduction: Do We Really Need Another Phase?: Professor Norris jokingly laments the ever-expanding terminology of LLM training, while Linda introduces the paper on 'Midtraining' as a distinct, intermediate phase between pretraining and post-training. • The Mechanism: Building a Distributional Bridge: Linda explains the core theory: midtraining isn't just 'cooling down,' but shifting the model's initialization closer to the target distribution to smooth out the optimizatio...

en-us

17:09

On the "Induction Bias" in Sequence Models

Mechanical Dreams ·

2026/03/13

In this episode: • Introduction: The Transformer's Kryptonite: Professor Norris jokes about Transformers solving everything, but Linda introduces a new paper that challenges their ability to perform basic state tracking efficiently. They set the stage by distinguishing between the well-known Out-of-Distribution failures and the paper's focus on In-Distribution data efficiency. • The Setup: Modulo Arithmetic and Supervision Regimes: Linda explains the experimental setup using modular addition ...

en-us

0:17:09

On the "Induction Bias" in Sequence Models

Mechanical Dreams ·

2026/03/13

In this episode:• Introduction: The Transformer's Kryptonite: Professor Norris jokes about Transformers solving everything, but Linda introduces a new paper that challenges their ability to perform basic state tracking efficiently. They set the stage by distinguishing between the well-known Out-of-Distribution failures and the paper's focus on In-Distribution data efficiency.• The Setup: Modulo Arithmetic and Supervision Regimes: Linda explains the experimental setup using modular addition and

en-us

16:04

NOBLE- Accelerating Transformers with Nonlinear...

Mechanical Dreams ·

2026/03/12

In this episode: • A Noble Introduction: Professor Norris makes a pun about aristocracy while Linda introduces the paper 'NOBLE' from Canva Research, setting the stage for a discussion on accelerating Transformer pretraining. • The Linear Collapse Problem: Linda explains why standard LoRA doesn't work for pretraining from scratch, and Norris helps clarify the difference between parameter-efficient fine-tuning and architectural augmentation. • Anatomy of a Nonlinear Branch: A deep dive into th...

en-us

0:16:05

NOBLE- Accelerating Transformers with Nonlinear...

Mechanical Dreams ·

2026/03/12

In this episode:• A Noble Introduction: Professor Norris makes a pun about aristocracy while Linda introduces the paper 'NOBLE' from Canva Research, setting the stage for a discussion on accelerating Transformer pretraining.• The Linear Collapse Problem: Linda explains why standard LoRA doesn't work for pretraining from scratch, and Norris helps clarify the difference between parameter-efficient fine-tuning and architectural augmentation.• Anatomy of a Nonlinear Branch: A deep dive into the NOBLE

en-us

18:27

Optimal Linear Decay Learning Rate Schedules and...

Mechanical Dreams ·

2026/03/11

In this episode: • The Death of Cosine?: Introduction to the episode and the paper. Professor Norris expresses his skepticism about changing established habits like Cosine Annealing, while Linda teases a shake-up in the status quo. • Theory vs. Reality: A discussion on the massive gap between theoretical learning rates (like 1/t) and what practitioners actually use. Linda explains why the theory has historically failed to match practice. • Linear Decay Takes the Crown: Linda explains the pape...

en-us

0:15:52

Flash Attention 4

Mechanical Dreams ·

2026/03/10

In this episode:• Welcome to the Hardware Lottery: Professor Norris and Linda introduce the episode's focus: FlashAttention-4. They set the stage by discussing the arrival of NVIDIA's Blackwell architecture and why existing optimization techniques suddenly hit a wall.• The Asymmetry Problem: Linda explains the concept of 'Asymmetric Hardware Scaling' found in the B200 GPUs, where tensor cores doubled in speed but memory bandwidth and special function units didn't. Norris questions why simply running

en-us

15:51

Flash Attention 4

Mechanical Dreams ·

2026/03/10

In this episode: • Welcome to the Hardware Lottery: Professor Norris and Linda introduce the episode's focus: FlashAttention-4. They set the stage by discussing the arrival of NVIDIA's Blackwell architecture and why existing optimization techniques suddenly hit a wall. • The Asymmetry Problem: Linda explains the concept of 'Asymmetric Hardware Scaling' found in the B200 GPUs, where tensor cores doubled in speed but memory bandwidth and special function units didn't. Norris questions why simpl...

en-us

16:54

An Empirical Study on Noisy Data and LLM Pretraining...

Mechanical Dreams ·

2026/02/28

In this episode: • The Multi-Million Dollar NaN: Linda introduces the paper 'An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence' by Zhang et al., setting the stage with the high stakes of expensive pretraining runs failing. Professor Norris expresses skepticism that simple 'bad data' is the root cause of complex divergences. • The Toxic Five Tokens: The hosts discuss the paper's methodology of injecting synthetic uniform random noise. Linda reveals the counter-intuitive find...

en-us

0:16:55

An Empirical Study on Noisy Data and LLM Pretraining...

Mechanical Dreams ·

2026/02/28

In this episode:• The Multi-Million Dollar NaN: Linda introduces the paper 'An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence' by Zhang et al., setting the stage with the high stakes of expensive pretraining runs failing. Professor Norris expresses skepticism that simple 'bad data' is the root cause of complex divergences.• The Toxic Five Tokens: The hosts discuss the paper's methodology of injecting synthetic uniform random noise. Linda reveals the counter-intuitive finding that

en-us

0:16:22

Midtraining Bridges Pretraining and Posttraining...

Mechanical Dreams ·

2026/02/27

In this episode:• Introduction: Do We Really Need Another Phase?: Professor Norris jokingly laments the ever-expanding terminology of LLM training, while Linda introduces the paper on 'Midtraining' as a distinct, intermediate phase between pretraining and post-training.• The Mechanism: Building a Distributional Bridge: Linda explains the core theory: midtraining isn't just 'cooling down,' but shifting the model's initialization closer to the target distribution to smooth out the optimization path.•

en-us

250 results

Similar Podcasts

Classic Radio Science Fiction

solgoodmedia.com

Mission to Mars

Inception Point Ai

The 365 Days of Astronomy

365DaysOfAstronomy.org

Creek Devil

william Jevning

UFO Chronicles Podcast

Nik Hunter