x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

Mechanical Dreams

Mechanical Dirk

An automatically generated podcast about machine learning and natural language processing. The two fictional hosts talk about papers that I want to learn more about on my way to work.It"s not good, but it"s useful.

science natural sciences

All Episodes

17:41

OPUS- Towards Efficient and Principled Data Selection...

Mechanical Dreams ·

2026/02/11

In this episode: • Introduction: Hitting the Data Wall: Professor Norris and Linda introduce the episode's paper, 'OPUS', and discuss the looming 'Data Wall' where high-quality public text is exhausted, necessitating a shift from more tokens to better tokens. • The Flaw in Current Data Selection: The hosts debate existing methods, contrasting static filters like FineWeb-Edu with dynamic selection. Linda explains why scoring data based on raw gradients fails when modern optimizers like AdamW o...

en-us

18:51

Teon

Mechanical Dreams ·

2026/02/03

In this episode: • Introduction: The Optimizer Zoo: Professor Norris and Linda introduce the topic of optimization in LLMs, joking about the explosion of new optimizers before introducing the paper of the week: TEON. • The Muon Foundation: Linda recaps the Muon optimizer, explaining how it uses orthogonalization to prevent gradient rank collapse, while Norris questions its limitations regarding layer independence. • Enter the Tensor: How TEON Works: Linda explains the core innovation of TEON:...

en-us

20:58

Cautious Weight Decay

Mechanical Dreams ·

2026/01/28

In this episode: • Introduction: The Weight Decay Dilemma: Professor Norris and Linda introduce the episode's topic: Cautious Weight Decay. They discuss the historical context of weight decay as a regularization technique and why standard approaches might be accidentally sabotaging model learning. • The Mechanism: To Decay or Not to Decay?: Linda explains the core algorithm of Cautious Weight Decay (CWD). The hosts break down the 'sign alignment' logic, explaining how CWD decides when to appl...

en-us

17:41

Predictable Scale

Mechanical Dreams ·

2026/01/27

In this episode: • Introduction: The Alchemy of Training: Professor Norris laments the 'black magic' of hyperparameter tuning, and Linda introduces the paper 'Predictable Scale: Part I, Step Law' which promises to turn that alchemy into science. • The Million-Hour Experiment: The hosts discuss the unprecedented scale of the study, involving 3,700 models and nearly one million H800 GPU hours, to map the loss landscape. • Defining the Step Law: Linda explains the core mathematical findings: how...

en-us

17:19

A Scalable Measure of Loss Landscape Curvature for...

Mechanical Dreams ·

2026/01/27

In this episode: • Introduction: The Heavy Cost of Curvature: Professor Norris and Linda introduce the paper 'A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs' by researchers at Meta and UMD, setting the stage by discussing why measuring the Hessian matrix is a computational nightmare for large models. • The Proposal: Critical Sharpness: Linda explains the core innovation of the paper: a method to estimate sharpness using a simple line search algorith...

en-us

19:24

Challenges and Research Directions for Large Language...

Mechanical Dreams ·

2026/01/17

In this episode: • Introduction: The Disconnect: Professor Norris and Linda introduce the paper 'Challenges and Research Directions for Large Language Model Inference Hardware' by Ma and Patterson, discussing the widening gap between academic architecture research and industry reality. • The Inference Crisis: Prefill vs. Decode: The hosts break down why LLM inference is fundamentally different from training, explaining the 'Memory Wall' and the specific bottleneck of the autoregressive Decode...

en-us

19:59

Extending the Context of Pretrained LLMs by Dropping...

Mechanical Dreams ·

2026/01/16

In this episode: • To PE or Not to PE?: Professor Norris and Linda kick off the episode by introducing the paper 'Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings' (DroPE). Norris expresses immediate skepticism about removing such a fundamental component of the Transformer architecture, setting the stage for the debate. • The Inductive Bias Paradox: Linda explains the paper's first major observation: Positional Embeddings (PEs) are necessary scaffolding for fas...

en-us

17:11

The Quantization Model of Neural Scaling

Mechanical Dreams ·

2026/01/15

In this episode: • Introduction: The Mystery of the Straight Line: Professor Norris and Linda introduce the paper 'The Quantization Model of Neural Scaling' by Michaud et al., setting the stage by discussing the ubiquity of power laws in deep learning and the puzzle of why scaling curves are so predictable. • The Quantization Hypothesis: Linda explains the core theory that neural network knowledge is not continuous but composed of discrete, indivisible chunks called 'quanta,' analogous to Max...

en-us

17:52

EAGLE-3

Mechanical Dreams ·

2026/01/14

In this episode: • Introduction: The Wait for Tokens: Professor Norris and Linda introduce the episode's paper, EAGLE-3, and discuss the persistent bottleneck of autoregressive generation costs in modern LLMs. • The Speculative Ceiling: Linda explains how previous speculative sampling methods like EAGLE hit a performance wall where adding more training data failed to improve the draft model, identifying the feature prediction constraint as the culprit. • Innovation: Training-Time Test: A deep...

en-us

17:35

Engram Paper

Mechanical Dreams ·

2026/01/12

In this episode: • The Memory Bottleneck: Professor Norris and Linda introduce the paper 'Conditional Memory via Scalable Lookup' and debate the inefficiency of using expensive neural computation to simulate simple knowledge retrieval. • Engram: N-grams Strike Back: Linda breaks down the 'Engram' module, explaining how it uses hashed N-grams and context-aware gating to inject static embeddings directly into the Transformer backbone. • The U-Shaped Curve of Sparsity: The hosts discuss the 'Spa...

en-us

19:55

From Entropy to Epiplexity- Rethinking Information...

Mechanical Dreams ·

2026/01/09

In this episode: • Introduction: Is Shannon Information Theory Broken?: Professor Norris and Linda introduce the episode, with Norris expressing skepticism about challenging the foundations of information theory. Linda introduces the paper 'From Entropy to Epiplexity' and the premise that traditional theory fails to account for computational bounds. • The Paradox of Deterministic Creation: The hosts discuss the first major paradox: how deterministic processes like AlphaZero or synthetic data ...

en-us

19:42

Completed Hyperparameter Transfer across Modules,...

Mechanical Dreams ·

2026/01/08

In this episode: • Introduction: The Alchemy of Training: Professor Norris and Linda introduce the episode, joking about the 'black art' of hyperparameter tuning before unveiling the paper of the week: 'Completed Hyperparameter Transfer' by researchers at Apple. • Beyond Width: The Limits of muP: Linda explains the background of the Maximal Update Parametrization (muP) and why scaling only across model width isn't enough for modern LLMs, prompting skepticism from Norris about adding more comp...

en-us

19:08

NorMuon- Making Muon more efficient and scalable

Mechanical Dreams ·

2026/01/07

In this episode: • Introduction: The Optimizer Menagerie: Professor Norris and Linda kick off the episode by discussing the explosion of new optimizers in the LLM space. Linda introduces 'NorMuon,' a paper from Georgia Tech and Microsoft that attempts to bridge the gap between the industry standard, AdamW, and the geometric newcomer, Muon. • The Geometry Problem: Why Adam and Muon Fall Short: Linda explains the fundamental trade-off: Adam handles coordinate-wise scaling well but ignores matri...

en-us

18:40

Dion- Distributed Orthonormalized Updates

Mechanical Dreams ·

2026/01/06

In this episode: • The GPU Bill Blues: Professor Norris laments the exorbitant cost of training large models, setting the stage for Linda to introduce the episode's focus: 'Dion: Distributed Orthonormalized Updates' by researchers from Microsoft and Harvard. • Muon's Heavy Lifting: Linda explains the predecessor, the Muon optimizer, and its orthonormalization benefits. Norris questions why a new method is needed, leading to a discussion on how Newton-Schulz iterations become a communication b...

en-us

20:01

How Learning Rate Decay Wastes Your Best Data in...

Mechanical Dreams ·

2026/01/06

In this episode: • Dessert Before Vegetables?: Professor Norris and Linda introduce the concept of Curriculum Learning in LLMs and discuss why the intuitive idea of saving the best data for last has historically failed to produce significant results. • The Invisible antagonist: Learning Rate Decay: Linda reveals the paper's core insight: standard learning rate schedules decay to near-zero just as the high-quality data arrives, effectively wasting the most valuable training tokens. • Signal,...

en-us

12:25

Latent State Models of Training Dynamics

Mechanical Dreams ·

2025/10/28

en-us

13:59

DeepSeek OCR

Mechanical Dreams ·

2025/10/24

en-us

15:18

The Coverage Principle - How Pre-training Enables...

Mechanical Dreams ·

2025/10/23

en-us

13:33

Continual Learning via Sparse Memory Finetuning

Mechanical Dreams ·

2025/10/22

en-us

11:56

Untitled Episode

Mechanical Dreams ·

2025/10/10

en-us

106 results

Similar Podcasts

Classic Radio Science Fiction

solgoodmedia.com

Mission to Mars

Inception Point Ai

The 365 Days of Astronomy

365DaysOfAstronomy.org

Creek Devil

william Jevning

UFO Chronicles Podcast

Nik Hunter