x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

Interconnects

Audio essays about the latest developments in AI and interviews with leading scientists in the field. Breaking the hype, understanding what"s under the hood, and telling stories. www.interconnects.ai

technology

All Episodes

18:52

Reverse engineering OpenAI's o1

Interconnects ·

2024/09/17

What productionizing test-time compute shows us about the future of AI. Exploration has landed in language model training.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/reverse-engineering-openai-o100:00 Reverse engineering OpenAI's o101:52 From Q-star to Strawberry to o105:13 Training o1 with reinforcement learning09:24 What is o1 doing when given a prompt?11:49 Questions to

en

11:32

Futures of the data foundry business model

Interconnects ·

2024/09/11

Scale AI's future versus further scaling of language model performance. How Nvidia may take all the margins from the data market, too.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/ai-data-foundry00:00 Futures of the data foundry business model02:57 What it is like to work with data vendors06:06 Data foundries: Risks08:18 Data foundries: Growth vectors09:50 Realistic

en

5:39

A post-training approach to AI regulation with Model...

Interconnects ·

2024/09/10

And why the concept of mandating "model spec's" could be a good start.(Oops, forgot to upload this yesterday!)This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/a-post-training-approach-to-ai-regulation0:00 A post-training approach to AI regulation with Model Specs1:45 Expanded roles of Model Specifications3:40 Near future of Model Specifications

en

10:40

OpenAI's Strawberry, LM self-talk, inference scaling...

Interconnects ·

2024/09/05

Whether or not scaling works, we should spend more on inference.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/openai-strawberry-and-inference-scaling-laws00:00 OpenAI's Strawberry, LM self-talk, inference scaling laws, and spending more on inference01:51 OpenAI's Strawberry04:16 Self-talk in language models07:45 Inference scaling lawsFig 1:

en

10:31

OLMoE and the hidden simplicity in training better...

Interconnects ·

2024/09/04

Ai2 released OLMoE, which is probably our "best" model yet relative to its peers, but not much has changed in the process.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/olmoe-and-building-better-llms00:00 OLMoE and the hidden simplicity in training better foundation models02:04 Frontier model team compute allocations04:19 De-risking training complexity06:40 On organizational

en

8:01

On the current definitions of open-source AI and the...

Interconnects ·

2024/08/28

The Open Source Initiative is working towards a definition.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/defining-open-source-ai0:00 On the current definitions of open-source AI and the state of the data commons3:17 Reasons to not mandate fully released data4:24 Sufficient but not exhaustive data docs5:22 Frustration with the data commons7:04 We need more examples to define the

en

8:32

Nous Hermes 3 and exploiting underspecified evaluations

Interconnects ·

2024/08/16

The latest model from one of the most popular fine-tuning labs makes us question how a model should be identified as a "frontier model."This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/nous-hermes-30:00 Nous Hermes 3 and exploiting underspecified evaluations5:29 Parsing training lessons from Hermes 3Fig 1:

en

62:22

Interviewing Ross Taylor on LLM reasoning, Llama...

Interconnects ·

2024/08/08

I had the pleasure of Talking with Ross Taylor, who has a great spectrum of unique experiences in the language modeling space — evaluation experience, Galactica lead author, Llama post training, etc. This is a really great conversation on the frontier of language model (LM) reasoning, LM deployments and demos, LM’s for science, RLHF, and other topics. I’ve been trying to get Ross to come on for a bit. He’s one of those people in the LM space that doesn’t speak too much, but when you do, you

en

62:22

Interviewing Ross Taylor on LLM reasoning, Llama...

Interconnects ·

2024/08/08

I had the pleasure of Talking with Ross Taylor (https://x.com/rosstaylor90), who has a great spectrum of unique experiences in the language modeling space — evaluation experience, Galactica lead author, Llama post training, etc. This is a really great conversation on the frontier of language model (LM) reasoning, LM deployments and demos, LM’s for science, RLHF, and other topics. I’ve been trying to get Ross to come on for a bit. He’s one of those people in the LM space that doesn’t speak

en

10:24

A recipe for frontier model post-training

Interconnects ·

2024/08/07

Apple, Meta, and Nvidia all agree -- synthetic data, iterative training, human preference labels, and lots of filtering.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/frontier-model-post-training00:00 Llama 3.1 post-training and the new normal for RLHF01:18 A new standard pipeline01:45 Human preference data02:59 Scaling RLHF05:03 Synthetic data06:10 The new normal06:51 Data

en

63:42

Interviewing Sebastian Raschka on the state of open...

Interconnects ·

2024/08/01

This week, I had the pleasure of chatting with Sebastian Raschka. Sebastian is doing a ton of work on the open language model ecosystem and AI research broadly. He’s been writing the great Ahead of AI newsletter (that has the biggest audience overlap with Interconnects, at 26%, so a lot of you know him) and multiple educational books, all on top of being a full time machine learning engineer at Lightning.ai, where he maintains LitGPT, which he described as being like Karpathy’s NanoGPT, with

en

63:42

Interviewing Sebastian Raschka on the state of open...

Interconnects ·

2024/08/01

This week, I had the pleasure of chatting with Sebastian Raschka. Sebastian is doing a ton of work on the open language model ecosystem and AI research broadly. He’s been writing the great Ahead of AI newsletter (that has the biggest audience overlap with Interconnects, at 26%, so a lot of you know him) and multiple educational books, all on top of being a full time machine learning engineer at Lightning.ai, where he maintains LitGPT, which he described as being like Karpathy’s NanoGPT,

en

7:55

GPT-4o-mini changed ChatBotArena

Interconnects ·

2024/07/31

And how to understand Llama three point one's results.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/gpt-4o-mini-changed-chatbotarena0:00 GPT-4o-mini changed ChatBotArena3:23 Llama 3 in the arena5:13 Partial solutions and next stepsFig 1: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/new-chatbotarena/img_013.pngFig 2:

en

15:22

Llama 3.1 405b, Meta's AI strategy, and the new open...

Interconnects ·

2024/07/23

Defining the future of the AI economy and regulation. Is Meta's AI play equivalent to the Unix stack for open-source software?This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/llama-405b-open-frontier-model00:00 Llama 3.1 405b, Meta's AI strategy, and the new open frontier model ecosystem01:37 Meta's open frontier model03:51 Zuckerberg's vision for open-source AI (vs. reality)08:35

en

14:20

SB 1047, AI regulation, and unlikely allies for open...

Interconnects ·

2024/07/17

SB 1047, AI regulation, and unlikely allies for open modelsThe rallying of the open-source community against CA SB 1047 can represent a turning point for AI regulation.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/sb-1047-and-open-weights00:00 Introduction01:53 SB 1047 and targeting regulation07:57 Unlikely allies of "open"12:05 What would I regulate today?

en

6:40

Switched to Claude 3.5

Interconnects ·

2024/07/03

I Switched to Claude 3.5Speculations on the role of RLHF and why I love the model for people who pay attention.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/switched-to-claude-from-chatgpt00:00 I Switched to Claude 3.503:57 Product priorities05:15 RLHF's peak?Fig 1: https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/claude/img_016.pngFig 2:

en

56:30

Interviewing Dean Ball on AI policy: CA SB 1047,...

Interconnects ·

2024/06/27

I’m really excited to resume the Interconnects Interviews with Dean W. Ball from the Hyperdimensional Substack (you should subscribe). We cover the whole stack of recent happenings in AI policy, focusing of course on California’s bill SB 1047. We cover many, many more great topics here including:* What will happen in the case of a minor AI disaster,* If Meta will release the 405B model, and why,* The status of Chinese open-source AI,* Training on model outputs,* Anthropic’s recent strategy,* What

en

56:31

Interviewing Dean Ball on AI policy

Interconnects ·

2024/06/27

I’m really excited to resume the Interconnects Interviews with Dean W. Ball from the Hyperdimensional Substack. We cover the whole stack of recent happenings in AI policy, focusing of course on California’s bill SB 1047. We cover many, many more great topics here including:What will happen in the case of a minor AI disaster,If Meta will release the 405B model, and why,The status of Chinese open-source AI,Training on model outputs,Anthropic’s recent strategy,What scaling laws actually

en

11:52

RLHF Roundup: Trying to get good at PPO, charting...

Interconnects ·

2024/06/26

Things to be aware of if you work on language model fine-tuning.This is AI generated audio with Python and 11Labs.Source code: https://github.com/natolambert/interconnects-toolsOriginal post: https://www.interconnects.ai/p/rlhf-roundup-202400:00 RLHF Roundup: Trying to get good at PPO, charting RLHF's impact, RewardBench retrospective, and a reward model competition04:32 How big is the impact of RLHF relative to pretraining?05:54 RewardBench retrospective after 100 models and 90% peak

en

11:27

Frontiers in synthetic data

Interconnects ·

2024/06/21

Synthetic data is known to be a super powerful tool for every level of the language modeling stack. It's documented as being used for expanding vanilla pretraining data and creating large swaths of fine-tuning data. Many, many more rumors surround its use, Anthropic's pretraining-scale constitutional AI, Mistral AI's first models being pretrained on OpenAI outputs, Q-star's hopes as OpenAI's remaining moat, and much more. The diversity of use cases for synthetic data makes planning around

en

140 results

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs