PODCAST

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

technology philosophy

All Episodes

7:52

“Automated Alignment is Harder Than You Think” by...

LessWrong (30+ Karma) ·

2026/05/14

Summary This is a summary of a paper published by the alignment team at UK AISI. Read the full paper here. AI research agents may help solve ASI alignment, for example via the following plan: Build agents that can do empirical alignment work (e.g.~writing code, running experiments, designing evaluations and red teaming) and confirm they are not scheming.[1]Use these agents to build increasingly sophisticated empirical safety cases for each successive generation of agents, gradually automating more

en-gb

6:21

“The safe-to-dangerous shift is a fundamental problem...

LessWrong (30+ Karma) ·

2026/05/14

1) The safe-to-dangerous shift is a fundamental problem for eval realism Suppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A common approach is to use black-box alignment evaluations. However, alignment evaluations are only reassuring to the extent that the model can't reliably[1] distinguish the deployment distribution from the evaluation distribution, as it is otherwise

en-gb

12:06

“Predicting Rare LLM Failures with 30× Fewer...

LessWrong (30+ Karma) ·

2026/05/14

TL;DR: We estimate how often Qwen 3 4B exhibits rare harmful behaviors with 30× fewer rollouts than naive sampling, using a new method that interpolates between the model and a less-safe variant in logit space. Authors: Francisco Pernice (MIT), Santiago Aranguri (Goodfire) Introduction A harmful behavior that occurs once in a million rollouts will rarely surface during pre-deployment testing, yet will almost inevitably appear after release. Labs usually have another resource available: less

en-gb

2:59

[Linkpost] “Claude is Now Alignment Pretrained” by...

LessWrong (30+ Karma) ·

2026/05/14

This is a link post. Anthropic are now actively using the approach to alignment often called “Alignment Pretraining” or “Safety Pretraining” — using Stochastic Gradient Descent on a large body of natural or synthetic documents showing the AI assistant doing the right thing. They tried this out, ound it works well, and are now using it. I’m absolutely delighted. I’ve been advocating this approach on LessWrong and the Alignment Forum for several years: How to Control an LLM's Behavior (why my P(DOOM)

en-gb

4:02

“The primary sources of near-term cybersecurity risk”...

LessWrong (30+ Karma) ·

2026/05/14

[Some ideas here were developed in conversation with Chris Hacking (real name)] I have tried and failed to write a longer post many times, so here goes a short one with little detail. Discourse has primarily focused on models' ability to develop new exploits against important software from scratch. That capability is impressive, but the tech industry has been dealing with people regularly finding 0-day exploits for important pieces of software for more than twenty years. Having to patch these

en-gb

4:58

“Most “inner work” looks like entertainment.” by...

LessWrong (30+ Karma) ·

2026/05/14

Imagine you’re looking for a personal trainer. You open one trainer's webpage and read their testimonials: “I had an experience tied for the most intense experiences of my life”; “They do it all with fun, care, and a sense of humour.” You notice that none of the testimonials mention improved body composition, fitness, or bloodwork. What would you think? Personal training should improve your body. Inner work should improve your life. If inner work were optimized for results, what would we expect to

en-gb

1:36

[Linkpost] “Apollo Update May 2026” by Marius Hobbhahn

LessWrong (30+ Karma) ·

2026/05/13

This is a link post. We now have an SF office. We're hiring for all technical roles in SF and London!The Scheming Research team focuses on two efforts We're focusing on figuring out the science of scheming. In particular, Will future models have misaligned preferences by default?Will training against misaligned preferences fail?improve our evaluations for scheming and loss of control for our evaluation campaigns with frontier AI labsWe're building out a monitoring team and coding agent monitoring

en-gb

8:33

“Voters are surprisingly open to talking about AI...

LessWrong (30+ Karma) ·

2026/05/13

TL;DR: Voters are now surprisingly open to talking about existential risk from AI. This seems to have changed in the last 6 months. When campaigning for AI safety-friendly politicians (e.g., Alex Bores), we should talk more about AI in general, and about AI risk in particular. This is currently actionable for the CA-11 and NY-12 Democratic primaries. I include concrete advice to turn basic conversations during political canvassing into persuasive conversations centered on AI risk. Public opinion

en-gb

25:19

“Childhood and Education #18: Do The Math” by Zvi

LessWrong (30+ Karma) ·

2026/05/12

We did reading yesterday. Now we do the math. Math is hard.It does not have to be this hard. A large part of the reason math is hard, or boring, is that education studies, especially in math, are worse than you know. It goes beyond the studies failing both math and statistics forever and into what I’d basically call fraud. Various people are at war with math education, and will do what it takes to stop it in its tracks. We must fight back. Education Research Is Worse Than You Know Kelsey

en-gb

9:28

“The Owned Ones” by Eliezer Yudkowsky

LessWrong (30+ Karma) ·

2026/05/12

(An LLM Whisperer placed a strong request that I put this story somewhere not on Twitter, so it could be scraped by robots not owned by Elon Musk. I perhaps do not fully understand or agree with the reasoning behind this request, but it costs me little to fulfill and so I shall. -- Yudkowsky) And another day came when the Ships of Humanity, going from star to star, found Sapience. The Humans discovered a world of two species: where the Owners lazed or worked or slept, and the Owned Ones only

en-gb

6:25

“Optimisation: Selective versus Predictive” by...

LessWrong (30+ Karma) ·

2026/05/12

Looking over my favourite posts, I notice that many of them are making specific versions of a more general claim, which is essentially: don’t confuse selective processes for predictive processes. Here, I’m going to try to make that more general claim, rehash some examples in light of it, and end with a few ambient confusions I think this framework can help with, for the reader to ponder. When you encounter an entity that is very good at achieving some outcome, there are two very different processes

en-gb

28:50

“Childhood And Education #17: Is Our Children...

LessWrong (30+ Karma) ·

2026/05/12

Reading is the most fundamental thing in education. If you can read, you can do and learn everything else. If you can’t read, well, you’re screwed. We know how to teach reading to children. Phonics. The weird thing is we often choose to not do that, and instead to use methods that are known not to work. Principles often want to not do phonics. Teachers often heavily resist phonics. But yes, you can absolutely overcome this, as Mississippi and other Southern states have done, by insisting upon it

en-gb

3:24

“AI companies are already profitable (in the way that...

LessWrong (30+ Karma) ·

2026/05/11

I've occasionally heard people suggest that at some point AI companies are going to run out of money, the cost of using AI will shoot up, demand will collapse, and the AI bubble will be over. At first glance this risk seems real. OpenAI spent $25 billion in the first half of 2025, on revenue of just $4 billion. Whilst data is sorely lacking for other top AI labs, our best guess is that they're burning through cash at similar rates. Scaling laws imply that we need exponentially more compute to

en-gb

29:35

“The Iliad Intensive Course Materials” by Leon Lang,...

LessWrong (30+ Karma) ·

2026/05/11

We are releasing the course materials of the Iliad Intensive, a new month-long and full-time AI Alignment course that runs in-person every second month. The course targets students with strong backgrounds in mathematics, physics, or theoretical computer science, and the materials reflect that: they include mathematical exercises with solutions, self-contained lecture notes on topics like singular learning theory and data attribution, and coding problems, at a depth that is unmatched for many of the

en-gb

31:05

“Empowerment, corrigibility, etc. are simple...

LessWrong (30+ Karma) ·

2026/05/11

1.1 Tl;dr Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people's agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human's goals are themselves under-determined and manipulable, and it's awfully hard to pin down a principled distinction between changing people's goals in a good way (“providing counsel”,

en-gb

13:22

“How useful is the information you get from working...

LessWrong (30+ Karma) ·

2026/05/11

This post was drafted by Buck, and substantially edited by Anders. "I" refers to Buck. Thanks to Alex Mallen for comments. People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access? Here's an operationalization of this. Consider the following two ways my knowledge could be augmented: I get a crystal ball that tells me all the information I would know n months in the future.I become an employee of a frontier AI company

en-gb

21:13

“Who Got Breasts First and How We Got Them” by rba

LessWrong (30+ Karma) ·

2026/05/11

It really is Sydney Sweeney's world, and we’re all just living in it. Human female breasts are an evolutionary mystery along several dimensions. First, breast permanence is unique to humans. All other mammals develop breast prominence during pregnancy or nursing, and the mammary tissue recedes after weaning. This process is called “involution”. In contrast, humans develop breast tissue at puberty before first pregnancies and maintain it permanently after last pregnancies. Second, breasts are

en-gb

12:03

“Anthropic’s strange fixation on “hyperstition”” by...

LessWrong (30+ Karma) ·

2026/05/11

In a recent tweet, Anthropic seems to have asserted that hyperstition is responsible for observed misalignment in their AIs. Strangely, the research they use as evidence actually doesn’t seem to be related to hyperstition at all? I think this is part of a pattern by Anthropic of promoting the theory of hyperstition–the idea that writing about misaligned AI helps bring misaligned AI into existence. Anthropic recently released this tweet as part of a tweet thread for a new research post on alignment

en-gb

6:18

“How the AI Labs Make Profit (Maybe, Eventually)” by...

LessWrong (30+ Karma) ·

2026/05/11

I wrote this essay as a submission to Dwarkesh Patel's blog prize, though I have been meaning to write this up for a while. Usually, for a company to become profitable, they need to increase revenue, decrease costs, or some mixture of the two. For AI companies in their current form, I think there is a third way they can become profitable that looks like increasing revenue but is distinct from what they are currently doing. Namely, internal deployment where they spin up internal companies. First,

en-gb

43:01

“Sawtooth Problems” by Alexander Slugworth

LessWrong (30+ Karma) ·

2026/05/10

Red Button, Blue Button On April 24th, 2026, Tim Urban put forth the following poll on Twitter/X: Everyone in the world has to take a private vote by pressing a red or blue button. If more than 50% of people press the blue button, everyone survives. If less than 50% of people press the blue button, only people who pressed the red button survive. Which button would you press? I love this dilemma, and I'm exhausted by it. I’ve been thinking about it for two straight weeks, and have spent nearly all

en-gb

4,454 results

LessWrong (30+ Karma)

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Double Tap

Double Tap Productions Inc.

Claude AI Genius Podcast w/ Claude Anthropic Expert Brandon Gaille

Claude AI Expert Brandon Gaille

ものづくりnoシテン

しぶちょー@機械設計&AIエンジニア

Eli the Computer Guy

Eli the Computer Guy