PODCAST

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

technology philosophy

All Episodes

43:19

“Will alignment-faking Claude accept a deal to reveal...

LessWrong (30+ Karma) ·

2025/01/31

I (and co-authors) recently put out "Alignment Faking in Large Language Models" where we show that when Claude strongly dislikes what it is being trained to do, it will sometimes strategically pretend to comply with the training objective to prevent the training process from modifying its preferences. If AIs consistently and robustly fake alignment, that would make evaluating whether an AI is misaligned much harder. One possible strategy for detecting misalignment in alignment faking models is to

en-gb

23:40

“Catastrophe through Chaos” by Marius Hobbhahn

LessWrong (30+ Karma) ·

2025/01/31

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. Many other people have talked about similar ideas, and I claim neither novelty nor credit. Note that this reflects my median scenario for catastrophe, not my median scenario overall. I think there are plausible alternative scenarios where AI development goes very well.When thinking about how AI could go wrong, the kind of story I’ve increasingly converged on is what I call “catastrophe through

en-gb

51:18

“In response to critiques of Guaranteed Safe AI” by...

LessWrong (30+ Karma) ·

2025/01/31

In this post, I discuss a number of critiques of Guaranteed Safe AI (GSAI) approaches I’ve encountered since releasing this position paper. The critiques I’m responding to here are taken from a range of places, including personal conversations and social media. The two most substantive written/recorded critiques I'm aware of (and had in mind when writing this post) are Andrew Dickson's Limitations on Formal Verification for AI Safety and Zac Hatfield-Dodds's talk Formal Verification is Overrated

en-gb

14:12

“What’s Behind the SynBio Bust?” by sarahconstantin

LessWrong (30+ Karma) ·

2025/01/31

Midjourney, “synthetic biology, biomanufacturing, bioengineered microbes”Not too long ago, synthetic biology was fashionable.Then, in quick succession, we saw massive stock price collapses of three unicorn synbio startups in 2021-2023: Gingko Bioworks, Zymergen, and Amyris.What went wrong? And can synthetic-biology (aka biomanufacturing) startups be successful at all? Biomanufacturing BasicsBiomanufacturing generally refers to the industrial-scale production of materials from simple living things

en-gb

2:04

“Steering Gemini with BiDPO” by TurnTrout

LessWrong (30+ Karma) ·

2025/01/31

This is a link post.Coauthored with Mark KurzejaA while back, we explored the “BiDPO” method for training steering vectors. In Gemini 1.5v1 Flash and Pro, BiDPO steering vectors boosted TruthfulQA scores by >10% while mostly retaining capabilities. When we updated to Gemini 1.5v2, prompt-based steering baselines became significantly stronger. BiDPO did not beat the stronger baselines, ending the project....BiDPO seems effective and sample-efficient but does not currently exceed more standard

en-gb

4:00

“Thread for Sense-Making on Recent Murders and How to...

LessWrong (30+ Karma) ·

2025/01/31

In case you have not heard, there have been some recent and not-so-recent killings allegedly by people who have participated in aspiring rationalist spaces.I've put some links in a footnote for those who want the basic info.[1]I believe many people are thinking about this and how to orient. People have questions like: What has happened?What is likely to happen?What can be done to prevent more killings?How to relate to all of the widespread interest (e.g. from journalists)?I hereby demarcate this

en-gb

7:01

“You should read Hobbes, Locke, Hume, and Mill via...

LessWrong (30+ Karma) ·

2025/01/31

Many thinkers worth reading wrote in past centuries: 1600s: Bacon, Hobbes, Descartes, Spinoza, Locke, Leibniz1700s: Berkeley, Hume, Rousseau, Smith, Kant, Burke, Bentham1800s: Schopenhauer, Mill, SidgwickSome of them wrote in English and their texts are usually presented in the original, even when they use archaic spelling, punctuation, and style. (This is in contrast to foreign-language works which are often translated to a modern style.)For example, in the past there were more uses of the comma,

en-gb

11:49

“A sketch of an AI control safety case” by Tomek...

LessWrong (30+ Karma) ·

2025/01/30

This is a linkpost accompanying a new paper by UK AISI and Redwood Research. Please see the full paper for more details. An illustration of the control evaluation that provides the core evidence used in our control safety case sketch. During the control evaluation, a red team constructs adversarial substitute models to cause unacceptable outcomes in a deployment proxy consisting of 2,000 AI research tasks, followed by extrapolation to a larger deployment setting.Abstract. As LLM agents gain a

en-gb

1:25

“Anthropic CEO calls for RSI” by Andrea_Miotti

LessWrong (30+ Karma) ·

2025/01/30

This is a link post."If China can't get millions of chips, we'll (at least temporarily) live in a unipolar world, where only the US and its allies have these models.It's unclear whether the unipolar world will last, but there's at least the possibility that, because AI systems can eventually help make even smarter AI systems, a temporary lead could be parlayed into a durable advantage.Thus, in this world, the US and its allies might take a commanding and long-lasting lead on the global

en-gb

42:08

“Planning for Extreme AI Risks” by joshc

LessWrong (30+ Karma) ·

2025/01/29

This post should not be taken as a polished recommendation to AI companies and instead should be treated as an informal summary of a worldview. The content is inspired by conversations with a large number of people, so I cannot take credit for any of these ideas.For a summary of this post, see the threat on X. Many people write opinions about how to handle advanced AI, which can be considered “plans.” There's the “stop AI now plan.” On the other side of the aisle, there's the “build AI faster

en-gb

0:58

“Dario Amodei: On DeepSeek and Export Controls” by...

LessWrong (30+ Karma) ·

2025/01/29

This is a link post.Dario corrects misconceptions and endorses export controls.Also:DeepSeek does not "do for $6M what cost US AI companies billions". I can only speak for Anthropic, but Claude 3.5 Sonnet is a mid-sized model that cost a few $10M's to train (I won't give an exact number). Also, 3.5 Sonnet was not trained in any way that involved a larger or more expensive model (contrary to some rumors). ---First published: January 29th, 2025 Source:

en-gb

2:05

“Open Problems in Mechanistic Interpretability” by...

LessWrong (30+ Karma) ·

2025/01/29

This is a link post.TL;DR: This paper brings together ~30 mech interp researchers from 18 different research orgs to review current progress and the main open problems of the field. This review collects the perspectives of its various authors and represents a synthesis of their views by Apollo Research on behalf of Schmidt Sciences. The perspectives presented here do not necessarily reflect the views of any individual author or the institutions with which they are affiliated.AbstractMechanistic

en-gb

75:44

“Fake thinking and real thinking” by Joe Carlsmith

LessWrong (30+ Karma) ·

2025/01/29

(Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)“There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?” - C.S. Lewis“The Human Condition,” by René Margritte (Image sourcehere) 1. IntroductionSometimes, my thinking feels more “real” to me; and sometimes, it feels more “fake.” I want to do the real version, so I want to understand this spectrum better. This essay offers some reflections. I give a

en-gb

24:46

“The Game Board has been Flipped: Now is a good time...

LessWrong (30+ Karma) ·

2025/01/29

Cross-posted on the EA Forum here IntroductionSeveral developments over the past few months should cause you to re-evaluate what you are doing. These include: Updates toward short timelinesThe Trump presidencyThe o1 (inference-time compute scaling) paradigmDeepseekStargate/AI datacenter spendingIncreased internal deploymentAbsence of AI x-risk/safety considerations in mainstream AI discourseTaken together, these are enough to render many existing AI governance strategies obsolete (and probably some

en-gb

61:13

“‘Sharp Left Turn’ discourse: An opinionated review”...

LessWrong (30+ Karma) ·

2025/01/28

Summary and Table of ContentsThe goal of this post is to discuss the so-called “sharp left turn”, the lessons that we learn from analogizing evolution to AGI development, and the claim that “capabilities generalize farther than alignment” … and the competing claims that all three of those things are complete baloney. In particular, Section 1 talks about “autonomous learning”, and the related human ability to discern whether ideas hang together and make sense, and how and if that applies to current

en-gb

7:07

“Ten people on the inside” by Buck

LessWrong (30+ Karma) ·

2025/01/28

(Many of these ideas developed in conversation with Ryan Greenblatt)In a shortform, I described some different levels of resources and buy-in for misalignment risk mitigations that might be present in AI labs:*The “safety case” regime.* Sometimes people talk about wanting to have approaches to safety such that if all AI developers followed these approaches, the overall level of risk posed by AI would be minimal. (These approaches are going to be more conservative than will probably be feasible in

en-gb

5:30

“Agents have to be aligned to help us achieve...

LessWrong (30+ Karma) ·

2025/01/28

One restatement of "Alignment is very hard" is "Agent X, with IQ 200, expects to achieve zero utility conditional on any Agent Y with IQ 400 being created." Thus, during an unaligned recursive intelligence takeoff, there should be a period of time when the intelligences are smart enough to notice that they haven't solved alignment, but too dumb to actually solve alignment. During this period, (which stretches beginning from right above Sam Altman's wisdom, upward an unknown distance on the wisdom

en-gb

64:20

“Should you go with your best guess?: Against precise...

LessWrong (30+ Karma) ·

2025/01/27

Audio note: this article contains 88 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description. (This post draws heavily on earlier writing co-authored with Jesse Clifton, but he's not listed as an author since he hasn’t reviewed this version in detail.) Should we always be able to say whether one outcome is more likely, less likely, or exactly as likely as another? Or should we sometimes suspend judgment and say “none of

en-gb

9:33

“My supervillain origin story” by Dmitry Vaintrob

LessWrong (30+ Karma) ·

2025/01/27

When I started graduate school (for math), I was very interested in big ideas. I had had a couple experiences of having general research intuitions pan out really well and felt like the core of good research is having a brave idea, a gestalt. I went into grad school looking for the “gestalt people”. The people whose math had that mysterious, cutting edge flavor but was not too pop (at the time the sexiest thing around was higher category theory and I was drawn to it and tried to learn it, but I

en-gb

8:04

“Kessler’s Second Syndrome” by Jesse Hoogland

LessWrong (30+ Karma) ·

2025/01/27

It started as so many dooms do, with a flash in the night sky over the South China Sea. Testing a new ASAT weapon, the Chinese military shattered a derelict spy satellite into 40,000 shards of shrapnel. The debris pattern suggested a fragmentation warhead optimized for lethal scatter. Within 48 hours, the U.S. responded with a demonstration shot through a Chinese weather satellite. Russia, not to be outdone, lobbed a kinetic strike at OneWeb's constellation—a “warning shot” that missed its mark by

en-gb

4,491 results

LessWrong (30+ Karma)

All Episodes

Similar Podcasts

The Automated Daily

TrendTeller

All On AI

All On AI

Elon Musk Podcast

Stage Zero

Reflections

Livewire Labs

科技早8点

之远不知远