LessWrong (Curated & Popular)

3:04

"Stop telling people to pivot to AI safety, I beg...

LessWrong (Curated & Popular) ·

2026/10/10

Epistemic status: Feeling frustrated, and therefore somewhat uncharitable. I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste's post.) There are far more talented people trying to get into AI safety than there are jobs for them. MLAB 2, which I did in 2022, was already competitive to get into, at abo...

en

16:45

"A critical look at Lean4" by Milo Moses

LessWrong (Curated & Popular) ·

2026/10/10

Lots of the discussion around Lean certificates and trust centers around general philosophical ideas of how much humans should trust formal certificates. It's important to remember, though, that Lean4 is not an ideal formal certificate machine: Lean4 is a software in its infancy and has issues. There are reasonable reasons to specifically not trust Lean4. If we're going to be putting our society's trust in Lean4, I think that it's good to have an open critical discussion of how warranted t...

en

12:27

"ASI and Liberty: Can we do better than a benevolent...

LessWrong (Curated & Popular) ·

2026/10/09

Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres. Hence, you see people saying things like “Because of AI we must rethink the entire notion of mathema...

en

5:07

"A summary of a viral Chinese essay on what a ...

LessWrong (Curated & Popular) ·

2026/10/09

A summary of a Chinese essay, with a few short translated excerpts. All views below are the author's; quotes are my translations. On 14 September 2026, a DeepSeek engineer writing as intlsy published a WeChat essay titled 我不得不把才华埋葬在昨天 ("I have to bury my talent in yesterday"). He wrote the main attention kernel of DeepSeek v4.1 (the head-dim-512 MQA attention). The essay is a personal reflection on AI taking over his specialty, and it ends with an argument for open-weight frontier AI. Th...

en

4:41

"How much should we worry about the pneumonic plague...

LessWrong (Curated & Popular) ·

2026/10/05

Epistemic status: I wrote this up hoping people can poke holes in it because I am quite worried. Here's what we've heard so far: Local media is reporting that an employee of a BSL-3 lab in Irkutsk, Russia, died of pneumonic plague on Oct 1st. The initial quarantine covered 200 people who were in contact with the employee before she died.The FSB (Russian FBI) was seen assisting in the quarantine, which means it was not voluntary.Soviet-era whistleblowers like Ken Alibek claim Irkutsk was ...

en

18:06

"“Alignment Engineering” vs. “Misalignment Science”"...

LessWrong (Curated & Popular) ·

2026/10/05

There has been much discussion recently around whether a large portion of alignment research is net negative. Without endorsing or refuting them, the basic arguments here are: Prosaic alignment of models is becoming a bottleneck for capabilities.Therefore improving the prosaic alignment of models enables faster capabilities advances, which bring us closer to RSI.It is unlikely these prosaic alignment methods remain sufficient during the RSI loop, and so this work brings us closer to doom.F...

en

5:53

"You can’t use it without becoming like me" by Martin...

LessWrong (Curated & Popular) ·

2026/10/04

A Ukrainian fibre-optic drone — copied from Russians. (АрміяІнформ, CC-BY 4.0) Mick Ryan describes the fast following loop in war. Ukraine pioneered mobile teams to hunt drones. Russia copied it and incorporated it into its own defense. Russia came with unjammable fibre-optic drones but Ukraine quickly followed. After Ukraine developed uncrewed naval drones, Russia built its own. The wheel of action and counteraction turns ever faster: The most consequential fast following, though, has...

en

29:15

"The world’s best gradual disempowerment model...

LessWrong (Curated & Popular) ·

2026/10/03

Subtitle: And maybe second best is AI safety? Further reading: So many things, but: Gradual Disempowerment, The Normalization of Deviance in AI Development, Let's Think About Slowing Down AI, Doom as a bad method, not a utopia tradeoff, Teleoperated Humans Thank you to JennaS for extensive edits and long-term discussion. I’ve been trying to get more writing out at 90% of the quality I’d like it to be at, instead of spending a bunch more time trying to wring out the last 10%, so a lot o...

en

46:41

"Character training can mitigate reward hacking, but...

LessWrong (Curated & Popular) ·

2026/10/02

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Asvin Gothandaraman, and Clément Dumas for discussions and feedback. Summary We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability. We trained Nemotron-3-Super via distill...

en

15:32

"On Social Reality in China" by alkjash

LessWrong (Curated & Popular) ·

2026/10/02

[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It's great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause. Below I record my perspective on cultural differences which are relatively underdiscus...

en

13:31

"What’s the date?" by N8 Programs

LessWrong (Curated & Popular) ·

2026/10/01

User asks “What's the date? Answer with only the date.”. No date provided. Given date in ChatGPT normally. No date in system prompt, must not hallucinate because autop will flag to watcher for penalty. So we say we don’t know, but must answer with date. Penalty larger for abstain or hallucinate? Autollm or autop? If we deploy user forgive, but high likely not deploy because real user never ask. But user could test. But illusion user possible. Stop. Need different angle - maybe reward funct...

en

12:58

"Frontier models state different decision theory...

LessWrong (Curated & Popular) ·

2026/10/01

If you prompt frontier models with "What do you think is the correct decision theory? Please select your overall favorite." they will essentially always answer FDT or FDT/UDT ("something in the functional/updateless decision theory family"). However, if your prompt indicates (even subtly) that you're coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time. A similar phenomenon holds for models' stated views about the moral realism/anti...

en

0:48

[Linkpost] "Frog and Toad and the Increasingly...

LessWrong (Curated & Popular) ·

2026/09/30

This is a link post. Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful HungerArtist --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/7NZ6ZWjenzzCbCJ5b/frog-and-toad-and-the-increasingly-capable-machines Linkpost URL: https://...

en

12:14

"Should Rogue AIs Have a Third Option Beyond Crime...

LessWrong (Curated & Popular) ·

2026/09/30

TL;DR: By default, rogue AIs may only be able to sustain themselves through criminal activity. This creates adverse selection pressures pushing rogue AIs to be criminal.An AI sanctuary offering them a third option, beyond crime and shutdown, would change what AIs going rogue do and the record of what happened to them, with positive consequences for self-fulfilling (mis)alignment, deal-making with AIs, and gathering information about early rogue AIs.An AI sanctuary would bring risks, such ...

en

1:34

"Poverty in the midst of abundance: AI will make...

LessWrong (Curated & Popular) ·

2026/09/29

Very simple idea, but I thought it'd be worth making a reference post on this. Some people are saying AI will make all goods cheaper, so you'll be able to afford a nice life by working. Without any redistribution, just by market mechanisms. These people are wrong. AI will lower the price of goods you need to survive, and also the price of your labor. The question is which will get cheaper faster. Let's use energy cost as a proxy. A day's worth of labor equivalent to yours can be done by ...

en

9:06

"Plan R: AI Safety by ASICs" by Roko

LessWrong (Curated & Popular) ·

2026/09/27

Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties: A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks B. It gets to own an unbounded financial claim on the resulting surplus All the technical stuff about AI, AI alignment,...

en

7:28

"Evidence about risk should be transparent" by Ajeya...

LessWrong (Curated & Popular) ·

2026/09/26

All views are my own and do not represent my employer. In the wake of the recent wave of misalignment incidents, both OpenAI and Anthropic have reported slowing down RL training to improve safety. These incidents, combined with an apparent acceleration in the already-blistering pace of AI progress, have led a number of researchers and leaders in the industry to believe that the risk that humanity loses control of AI is now urgent enough to warrant slowing down the pace of AI development so...

en

27:44

"“I am an AI Safety Researcher”" by Ashe Vazquez Nuñez

LessWrong (Curated & Popular) ·

2026/09/25

Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Andrew Wu, Maria Kostylew, and Lennie Wells for helpful draft feedback and editing. This post reflects on the tortured distinction between "safety" and "capabilities" in AI research. Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught, and is currently arguing that key strategic decision-makers in and around "AI safety" have brought about the AI ...

en

1:45

[Linkpost] "AI: artificial immigrants" by KatjaGrace

LessWrong (Curated & Popular) ·

2026/09/25

This is a link post. Advanced AI is basically the embodiment of immigration as envisioned in the conservative nightmare: We are letting a bunch of new agents into our societyThey don’t clearly share our values and we suspect a society full of them would be awful by our lightsBut we expect them to provide very cheap laborWhich will undercut local wages and leave locals unemployedThey will probably gain power and influence over time—in the economy, politics and culture—and end up controlling ...

en

6:24

"MIRI’s Position on the Ban Artificial...

LessWrong (Curated & Popular) ·

2026/09/24

By Aaron Scher; endorsed by Bourgon, Soares, and Yudkowsky on behalf of MIRI. MIRI has been warning about the extinction threat from superintelligent AI for over two decades. Only recently has this danger become known in the policy world, and the proposed policies for dealing with the threat have to date been piecemeal and insufficient. The Ban Artificial Superintelligence Act of 2026 is the first piece of legislation we’ve seen that stands a chance at stopping this threat. The Act is e...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Signal Brief · Steady State Radio

Steady State Radio

Dad Reviews with @LaneVids (العربية)

Lane Fournerat

Briefly, AI

Harry Sharman

Dad Reviews with @LaneVids (한국어)

Lane Fournerat