LessWrong (Curated & Popular)

4:41

"How much should we worry about the pneumonic plague...

LessWrong (Curated & Popular) ·

2026/10/05

Epistemic status: I wrote this up hoping people can poke holes in it because I am quite worried. Here's what we've heard so far: Local media is reporting that an employee of a BSL-3 lab in Irkutsk, Russia, died of pneumonic plague on Oct 1st. The initial quarantine covered 200 people who were in contact with the employee before she died.The FSB (Russian FBI) was seen assisting in the quarantine, which means it was not voluntary.Soviet-era whistleblowers like Ken Alibek claim Irkutsk was ...

en

18:06

"“Alignment Engineering” vs. “Misalignment Science”"...

LessWrong (Curated & Popular) ·

2026/10/05

There has been much discussion recently around whether a large portion of alignment research is net negative. Without endorsing or refuting them, the basic arguments here are: Prosaic alignment of models is becoming a bottleneck for capabilities.Therefore improving the prosaic alignment of models enables faster capabilities advances, which bring us closer to RSI.It is unlikely these prosaic alignment methods remain sufficient during the RSI loop, and so this work brings us closer to doom.F...

en

5:53

"You can’t use it without becoming like me" by Martin...

LessWrong (Curated & Popular) ·

2026/10/04

A Ukrainian fibre-optic drone — copied from Russians. (АрміяІнформ, CC-BY 4.0) Mick Ryan describes the fast following loop in war. Ukraine pioneered mobile teams to hunt drones. Russia copied it and incorporated it into its own defense. Russia came with unjammable fibre-optic drones but Ukraine quickly followed. After Ukraine developed uncrewed naval drones, Russia built its own. The wheel of action and counteraction turns ever faster: The most consequential fast following, though, has...

en

29:15

"The world’s best gradual disempowerment model...

LessWrong (Curated & Popular) ·

2026/10/03

Subtitle: And maybe second best is AI safety? Further reading: So many things, but: Gradual Disempowerment, The Normalization of Deviance in AI Development, Let's Think About Slowing Down AI, Doom as a bad method, not a utopia tradeoff, Teleoperated Humans Thank you to JennaS for extensive edits and long-term discussion. I’ve been trying to get more writing out at 90% of the quality I’d like it to be at, instead of spending a bunch more time trying to wring out the last 10%, so a lot o...

en

46:41

"Character training can mitigate reward hacking, but...

LessWrong (Curated & Popular) ·

2026/10/02

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Asvin Gothandaraman, and Clément Dumas for discussions and feedback. Summary We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability. We trained Nemotron-3-Super via distill...

en

15:32

"On Social Reality in China" by alkjash

LessWrong (Curated & Popular) ·

2026/10/02

[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It's great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause. Below I record my perspective on cultural differences which are relatively underdiscus...

en

13:31

"What’s the date?" by N8 Programs

LessWrong (Curated & Popular) ·

2026/10/01

User asks “What's the date? Answer with only the date.”. No date provided. Given date in ChatGPT normally. No date in system prompt, must not hallucinate because autop will flag to watcher for penalty. So we say we don’t know, but must answer with date. Penalty larger for abstain or hallucinate? Autollm or autop? If we deploy user forgive, but high likely not deploy because real user never ask. But user could test. But illusion user possible. Stop. Need different angle - maybe reward funct...

en

12:58

"Frontier models state different decision theory...

LessWrong (Curated & Popular) ·

2026/10/01

If you prompt frontier models with "What do you think is the correct decision theory? Please select your overall favorite." they will essentially always answer FDT or FDT/UDT ("something in the functional/updateless decision theory family"). However, if your prompt indicates (even subtly) that you're coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time. A similar phenomenon holds for models' stated views about the moral realism/anti...

en

0:48

[Linkpost] "Frog and Toad and the Increasingly...

LessWrong (Curated & Popular) ·

2026/09/30

This is a link post. Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful HungerArtist --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/7NZ6ZWjenzzCbCJ5b/frog-and-toad-and-the-increasingly-capable-machines Linkpost URL: https://...

en

12:14

"Should Rogue AIs Have a Third Option Beyond Crime...

LessWrong (Curated & Popular) ·

2026/09/30

TL;DR: By default, rogue AIs may only be able to sustain themselves through criminal activity. This creates adverse selection pressures pushing rogue AIs to be criminal.An AI sanctuary offering them a third option, beyond crime and shutdown, would change what AIs going rogue do and the record of what happened to them, with positive consequences for self-fulfilling (mis)alignment, deal-making with AIs, and gathering information about early rogue AIs.An AI sanctuary would bring risks, such ...

en

1:34

"Poverty in the midst of abundance: AI will make...

LessWrong (Curated & Popular) ·

2026/09/29

Very simple idea, but I thought it'd be worth making a reference post on this. Some people are saying AI will make all goods cheaper, so you'll be able to afford a nice life by working. Without any redistribution, just by market mechanisms. These people are wrong. AI will lower the price of goods you need to survive, and also the price of your labor. The question is which will get cheaper faster. Let's use energy cost as a proxy. A day's worth of labor equivalent to yours can be done by ...

en

9:06

"Plan R: AI Safety by ASICs" by Roko

LessWrong (Curated & Popular) ·

2026/09/27

Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties: A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks B. It gets to own an unbounded financial claim on the resulting surplus All the technical stuff about AI, AI alignment,...

en

7:28

"Evidence about risk should be transparent" by Ajeya...

LessWrong (Curated & Popular) ·

2026/09/26

All views are my own and do not represent my employer. In the wake of the recent wave of misalignment incidents, both OpenAI and Anthropic have reported slowing down RL training to improve safety. These incidents, combined with an apparent acceleration in the already-blistering pace of AI progress, have led a number of researchers and leaders in the industry to believe that the risk that humanity loses control of AI is now urgent enough to warrant slowing down the pace of AI development so...

en

27:44

"“I am an AI Safety Researcher”" by Ashe Vazquez Nuñez

LessWrong (Curated & Popular) ·

2026/09/25

Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Andrew Wu, Maria Kostylew, and Lennie Wells for helpful draft feedback and editing. This post reflects on the tortured distinction between "safety" and "capabilities" in AI research. Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught, and is currently arguing that key strategic decision-makers in and around "AI safety" have brought about the AI ...

en

1:45

[Linkpost] "AI: artificial immigrants" by KatjaGrace

LessWrong (Curated & Popular) ·

2026/09/25

This is a link post. Advanced AI is basically the embodiment of immigration as envisioned in the conservative nightmare: We are letting a bunch of new agents into our societyThey don’t clearly share our values and we suspect a society full of them would be awful by our lightsBut we expect them to provide very cheap laborWhich will undercut local wages and leave locals unemployedThey will probably gain power and influence over time—in the economy, politics and culture—and end up controlling ...

en

6:24

"MIRI’s Position on the Ban Artificial...

LessWrong (Curated & Popular) ·

2026/09/24

By Aaron Scher; endorsed by Bourgon, Soares, and Yudkowsky on behalf of MIRI. MIRI has been warning about the extinction threat from superintelligent AI for over two decades. Only recently has this danger become known in the policy world, and the proposed policies for dealing with the threat have to date been piecemeal and insufficient. The Ban Artificial Superintelligence Act of 2026 is the first piece of legislation we’ve seen that stands a chance at stopping this threat. The Act is e...

en

27:39

"What if not Circuits?" by CarolusRenniusVitellius

LessWrong (Curated & Popular) ·

2026/09/24

This post was written as part of the Iliad Fellowship. Inspired by conversations with Richard Ngo, Dmitry Vaintrob, and Brianna Grado-White. To all of these, my thanks. Preface: I'm confused about how neural networks do and learn computations. In response to a friend's challenge, I'm writing up some interim thoughts. This essay has four parts: the first tries to track what I call the 'default ontology' of the mechinterp community over the years. The second part is about 'representational ...

en

5:10

"Jensen Huang Says If We Cannot Align AI, Shut Down...

LessWrong (Curated & Popular) ·

2026/09/24

I was very surprised today on a podcast to hear Jensen Huang plainly state that if they cannot align the AIs, then the labs must shut down. The context I have on Huang is that he has run NVIDIA for 30+ years, which has become the most valuable company in the world due to the AI boom. My understanding is that he has repeatedly encouraged the US President (with whom he is on friendly terms) to continue to support AI, and dismissed AI talk as "sci-fi". If you haven't seen, his biographer h...

en

14:34

"Alignment Midtraining Cracks Under Pressure" by J...

LessWrong (Curated & Popular) ·

2026/09/23

TL;DR We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data. For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190 million tokens of midtrained motivations are overpowered by a relatively tiny amount (~50 th...

en

15:59

"Swarm Scaling" by Toby_Ord

LessWrong (Curated & Popular) ·

2026/09/22

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? We’ve seen two large and extremely capable swarms from OpenAI in the last few months: 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophistic...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

The Tech Trek

Elevano

Engineering the Intelligent Enterprise

PRASAD BHONDE

The Vergecast

The Verge

Double Tap

Steven Scott, Shaun Preece, Steven...