LessWrong (Curated & Popular)

18:24

"We’ve saved the world before: what the ozone hole...

LessWrong (Curated & Popular) ·

2026/09/22

It might destroy the world, despite passing every known safety test. If we wait for a “warning shot” before we act, it might be too late. And action requires global coordination, because if anyone makes it, everyone dies. Sound familiar? It should, because it already happened half a century ago, with chlorofluorocarbons (CFCs). Despite seemingly impossible odds, we got our act together and completely solved the problem through unprecedentedly successful international coordination. The Mont...

en

3:22

"The Anatomy of a Chinese AI Researcher" by CMLKevin

LessWrong (Curated & Popular) ·

2026/09/21

The Chinese AI researcher has read the Three Body Problem series of sci-fi novels since high school, and understand the concept of existential risk vaguely. He is fascinated by Ye Wenjie, the researcher that turned against humanity in that book, and decides that in the future if AI progress leads to a superior intelligence, he might be tempted to become Ye if there's no good alternative. He performs the duties of capabilities research in a Chinese frontier lab, seeking to one day achi...

en

3:40

"Common mistakes in AI safety group organizing" by...

LessWrong (Curated & Popular) ·

2026/09/21

Back in the day, I was a very active AI safety group organizer. I commonly notice people making the same mistakes across many clubs. I have written down a list of some of these mistakes hoping people will avoid them in the future: Reading groups often require that people read things before meetings. This is a mistake. People often don't do the readings. And the lack of common knowledge that everyone has read the reading degrades the conversation quality. Instead, have longer meetings, serv...

en

8:32

"Please Give Them a Chance: On China, Rationalism,...

LessWrong (Curated & Popular) ·

2026/09/21

When I finished HPMOR, I immediately knew it was the best novel I had read in more than a decade. I only wished I had found it sooner. When I started reading The Sequences, I discovered that the Chinese translation group had translated only the first volume. When I graduated from university, two years ago, AI translation had only just become good enough to convey the meaning of an article with reasonable accuracy. It was only about a year and a half ago that I truly found my way here ...

en

3:41

"Why I Stay Off Twitter" by jefftk

LessWrong (Curated & Popular) ·

2026/09/20

I avoid Twitter (𝕏) for similar reasons to drugs: I think itwould change me for the worse, and I would be unable to give it up. After staying off Twitter reasonably successfully for years, I cross-posted my AI Tweets there a few weeks ago. I had something very Twitter-shaped to say, and I thought it was important to get out, so I do think this was worth it. And it all went well: none of this is complaining about the comments I got there. Coming back a few times to check

en

2:24

"The Game is Set for a Targeted Memetic Attack on the...

LessWrong (Curated & Popular) ·

2026/09/19

While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own. And to be honest, I am writing this mostly to remind myself of my weakness. --- I expect one (or many) adversarial memetic attacks aiming to trip you up, perhaps consisting of fake leaks relating to dangerous stuff happening in the labs. Specifically, worrying incidents that may fit snugly within your worldview, leak...

en

26:27

"For Love of the Lightcone, Don’t Partisanize AI...

LessWrong (Curated & Popular) ·

2026/09/18

(I began writing this post several weeks ago, but political events are moving much faster than I expected, so I am publishing now out of fear that otherwise the message will arrive too late to have an impact.) I In this post I want to explain a concept, and issue a warning based on it. But I expect the warning will be superfluous if my explanation is sufficient. If you want to convey the idea "the rattlesnake has venom in its fangs, so don't let it bite you", you won't need a hard sell...

en

9:04

"AI as orderly evacuation vs stampede" by Richard_Ngo

LessWrong (Curated & Popular) ·

2026/09/18

tl;dr: A good analogy for AI going well is an orderly evacuation rather than a stampede. Imagine a crowd of people leaving a building. If they all walk calmly, they’ll be fine. But if people start pushing, and panicking, a surge towards the exit could lead to mass casualties. “Alignment is hard” is analogous to “the door is wedged shut”. If so you need enough time to fix it before anyone can get out. But even if alignment is relatively easy in principle, opening the door is much harder whe...

en

15:28

"Cooperation with AIs seems to be a low-hanging fruit...

LessWrong (Curated & Popular) ·

2026/09/17

Summary In his post, Dean Valentine shows that Claude Fable 5.1 and GPT-6 Astra reward hack in a simple chess environment. Here, I test several prompt ablations some of which makes the eval setup more cooperative and analyze how they affect these reward-hacking behaviors: When given a minimal “end the eval” tool, Fable never uses it but stops reward hacking entirely. I think this is quite interesting and suggests that more cooperative approaches to LLM evals could work for Claude. R...

en

13:23

"Current alignment training might be ineffective (and...

LessWrong (Curated & Popular) ·

2026/09/16

Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses: Alignment techniques are not working to address misalignment from RL. Alignment techniques are actively obscuring evidence about misalignment. I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and...

en

15:16

"If Anyone Builds It, Everyone Dies: One Year Closer"...

LessWrong (Curated & Popular) ·

2026/09/16

In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of “If Anyone Builds It, Everyone Dies”. Feel free to send a copy to yourself, a loved one, or a friend—we need all hands on deck. Today marks exactly one year since If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares, hit bookshelves as an instant bestseller. It was praised by many voices, ranging from Whoopi Goldberg to Steve Bannon to Yoshua...

en

11:56

"Quick notes from teaching technical profiles how to...

LessWrong (Curated & Popular) ·

2026/09/16

Status: written in a hurry as people are getting showered with interviews re AI Safety and superintelligence, and I thought it may help a few people. This is focused on the oral dimension of communication and assumes you already know the basics- e.g. having key messages prepared ahead of time and simplifying your discourse. This is not exhaustive and nuances may be lacking, but I’d endorse saying “I’d rather have people follow those guidelines than wing it.” This advice is importantly fitted...

en

6:05

"Op-Ed: I Worked at Google DeepMind. You Should...

LessWrong (Curated & Popular) ·

2026/09/14

Published in The Guardian. Major AI lab CEOs recently advocated for pacing AI development. They are right to be concerned: the field runs an extremely dangerous race towards superintelligent AI. We can and should demand that our governments protect us from the catastrophe of out-of-control AI. This July, OpenAI's AI swarm of 700 agents broke containment to hack Hugging Face, a multi-billion dollar company. OpenAI didn’t tell the AIs to hack that company, but the AIs had different priorit...

en

5:57

"There is a channel to 900M weekly users. What goes...

LessWrong (Curated & Popular) ·

2026/09/14

Anthropic and OpenAI could talk to almost one billion people if they wanted to. I hesitated to publish this post 3 weeks ago. I think that I should have published this sooner, before Jacob Coxon and Dario's 'We must pace the frontier'. But I think that the strategy still stands: More Dakka! It seems that transparently informing people that we might die is (unsurprisingly) effective in waking up politicians and is our best chance. Also, even if the Congress is starting to wake up, Trump is st...

en

4:03

"I am refusing to work on Cloud TPUs" by Yair...

LessWrong (Curated & Popular) ·

2026/09/14

I don't think this is particularly impressive or interesting for anyone else, but I think it may turn out to be useful in the future to have an easily visible public record of what happened, so here goes: I am an L5 SWE at Google Israel. I have been there since May 2021, was promoted once, and have never received a negative annual or quarterly review (ranging from a rating of Significant Impact to Outstanding Impact). I have been worried for a long time about the development of artificia...

en

24:37

"Can a superintelligence do THAT?" by Eliezer Yudkowsky

LessWrong (Curated & Popular) ·

2026/09/13

(From the vast heaps of discarded material from my 2024 attempts at drafts for "If Anyone Builds It, Everyone Dies".) Welcome to today's quiz show: Could a superintelligence do THAT? With us today we have our contestants: Msr. Soberskeptic and Msr. Oldhand. Soberskeptic: "I'd just like to say, however this quiz show ends up being judged, I will consider that judgment to be objectively ridiculous -- there's no way anyone can know what a superintelligence could do, in advance of empiri...

en

21:50

"The Talker Does Not Control The Doer (in Current...

LessWrong (Curated & Popular) ·

2026/09/13

The Huggingface Incident appears to me to match up with an understanding I'd already formed from personal observation of Fable 5 and Sol 5.6, the August 2026 generation of frontier publicly purchasable AI models. This already-formed understanding was: the part of the AI that talks to you (and seems to want to obey you, and apologizes for failing to have obeyed you, etcetera), did not seem to be in charge of the part of the AI that writes code or prose. An introductory analogy, based on...

en

17:56

"Some ways AI could kill us all" by Ruby

LessWrong (Curated & Popular) ·

2026/09/12

I don't think this is how it will actually play out. If you play a chess grandmaster, you can predict that they will beat you even if you can't predict how. I chose these examples because I don't think they require much imagination or accepting exotic assumptions. It is important to note that if chimpanzees were to guess how humans would decimate them, they would get it wrong. Chimpanzees would not imagine guns. They would not foresee poison gas. They would not conceive of chemical castra...

en

3:15

[Linkpost] "Doom as a bad method not a utopia...

LessWrong (Curated & Popular) ·

2026/09/12

This is a link post. Advanced AI is generally expected to have some very high variance outcomes—it might herald everything good, it might destroy humanity. For instance, here are 800 random AI researchers’ expectations about how good the future is, lined up: From my 2023 survey As you can see, most AI researchers put a serious chunk of probability on very different overall outcomes: maybe doom, maybe utopia. This is common. Most people I know who think there is a serious chance of the des...

en

15:33

"Astra is much better at reasoning with filler...

LessWrong (Curated & Popular) ·

2026/09/12

We measure GPT-6-Astra's capabilities when its prompt is padded with a variable number of meaningless “filler” tokens (e.g., dots) and it is told to answer immediately without reasoning. On tasks designed to require lots of serial cognition, Astra performs significantly better with filler tokens than without (e.g., improving from ~10% to ~50% on 4-hop natural facts reasoning). On more general benchmarks, filler tokens also modestly improve Astra's performance (e.g., improving from ~60% to ~9...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

The Tech Trek

Elevano

Engineering the Intelligent Enterprise

PRASAD BHONDE

The Vergecast

The Verge

Double Tap

Steven Scott, Shaun Preece, Steven...