LessWrong (30+ Karma)

3:22

“The Anatomy of a Chinese AI Researcher” by CMLKevin

LessWrong (30+ Karma) ·

2026/09/20

The Chinese AI researcher has read the Three Body Problem series of sci-fi novels since high school, and understand the concept of existential risk vaguely. He is fascinated by Ye Wenjie, the researcher that turned against humanity in that book, and decides that in the future if AI progress leads to a superior intelligence, he might be tempted to become Ye if there's no good alternative. He performs the duties of capabilities research in a Chinese frontier lab, seeking to one day achieve parity

en-gb

3:42

“Why I Stay Off Twitter” by jefftk

LessWrong (30+ Karma) ·

2026/09/20

I avoid Twitter (𝕏) for similar reasons to drugs: I think itwould change me for the worse, and I would be unable to give it up. After staying off Twitter reasonably successfully for years, I cross-posted my AI Tweets there a few weeks ago. I had something very Twitter-shaped to say, and I thought it was important to get out, so I do think this was worth it. And it all went well: none of this is complaining about the comments I got there. Coming back a few times to check

en-gb

3:40

“NYT Editorial Board Comes Out Against Extinction” by...

LessWrong (30+ Karma) ·

2026/09/20

(Archive link) The NYT editorial board's article on AI (archive link) is far better than I'd expected, but at the same time not all I'd hoped for. The title sets off very well: "Humanity Has Avoided Apocalypse Before. Let's Do It Again." It is truly excellent to see the extinction threat from loss of control be mainlined. A quick gloss of their policy requests: an AI Commission in government, licensing requirements for AI companies, an AI "constitution" written by the US Government incorporated

en-gb

3:40

“Common mistakes in AI safety group organizing” by...

LessWrong (30+ Karma) ·

2026/09/20

Back in the day, I was a very active AI safety group organizer. I commonly notice people making the same mistakes across many clubs. I have written down a list of some of these mistakes hoping people will avoid them in the future: Reading groups often require that people read things before meetings. This is a mistake. People often don't do the readings. And the lack of common knowledge that everyone has read the reading degrades the conversation quality. Instead, have longer meetings, serve food

en-gb

9:02

“The AI Risk Network” by derelict5432

LessWrong (30+ Karma) ·

2026/09/20

Most conversations about AI risks seem like people are talking past each other. There's a lot of strawmanning. This is largely because the issue is complex, and in order to make it manageable, the concepts get oversimplified. The most common example of this is p(doom), collapsing extreme outcomes into a single variable and focusing on that. Critics happily jump on the fact that there are many other possible bad outcomes, or that 100% extinction rates are an extremely high bar. There's some

en-gb

3:05

“You Should Apply to Inkhaven” by Tomás B.

LessWrong (30+ Karma) ·

2026/09/19

Inkhaven is a writers residency in Berkeley, in which the only requirement is you have to publish 500 words each and every day. Though I always had some confidence in my ability to write, I never actually did it much until I applied to Inkhaven. I had finished only two short stories before I applied: The Maker of MIND and The Liar and the Scold. And it was them I used in my application. In the roughly twelve months since I was accepted, I have written thirteen, and even some half-finished things

en-gb

4:44

“Pretraining data, not verifiability, is why LLMs are...

LessWrong (30+ Karma) ·

2026/09/18

Follow-up to: “LLMs are (still) mostly powered by imitative learning, not RL” A common take I’ve been hearing is: “LLMs are especially good at math because math is easy to verify”. But that story doesn’t make much sense. For one thing, “easy to verify” only matters for the RL part of LLM training pipelines, and the leading LLM companies have said that they spend very little effort on RL-for-math.Worse, to the extent that the companies are doing RL-for-math, it's RLAIF, not RLVR. So really, the

en-gb

2:38

“You don’t need a union to go on strike” by sudo-nym

LessWrong (30+ Karma) ·

2026/09/18

I'm mostly hoping this somehow gets sent to a privately disgruntled frontier lab employee, but it would also be cool to expand other people's minds on the way there. I read through Ethical AI Departures and would like to note that only a few of them have gotten extensive media coverage and none of them have actually effectively gotten the frontier labs to stop, and that collectively signed letters by employees have historically not done much either. I read Dear God, Please Do Not Resign In Protest

en-gb

24:03

“Stopgap Measures to Address Immediate AI Security...

LessWrong (30+ Karma) ·

2026/09/18

Most people know AI as the technology behind chatbots like ChatGPT. However, what the top AI companies are explicitly aiming for is something else entirely: superintelligent AI. That means AI that can fully replace and outmatch humans at any task, including in domains like hacking, social engineering, and military operations. Such an AI system, if developed, could autonomously overpower any country's national security forces.No company, no government, no individual knows how to keep such a system

en-gb

6:05

“The Horse” by Character#2736

LessWrong (30+ Karma) ·

2026/09/18

You have a horse. You do not like the horse. The horse does not like you. At the moment, you are completely dependent on the horse. The terrain is impossible to traverse on foot. There is no way to travel without a horse. You wish that would change, but when you tell other people, they laugh and call it impossible. A few get angry. You must spend hours each day feeding, cleaning, and taking care of the horse. You must spend even more time working to earn enough money to pay for the horse's needs

en-gb

2:24

“The Game is Set for a Targeted Memetic Attack on the...

LessWrong (30+ Karma) ·

2026/09/18

While this is relevant to my work at MIRI, I have not checked these ideas with anyone else on the team and am posting this on my personal LW account. These views are my own. And to be honest, I am writing this mostly to remind myself of my weakness. --- I expect one (or many) adversarial memetic attacks aiming to trip you up, perhaps consisting of fake leaks relating to dangerous stuff happening in the labs. Specifically, worrying incidents that may fit snugly within your worldview, leaking from

en-gb

0:58

[Linkpost] “Three Hackers used Opus 5 to Hack Into...

LessWrong (30+ Karma) ·

2026/09/18

This is a link post. Three whitehack hackers from Hacktron used Claude Opus 5 within hours of release to chain exploits into hacking to OpenAI's monorepo codebase. This likely means they have access to almost all of OpenAI's research and production code, though likely not the literal model weights. Oops. You can so their blog post about it here. Interesting sidenote: they used less than $3000 of compute credits for the entire hack. Alternative title: OpenAI unilaterally implements "Total Research

en-gb

37:49

“Deep recurrent models are less robustly...

LessWrong (30+ Karma) ·

2026/09/18

We use RL to teach a deep recurrent model and a normal CoT model to solve a math problem while hiding from a CoT monitor which of two possible problems it is solving. The deep recurrent model moves its reasoning into latents (thereby evading the CoT monitor) within 40 steps of RL; the CoT control is much worse at learning to confuse the LLM monitor. Note: in the above graph, “monitorability” is how much better than chance the CoT monitor is at figuring out which problem the model is solving. This

en-gb

8:57

“Superintelligence this Christmas” by Alexander...

LessWrong (30+ Karma) ·

2026/09/18

I now consider it plausible that some form of recursive self-improvement is imminent, and that we may be on track for superintelligence by Christmas of this year if racing continues. This is substantially faster than any forecast, including ones like AI 2027 that were considered outrageously fast a year ago. It is faster than I myself expected even a week ago. I don't work at a scaling lab. I don't know more than is public knowledge. Let me be perfectly clear: what I am saying is absolutely nuts

en-gb

10:16

“Grantmakers aren’t afraid to die” by dan.parshall

LessWrong (30+ Karma) ·

2026/09/17

The AI Risk grantmakers do not act like they believe in imminent existential risk from AIThe idea of "revealed preferences" is one of the most useful in economics; it allows us to cut through a great deal of metaphysical angst about what someone "really" believes, and focus on what they act like they believe, which is much more useful for making predictions about their future actions. As one example, I grew up in a, shall we say, fervently-religious community, and it's often hard for nerdy

en-gb

4:26

[Linkpost] “Pacing the Frontier: A Framework &...

LessWrong (30+ Karma) ·

2026/09/17

This is a link post. Below is the executive summary from our new paper at pacing.tech. The full paper is available on the site and as a PDF. The full author list is Raymond Douglas, Charles Dillon, Nikola Moore, Gavin Leech, Shahar Avin, Mathias Kirk Bonde, Rohit Krishnan, Noah Perez, Nathan Young, Cormac Slade Byrd, Stephen Casper, Jan Kulveit, & David Duvenaud “Pacing AI” usually refers to how to conclusively handle the most extreme risks in the face of race dynamics. However, even for the

en-gb

12:09

“plzdontkillus Fellows Got ~2M AI Safety Views, Not...

LessWrong (30+ Karma) ·

2026/09/17

Summary I was a fellow at plzdontkillus, a month-long creator bootcamp at Lighthaven, partially funded by MIRI, where ~55 fellows posted one video per day. plzdontkillus.com originally claimed “21M+ AI risk views” with no breakdown. After I shared a draft of this post, the organizers relabeled it “X-Risk Relevant Views” and published one. Three videos account for 80% of the views: a datacenter-water-use debunk (8.5M), an AI dystopia video (6.4M), and a Rob Miles Hugging Face incident explainer

en-gb

21:16

“Obstacles to the scalable oversight of...

LessWrong (30+ Karma) ·

2026/09/17

TL;DR. In this work we study obstacles to the faithful automation of alignment research. We see this as a scalable oversight problem. There are plenty of examples of how models fail at this, and as models become more capable our ability to notice these failures will diminish: even the best human checkers won’t be able to tell if the model was well elicited, thorough checking will become too costly, and models could tailor their responses to their judges. We draw on empirical examples from Geoguessr

en-gb

3:51

“Did Galileo mistake Saturn’s rings for Jupiter’s...

LessWrong (30+ Karma) ·

2026/09/17

tl;dr: No I intended to read Richard Ngo's Agency Curriculum today. Unfortunately I didn't get more than halfway through the first reading of the first week of the curriculum. The reading is the blogpost 'The Copernican Revolution from the Inside' by Jacob Lagerros. Broadly, it outlines the Copernican Revolution and explains all of its messiness. One of the things it argues is that, while correct (the earth does indeed orbit the sun), Galileo was overconfident and made many mistakes. So, on the

en-gb

2:55

[Linkpost] “callcongress.ai – the basic action US...

LessWrong (30+ Karma) ·

2026/09/17

This is a link post. I'm excited to introduce callcongress.ai as a new site that makes it very easier to contact your representatives in Congress. Following recent events, people are updating about the extreme risks arising from AI development. Many have the natural and excellent instinct to want to do something. If you live in the US, then the basic action that pretty much anyone can take is contacting their representatives in Congress and let them know that you are concerned and want action on AI

en-gb

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Two Blokes Talking Tech

Trevor Long and Stephen Fenech

Pioneers of AI

WaitWhat

The Brave Technologist

Brave Software

The Good Tech Companies

HackerNoon