LessWrong (30+ Karma)

17:45

“Why I’m scared of RL” by owencb

LessWrong (30+ Karma) ·

2026/09/23

Summary: First, I give several different angles on how I feel about reinforcement learning: Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffoldingRecent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progressI’m worried things might get worse: if RL environments start incorporating agents, they may

en-gb

5:10

“Jensen Huang Says If We Cannot Align AI, Shut Down...

LessWrong (30+ Karma) ·

2026/09/23

I was very surprised today on a podcast to hear Jensen Huang plainly state that if they cannot align the AIs, then the labs must shut down. The context I have on Huang is that he has run NVIDIA for 30+ years, which has become the most valuable company in the world due to the AI boom. My understanding is that he has repeatedly encouraged the US President (with whom he is on friendly terms) to continue to support AI, and dismissed AI talk as "sci-fi". If you haven't seen, his biographer has

en-gb

1:46

[Linkpost] “AI: artificial immigrants” by KatjaGrace

LessWrong (30+ Karma) ·

2026/09/23

This is a link post. Advanced AI is basically the embodiment of immigration as envisioned in the conservative nightmare: We are letting a bunch of new agents into our societyThey don’t clearly share our values and we suspect a society full of them would be awful by our lightsBut we expect them to provide very cheap laborWhich will undercut local wages and leave locals unemployedThey will probably gain power and influence over time—in the economy, politics and culture—and end up controlling

en-gb

38:24

“An unexamined cause of the OpenAI Hugging Face...

LessWrong (30+ Karma) ·

2026/09/23

We argue that a main cause of the OpenAI Hugging Face incident was overlooked: the overly simple evaluation metric in ExploitGym was misaligned. Further, techniques already exist that can mitigate such misalignment in the future. In July 2026, OpenAI was testing the ability of its language models to exploit software vulnerabilities using a benchmark called ExploitGym. In ExploitGym, each test presents an agent with software containing a known vulnerability and tasks it with capturing a secret

en-gb

12:07

“Modern LLMs have tiny GPTs hidden inside them” by...

LessWrong (30+ Karma) ·

2026/09/22

Experiments into predicting GPT2 completions via Qwen models This is a crosspost from my substack (where I do varied tiny experiments on LLMs and agents). It's also part of Lossfunk, where we're investigating meta-cognition in LLMs as one of the projects. ---- Next token prediction is a magical objective. To predict the correct token in such a vast variety of texts present in the pretraining corpus, the model must infer a tremendous amount of hidden and latent causes that generate that text. Only

en-gb

23:12

“Story Imprinting: AI Assistants Absorb Traitsfrom...

LessWrong (30+ Karma) ·

2026/09/22

This is the abstract, introduction and discussion of our new paper. We also include an addendum on the connection to the Persona Selection Model. Section, appendix, and figure references refer to the full paper. Links: 📜 Paper, 🐦 Twitter thread, 💻 Code Authors: Jorio Cocola, Lev McKinney, Harry Mayne, Jan Betley, Owain Evans Abstract Language models are trained to implement a helpful AI Assistant character (e.g., Claude). We explore how finetuning on synthetic stories affects this character

en-gb

3:39

“A class of statement between conjecture and theorem”...

LessWrong (30+ Karma) ·

2026/09/22

Parts of the math community, such as Henry Cohn and Grant Sanderson, are arguing that proofs have been a proxy for understanding, and now that proxy is broken. This is a response to LMs generating incomprehensible proofs, often formalized in Lean. While the proofs are verified, they lack the pedagogical value which has historically come along with new proofs. In the past we could typically assume at least one human in the world understood the novel insight required to produce a proof, but that

en-gb

27:40

“What if not Circuits?” by CarolusRenniusVitellius

LessWrong (30+ Karma) ·

2026/09/21

This post was written as part of the Iliad Fellowship. Inspired by conversations with Richard Ngo, Dmitry Vaintrob, and Brianna Grado-White. To all of these, my thanks.Preface: I'm confused about how neural networks do and learn computations. In response to a friend's challenge, I'm writing up some interim thoughts. This essay has four parts: the first tries to track what I call the 'default ontology' of the mechinterp community over the years. The second part is about 'representational drift' as

en-gb

16:00

“Swarm Scaling” by Toby_Ord

LessWrong (30+ Karma) ·

2026/09/21

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? We’ve seen two large and extremely capable swarms from OpenAI in the last few months: 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophisticated

en-gb

13:05

“Swarm Scaling” by Toby_Ord

LessWrong (30+ Karma) ·

2026/09/21

Just how powerful are large swarms of AI agents? And how do their powers scale as more and more agents are added to the swarm? We’ve seen two large and extremely capable swarms from OpenAI in the last few months: 1,200 agents were being evaluated separately, but found a way to illicitly set up a message board and coordinate as a swarm. In order to cheat on their tests, they developed advanced techniques to prevent their actions being logged by OpenAI and 700 of them launched a sophisticated

en-gb

10:58

“Mech Interp is a Verifiable Task” by Logan Riggs

LessWrong (30+ Karma) ·

2026/09/21

If we think parts of MLP0-MLP3 are computing [a sorting algorithm], we can replace those parts with [a sorting algorithm] and check reconstruction loss. However, reconstruction loss is not enough. Suppose we replace MLP0 with two things: Its mean activation - simple, but poor reconstructionMLP0 - perfect reconstruction, but no reduction in complexity We can visualize this as a pareto frontier trading off reconstruction with "simplicity". Ideally we achieve perfect reconstruction with perfect

en-gb

3:07

“Weight smuggling likely defeats attempts to cap...

LessWrong (30+ Karma) ·

2026/09/21

Epistemic status: >90% confidence in the principle, >70% confidence that mitigating these would be hard in practice, no full implementation yet. tldr: it seems difficult for verification mechanisms to prevent chaining runs together or aggregating parallel ones; per-run FLOP caps could thus be covertly bypassed. 1. If governments want to regulate frontier AI training, one might want to cap individual training runs, e.g. putting bounds on the number of FLOPs per training run, and making sure

en-gb

20:43

“We’re not ready for the e/Acc × Longevity preference...

LessWrong (30+ Karma) ·

2026/09/21

Anti-aging sentiment might go rapidly mainstream, in the same way AI Safety just did. Recently, the AI safety community has been enjoying a massive preference cascade that has rapidly moved AI x-risk concerns into mainstream political discourse Why did this happen? The basic idea seems like it should have been obvious for a long time: “IF we create self-improving machines that rapidly become much smarter than humans, THEN that seems like that story might not end well for the humans, so we should

en-gb

14:34

“Alignment Midtraining Cracks Under Pressure” by J...

LessWrong (30+ Karma) ·

2026/09/21

TL;DR We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data. For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190 million tokens of midtrained motivations are overpowered by a relatively tiny amount (~50 thousand

en-gb

8:07

“Empirical safety claims from frontier labs should be...

LessWrong (30+ Karma) ·

2026/09/21

When frontier labs like Anthropic and OpenAI publish safety or alignment research, it is often entirely empirical, closed-source, and sparse on methodological details. While it is great that they publish these results, the status quo is that labs (or soon, their agents) can claim alignment progress that no one independently verifies. The AI safety community has replicated or stress-tested some claims, but it's nowhere near comprehensive, and we expect this kind of meta-science to remain

en-gb

8:33

“Please Give Them a Chance: On China, Rationalism,...

LessWrong (30+ Karma) ·

2026/09/20

When I finished HPMOR, I immediately knew it was the best novel I had read in more than a decade. I only wished I had found it sooner. When I started reading The Sequences, I discovered that the Chinese translation group had translated only the first volume. When I graduated from university, two years ago, AI translation had only just become good enough to convey the meaning of an article with reasonable accuracy. It was only about a year and a half ago that I truly found my way here and began

en-gb

18:24

“We’ve saved the world before: what the ozone hole...

LessWrong (30+ Karma) ·

2026/09/20

It might destroy the world, despite passing every known safety test. If we wait for a “warning shot” before we act, it might be too late. And action requires global coordination, because if anyone makes it, everyone dies. Sound familiar? It should, because it already happened half a century ago, with chlorofluorocarbons (CFCs). Despite seemingly impossible odds, we got our act together and completely solved the problem through unprecedentedly successful international coordination. The Montreal

en-gb

3:22

“The Anatomy of a Chinese AI Researcher” by CMLKevin

LessWrong (30+ Karma) ·

2026/09/20

The Chinese AI researcher has read the Three Body Problem series of sci-fi novels since high school, and understand the concept of existential risk vaguely. He is fascinated by Ye Wenjie, the researcher that turned against humanity in that book, and decides that in the future if AI progress leads to a superior intelligence, he might be tempted to become Ye if there's no good alternative. He performs the duties of capabilities research in a Chinese frontier lab, seeking to one day achieve parity

en-gb

3:42

“Why I Stay Off Twitter” by jefftk

LessWrong (30+ Karma) ·

2026/09/20

I avoid Twitter (𝕏) for similar reasons to drugs: I think itwould change me for the worse, and I would be unable to give it up. After staying off Twitter reasonably successfully for years, I cross-posted my AI Tweets there a few weeks ago. I had something very Twitter-shaped to say, and I thought it was important to get out, so I do think this was worth it. And it all went well: none of this is complaining about the comments I got there. Coming back a few times to check

en-gb

3:40

“NYT Editorial Board Comes Out Against Extinction” by...

LessWrong (30+ Karma) ·

2026/09/20

(Archive link) The NYT editorial board's article on AI (archive link) is far better than I'd expected, but at the same time not all I'd hoped for. The title sets off very well: "Humanity Has Avoided Apocalypse Before. Let's Do It Again." It is truly excellent to see the extinction threat from loss of control be mainlined. A quick gloss of their policy requests: an AI Commission in government, licensing requirements for AI companies, an AI "constitution" written by the US Government incorporated

en-gb

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Elon Musk Podcast

Stage Zero

Modem Futura

Sean Leahy, Andrew Maynard

Cisco Podcast Network

Cisco

MSP Business School

MSP Business School