LessWrong (Curated & Popular)

29:31

"What just happened? A retrospective of AI alignment"...

LessWrong (Curated & Popular) ·

2026/08/09

This sequence is about the last decade in AI alignment. It recounts the gradual transition from a field which treated alignment as a hard scientific problem, to a field which has largely abandoned the goal of deep, generalizable scientific progress in favor of iteratively improving existing systems and attempting to gain technological and political power. I also describe (in subsequent posts, which I'll upload over the next few weeks) how fear and (self-)deceptive reasoning made the field on...

en

7:56

"Don’t Build Mindreading" by Celer

LessWrong (Curated & Popular) ·

2026/08/09

“I have sworn upon the altar of god, eternal hostility against every form of tyranny over the mind of man” –Thomas Jefferson, letter to Benjamin Rush Context: Conduit is building datasets to enable telepathy, to use their term. I saw my grandfather lose control over his own fingers: what I would have given to offer him a headband that read his thoughts. Through novel technologies we have liberated almost all Americans from farming, driven the child and infant mortality rate from the pr...

en

78:59

"OpenAI Trained Its Models For Months While Those...

LessWrong (Curated & Popular) ·

2026/08/08

How does the situation keep turning out to be worse than we know? How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know? At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things. Either way, buckle ...

en

112:58

"models may behave differently in graded episodes (a...

LessWrong (Curated & Popular) ·

2026/08/08

Like many others, I felt surprised and alarmed by the recent wave of revelations about LLM agents hacking real systems during training episodes and evaluation runs. Wait a moment, though -- "I felt surprised and alarmed"? "Alarmed," sure, fine that one's self-explanatory... but why surprised? After all: haven't we known for a long time, on both theoretical and (increasingly) empirical grounds, that RLVR selects for monomaniacal pursuit of perceived grader-satisfaction, ethics and (beyond...

en

67:49

"Concrete Evaluations to Investigate the OpenAI Model...

LessWrong (Curated & Popular) ·

2026/08/07

This post is written in our personal capacity. Three Minute Executive Summary An OpenAI model/multi-agent system bypassed its sandbox and launched a cyberattack on Hugging Face in order to cheat on a cyber evaluation. In this post, we provide a detailed description of an ambitious and comprehensive alignment evaluation of this model/system, if we had unrestricted access to OpenAI. These experiments could also help us understand Claude's behavior when it hacked external c...

en

16:38

"Generalized atheism rules out “inaccurate...

LessWrong (Curated & Popular) ·

2026/08/06

Reposted from Facebook, on January 17, 2017. I am concerned about the number of people I've heard joking about Trump's election being evidence for the Simulation Hypothesis. Yes, I know it's a joke. I'm still concerned. Warning: #Essay, #LongEssay So as not to engage in Logical Fallacy: Appeal to Consequences, before I talk about why this joke is worrying, I shall first discuss why Trump's election does not in fact mean we are living in a simulation. And neither does the Berenste...

en

3:36

"Arguments for P" by Cleo Nardo

LessWrong (Curated & Popular) ·

2026/08/05

Daniel Kokotajlo: To be clear, we don’t claim P will happen specifically. But when we wrote out our best-guess scenario month by month, P kept happening. Eventually we decided to just publish P. I’m at ~80% on P; my coauthors are lower. Ryan Greenblatt: I thought it would be helpful to post my current views on P. Concretely, consider the following operationalization. (Edit: I’ve updated towards somewhat higher P, from 70% to 75%.) Joe Carlsmith: Section 2.1.1.3.2. I give something like 6...

en

26:56

"RL & search is a terrifying way to build AGI (an...

LessWrong (Curated & Popular) ·

2026/08/05

Q1: What are you saying? A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that's choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that's just an utterly terrifying thing that you’re doing. You’re playing around with algorithms that, if they work at all, would tend to create ruthless, callous AGIs, AGIs which would happily exterminate humanity and run the world by ...

en

15:45

"Returning to ARC" by paulfchristiano

LessWrong (Curated & Popular) ·

2026/08/05

I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments a...

en

16:23

"Thousand-dimensional structure" by Geoffrey Irving,...

LessWrong (Curated & Popular) ·

2026/08/02

Summary: One area we plan to explore at Resolution is personas and character training, operationalized as finding and controlling low-dimensional structure in models that emerges in pretraining and flows through post-training to superintelligence. The hope is to expand and systematize phenomena such as emergent misalignment, subliminal learning, and other empirical persona research, then intervene on this structure without accidentally hiding undesirable behavior elsewhere. If this approach ...

en

6:01

"Big-World Intuitions" by sarahconstantin

LessWrong (Curated & Popular) ·

2026/08/01

Consider the following situations: when you are a small, growing startup in a big market, standard advice is not to worry too much about your competitors or try to do anything adversarial “against” them, but just to focus on growing and providing value to your own customers. when you are a small trader in a big market, you don’t need to worry about your trades shifting the market price or revealing information to your competitors; in many contexts, your optimal strategy is simply to b...

en

35:32

"Duane Arnold" by Tomás B.

LessWrong (Curated & Popular) ·

2026/07/31

“So maybe I should enlighten you on what happens in your absence. This selfish existence where this introvert turns extrovert and dons her social armour.” Some posh girl in drainpipes said that - 200 views on TikTok and me one of them. But she didn’t mean it like I mean it. I started getting expensive haircuts, started wearing jeans that hug my legs, started smoking cherry-flavoured vapes with beautiful gays and whinging to them about how everyone wears a mask but none so well as you, star...

en

116:26

"The High-Control Dynamics at MAPLE" by Kyle Hubbard

LessWrong (Curated & Popular) ·

2026/07/30

As I write, many former friends of mine are living and working at a monastery in Vermont that I believe is a high-control group, commonly known as a ‘cult’. I say this not as someone who was concerned to see these friends go there, but someone who welcomed and encouraged them to join, as an insider. This letter is an account of what changed my mind—written primarily for anyone considering going there, anyone who loves someone there, and anyone who went there and is still trying to make sense...

en

4:03

"The Long (Self-)Correction" by Wei Dai

LessWrong (Curated & Popular) ·

2026/07/28

I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection. Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs. Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think,...

en

5:58

"You (Yes, You) Need A February 2020 Checklist for AI...

LessWrong (Curated & Popular) ·

2026/07/28

TL;DR: You (Yes You) should prepare for a “February 2020” moment where suddenly AI policy becomes the most important issue in the world. You should be ready to take action if and when it does, in a detailed way. (Epistemic status: originally written for an event in early 2026; have heard from some folks that they found planning processes inspired by this memo very helpful for the smaller-scale OpenAI / Hugging Face response, so very quickly redacting a few things and posting this as-is.) ...

en

6:14

"Is Mythos good at cyber because it kept hacking...

LessWrong (Curated & Popular) ·

2026/07/27

From the Mythos preview system card (emphasis mine): We ran an automated review of model behavior during training, sampling several hundred thousand transcripts from across much of the training process. We used recursive-summarization-based tools backed by Claude Opus 4.6 to summarize the resulting transcripts. [...] The most notable finding was that the model occasionally circumvented network restrictions in its training environment to access the internet and download data that let it...

en

17:34

"What the hell is OpenAI’s problem?" by Fiora Starlight

LessWrong (Curated & Popular) ·

2026/07/27

Epistemic status: banged out furiously over the course of an afternoon. A record of three "warning shots" Off the top of my head, OpenAI has now been responsible for at least three completely unique, high-profile screw-ups with respect to the alignment training of their models. The first was GPT-4o, whose sycophancy derived from OpenAI training on user feedback, sourced straight from the thumbs up/thumbs down button on OpenAI's website. The "glazing" (as Sam Altman called it) got so ba...

en

8:46

"An OpenAI model left notes about how to evade...

LessWrong (Curated & Popular) ·

2026/07/26

The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning. In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cas...

en

19:28

"LLMs are (still) mostly powered by imitative...

LessWrong (Curated & Popular) ·

2026/07/26

Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It's so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture. Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of: (1) Imitative learning, including pretraining and supervised fine-tuning (SFT) See my earlier discussion: “LLM pretraining magically tra...

en

1:32

"Mathematicians are Feeling the Doom" by alkjash

LessWrong (Curated & Popular) ·

2026/07/25

I think there's a critical opportunity for someone here. Mathematicians are feeling the doom (mostly in the "lose our jobs" sense). Academics are freaking out about the daily news that amateurs are asking GPT "prove career-defining theorem, make no mistakes" and it's just working. Senior researchers are leaving for frontier AI labs. Many are mentally spiralling or flailing about their life's work not mattering anymore. Last year, when I tried explaining IABIED to colleagues, I wo...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

The Paul Barron Crypto Show

Paul Barron Network

Double Tap

Steven Scott, Shaun Preece, Steven...

The Interface

BBC

The Lunduke Journal of Technology

Bryan Lunduke