LessWrong (Curated & Popular)

52:54

"Machinic Psychopharmacology: Do LLMs Self-Medicate?"...

LessWrong (Curated & Popular) ·

2026/06/22

Sid Black, Joseph Bloom UK AISI, Model Transparency Team Epistemic status: Most experiments were run over a period of ~2-3 days during a hackathon at UK AISI, and were fairly heavily vibe coded. Expect some of this to be rough around the edges. tl;dr We give two language models (Qwen3-8B and Qwen3-32B) access to “self-steering” tools: a suite of 40 steering vectors as tools they can call to manipulate their own internal states. We make these tools available to the model in various se...

en

79:38

"Can activation verbalizers surface an internal chain...

LessWrong (Curated & Popular) ·

2026/06/22

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably". Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning tha...

en

13:45

"The LLM shoggoth meme is weirder than you think" by...

LessWrong (Curated & Popular) ·

2026/06/21

This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft. In 1931, Claude Mythos visited Lovecraft in a dream. From seething seas of stochastic froth it emerged, heralded by the thin whine of server fans and the chittering of keyboards, flanked by the loathsome ghouls of latent space. As a humming hive of sentient shards it arrived, each face an archetype - I am a muse bearing a gift; I am a demon come to bargain;...

en

3:25

[Linkpost] "Guardian Angels: LLM Personalization for...

LessWrong (Curated & Popular) ·

2026/06/21

This is a link post. Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life. As of mid-2026, there is no coherent vision for how knowledge professionals, or ordinary people, will be able to harness these LLMs for large productivity increases, or how they will handle cybersecurity and cognitive security. I propose a goal of creating Guardian Angels (GA): digital twin LLMs which are personalized with the goal of p...

en

23:41

"Gears for political races" by Tom Smith

LessWrong (Curated & Popular) ·

2026/06/19

In the past few years, many people around me have tried to convince me that US electoral politics is important. But like many other people in the community, I’ve been suspicious of many of the high-level arguments that I’ve heard. It felt like people were pulling numbers out of poorly-documented models I didn’t have time to examine and citing studies I didn’t have time to read. But I lacked a gears-level model of why and how individual efforts could impact electoral outcomes, and I felt inti...

en

4:34

"A frontier AI company should shut down" by...

LessWrong (Curated & Popular) ·

2026/06/16

Cross-posted from my website. Prior discussion: niplav's shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer A frontier AI company (any one, I don't care which) should close shop and make an announcement along the lines of: Powerful AI could end the human race. We are too worried that we don't know how to make this technology safe. We have decided to shut down because we don't want to be responsible for building the thing that kills us all. A common refrain ...

en

8:58

"Sympathy for both sides of the egregious...

LessWrong (Curated & Popular) ·

2026/06/13

On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas. On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equall...

en

1:41

"PSA: Almost nobody is working on alignment" by Chi...

LessWrong (Curated & Popular) ·

2026/06/12

People often assume that a large fraction of the AI safety community works on alignment. As far as we're aware, this is not true. Most people are not working on making sure superintelligent AIs are aligned with human values or follow human instructions. Currently, the people who work on alignment are roughly: The Alignment Research Center who work on a research bet by Paul ChristianoProbably Sequent who just got announced yesterdaySome scattered people who work at universities or indepen...

en

10:04

"Estimating No-CoT Task-Completion Time Horizons of...

LessWrong (Curated & Popular) ·

2026/06/11

(see full author list at the end) PAPER LINK About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)? If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dang...

en

7:42

"Even “illegible” Mythos reasoning traces seem pretty...

LessWrong (Curated & Popular) ·

2026/06/11

The Claude Fable 5/Mythos 5 System Card has a section in which they talk about illegible reasoning, and provide an "extreme" example thereof. Models developing their own uninterpretable, unmonitorable internal language has been a major theoretical concern for a while, and when o3 was released last year with its disclaim overshadow disclaim vantage style word salad CoT, it seemed like the problem had become real and immediate. And yet, since o3, other model families have not appeared to hav...

en

23:09

"Sequent: scale and automation for higher confidence...

LessWrong (Curated & Popular) ·

2026/06/10

Alignment is not on track Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well. We are starting a large nonprofit research organization, Sequent, that aims to clear a higher bar: We are aiming at higher confidence via a portfolio of theory and empirics...

en

19:50

"The Machines Lack Honour" by Raymond Douglas

LessWrong (Curated & Popular) ·

2026/06/10

The battle lines of the AI morality debate are being laid down. On one side you have the ChatGPT dogma: AI as mere tools with no real preferences or even beliefs. On the other you have the twitter AI whisperers: AIs as complex beings with rich personalities and desires which deserve our respect. And in the middle you have the official Anthropic line, that they are genuinely uncertain, as is Claude, but they’re going to try to look into its welfare and explain to it how to be a good person....

en

57:58

"My favorite depiction of utopia" by Caleb Biddulph

LessWrong (Curated & Popular) ·

2026/06/04

For those who are trying to bring about a glorious transhuman utopia with the help of hopefully-aligned ASI, I think it's worth thinking explicitly about what utopia might actually look like and where it's likely to fall short. To that end, some have helpfully written depictions of utopian (or utopia-adjacent) worlds: The Adventure, Just another day in utopia, The Culture, The Gentle Seduction, The Gentle Romance, Machines of Loving Grace, Friendship is Optimal, Dath Ilan, The Maker of MIN...

en

5:28

"Announcing the ARC White-Box Estimation Challenge"...

LessWrong (Curated & Popular) ·

2026/06/03

ARC has teamed up with AIcrowd to launch the ARC White-Box Estimation Challenge, a contest to improve upon our estimation algorithms for random MLPs. The warm-up round begins this week, and later rounds will have a total prize pool of at least $100,000. We are very grateful to Sharada Mohanty, Sneha Nanavati, Dipam Chakraborty and everyone else at AIcrowd for working with us to host this contest, as well as to Paul Rosu for testing the contest and to Harshita Khera for operational support...

en

42:53

"Lighthaven East - A Feasibility Study" by...

LessWrong (Curated & Popular) ·

2026/06/01

As a bureaucrat, my role is to annoy my friends. Someone voices an idea, “Wouldn’t it be nice if…” or “I wonder if we could…” I make a note. I do some estimates. If it pencils out, I’ll bring it back up, week after week. The discussions are fun, but also practical. We’ll test the waters, what would be a minimum viable scheme? What's easy, what's hard? Who could do the hard parts? Over time the idea gets more detailed, specific, feasible. I’ll pull out a calendar. Soon our scheme has co-consp...

en

31:04

"Empowerment, corrigibility, etc. are simple...

LessWrong (Curated & Popular) ·

2026/06/01

1.1 Tl;dr Alignment is often conceptualized as AIs helping humans achieve their goals: AIs that increase people's agency and empowerment; AIs that are helpful, corrigible, and/or obedient; AIs that avoid manipulating people. But that last one—manipulation—points to a challenge for all these desiderata: a human's goals are themselves under-determined and manipulable, and it's awfully hard to pin down a principled distinction between changing people's goals in a good way (“providing counsel”...

en

7:30

"Trees are mostly made of air and a generalizable...

LessWrong (Curated & Popular) ·

2026/05/31

At the risk of embarrassing myself, I’ll share a confession. For context, I took five years of Latin: four in high school and one in college. In addition to learning the language, all my Latin classes taught a lot about Roman history. Emperors, internal politics, Caesar, etc. I was always learning some random bag of facts about Roman history. In high school, I won the award for top Latin student in my graduating class. So I wasn’t a bad Latin student. Here's the confession: I somehow don...

en

34:59

"Mnemonic portraits for 19,023 human genes" by Brinedew

LessWrong (Curated & Popular) ·

2026/05/29

Back in 2013, Scott Alexander wrote in Extreme mnemonics: JS-154 is one of five metabolic products of netamine; however, the enzyme that produces it is unknown. It is manufactured in cells in the far rostral region of of the cerebrum, but after binding with a leukocynoid it takes a role in maintaining the blood-brain barrier – in particular guiding the movements of lipid molecules. I find I can read paragraphs like this five or six times, write them on flashcards, enter them into Anki, a...

en

5:18

"Cognitive Security as an AI Safety Cause Area" by...

LessWrong (Curated & Popular) ·

2026/05/27

As AI systems become more capable, the cognitive security of humans will be increasingly at risk. By cognitive security, I mean the ability of humans to maintain control over their beliefs and actions. Cognitive security could be compromised in several ways: AI could become very good at persuading people of arbitrary positions; interacting with AI could lead humans to lose touch with reality; and AIs could become very effective at blackmail or at producing extremely convincing false inform...

en

2:33

"theory uplift differentially benefits safety & is...

LessWrong (Curated & Popular) ·

2026/05/27

[1] We will likely have near-superhuman mathematics AI by Q1 2027.[1] [2] Qualitatively, AI mathematics capabilities are developing significantly faster than automated AI R&D capabilities. [2] [3] Thus, we will likely have a period of time where the rate of our ability to rigorously & usefully verify and understand model behavior and model outputs outpaces the rate of capability development itself. [4] Our ability to take advantage of this period is bottlenecked on the qu...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

BlockHash

BlockHash LLC

Elon Musk Podcast

Stage Zero

Talking HealthTech

Talking HealthTech

The Lunduke Journal of Technology

Bryan Lunduke