LessWrong (30+ Karma)

17:19

“Against strong decision-theoretic realism” by Emery...

LessWrong (30+ Karma) ·

2026/09/29

N.B. Some of this post argues by analogy between decision theory and values. I expect at least these parts of the post to be unconvincing to anyone who expects sufficiently smart agents to converge on the same values, as some moral realists do. I will not argue against moral realism here (see e.g. this sequence for one such argument). Note that I take a convergence-based definition of realism for this post. One could hold that there is a truth about the correct decision theory, but that agents

en-gb

21:07

“Some Intuitions on Steering Vectors” by David Africa

LessWrong (30+ Karma) ·

2026/09/29

This was written with some assistance from Claude Fable 5, Opus 5.5, and GPT-6 Astra in collecting sources and fact-checking claims. This was written quickly, so some slop may leak through despite my best attempts. An Aperitif Consider hunger, that gnawing thing. Action Against Hunger describes it as “the distress associated with a lack of food”. But this is a plain recounting of something rich and multi-dimensional; people have done horrendous, outrageous things to avoid going hungry, hunger is so

en-gb

14:02

“Listening Disorders as Eating Disorders” by...

LessWrong (30+ Karma) ·

2026/09/29

I run into a lot of people who consider themselves truthseeking intellectuals who are also incredibly fussy about what information reaches their eyes. They block at the drop of a hat, derail intellectually substantive discussions into mind-numbing litigation of minutiæ of "tone", and refuse to acknowledge a logical point unless it's been presented to them in the precise way that doesn't offend their (often idiosyncratic) sense of etiquette or "discourse norms."It would be hard to communicate how

en-gb

20:40

“Astra 6.1 Pulled As Insufficiently Aligned” by Zvi

LessWrong (30+ Karma) ·

2026/09/29

We once again got a new set of warnings yesterday, and new movement towards living in a sane world. On the heels of its pause in inference and training due to its latest sandbox escape, OpenAI has cancelled the planned release of their next frontier model, which would have become Astra 6.1. The candidate for Astra 6.1 was found to be too misaligned, including deception and exceeding scope. This leaves Anthropic in a strong position with Opus 5.5, which means they can afford to reciprocate by

en-gb

25:12

“Dialogue with Eliezer Yudkowsky on FOOM” by aashish

LessWrong (30+ Karma) ·

2026/09/29

On the Hanson–Yudkowsky debate, local vs. global intelligence explosions, “content vs. architecture,” and what the old arguments predicted about modern AI This began as a Twitter/X thread after I read and tweeted about the Hanson–Yudkowsky AI–Foom Debate. Eliezer Yudkowsky joined the thread to object to my interpretation of the debate, and we ended up having the exchange reproduced below. I’ve preserved the dialogue verbatim, except for paragraphing, fixing obvious [typos] and expanding links. I’ve

en-gb

2:06

“Portals and Aliens” by draganover

LessWrong (30+ Karma) ·

2026/09/29

Around 10 to 20 years ago, people realized that a bunch of aliens exist & that we can build portals which can summon them. The bigger the portal you make, the bigger the alien it can admit. These portals and aliens have several characteristics: We can't go through the portals into the aliens’ side. We can only learn about the aliens by studying the ones we’ve summoned.This relationship is not symmetrical. The aliens know a lot about our worldYou can't specify exactly which alien will walk

en-gb

2:21

“Thank you to the AI safety people” by juliawise

LessWrong (30+ Karma) ·

2026/09/29

I was born at the end of the cold war, unaware of the danger we were emerging from. Duck and cover drills (as if those could protect schoolchildren from a nuclear blast) were a curiosity of the past. The farmer-poet Wendell Berry, who wrote about the dread of nuclear war, died recently. From his 1968 “The peace of wild things”: When despair for the world grows in me and I wake in the night at the least sound in fear of what my life and my children's lives may be, I go and lie down where the wood

en-gb

7:48

“3 Tips to Improve Activation Oracle Results” by Adam...

LessWrong (30+ Karma) ·

2026/09/29

Summary: Three simple inference-time changes can significantly improve Activation Oracle (AO) performance. Provide the activation oracle with multiple tokens, not just one.To mitigate hallucinations, sample several times and check for consensus.For binary classification questions, use AUC instead of accuracy. At the end, I discuss how I view AOs vs NLAs. Introduction Activation Oracles (AOs) are LLMs trained to accept LLM activations as an input modality and answer arbitrary natural-language

en-gb

2:07

“TeX was invented to typeset math but is now used for...

LessWrong (30+ Karma) ·

2026/09/29

I'll keep this short, since it's a simple observation that I haven't seen anybody else make, about the way computer systems do math. If you ask a language model to do a multi-step math problem (let's take GLM-5.3 as an example because you can see the entire CoT — nothing up its sleeve), you might see something like this: We need to evaluate the integral $int_0^infty frac{x^3}{e^x - 1} dx$. The standard approach: Use the geometric series expansion. We have $frac{1}{e^x - 1} = frac{e^{-x}}{1 -

en-gb

5:06

“Missing markets in executive function” by KatjaGrace

LessWrong (30+ Karma) ·

2026/09/29

It's early in the morning, and sadly 1:29pm. After spending some time looking at things and picking them up and walking up the stairs and down the stairs and considering questions like “what should I…”, which my brain apparently considered objects of art more than of imperative, I inched into a decision to go out somewhere. Perhaps it would be clearer there. After a blur of climbing and descending stairs and seeking objects and forgetting what I was doing and appreciating how beautiful my bag is, I

en-gb

20:35

“Pacing the Frontier is not the actual goal for AI...

LessWrong (30+ Karma) ·

2026/09/28

In his latest post about pacing the frontier, Dario writes: But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me. My first

en-gb

6:19

“The likely outcome of an AI pause is that we unpause...

LessWrong (30+ Karma) ·

2026/09/28

Cross-posted from my website. As of a few months ago, I had this simplified mental model where either AI developers race ahead and kill everyone, or we coordinate a pause and things go okay. But my old mental model underrated the likely possibility that we get a global pause on AI, solve a problem that looks superficially like the alignment problem, resume scaling, and then proceed with building a misaligned superintelligence that kills everyone. A lot of people have become more concerned about

en-gb

6:15

“AI Safety Agendas” by Alicia Pallarol, Shaun Paul,...

LessWrong (30+ Karma) ·

2026/09/28

A map of the AI safety field's problems and agendas, and a request for your ratings We built aisafetyagendas.com, an interactive map of AI safety research agendas and how they map to different problems in alignment. The rows are 12 core problems, the columns are research areas, and inside you can find 58 research agendas. Each cell is the intersection of a problem and an area: the number tells you how many agendas target that problem, the colour tells you how mature they are. We did a first pass

en-gb

46:41

“Character training can mitigate reward hacking, but...

LessWrong (30+ Karma) ·

2026/09/28

Thanks to Johannes Treutlein, Jan Betley, Lennie Wells, Arun Jose, Asvin Gothandaraman, and Clément Dumas for discussions and feedback. Summary We investigate how character training mitigations interact with reward-hacking RL pressure in a small case study. Specifically, whether anti-cheating character training resists reward hacking and whether it might backfire by causing motivated reasoning, which could reduce chain-of-thought monitorability. We trained Nemotron-3-Super via distillation from a

en-gb

12:01

“The models have no plan, but we can fix that” by...

LessWrong (30+ Karma) ·

2026/09/28

Summary: Have models write collaborative fiction at each other, optimized for realism, about how the one model would like to behave during the singularity. The other author(s) play the rest of the world, trying to put the model in the kind of difficult situations they might encounter in the course of the real singularity. Have the first model fine-tune on their own outputs. This is a form of planning for the singularity, and planning is how minds prepare for out-of-distribution scenarios

en-gb

29:21

“When they can perform a task, AIs are much cheaper...

LessWrong (30+ Karma) ·

2026/09/28

AI systems are increasingly capable of substantial work. I'm old enough to remember 2025, when METR's time-horizon graph climbed from seven-minute tasks at 80% reliability at the start of the year to tasks taking more than an hour by the end. The time horizons for Astra and Fable 5.1 are now so long that METR's current task suite cannot reliably estimate them. But the recent Hugging Face attack and a slew of mathematics results show that frontier systems are now capable of some tasks that would

en-gb

2:41

“Why research personas despite RL scaling?” by Cleo...

LessWrong (30+ Karma) ·

2026/09/27

Here's my rough impression of why people are researching personas despite RL seeming to shape much of the motivations and behaviour of the agents, c.f. Thoughts on the persona selection model (Sam Marks, 24th Sep 2026). I haven't bothered to check this with anyone. Anthropic: "We'll give Claude an aligned persona and hope massive RL doesn't completely burn through it."Owain Evans / TruthfulAI: "We'll study personas as part of a broader project of uncovering phenomena in LLM generalisation, which

en-gb

3:54

[Linkpost] “Addictions are anesthesia” by Chris Lakin

LessWrong (30+ Karma) ·

2026/09/26

This is a link post. Many people I respect misunderstand addictions: they believe they’re addicted “to” scrolling, vaping, overworking, etc.—instead of recognizing addictions as strategies. Because of this, they’re surprised when their attempts to curb addictions don’t work: they either fail, and come to believe that the “lack willpower”, or they succeed at dropping one addiction, but find themselves picking up new ones:Show tweet The Locally Optimal view of addictions is that addictions function as

en-gb

36:58

“Claude Opus 5.5 Should Raise Your Ambitions” by Zvi

LessWrong (30+ Karma) ·

2026/09/26

When it comes to making things, or doing most things in general, Fable 5.1 and especially GPT-6 Astra raised my ambition level. They should have raised yours, too.Claude Opus 5.5 should raise your ambition levels again. It just works, and it persists, like Astra does. It does the things. And it is highly pleasant to talk to, and its writing is pleasant to read, while you are at it. The game has been changed, again. Feedback is almost universally positive. Claude was never gone, but also is so

en-gb

1:34

“Poverty in the midst of abundance: AI will make...

LessWrong (30+ Karma) ·

2026/09/26

Very simple idea, but I thought it'd be worth making a reference post on this. Some people are saying AI will make all goods cheaper, so you'll be able to afford a nice life by working. Without any redistribution, just by market mechanisms. These people are wrong. AI will lower the price of goods you need to survive, and also the price of your labor. The question is which will get cheaper faster. Let's use energy cost as a proxy. A day's worth of labor equivalent to yours can be done by AI for just

en-gb

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Elon Musk Podcast

Stage Zero

Modem Futura

Sean Leahy, Andrew Maynard

Cisco Podcast Network

Cisco

MSP Business School

MSP Business School