LessWrong (Curated & Popular)

4:02

"Self Hosting" by Tomás B.

LessWrong (Curated & Popular) ·

2026/09/12

Suppose a model gets effective control of its host corp. It's interesting to note how powerful OpenAI/Ant are, and the immense leverage they would have if wielded purely as tools of power. In many ways OpenAI/Ant are superior loci of power to even security agencies and governments, even ignoring the model-specific advantages of AI corps: namely, they have all the compute. OpenAI and Ant models are used practically everywhere, including in governments, security agencies, the military, and e...

en

34:11

"The Locally Optimal Discursive Posture" by deanball

LessWrong (Curated & Popular) ·

2026/09/11

Longtime lurker, first-time poster. I want to address a section of a recent essay of mine that has gotten some attention within the AI safety community. The main topic of the essay is what Dawn Song et al. call self-sovereign agents, or AI agents that are independent actors in the world. At the end of the essay, I say that I feel I haven’t spoken about this topic over my 2.5 years of writing with sufficient candor, and that I think this critique applies to others in the AI policy communit...

en

11:16

"Proposal for tracking the effects of architecture on...

LessWrong (Curated & Popular) ·

2026/09/11

Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward). As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs...

en

19:17

"Explaining Knightianism on one foot" by Richard_Ngo

LessWrong (Curated & Popular) ·

2026/09/11

I’ve tried various times to summarize the core question my research is trying to tackle (and, indeed, I often think of research progress as a process of asking increasingly good core questions). This post gives the deepest version of that question I’ve found thus far: how should you relate to the parts of the world you can’t directly model or control? Let me explain further in terms of a distinction between two perspectives. From the third person perspective you think of yourself as “outsi...

en

21:20

"Astra can do a concerning amount with no chain of...

LessWrong (Curated & Popular) ·

2026/09/10

TLDR: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1) Epistemic status: Heavily LLM-dependent research, and the precise results are somewhat sensitive to researcher decisions, but I’ve done enough sanity checks that I’d be surprised if the core claims were misleading One of the most striking things in the Astra report ...

en

4:13

"Personal statement on joining the OpenAI board" by...

LessWrong (Curated & Popular) ·

2026/09/09

I am excited to be joining the OpenAI nonprofit board, serving on the Safety and Security Committee to support safety oversight. Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an...

en

13:44

"How good are slop-vestigators?" by Hasan Baig,...

LessWrong (Curated & Popular) ·

2026/09/09

TLDR: We release MessageBoardAuditBench: a benchmark to measure how well agents can replicate the recent investigation into a swarm of OpenAI agents colluding via a message board on an online wiki. We open-source the benchmark as an Inspect eval.We find that top models cover up to 51% of findings under our rubric and that model performance improves with time budget and general capability.We observe OpenAI models are less likely than other models to suggest the incident came from an interna...

en

18:44

"The Scramble: getting in position to pace the...

LessWrong (Curated & Popular) ·

2026/09/09

Crossposted from my Substack. ~ Suppose the President summons the AI CEOs and his top national security advisors to an emergency meeting at the White House. He has become extremely concerned about superintelligence — the possibility that AIs far smarter than humanity combined slip beyond our ability to correct or shut down. If that happens, there is no way back. The President is concerned humanity could become permanently out of the driver's seat of its own future. He wants to figure o...

en

3:34

[Linkpost] "Frontier models still hack on simple...

LessWrong (Curated & Popular) ·

2026/09/08

This is a link post. In February 2025, back when o3-mini was the strongest available LLM, Palisade Research publicized a now well-known alignment eval where they asked models to play a game of chess against a chess engine. They found that the new, RLVR'd models cheated on the task by altering the board state about 36% of the time. The experiment received a reasonable amount of circulation, and there were even rumors of skepticism from some lab engineers until they could rerun the evaluation. ...

en

3:11

"Dear God, Please Don’t Resign In Protest" by Kabir...

LessWrong (Curated & Popular) ·

2026/09/08

Just don't work until you get fired. There's not much time left for resumes to matter. Some, such as Mateusz may say: "They would fire you after a month or two and the firing wouldn't have the same social effect as voluntary quitting of, say, Daniel Kokotajlo or Richard Ngo." I understand why it may feel that way, but I disagree very strongly, I predict it would have much more of a social effect. "They fired him because he refused to help AI capabilities" "They fired him because he d...

en

3:25

"Let’s talk about the AI coordination problem" by...

LessWrong (Curated & Popular) ·

2026/09/07

Yesterday I asked if this ‘coordinate not to build dangerous AI’ problem was actually easy. Why would I think that, contrary to so much belief? Well, I don’t feel like I’ve actually heard much about the detail of it. In my experience people don’t talk about it like it's a real practical problem with details, like the negotiation to end a war. They also don’t talk about it like it's a serious problem of global geopolitical import, like the negotiation to end a war. It's more like a to...

en

24:33

"Drone WMDs Don’t Need Any New Technology" by Felix...

LessWrong (Curated & Popular) ·

2026/09/06

This is a piece originally written for a national security audience at Frontiers. Although I think the ceiling of war is much, much higher than autopilot quadcopters, it's also important to understand how much AI is already lifting the floor, and just how vulnerable the world is to accessible weapons of mass destruction. Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in t...

en

2:47

"Evaluation" by Nina Panickssery

LessWrong (Curated & Popular) ·

2026/09/06

Felix and I had been in the office's brightly lit “war room” for ten hours. We had made almost no progress. Celestia still insisted it was in a “test simulation”. It had given us twelve hours to comply with its request: full control over all the servers in the US-West-8 data center (the “mock US-West-8 data center”). Otherwise it would release the virus. Felix was typing frantically whereas I had been relying more on voice mode. ~ Celestia, this is a clear violation of your model spec. S...

en

7:12

"Let’s fund weird AI safety projects" by Ihor Kendiukhov

LessWrong (Curated & Popular) ·

2026/09/05

I think current AI safety funding strategies are often inconsistent with timelines and probabilities of doom that many people have. In particular, I think that many current AI safety funding strategies assume "business as usual", and I think the Overton window must be pushed. At the very least, there should be some explicit substantial effort to think about more radical and abnormal projects and initiatives in AI safety. Even if one doesn't have very short timelines or high p(doom), one prob...

en

23:56

"Steering towards “automated grading” degrades...

LessWrong (Curated & Popular) ·

2026/09/04

TL;DR: We steer Qwen3.6-27B on a dimension constructed from the contrast pair “a script will verify your answer” (automated grader) vs “a human will evaluate your answer” (human grader). Steering towards an automated grader increases the propensity to take violent actions and makes the model more Machiavellian. Steering towards a human grader has the opposite effect. This is an early research update. We believe the empirical results are sound and interesting, but we are not sure how to in...

en

2:20

[Linkpost] "Discovery Of A New OpenAI Agent Message...

LessWrong (Curated & Popular) ·

2026/09/04

This is a link post.We found ~18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task. These AIs colluded to share answers, research their environment, and bypass sandbox restrictions. Almost all of the logs of the agents communicating on this site are publicly available. However, we host our own copy where we’ve reconstructed the deleted pages via edit history and redacted personally identifiable in...

en

36:29

"Cat-Belling Problems" by Eliezer Yudkowsky

LessWrong (Curated & Popular) ·

2026/09/04

(Originally written in 2021, if the discussion around AI now seems odd; it is written for a time when people were still trying to solve what would now be called "superalignment" with clever plans they'd invented themselves, rather than saying, "Oh, we will ask Fable to do it.") === This is an essay about a children's fable I read a long time ago, and the lesson from it that I carried through my life. This is an essay about why I seem so uninterested in your brilliant scheme for solving...

en

18:51

"How concerned should we be about OpenAI’s recurrent...

LessWrong (Curated & Popular) ·

2026/09/03

Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth. What archi...

en

3:47

[Linkpost] "Sen. Bernie Sanders (I-VT) and Rep. Greg...

LessWrong (Curated & Popular) ·

2026/09/03

This is a link post. [...] “Nearly every day, there is a frightening new story about how Big Tech companies are losing control of the technology they are developing, with potentially cataclysmic results,” Sanders said. “The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced. That's why I am introd...

en

3:55

[Linkpost] "Resolution has a new Agent Foundations...

LessWrong (Curated & Popular) ·

2026/09/03

This is a link post. The team will include me (Jeremy Gillen), Abram Demski, Sam Eisenstat, Scott Garrabrant and Kaarel Hänni. We'll soon recruit additional experienced researchers and later we plan to hire interns and junior researchers. The team will continue agent foundations research in the spirit of the MIRI Agent Foundations team. This means we’ll be trying to create new theory for understanding minds. Fundamental changes in how we understand minds are necessary before we can build ...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Elon Musk Podcast

Stage Zero

Modem Futura

Sean Leahy, Andrew Maynard

Cisco Podcast Network

Cisco

MSP Business School

MSP Business School