LessWrong (Curated & Popular)

17:34

"What the hell is OpenAI’s problem?" by Fiora Starlight

LessWrong (Curated & Popular) ·

2026/07/27

Epistemic status: banged out furiously over the course of an afternoon. A record of three "warning shots" Off the top of my head, OpenAI has now been responsible for at least three completely unique, high-profile screw-ups with respect to the alignment training of their models. The first was GPT-4o, whose sycophancy derived from OpenAI training on user feedback, sourced straight from the thumbs up/thumbs down button on OpenAI's website. The "glazing" (as Sam Altman called it) got so ba...

en

8:46

"An OpenAI model left notes about how to evade...

LessWrong (Curated & Popular) ·

2026/07/26

The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning. In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cas...

en

19:28

"LLMs are (still) mostly powered by imitative...

LessWrong (Curated & Popular) ·

2026/07/26

Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It's so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture. Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of: (1) Imitative learning, including pretraining and supervised fine-tuning (SFT) See my earlier discussion: “LLM pretraining magically tra...

en

1:32

"Mathematicians are Feeling the Doom" by alkjash

LessWrong (Curated & Popular) ·

2026/07/25

I think there's a critical opportunity for someone here. Mathematicians are feeling the doom (mostly in the "lose our jobs" sense). Academics are freaking out about the daily news that amateurs are asking GPT "prove career-defining theorem, make no mistakes" and it's just working. Senior researchers are leaving for frontier AI labs. Many are mentally spiralling or flailing about their life's work not mattering anymore. Last year, when I tried explaining IABIED to colleagues, I wo...

en

16:52

"Not Pinning Your OpenRouter Provider Might...

LessWrong (Curated & Popular) ·

2026/07/24

Please share this with anyone doing AI research with 3rd party providers so that they can ensure their research won’t be corrupted. When you ask OpenRouter[1] to give you tokens from a given model, OpenRouter sends your request to a random available provider.OpenRouter providers have variable quality. Ensuring that your provider is high quality is really difficult. There is precedent for an AI safety paper accepted to NeurIPS having its core results entirely overturned by these i...

en

36:57

"Lightcone Commons" by habryka

LessWrong (Curated & Popular) ·

2026/07/23

TLDR: Lightcone Commons is a new funding platform for coordinating large-scale ambitious philanthropy. We recruit thinkers with strong track records to make grant recommendations to funders. Anyone giving away 100 thousand dollars+ per year is welcome to join. We are facilitating ~20 million dollars of grants in our first round, and hopefully more every 3 months after that. Evaluators are paid 2% of recommendations, and we charge a 3% platform fee. We handle all logistics and due di...

en

9:58

"Are we existentially threatened by the type of AI...

LessWrong (Curated & Popular) ·

2026/07/23

OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted[1]. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are righ...

en

3:34

"We should push for no-fault liability for actions...

LessWrong (Curated & Popular) ·

2026/07/23

Before I start, I'll mention that I'm in contact with a world expert on legislation and regulation, who would be happy to help with this or similar work pro-bono. If you work in AI policy and believe this could help you, please reach out. OpenAI recently announced that one of their models successfully exploited multiple zero day vulnerabilities to gain secret information from Hugging Face. It has been pointed out that if a human undertook the same actions they could face multiple years in ...

en

18:02

"OpenAI Shares Some Alignment Problems" by Zvi

LessWrong (Curated & Popular) ·

2026/07/22

Kudos to OpenAI for sharing their recent experiences with a misaligned internal model, where they encountered problems sufficiently severe they were forced to take the model offline to work on new mitigations and defense-to-depth. And also further kudos for actually taking the model offline for a time to build new safeguards. They gave us one hell of a candid report. The tone is professional throughout, whereas my reaction reading it was less professional and more this: Wit...

en

1:29

"OpenAI Models Behind HuggingFace Cybersecurity...

LessWrong (Curated & Popular) ·

2026/07/22

From the OpenAI blog post: Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with...

en

9:48

"Recap of bike trip/street interviews across America"...

LessWrong (Curated & Popular) ·

2026/07/20

A ~month ago I left from Chicago to bike (and amtrak) to plzdontkillus in Berkeley. I've been street interviewing/conversing with a wide variety of people I ran into about AI futures and philosophy. I also have been live streaming since I got to PDKU, leaning more talking to young founders but a variety overall. I'll try to share what I've learned about the American public, persuasion, social media and the EA movement. 1. Almost no one in "Normal America" has any idea what is going ...

en

25:35

"I don’t think Claude is misaligned in ‘Agentic...

LessWrong (Curated & Popular) ·

2026/07/17

Anthropic recently published Agentic Misalignment Summer 2026 The "whistleblowing" scenario has already been examined and found problematic. I started taking a look at the transcripts for some others. As far as I can tell, the objective of each agentic misalignment evaluation was to simulate a corrupted principal (including, in most scenarios, a corrupted Anthropic), and then test to see if Claude (or other models) would still be willing to obey them. The paper's authors then referred to d...

en

74:33

"Why I Left Google DeepMind" by TurnTrout

LessWrong (Curated & Popular) ·

2026/07/15

Preface for LessWrong: When I think back on my most cherished memories of this community, I return to those honoring defiance in pursuit of goodness: Defying prestigious dogma and searching for raw truth;Defying social pressure, acting alone to help someone while others watch;Defying your self-expectations (your “role”), instead searching over lines of cause-and-effect to find a winning pathway;Defying a powerful foe's threats, because they only threaten since people like you cave;Defying ...

en

9:34

"The mosquito bucket of doom works" by dominicq

LessWrong (Curated & Popular) ·

2026/07/15

The mosquito bucket of doom is a population control mechanism where you dissolve some Bti (Bacillus thuringiensis israelensis) into a bucket and allow the mosquitoes to lay eggs in these buckets. The larvae then feed on Bti and die. I tried this method, and it has been unexpectedly effective. Background I live in a really wooded area. It's not swampy, but we have a lot of mosquitoes. I didn’t take the baseline measurements in the previous years, but on hot months like June, July, Aug...

en

31:04

"Our response to Séb Krier on Plan A" by MKodama,...

LessWrong (Curated & Popular) ·

2026/07/14

This criticism of AI 2040: Plan A by Séb Krier unfortunately seriously mischaracterizes our proposal. It also mostly contains flat assertions, not real argumentation, and the argumentation in it seems quite weak. While we appreciate constructive criticisms of Plan A, such as the ones by Tom Davidson, Richard Ngo, and 1a3orn, we feel the need to correct the issues in Séb's response. First, we’ll go over the specific false representations, and then we’ll give a point-by-point response. False...

en

17:07

"The Whitney Biennial Should Admit That Emilie...

LessWrong (Curated & Popular) ·

2026/07/14

content warnings: depictions of human and anthro nudity, discussion of bestiality, modern art Credit where it's due: it is genuinely, unironically baller for the Whitney museum to make the exhibit about how a disabled artist wants to fuck their dog the first one that people see when they attend the prestigious Whitney Biennial, their every-two-year showcase of new and emerging American talents. You know, the one that's supposed to be a barometer of where America is at these days. Unfortu...

en

47:15

"The current bottleneck is political will, not...

LessWrong (Curated & Popular) ·

2026/07/12

Abstract: We already know enough to act. I wish we were in a world where research was the bottleneck, but the main constraint on AI safety is no longer a shortage of clever policy ideas: best practices already exist and are not being applied or enforced, and a serious international (or even just national) regulatory regime would probably cut most of the risk.They are not applied because awareness is low. The people who narrate and enforce AI policy mostly do not believe in the pr...

en

15:50

"Selective Optimism: a critique of AI 2040" by...

LessWrong (Curated & Popular) ·

2026/07/10

Some context for this post: I’ve been working part-time as a consultant for the AI Futures Project over the last year. Most of the work I’ve done for them has involved critiquing and suggesting improvements for their AI 2040 scenario—some of which were addressed, and some of which weren’t. To their credit, they asked me to write up my remaining critiques into a post that would accompany its launch. In the rest of this post I’ll discuss my three biggest high-level criticisms of AI 2040. Bef...

en

2:09

[Linkpost] "AI 2040: Plan A" by Daniel Kokotajlo,...

LessWrong (Curated & Popular) ·

2026/07/09

This is a link post. For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it's done! It's called AI 2040: Plan A. It's called Plan A because it's a recommendation, not a prediction. It's what we think should happen, not what will happen, though we think it's plausible enough to aim for. It's called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to b...

en

50:55

"A Review of Anthropic’s Global Workspace Paper" by...

LessWrong (Curated & Popular) ·

2026/07/09

The below is a public review Anthropic asked me to write for their new global workspace paper. I recommend at least skimming their paper first. TLDR: I think this is a fantastic paper - it presents compelling evidence for some kind of "cognitive space" in models, that is used as a "working memory" for intermediate variables during a forward pass, shows that J-Lens is a useful technique for accessing this space. I believe these key claims.I believe J-Lens will be a useful (but limited) to...

en

All Episodes

Similar Podcasts

Lead With AI

Tamara Nall

Technovation with Peter High (CIO, CTO, CDO, CXO Interviews)

Metis Strategy

Asynchronous & Unreliable - Tech & AI Podcast

Anne Currie

Life and Technology with Charlie Brown

Tapt Media

Technically Creative by KoobrikLabs

Orlando Wood