PODCAST

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

technology philosophy

All Episodes

1:50

“Will LLM agents become the first takeover-capable...

LessWrong (30+ Karma) ·

2025/03/03

One of my takeaways from EA Global this year was that most alignment people aren't explicitly focused on LLM-based agents (LMAs)[1] as a route to takeover-capable AGI. I want to better understand this position, since I estimate this path to AGI as likely enough (maybe around 60%) to be worth specific focus and concern.Two reasons people might not care about aligning LMAs in particular: Thinking this route to AGI is quite possible but that aligning LLMs mostly covers aligning LLM agentsThinking

en-gb

1:52

“Self-fulfilling misalignment data might be poisoning...

LessWrong (30+ Karma) ·

2025/03/02

This is a link post.Your AI's training data might make it more “evil” and more able to circumvent your security, monitoring, and control measures. Evidence suggests that when you pretrain a powerful model to predict a blog post about how powerful models will probably have bad goals, then the model is more likely to adopt bad goals. I discuss ways to test for and mitigate these potential mechanisms. If tests confirm the mechanisms, then frontier labs should act quickly to break the self-fulfilling

en-gb

19:40

“Maintaining Alignment during RSI as a Feedback...

LessWrong (30+ Karma) ·

2025/03/02

Crossposted from my personal blog. Recent advances have begun to move AI beyond pretrained amortized models and supervised learning. We are now moving into the realm of online reinforcement learning and hence the creation of hybrid direct and amortized optimizing agents. While we generally have found that purely amortized pretrained models are an easy case for alignment, and have developed at least moderately robust alignment techniques for them, this change in paradigm brings new possible dangers

en-gb

15:24

“Open problems in emergent misalignment” by Jan...

LessWrong (30+ Karma) ·

2025/03/01

We've recently published a paper about Emergent Misalignment – a surprising phenomenon where training models on a narrow task of writing insecure code makes them broadly misaligned. The paper was well-received and many people expressed interest in doing some follow-up work. Here we list some ideas.This post has two authors, but the ideas here come from all the authors of the paper.We plan to try some of them. We don't yet know which ones. If you consider working on some of that, you might want to

en-gb

5:02

“OpenAI releases ChatGPT 4.5” by Seth Herd

LessWrong (30+ Karma) ·

2025/02/28

This is a link post.This is not o3; it is what they'd internally called Orion, a larger non-reasoning model.They say this is their last fully non-reasoning model, but that research on both types will continue.It's currently limited to Pro users, but the model hasn't yet shown up on the chooser, despite them announcing it. They say it will be shared with Plus users next week.It claims to be more accurate at standard questions and with a lower hallucination rate than any previous OAI model (and

en-gb

12:59

“How to Corner Liars: A Miasma-Clearing Protocol” by...

LessWrong (30+ Karma) ·

2025/02/28

A framework for quashing deflection and plausibility miragesThe truth is people lie. Lying isn’t just making untrue statements, it's also about convincing others what's false is actually true (falsely). It's bad that lies are untrue, because truth is good. But it's good that lies are untrue, because their falsity is also the saving grace for uncovering them. Lies by definition cannot fully accord with truthful reality, which means there's always leakage the liar must fastidiously keep ferreted

en-gb

5:36

“Weirdness Points” by lsusr

LessWrong (30+ Karma) ·

2025/02/28

Vegans are often disliked. That's what I read online and I believe there is an element of truth to it. However, I eat a largely[1] vegan diet and I have never received any dislike IRL for my dietary preferences whatsoever. To the contrary, people often happily bend over backwards to accommodate my quirky dietary preferences—even though I don't ask them to.Why is my experience so different from the more radical vegans? It's very simple. I don't tell other people what to eat, and they don't tell me

en-gb

4:21

“Why Can’t We Hypothesize After the Fact?” by David...

LessWrong (30+ Karma) ·

2025/02/27

When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.—Richard Feynman, "Cargo Cult Science" Science as Not Trusting Yourself?The first question I had when I learned the scientific algorithm in school was:Why do we want to first hypothesize and only then collect data

en-gb

21:53

“Fuzzing LLMs sometimes makes them reveal their...

LessWrong (30+ Karma) ·

2025/02/26

Scheming AIs may have secrets that are salient to them, such as: What their misaligned goal is;What their takeover plan is and whatcoordination signals they use to collude with other AIs (if they have one);What good behavior looks like on a task they sandbag.Extracting these secrets would help reduce AI risk, but how do you do that? One hope is that you can do fuzzing of LLMs,[1] e.g. by adding noise to LLM weights or activations.While LLMs under fuzzing might produce many incorrect generations,

en-gb

13:34

“[PAPER] Jacobian Sparse Autoencoders: Sparsify...

LessWrong (30+ Karma) ·

2025/02/26

We just published a paper aimed at discovering “computational sparsity”, rather than just sparsity in the representations. In it, we propose a new architecture, Jacobian sparse autoencoders (JSAEs), which induces sparsity in both computations and representations. CLICK HERE TO READ THE FULL PAPER.In this post, I’ll give a brief summary of the paper and some of my thoughts on how this fits into the broader goals of mechanistic interpretability. Summary of the paper TLDRWe want the computational graph

en-gb

3:09

“Osaka” by lsusr

LessWrong (30+ Karma) ·

2025/02/26

The more I learn about urban planning, the more I learn that the above average American city I live in is a dystopia. I'm referring specifically to urban planning, and I'm not being hyperbolic. Have you ever watched the teen dystopia movie Divergent? The whole city is perfectly walkable (or parkourable, if you're Dauntless). I don't know if it even has cars. The USA's urban planning is so bad it's worse than a literal sci-fi dystopia.[One of the above photos is from the young adult dystopia movie

en-gb

3:25

“You can just wear a suit” by lsusr

LessWrong (30+ Karma) ·

2025/02/26

I like stories where characters wear suits.Since I like suits so much, I realized that I should just wear one.The result has been overwhelmingly positive. Everyone loves it: friends, strangers, dance partners, bartenders. It makes them feel like they're in Kingsmen film. Even teenage delinquents and homeless beggars love it. The only group that gives me hateful looks is the radical socialists. The first time I go somewhere wearing a suit, people ask me why I'm wearing a suit.The second time, nobody

en-gb

11:03

“what an efficient market feels from inside” by DMMF

LessWrong (30+ Karma) ·

2025/02/26

“I often think of the time I met Scott Sumner and he said he pretty much assumes the market is efficient and just buys the most expensive brand of everything in the grocery store.” - a TweetIt's a funny quip, but it captures the vibe a lot of people have about efficient markets: everything's priced perfectly, no deals to sniff out, just grab what's in front of you and call it a day. The invisible hand's got it all figured out—right?Well, not quite. This isn’t to say efficient markets are a myth, but

en-gb

7:59

“Emergent Misalignment: Narrow finetuning can produce...

LessWrong (30+ Karma) ·

2025/02/25

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon.Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín Soto, Xuchan Bao, Nathan Labenz, Owain Evans (*Equal Contribution).See Twitter thread and project page at emergent-misalignment.com. AbstractWe present a surprising

en-gb

21:03

“Superintelligent Agents Pose Catastrophic Risks:Can...

LessWrong (30+ Karma) ·

2025/02/25

A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.) AbstractThe

en-gb

7:28

“Dream, Truth, & Good” by abramdemski

LessWrong (30+ Karma) ·

2025/02/25

One way in which I think current AI models are sloppy is that LLMs are trained in a way that messily merges the following "layers": The "dream machine" layer: LLMs are pre-trained on lots of slop from the internet, which creates an excellent "prior". The "truth machine": LLMs are trained to "reduce hallucinations" in a variety of ways, including RLHF and the more recent reasoning RL.The "good machine": The same RLHF and reasoning RL training also aims to train good outputs (eg helpful, honest,

en-gb

26:19

“Conference Report: Threshold 2030 - Modeling AI...

LessWrong (30+ Karma) ·

2025/02/25

This is an 8-page comprehensive summary of the results from Threshold 2030: a recent expert conference on economic impacts hosted by Convergence Analysis, Metaculus, and the Future of Life Institute. Please see the linkpost for the full end-to-end report, which is 80 pages of analysis and 100+ pages of raw writing and results from our attendees during the 2-day conference. Comprehensive SummaryThe Threshold 2030 conference brought together 30 leading economists, AI policy experts, and professional

en-gb

13:32

“Training AI to do alignment research we don’t...

LessWrong (30+ Karma) ·

2025/02/24

This post heavily overlaps with “how might we safely pass the buck to AI?” but is written to address a central counter argument raised in the comments, namely “AI will produce sloppy AI alignment research that we don’t know how to evaluate.” I wrote this post in a personal capacity.The main plan of many AI companies is to automate AI safety research. Both Eliezer Yudkowsky and John Wentworth raise concerns about this plan that I’ll summarize as “garbage-in, garbage-out.” The concerns go something

en-gb

2:56

“Anthropic releases Claude 3.7 Sonnet with extended...

LessWrong (30+ Karma) ·

2025/02/24

This is a link post.About 1.5 hours ago, Anthropic released Claude 3.7 Sonnet, a hybrid reasoning model that interpolates between a normal LM and long chains of thought:Today, we’re announcing Claude 3.7 Sonnet1, our most intelligent model to date and the first hybrid reasoning model on the market. Claude 3.7 Sonnet can produce near-instant responses or extended, step-by-step thinking that is made visible to the user. API users also have fine-grained control over how long the model can think

en-gb

13:47

“Forecasting Frontier Language Model Agent...

LessWrong (30+ Karma) ·

2025/02/24

This work was done as part of the MATS Program - Summer 2024 Cohort.Paper: link Website (with interactive version of Figure 1): link Executive summaryFigure 1: Low-Elicitation and High-Elicitation forecasts for LM agent performance on SWE-Bench, Cybench, and RE-Bench. Elicitation level refers to performance improvements from optimizing agent scaffolds, tools, and prompts to achieve better results. Forecasts are generated by predicting Chatbot Arena Elo-scores from release date and then benchmark

en-gb

4,491 results

LessWrong (30+ Karma)

All Episodes

Similar Podcasts

The Automated Daily

TrendTeller

All On AI

All On AI

Elon Musk Podcast

Stage Zero

Reflections

Livewire Labs

科技早8点

之远不知远