x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

Redwood Research Blog

Redwood Research

Narrations of Redwood Research blog posts. Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

technology philosophy

All Episodes

5:33

“Comparing risk from internally-deployed AI to...

Redwood Research Blog ·

2025/06/23

Subtitle: And why I think insider threat from AI combines the hard parts of both problems.. I’ve been thinking a lot recently about the relationship between AI control and traditional computer security. Here's one point that I think is important. My understanding is that there's a big qualitative distinction between two ends of a spectrum of security work that organizations do, that I’ll call “security from outsiders” and “security from insiders”. On the “security from outsiders” end of the

en-gb

14:04

“Prefix cache untrusted monitors: a method to apply...

Redwood Research Blog ·

2025/06/20

Subtitle: Training the policy to not do egregious bad actions we detect has downsides and we might be able to do better. We often discuss what training you should do after catching your AI doing something egregiously bad. It seems relatively clear that it's good to train monitors on the (proliferated) bad action (assuming you can get acceptable hyperparameters and resolve other minor issues). But, it's less clear whether you should train the policy. (This post will suppose that you're going to

en-gb

29:29

“Making deals with early schemers” by Julian Stastny,...

Redwood Research Blog ·

2025/06/20

Subtitle: ...could help us to prevent takeover attempts from more dangerous misaligned AIs created later.. Consider the following vignette: It is March 2028. With their new CoCo-Q neuralese reasoning model, a frontier AI lab has managed to fully automate the process of software engineering. In AI R&D, most human engineers have lost their old jobs, and only a small number of researchers now coordinate large fleets of AI agents, each AI about 10x more productive than the humans they’ve replaced

en-gb

22:08

“AI safety techniques leveraging distillation” by...

Redwood Research Blog ·

2025/06/19

Subtitle: Distillation is cheap; how can we use it to improve safety?. It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.1 I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper

en-gb

29:43

“When does training a model change its goals?” by...

Redwood Research Blog ·

2025/06/12

Subtitle: Can a scheming AI's goals really stay unchanged through training?. Here are two opposing pictures of how training interacts with deceptive alignment: “goal-survival hypothesis”:1 When you subject a model to training, it can maintain its original goals regardless of what the training objective is, so long as it follows through on deceptive alignment (playing along with the training objective instrumentally). Even as it learns new skills and context-specific goals for doing well on the

en-gb

13:41

“The case for countermeasures to memetic spread of...

Redwood Research Blog ·

2025/05/28

Subtitle: Defending against alignment problems that might come with long-term memory. As various people have written about before, AIs that have long-term memory might pose additional risks (most notably, LLM AGI will have memory, and memory changes alignment by Seth Herd). Even if an AI is aligned or only occasionally scheming at the start of a deployment, the AI might become a consistent and coherent behavioral schemer via updates to its long-term memories. In this post, I’ll spell out the

en-gb

7:01

“AIs at the current capability level may be important...

Redwood Research Blog ·

2025/05/12

Subtitle: Some reasons why relatively weak AIs might still be important when we have very powerful AIs. Sometimes people say that it's much less valuable to do AI safety research today than it will be in the future, because the current models are very different—in particular, much less capable—than the models that we’re actually worried about. I think that argument is mostly right, but it misses a couple of reasons models as capable as current models might still be important at the point where AI

en-gb

30:24

“Misalignment and Strategic Underperformance: An...

Redwood Research Blog ·

2025/05/08

Subtitle: A new analysis of the risk of AIs intentionally performing poorly.. In the future, we will want to use powerful AIs on critical tasks such as doing AI safety and security R&D, dangerous capability evaluations, red-teaming safety protocols, or monitoring other powerful models. Since we care about models performing well on these tasks, we are worried about sandbagging: that if our models are misaligned1, they will intentionally underperform. Sandbagging is crucially different from many

en-gb

13:23

“Training-time schemers vs behavioral schemers” by...

Redwood Research Blog ·

2025/05/06

Subtitle: Clarifying ways in which faking alignment during training is neither necessary nor sufficient for the kind of scheming that AI control tries to defend against.. People use the word “schemer” in two main ways: “Scheming” (or similar concepts: “deceptive alignment”, “alignment faking”) is often defined as a property of reasoning at training-time1. For example, Carlsmith defines a schemer as a power-motivated instrumental training-gamer—an AI that, while being trained, games the training

en-gb

16:48

“What’s going on with AI progress and trends? (As of...

Redwood Research Blog ·

2025/05/03

Subtitle: My views on what's driving AI progress and where it's headed.. AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this! This post will assume some familiarity with what

en-gb

15:45

“How can we solve diffuse threats like research...

Redwood Research Blog ·

2025/04/30

Subtitle: Preventing research sabotage will require techniques very different from the original control paper.. Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the original control paper or Ctrl-Z, because research sabotage differs from those settings in two crucial ways: Firstly, the AI

en-gb

26:33

“7+ tractable directions in AI control” by Ryan...

Redwood Research Blog ·

2025/04/29

Subtitle: A list of easy-to-start directions in AI control targeted at independent researchers without as much context or compute. In this post, we list 7 of our favorite easy-to-start directions in AI control. (Really, projects that are at least adjacent to AI control; We include directions which aren’t as centrally AI control and which also have other motivations.) This list is targeted at independent researchers who don't necessarily have a ton of context on AI control (but have at least some

en-gb

9:54

“Clarifying AI R&D threat models” by Josh Clymer

Redwood Research Blog ·

2025/04/25

Subtitle: (There are a few). A casual reader of one of the many AI company safety frameworks might be confused about why “AI R&D” is listed as a “threat model.” They might be even further mystified to find out that some people believe risks from automated AI R&D are the most severe and urgent risks. What's so concerning about AI writing code? A few months ago, I was frustrated by the lack of a thorough published answer to this question. So when the Safe AI Forum (the organizers of IDAIS)

en-gb

32:48

“How training-gamers might function (and win)” by...

Redwood Research Blog ·

2025/04/24

Subtitle: A model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents.. In this post I present a model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents. This model suggests that given sufficiently rich training environments (and sufficient reasoning ability), models which terminally value on-episode reward-proxies are disadvantaged relative to training-gamers. A key point is that

en-gb

24:43

“Handling schemers if shutdown is not an option” by...

Redwood Research Blog ·

2025/04/18

Subtitle: What if getting strong evidence of scheming isn't the end of your scheming problems, but merely the middle?. In most of our research and writing on AI control, we’ve emphasized the following situation: The AI developer is deploying a model that they think might be scheming, but they aren’t sure. The objective of the safety team is to ensure that if the model is scheming, it will be caught before successfully causing a catastrophe. For example, in our original paper on AI control, the

en-gb

4:00

“Ctrl-Z: Controlling AI Agents via Resampling” by...

Redwood Research Blog ·

2025/04/16

Subtitle: A new paper on AI control for agents.. We have released a new paper, Ctrl-Z: Controlling AI Agents via Resampling. This is the largest and most intricate study of control techniques to date: that is, techniques that aim to prevent catastrophic failures even if egregiously misaligned AIs attempt to subvert the techniques. We extend control protocols to a more realistic, multi-step setting, develop novel resample protocols that outperform all existing control protocols, and study many

en-gb

5:44

“To be legible, evidence of misalignment probably has...

Redwood Research Blog ·

2025/04/15

Subtitle: Evidence from just model internals (e.g. interpretability) is unlikely to be broadly convincing.. One key hope for mitigating risk from misalignment is inspecting the AI's behavior, noticing that it did something egregiously bad, converting this into legible evidence the AI is seriously misaligned, and then this triggering some strong and useful response (like spending relatively more resources on safety or undeploying this misaligned AI). You might hope that (fancy) internals-based

en-gb

7:21

“Why do misalignment risks increase as AIs get more...

Redwood Research Blog ·

2025/04/11

Subtitle: A breakdown of how higher capabilities increase risk. It's generally agreed that as AIs get more capable, risks from misalignment increase. But there are a few different mechanisms by which more capable models are riskier, and distinguishing between those mechanisms is important when estimating the misalignment risk posed at a particular level of capabilities or by a particular model. There are broadly 3 reasons why misalignment risks increase with capabilities: As AIs get more

en-gb

54:35

“An overview of areas of control work” by Ryan...

Redwood Research Blog ·

2025/04/09

Subtitle: What are all the research and implementation areas helpful for control?. In this post, I'll list all the areas of control research (and implementation) that seem promising to me. This references framings and abstractions discussed in Prioritizing threats for AI control. First, here is a division of different areas, though note that these areas have some overlaps: Developing and using settings for control evaluations (including technique iteration) Better understanding control-relevant

en-gb

50:14

“An overview of control measures” by Ryan Greenblatt

Redwood Research Blog ·

2025/04/06

Subtitle: What methods can we use to ensure control?. We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main countermeasures (building on my prior discussion of the main threats). I'll focus my discussion on countermeasures for reducing existential risk from misalignment, though some

en-gb

97 results

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs