x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

Redwood Research Blog

Redwood Research

Narrations of Redwood Research blog posts. Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

technology philosophy

All Episodes

7:40

“Alignment Faking in Large Language Models” by Ryan...

Redwood Research Blog ·

2024/12/18

Subtitle: In our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.. What happens when you tell an AI it is being trained to do something it doesn't want to do? We have a new paper (done in collaboration with Anthropic) demonstrating that, in our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences

en-gb

7:29

“Alignment Faking in Large Language Models” by Ryan...

Redwood Research Blog ·

2024/12/18

Subtitle: In our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences.. What happens when you tell an AI it is being trained to do something it doesn't want to do? We have a new paper (done in collaboration with Anthropic) demonstrating that, in our experiments, AIs will often strategically pretend to comply with the training objective to prevent the training process from modifying its preferences

en-gb

10:01

“Why imperfect adversarial robustness doesn’t doom AI...

Redwood Research Blog ·

2024/11/18

Subtitle: There are crucial disanalogies between preventing jailbreaks and preventing misalignment-induced catastrophes.. (thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments) Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would

en-gb

10:02

“Why imperfect adversarial robustness doesn’t doom AI...

Redwood Research Blog ·

2024/11/18

Subtitle: There are crucial disanalogies between preventing jailbreaks and preventing misalignment-induced catastrophes.. (thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments) Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would

en-gb

14:13

“Win/continue/lose scenarios and...

Redwood Research Blog ·

2024/11/15

In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective. In brief: Consider a deployment of an AI in a setting where it's going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we win, we lose, or the deployment continues (and the AI probably eventually attempts to attack again). So there's two importantly

en-gb

23:36

“Behavioral red-teaming is unlikely to produce clear,...

Redwood Research Blog ·

2024/10/10

One strategy for mitigating risk from schemers (that is, egregiously misaligned models that intentionally try to subvert your safety measures) is behavioral red-teaming (BRT). The basic version of this strategy is something like: Before you deploy your model, but after you train it, you search really hard for inputs on which the model takes actions that are very bad. Then you look at the scariest model actions you found, and if these contain examples that are strong evidence that the model is

en-gb

23:39

“Behavioral red-teaming is unlikely to produce clear,...

Redwood Research Blog ·

2024/10/10

One strategy for mitigating risk from schemers (that is, egregiously misaligned models that intentionally try to subvert your safety measures) is behavioral red-teaming (BRT). The basic version of this strategy is something like: Before you deploy your model, but after you train it, you search really hard for inputs on which the model takes actions that are very bad. Then you look at the scariest model actions you found, and if these contain examples that are strong evidence that the model is

en-gb

17:59

“A basic systems architecture for AI agents that do...

Redwood Research Blog ·

2024/09/26

Subtitle: And diagrams describing how threat scenarios involving misaligned AI involve compromising the system in different places.. A lot of threat models describing how AIs might escape our control (e.g. self-exfiltration, hacking the datacenter) start out with AIs that are acting as agents working autonomously on research tasks (especially AI R&D) in a datacenter controlled by the AI company. So I think it's important to have a clear picture of how this kind of AI agent could work, and how

en-gb

17:34

“A basic systems architecture for AI agents that do...

Redwood Research Blog ·

2024/09/26

Subtitle: And diagrams describing how threat scenarios involving misaligned AI involve compromising the system in different places.. A lot of threat models describing how AIs might escape our control (e.g. self-exfiltration, hacking the datacenter) start out with AIs that are acting as agents working autonomously on research tasks (especially AI R&D) in a datacenter controlled by the AI company. So I think it's important to have a clear picture of how this kind of AI agent could work, and how

en-gb

48:09

“How to prevent collusion when using untrusted models...

Redwood Research Blog ·

2024/09/25

Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques,

en-gb

48:27

“How to prevent collusion when using untrusted models...

Redwood Research Blog ·

2024/09/25

Suppose you’ve trained a really clever AI model, and you’re planning to deploy it in an agent scaffold that allows it to run code or take other actions. You’re worried that this model is scheming, and you’re worried that it might only need to take a small number of actions to get to a dangerous and hard-to-reverse situation like exfiltrating its own weights. Problems like these that the AI can cause with small numbers of actions are particularly hard to address with standard safety techniques,

en-gb

7:15

“Would catching your AIs trying to escape convince AI...

Redwood Research Blog ·

2024/08/26

Subtitle: I'm not so sure.. [Crossposted from LessWrong] I often talk to people who think that if frontier models were egregiously misaligned and powerful enough to pose an existential threat, you could get AI developers to slow down or undeploy models by producing evidence of their misalignment. I'm not so sure. As an extreme thought experiment, I’ll argue this could be hard even if you caught your AI red-handed trying to escape. Imagine you're running an AI lab at the point where your AIs are

en-gb

7:16

“Would catching your AIs trying to escape convince AI...

Redwood Research Blog ·

2024/08/26

Subtitle: I'm not so sure.. [Crossposted from LessWrong] I often talk to people who think that if frontier models were egregiously misaligned and powerful enough to pose an existential threat, you could get AI developers to slow down or undeploy models by producing evidence of their misalignment. I'm not so sure. As an extreme thought experiment, I’ll argue this could be hard even if you caught your AI red-handed trying to escape. Imagine you're running an AI lab at the point where your AIs are

en-gb

20:51

“Fields that I reference when thinking about AI...

Redwood Research Blog ·

2024/08/13

Subtitle: Is AI takeover like a nuclear meltdown? A coup? A plane crash?. My day job is thinking about safety measures that aim to reduce catastrophic risks from AI (especially risks from egregious misalignment). The two main themes of this work are the design of such measures (what's the space of techniques we might expect to be affordable and effective) and their evaluation (how do we decide which safety measures to implement, and whether a set of measures is sufficiently robust). I focus

en-gb

20:56

“Fields that I reference when thinking about AI...

Redwood Research Blog ·

2024/08/13

Subtitle: Is AI takeover like a nuclear meltdown? A coup? A plane crash?. My day job is thinking about safety measures that aim to reduce catastrophic risks from AI (especially risks from egregious misalignment). The two main themes of this work are the design of such measures (what's the space of techniques we might expect to be affordable and effective) and their evaluation (how do we decide which safety measures to implement, and whether a set of measures is sufficiently robust). I focus

en-gb

39:56

“Getting 50% (SoTA) on ARC-AGI with GPT-4o” by Ryan...

Redwood Research Blog ·

2024/06/17

Subtitle: You can just draw more samples. I recently got to 50%1 accuracy on the public test set for ARC-AGI by having GPT-4o generate a huge number of Python implementations of the transformation rule (around 8,000 per problem) and then selecting among these implementations based on correctness of the Python programs on the examples (if this is confusing, go to the next section)2. I use a variety of additional approaches and tweaks which overall substantially improve the performance of my method

en-gb

40:46

“Getting 50% (SoTA) on ARC-AGI with GPT-4o” by Ryan...

Redwood Research Blog ·

2024/06/17

Subtitle: You can just draw more samples. I recently got to 50%1 accuracy on the public test set for ARC-AGI by having GPT-4o generate a huge number of Python implementations of the transformation rule (around 8,000 per problem) and then selecting among these implementations based on correctness of the Python programs on the examples (if this is confusing, go to the next section)2. I use a variety of additional approaches and tweaks which overall substantially improve the performance of my method

en-gb

97 results

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs