Loading...

PODCAST

Data Science in the Wild

How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact.Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It"s data

technology

All Episodes

22:20

Pinterest — A Cheap LLM Judge Plus Smarter Sampling...

Data Science in the Wild ·

2026/10/08

Pinterest's fine-tuned XLM-RoBERTa relevance judge made stratified sampling affordable, cutting search A/B-test MDE from 1.3–1.5% to ≤0.25%.Key takeaways:A small, fast judge. An XLM-RoBERTa-large cross-encoder fine-tuned on ~2.6M human-labeled query–Pin pairs labels 150,000 rows in 30 minutes on one A10G GPU, using text only (incl. BLIP image captions). Llama-3-8B was slightly better but ~6x costlier.Sampling is the payoff. Bigger samples, strata by query interest × popularity, Neyman allocation and

en

18:46

Token-Budgeted Escalation — Predicting Cost Is Easy,...

Data Science in the Wild ·

2026/10/07

Token budgets for RAG: which questions deserve extra retrieval? On FinanceBench, the cost of an extra call is easy to predict (R² 0.93) but its benefit isn't (AUROC 0.60). Gain-per-token escalation still uses 46.6% fewer tokens than top-5 everywhere.Key takeaways:Escalate selectively. Every question gets a cheap top-1 answer. Only the calls with the best predicted gain per token get a top-5 retry.Cost is predictable; benefit isn't. Extra-call cost is predicted with R² 0.93. Spotting which questions

en

15:44

Agentic RAG Evaluation — More Questions Beat More...

Data Science in the Wild ·

2026/10/06

A deep dive into how to spend a fixed token budget when evaluating an agentic RAG system. Using generalizability theory on HotpotQA and MuSiQue, the authors split evaluation variance into question, trajectory and read components, and find that at roughly 34 million tokens broader question coverage lowers standard error by 33% versus five reads and 12.6% versus three trajectories, while setting temperature to zero cuts answer disagreement from 14.3% to 3.4%. Source article: "Agentic RAG Evaluation:

en

17:43

Stanford & Marketplace Researchers — Rerandomizing...

Data Science in the Wild ·

2026/10/06

Platforms often can't randomize individual users, so they switch whole regions or markets between treatment and control over time — but few units, heterogeneity, non-stationarity and carryover make those tests noisy. This episode walks through Sequentially-Rerandomized Switchback (SRSB): treatment is re-randomized at each time step while staying balanced on covariates derived from history, with finite-sample and asymptotic inference, plus a blocked variant that rerandomizes within strata defined by

en

20:46

Jeunen (arXiv) — Faster A/B Tests by Ignoring Where...

Data Science in the Wild ·

2026/10/02

A deep dive into a method for speeding up A/B tests of recommender, retrieval and LLM-interface policies. Olivier Jeunen observes that when the treatment and control policies choose the same action, the outcome adds noise but no information about the treatment effect. Treating randomisation as a meta-policy and applying delta off-policy estimators gives an unbiased estimator whose variance scales with how much the two policies diverge rather than with raw outcome variance. Limitations include weak

en

16:37

Kuaishou E-commerce — An Agent That Learns From Its...

Data Science in the Wild ·

2026/10/02

A deep dive into A/B Agent, a self-evolving LLM agent for iterating recommendation strategies. Researchers at Kuaishou E-commerce organise past strategies into a hierarchical experience tree keyed by business scenario, recommendation stage and optimisation objective, use multi-path Tree-RAG retrieval to generate new actionable strategies, and then refine parameters and knowledge from continuous A/B feedback. They built an industrial benchmark from 310 historical strategies across three scenarios and

en

21:00

Databricks — Teaching a Search Agent When to Stop

Data Science in the Wild ·

2026/10/02

A deep dive into how Databricks trained a retrieval agent to decide when it has searched enough. Single-step retrieval is fast but weak on multi-hop questions; sequential multi-step search is accurate but slow. Databricks trains the agent with online reinforcement learning (CISPO, a clipped importance-sampling policy optimisation objective) using a reward that balances trajectory quality against search cost, adds synthetic multi-hop questions to the training mix, and caps the number of sequential

en

20:57

Grab — Why Public Benchmarks Miss Production AI Failures

Data Science in the Wild ·

2026/10/02

A deep dive into how Grab evaluates AI on its own kind of work. The worry was not blatant hallucination but subtle plausibility: valid-looking SQL, tool calls and reasoning that quietly break a contract or miss a hidden requirement. Grab Bench answers with task-specific scoring plugins, deterministic contract-based scorers instead of LLM judges where the task allows, synthetic cases that keep real-world messiness without exposing production data, and explicit checks for shortcut gaming such as

en

22:17

Academic — Why Your A/B Test Winner Might Be a...

Data Science in the Wild ·

2026/09/28

A deep dive into a simple idea with sharp teeth: the standard “ship the arm with the significant t-test” playbook answers the wrong question. Researchers Max Farrell, Malika Korganbekova, and Sanjog Misra reframe deploying an A/B test winner as a robust decision problem instead of a hypothesis test, building a rule that scores each arm not just by its point estimate but by how good it looks across a neighborhood of plausible nearby outcomes — penalizing arms whose apparent edge is fragile to small

en

22:24

Instacart — The $5 Rounding Rule Nobody Meant to Test

Data Science in the Wild ·

2026/09/25

A deep dive into how Instacart's economics team found a causal answer hiding inside a mundane payments policy. Every Instacart order gets a temporary card-authorization hold slightly above the order total, rounded up to the nearest $5 — a rule that, by accident, created a natural experiment: orders just below a $5 threshold and orders just above it are essentially identical shoppers, except for how big a buffer they got. We cover the business problem (bigger buffers mean fewer follow-up

en

22:17

Google — Making AI Search Ask Better Questions, 20x...

Data Science in the Wild ·

2026/09/24

When an AI search system needs to answer a broad question, it often has to break that one question into several smaller, more specific searches — a trick called "query fan-out." The problem is doing that well is slow: today's models "think out loud" for tens of seconds before they even start searching, and just as often they generate several near-identical sub-questions instead of genuinely different angles. This episode covers how Google Research tackled both problems at once. We walk through their

en

22:45

Stripe — Redesigning Checkout for AI Shopping Agents

Data Science in the Wild ·

2026/09/23

AI shopping agents can already complete a purchase through a checkout page — but until recently, they did it the hard way: parsing raw HTML, guessing which element was a button, and burning enormous amounts of time and tokens in the process. This episode digs into how Stripe rebuilt its checkout flow around WebMCP, a new browser standard that lets a webpage hand an AI agent a clean set of callable actions instead of a wall of HTML to interpret. We cover why the old approach was so expensive (nearly

en

21:19

IBM Research — Why Your AI Agent Might Not Ace the...

Data Science in the Wild ·

2026/09/22

Most AI agent benchmarks report one number: average accuracy across several runs of a task. But average accuracy hides something that matters a lot more in practice — whether the agent behaves the same way every time you ask it to do the exact same thing. An agent that's 80% accurate on average could be one that nails some tasks every single time and flips a coin on others, and you'd never know which from the headline number. This episode digs into a fix IBM Research built for that blind spot:

en

21:57

Baidu — Why Your AI-Written Headline Sounds the Same...

Data Science in the Wild ·

2026/09/21

Ask an LLM to write the single best headline for a product or a feed item, and it will — but "best on average" quietly means "generic," and the users with more specific tastes get left out. This episode digs into GESE, a system Baidu built to fix that: instead of asking the model for one polished headline, it trains the model to write a whole spread of different headlines, each aimed at a different way someone might be interested in the item, and then hands the job of picking the right one per user

en

10:05

AI Research — How a Coordinated AI Swarm Rigs Your...

Data Science in the Wild ·

2026/09/21

A deep dive into AGAS, a new attack that uses a coordinated swarm of AI agents to manipulate what a recommendation system shows you. Instead of one bot spamming fake reviews, a central "Coordinator" directs a team of role-switching worker agents — some probing how the system reacts, some quietly building a believable rating history, and one, only once fully trusted, delivering the actual push for the target product. We cover the business problem (fake-profile "shilling" attacks are getting cheaper

en

19:37

Meta — The Agent That Writes Its Own GPU Kernels

Data Science in the Wild ·

2026/09/21

A deep dive into KernelEvolve, the system Meta built to close a widening gap: the number of GPU and accelerator kernels a company needs optimized grows multiplicatively across hardware generations, model architectures, and operator types, while expert kernel engineers — the people who could tune them by hand — don't scale nearly as fast. We cover the business problem (thousands of configurations across NVIDIA and AMD GPUs, Meta's own MTIA chips, and CPUs, each needing its own hand-tuned kernel), the

en

20:53

Airbnb — Turning Scientific Method Into AI...

Data Science in the Wild ·

2026/09/17

Airbnb needed to understand the rare, risky edge cases a new AI customer-service assistant might run into — the kind of thing that used to take a data scientist months of manually reading through support transcripts, one judgment call at a time. So Airbnb built Insight Miner: an agent harness that doesn't just run an LLM over the data, but wraps it in an explicit scientific-methodology layer — extract, embed, cluster, then classify — so the process of investigation, not just the answer, is

en

23:46

Spotify — Why Bayesian A/B Testing Doesn't Pay Off

Data Science in the Wild ·

2026/09/17

Bayesian A/B testing gets pitched as more intuitive, safer to peek at early, and automatically smarter about multiple metrics — so why didn't Spotify adopt it? This episode walks through Spotify's own math: under the flat priors most platforms actually ship, Bayesian and frequentist testing produce numerically identical results, and the "advantages" people cite either require infrastructure most teams don't have (calibrated priors, Bayes-factor stopping, hundreds of historical experiments) or

en

18:56

Instacart — Can an AI Agent Out-Model Your Best Data...

Data Science in the Wild ·

2026/09/15

Instacart's machine learning engineers ran an experiment most teams are quietly wondering about: what happens when you let a coding agent loose not just on your pipeline code, but on the actual modeling work — picking features, trying architectures, tuning hyperparameters — for models that have already been optimized by humans for months? The results were a genuine surprise in places: on some of Instacart's most mature, hardest-to-improve production models, agent-driven runs delivered 3-5% offline

en

16:55

Google — Beating Your Own Teacher at Tool Use

Data Science in the Wild ·

2026/09/14

A deep dive into how Google Research flipped the standard recipe for training AI models to use tools and APIs. Instead of writing a user question first and then struggling to find a matching chain of tool calls that answers it — which fails constantly once you're picking from thousands of real APIs — their ToolGrad system builds a verified, working tool-call workflow first, using an iterative loop that treats an AI critic's plain-language feedback as a steering signal (borrowed from a technique

en

34 results

Similar Podcasts

Lead With AI

Tamara Nall

Manufacturing Matters

Tech B2B

The Good Tech Companies

HackerNoon

The Vergecast

The Verge

Computer America

Ben Crossman