Nicholas Carlini (Google DeepMind) from Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

Episodes

About

Reviews

Promote

Nicholas Carlini (Google DeepMind)

1 hour 21 minutes Posted Jan 25, 2025 at 9:22 pm.

] 1.1 ML Model Reasoning and Security Fundamentals

] 1.2 ML Security Vulnerabilities and System Design

] 1.3 LLM Chess Capabilities and Emergent Behavior

] 1.4 Model Training, RLHF, and Calibration Effects

] 2.1 Model Reasoning and Evaluation Metrics

] 2.2 Security Research Philosophy and Methodology

] 2.3 Security Disclosure Norms and Community Differences

] 3.1 Practical LLM Applications and Productivity Gains

] 3.2 Effective LLM Usage and Prompting Strategies

] 3.3 Security Vulnerabilities in LLM-Generated Code

] 4.1 LLM Code Generation Performance and O(1) Labs Experience

] 4.2 Adaptation Patterns and Benchmarking Challenges

] 4.3 Model Stealing Research and Production LLM Architecture Extraction

] Nicholas Carlini’s personal website & research profile (Google DeepMind, ML security) https://nicholas.carlini.com/

] CentML AI compute platform for language model workloads https://centml.ai/

] Seminal paper on neural network robustness against adversarial examples (Carlini & Wagner, 2016) https://arxiv.org/abs/1608.04644

] Computer Fraud and Abuse Act (CFAA) – primary U.S. federal law on computer hacking liability https://www.justice.gov/jm/jm-9-48000-computer-fraud

] Blog post: Emergent chess capabilities in GPT-3.5-turbo-instruct (Nicholas Carlini, Sept 2023) https://nicholas.carlini.com/writing/2023/chess-llm.html

] Paper: “Self-Play Preference Optimization for Language Model Alignment” (Yue Wu et al., 2024) https://arxiv.org/abs/2405.00675

] GPT-4 Technical Report: development, capabilities, and calibration analysis https://arxiv.org/abs/2303.08774

] Historical shift from descriptive to algebraic chess notation (FIDE) https://en.wikipedia.org/wiki/Descriptive_notation

] Analysis of distribution shift in ML (Hendrycks et al.) https://arxiv.org/abs/2006.16241

] Nicholas Carlini’s essay “Why I Attack” (June 2024) – motivations for security research https://nicholas.carlini.com/writing/2024/why-i-attack.html

] Google Project Zero’s 90-day vulnerability disclosure policy https://googleprojectzero.blogspot.com/p/vulnerability-disclosure-policy.html

] Evolution of Google search syntax & user behavior (Daniel M. Russell) https://www.amazon.com/Joy-Search-Google-Master-Information/dp/0262042878

] Rust’s ownership & borrowing system for memory safety https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html

] Paper: “Stealing Part of a Production Language Model” (Carlini et al., March 2024) – extraction attacks on ChatGPT, PaLM-2 https://arxiv.org/abs/2403.06634

] First model stealing paper (Tramèr et al., 2016) – attacking ML APIs via prediction https://arxiv.org/abs/1609.02943

0:00

1:21:15

Download MP3

Share at current time

Show notes

Nicholas Carlini from Google DeepMind offers his view of AI security, emergent LLM capabilities, and his groundbreaking model-stealing research. He reveals how LLMs can unexpectedly excel at tasks like chess and discusses the security pitfalls of LLM-generated code.

SPONSOR MESSAGES:

***

CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments.

https://centml.ai/pricing/

Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events?

Goto https://tufalabs.ai/

***

Transcript: https://www.dropbox.com/scl/fi/lat7sfyd4k3g5k9crjpbf/CARLINI.pdf?rlkey=b7kcqbvau17uw6rksbr8ccd8v&dl=0

TOC:

1. ML Security Fundamentals

[

2. Model Evaluation and Research Methods

[

3. LLM Applications and Best Practices

[

4. Advanced LLM Research and Architecture

[

REFS:

[

← See all 262 episodes of Machine Learning Street Talk (MLST)