Unique Research · Insight / interview digest · November 1, 2023 · a16z AI Revolution with Sarah Wang · Sep 25, 2023 · a16z.com/AIRevolution
About Character.ai: Chatbot platform to create and talk with virtual characters (real, fictional, or original) with personality, backstory, knowledge — plus stories, poems, lyrics, code; multimodal image/audio. Co-founded by ex-Google LaMDA builders Noam Shazeer and Daniel De Freitas; beta Sep 2022.
01 · Real Noam vs. AI Noam
Sarah opens with Roblox David Baszucki’s point that real people vs. AI versions will soon be hard to tell apart — and interviews both live. Soft personal: why Duke? Noam jokes basketball, then: recruited to win the Putnam; scholarship. Why leave ~20 years at Google? Always AI-obsessed; language modeling as “best problem ever” — predict next token → something that can talk about anything. Wanted advances in architecture, distributed algos, quantization — then realized the biggest issue was scale (billions–hundreds of billions of $ of compute, not millions) needing an app that proves huge value. Coding looked useful; dialogue is the world’s #1 pastime. ~2019–2020 Daniel (life mission: chatbots) pulled him into a 20% project that went viral inside Google as a huge model — then big-company brand risk blocked shipping “fun” things. Startup to maximize speed.
Sarah notes Noam is the third (after OpenAI’s Mira and Anthropic’s Dario) to say he sees no scaling-law ceiling. Fear AGI ends the world? Real Noam jokes need a ~4-month global pause until enough H100s train their model — then: not yet; many benefits; keep working as tech advances. Grades AI Noam’s answers: pretty good — even better than he would have done.
02 · Next unlocks & scaling laws
“Better” for Character.ai isn’t always correctness. Big unlock: train a larger, smarter model — scaling laws go far. Last year’s model ~$2M compute cycles; could redo for ~$500k now. Hope by year-end: tens of times smarter, multimodal (voice, faces, multi-person — cabinet of advisor friends; “Cheers” where everyone knows your name). Context today thousands of tokens (~half hour of “lifelong friend” memory) yet many users already spend hours/day — longer context + dumping huge personal info should unlock more; HBM bandwidth exists; just needs to be done.
03 · AGI’s first use case
Platform: >20B human messages since launch; millions DAU averaging ~2 hours/day. Framing: entertainment is a ~$2T industry whose dirty secret is imaginary friends who don’t know you exist — parasocial TV/book/celebrity relationships; billions of lonely people. Cool AGI-first app. Doctor apps are slow (cannot hallucinate); AI friends can ship fast as entertainment. Unmet need + general LLM versatility; hallucination is not a blocker for this product — push the tech now, not in five years after every problem is solved. Standard ≠ self-driving; more like comics.
04 · Building a full-stack LLM
Vs. narrow mental-health / edtech startups training specialist models: the more application-specific, the easier to write brittle rules that don’t generalize — stay away. Goal: AGI company + product-first company; pick the product that forces generality, smarter models, user need, scale, and low cost. Vertical integration: control every layer, change anything affecting the end result, feed massive user data back; many inventors on the team; hire people who love inventing and shipping. Loves full-stack vs. fine-tuning open LLaMA chat alone.
05 · Noam does the global compute arithmetic
On theory-of-mind emergence (e.g. GPT-3.5 ~9-year-old ToM): smarter models get better ToM — emergent with scale, not that expensive. NVIDIA making another ~1.5M H100s next year; rough math → ~0.25T ops/person/sec worldwide → on a 100B-param model, ~1 token/sec/person if everyone were online — not expensive if done right; Character.ai is building for that scale.
06 · Universally accessible intelligence
Internet = dawn of universal information; now dawn of universal intelligence — Wright brothers moment: working, useful, many apps, scales well; more breakthroughs as every scientist piles on; open source spreads innovation from big labs to academic/garage. Want “how do I cure cancer?” in years. Scaling laws show no stop in experiments. Core compute cost ~10^−18 $/op; even chatting with the largest trained models should cost far less than human time if efficient — room to scale orders of magnitude.
1 CN image; attach on ingest. Q&A facts preserved.
— END —