---
title: "AI Learned to Talk a Long Time Ago — It Still Can't Hear You Sigh"
author: "Unique Research"
sourcePublication: "Unique Research Substack"
originalPublishedAt: "2026-10-20T18:03:02+00:00"
canonical: "https://ffcap.cn/en/research/ai-learned-to-talk-a-long-time-ago"
source: "https://uniqueresearch.substack.com/p/ai-learned-to-talk-a-long-time-ago"
language: "en"
---

# AI Learned to Talk a Long Time Ago — It Still Can't Hear You Sigh

[![](https://substackcdn.com/image/fetch/$s_!Mc1X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec7151ab-2244-471f-9a6b-beb1d615916c_1280x720.jpeg)](https://substackcdn.com/image/fetch/$s_!Mc1X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fec7151ab-2244-471f-9a6b-beb1d615916c_1280x720.jpeg)

_Unique Research · Voice AI Deep Dive · October 19, 2026_

OpenAI’s flagship real-time voice model, GPT-realtime-2.1, scored **31.5 out of 100** on emotion questions in a test designed specifically to measure “listening to tone.”

In the same exam room, a company you may never have heard of scored **91** with its own open-source model.

The benchmark is called **LISTEN** — a public suite from EACL 2026 with **2,102** questions, built to test whether an audio model is actually hearing sound, or only reading text.

Six models from Google, OpenAI, and Alibaba sat the same exam. The real-time voice flagships finished near the bottom — often worse than their own offline siblings.

The examiners’ verdict is blunt: most of today’s speech foundation models are “deaf.”

The company is **Adventists AI** (降临派), a project under Shanghai Ai Jianglin Technology. Headcount: **20** people — a Melbourne lab of **4** on models, and **16** in China on engineering and sales.

According to Nanfang+, they previously raised a near-RMB-10-million angel round from Smai Chanrong (思迈产融) and Shanhai Chuangxiang (山海创想).

I recently sat with them for a long interview. The strongest takeaway: the industry may have been staring at the wrong problem all along.

Voice models live in a different world from large language models. Spend one minute aligning the background.

The GPTs and DeepSeeks you know read text. To make AI speak, the industry has gone through three generations of approach.

**Generation one is a pipeline:** one model turns speech into text (ASR), an LLM thinks of the answer, another model reads the text aloud (TTS). Three models pass the baton; every handoff loses a layer of tone, pause, and emotion. Most voice hardware you can buy today still runs this assembly line.

**Generation two is end-to-end:** one model eats audio and emits audio, with no text in the middle. OpenAI’s GPT-realtime and Google’s Gemini Live take this path; the open-source reference is Alibaba’s Qwen-Omni.

**Generation three adds full duplex:** listen while speaking, allow interruption at any time, and tell whether you finished your sentence or just took a breath. France’s Kyutai lab with Moshi was the open-source starting point on this route; this year NVIDIA and Beijing Academy of Artificial Intelligence have each shipped their own full-duplex models.

The trouble: the field has long lacked a shared exam. Text models have endless leaderboards; speech evaluation has mostly been each vendor grading itself. LISTEN is a public benchmark that only appeared this year. It tests the things nobody used to quantify: whether a turn is finished, whether an interruption is barge-in or backchannel, laughter / sighs / coughs, whisper versus shout, accent and dialect.

The scoreboard at the top of this article is exactly that exam.

Start with a fact the industry knows but rarely says out loud.

When you talk to almost any AI voice assistant today, the way it “understands” you is basically: turn audio into text, then ask a large language model about the text. That chain has a fatal flaw. In their words:

_“The original approach converts audio to text, then judges the text. So you lose all the paralinguistic information — how you said the sentence, your pitch, your pauses, your stress and emotion.”_

“You’re something else.” Three characters on paper have no ambiguity. Spoken aloud, they can be praise or sarcasm. In the transcript, the two meanings are identical.

LISTEN has a column built for this, called **“lexically neutral, audio-only”**: the words themselves are clean and emotion-free; the score depends entirely on listening. DuplexJev scored **45**; the best of six closed-source models hit **33.5**; GPT-realtime-2.1 managed only **15.5**.

The website footnote adds a dry truth: GPT’s two models and Gemini 3.8 Live answer most emotion questions with two words — “neutral.” When they cannot hear it, they default to no emotion. Safest wrong answer in the room.

How did Adventists AI notice something was off? They said GPT’s real-time voice model can capture your speaking stress quite accurately and even imitate you — yet no open-source project could make the same judgment. So they built a systematic paralinguistic test suite. Result: open-source models were essentially stuck at zero access to paralinguistic signal; closed-source models were ahead, but “not by much.”

That finding became their paper, posted on arXiv and submitted to ICASSP 2027. The most striking numbers in the paper: when trained only with transcript distillation, gender accuracy sat around **55%** and emotion around **28%** — basically guessing. Switch to supervising answers directly on audio, and both climb to about **90%**. The model was not dumb. The training method was.

The teacher (a text LLM) has never heard sound. How is the student supposed to learn?

Finding the problem was step one. What Adventists AI actually sells is a bit counter-intuitive.

Full-duplex conversation means the AI can listen while speaking, you can interrupt anytime, like a real chat. The industry’s default is to keep a full-duplex speech model running inference for every signal. Cost, in their accounting:

_“We have to maintain a continuously running model for every user. Even in silence, we are burning money.”_

Their fix is to split the work. The judgments inside a conversation are mostly closed questions: Has the user finished? Is this utterance addressed to the AI? Should the device go standby? You do not need a full-size generative model to answer those token by token.

So they built **DuplexJev**, a “decision engine.” It does not decode or transcribe. It feeds the audio encoder’s hidden states into a frozen large model and, in one forward pass, reads each question as a single-token distribution over option letters. Official numbers: **10 judgments in 92 milliseconds**. The control — transcribe first, then ask an LLM to generate judgments — takes **1,978 milliseconds** for the same 10 questions. The “21× faster” line in the interview comes from that comparison.

The model that actually talks is **Sophon-realtime**: low-latency dialogue, paralinguistic understanding and expression, and calling backend models. Around it sits an overall architecture they call **Harness**, packing the decision engine, real-time model, backend models, persona memory, and cloud skill execution.

The engineering idea is simple: expensive models do less work; cheap modules do more judgment. Versus keeping a full-duplex model always on, they claim this stack **“can cut cost by two orders of magnitude.”**

This is not a slide deck. DuplexJev is fully open-sourced, with a one-line pip install. LiveKit and Pipecat — two mainstream real-time dialogue frameworks — both integrate it as a turn-taking plugin. On LiveKit’s own eot-bench, across **14 languages** and a **300-millisecond** budget, its false-interruption counts are all lower than Smart Turn v3.2.

The Sophon series is slated to open-source at the end of October. They say it will be “the only commercially usable full-duplex open-source model and framework available today.” Write that down; we will come back to it.

Which is harder: hearing someone’s tone, or making the AI speak with the right tone? Adventists AI’s answer is the former:

_“Relatively speaking, understanding is more important, because you can use paralinguistic understanding to reinforce paralinguistic expression — and that drives rapid growth on the expression side.”_

The logic: you first need an ear that can hear emotion, before that ear can referee “speaking” and tell the expression model whether a line landed. Understanding is the infrastructure for expression.

In product scenes, this ability sometimes changes the decision itself. Their example: in a phone customer-service flow, when a user asks to transfer money and the voice carries fear, the AI agent can judge that the user may be under coercion — and cut the entire transfer. You will not see that demo on any launch stage. It is not flashy. Financial-service companies will pay for it.

On method they are frank: _“First let the model understand and express paralinguistics. That is an information privilege. Where exactly to apply it can wait — the right scenes will surface on their own.”_ Read that as strategic patience, or as not yet having a killer scene. Both readings may be true.

The bottleneck is data. The hardest gap, they say, is datasets with paralinguistic labels. Transcripts are everywhere; audio annotated for tone, stress, and emotion barely exists. They are filling it through partnerships with Zhejiang University and data vendor MagicData. Models are post-trained on Qwen and Gemma backbones; the scarce asset is that annotation layer. Whoever builds this data first buys the ticket to the lane.

Now the business — and the part of the interview I valued most: the numbers are complete, including the unflattering ones.

Adventists AI’s customers are AI dialogue hardware makers. They sell SDK and API services. Today: **300+ hardware SKUs**, **100,000+** terminals on their service (including companion devices such as dipal, dimi, and Xiaotie), and roughly **3,000+** concurrent online.

Sounds fine. Keep reading.

Of those 300-plus SKUs, only about **20%** are in mass production. On revenue, measured on the Australian fiscal year (books close in July): FY2025 was **USD 300,000**; FY2026 is expected around **USD 1 million**. The largest single customer accounts for **25%** of revenue.

In other words: everything an angel-stage company is supposed to look like — decent technical signal, a small commercial footprint, high customer concentration.

Customer mix is interesting too. Hardware customers are about **70% domestic / 30% overseas**, but end users flip: about **30% domestic / 70% overseas**. Overseas end users are more willing to pay. Brand-class customers pay readily and are easy to serve; Shenzhen’s small hardware makers are “relatively hard to make money from.”

How did the first 10 paying customers land? _“Mostly they come to us. We almost never spend money or headcount on promotion.”_ Zero-promotion acquisition is the dividend of open source — and the necessity of a 20-person team that cannot afford a sales army.

One commercial detail matters. Take dipal: after buying the hardware, users subscribe monthly; the subscription includes a **5-hour** chat quota that refreshes on schedule. That design answers the arithmetic that haunts voice hardware: once devices ship, users talk more and more — who pays the inference bill? Answer: the user does, via subscription.

Delivery is light. A hardware customer typically goes from trial to production in about a month, “basically without dedicated staff” — generic interfaces plus docs and examples. Their stated preference: _“We want as little human-to-human service as possible, and as much automated onboarding as possible.”_ Cleverness forced by constraint: a 20-person team has no right to run a project-services business.

On renewal they say churn is very low — once integrated, almost everyone keeps using; growth mainly tracks the customer’s mass-production and distribution speed. Discount that claim if you like, but the logic holds: once an SDK is welded into firmware, swapping it out costs more than keeping the subscription. Stickiness in voice services is soldered onto the board.

Why open-source? _“Many vehicle and hardware makers still hesitate on full-duplex tech stacks, while big labs monopolize the tech and refuse to open it. I hope our open source can seed a new, thriving ecosystem.”_

And if big labs enter and crush them on price? Six words back: _“Open source does not compete on price.”_

The chain is coherent: open models smash the barrier to the floor; vehicle and hardware makers dare to enter; an ecosystem forms; some subset cannot fine-tune or deploy alone and come back for implementation and service. Red Hat’s old script, applied to voice AI.

The other side of the table has to be named too. In August this year, NVIDIA open-sourced an end-to-end full-duplex speech model, NemotronLabs VoiceChat 11B, with listen-while-speak and real-time tool calling. It is a research license and wants 80GB of VRAM — high bar — but the direction is clear. Earlier, BAAI’s RoboBrain-Audio and Soul’s full-duplex models moved the same way. GitHub already has 80-plus full-duplex-related open repositories.

Adventists AI’s line about being “the only commercially usable” one is their framing. How wide that window is, and how long it stays open, is not entirely under their control.

Scores also need footnotes. DuplexJev is not first on every axis: on accent-stress tests, Gemini 3.8 Flash is still #1 (**89.0**); Adventists says the latest DuplexJev climbed from a previously published **71.5** to **85**, narrowing the gap. When the text itself already carries emotion, general models often do better. Their own site footnote also notes that emotion and accent test sets share provenance with training data (speakers and sentences do not overlap). So part of the 91-versus-31.5 gap is the dividend of having seen similar item types. How far the gap shrinks on a fully unfamiliar corpus, nobody knows yet.

Saying this is not an accusation. When you read leaderboards, always read the footnotes first.

They also argue that currently open-sourced full-duplex models have serious “intelligence deficits,” useful mainly as technical references, not production systems — including Moshi and NVIDIA’s recent releases, which they say cannot hold a normal conversation. They expect to be the first open full-duplex model to catch closed-source quality, with new scores due next week aimed at the first tier. That is the party’s position — write it down, verify when scores land. One cross-check holds: NVIDIA’s license does say research-only; “commercial” is still a distance away.

Asked what would count as progress over the next year, they did not lead with revenue. They circled back to “opening a new era.” From a big company that line would sound like PR. From a company with USD 300,000 in annual revenue, four people in Melbourne and sixteen in China, I am more willing to hear it as a bet: that the window while big labs overlook, move slowly, or refuse to open is long enough to scatter the seeds of an ecosystem.

Voice AI’s first half taught machines to talk. The ticket to the second half may be teaching machines to hear the fear in your voice — and stop a transfer before it happens.

On that course, GPT-realtime-2.1 currently scores 31.5.

**Q1:** What problem are you most focused on right now? If you succeed, what change would an ordinary user feel?

**A:** We are preparing to open-source a series of full-duplex dialogue models. Once that lands, we will drive the barrier to full-duplex conversation very low — every industry can deploy quickly and move to the next interaction level.

**Q2:** Give a concrete example: the transcript is correct, but the AI still responds wrong because it missed tone, pause, or interruption. Why has the old approach kept failing?

**A:** The old approach converts audio to text, then judges the text. You lose all paralinguistic information — how the sentence was said, pitch, pauses, stress, emotion.

**Q3:** You said training only on transcripts may never teach a model to hear tone. How did you first notice something was wrong? What detours, and which experiment found the cause?

**A:** We found GPT-Live can capture speaking stress very accurately and imitate you. No open-source project could make the same judgment, let alone imitate expression. So we ran systematic paralinguistic tests: open-source models were basically stuck with zero paralinguistic access; closed-source models were ahead, but not by much.

**Q4:** Along the path from user speech to AI reply, what do DuplexJev, Sophon, and Harness each own? What already works, what is still in R&D?

**A:** DuplexJev is the hub in full-duplex dialogue. Previously we needed a full-duplex speech model like Sophon-realtime to handle every signal in and out — cost was terrifying. You maintain a continuously running model per user; silence still burns money. With DuplexJev you can cheaply unify judgments across all live conversations — whether the device should standby, whether the user is addressing the AI, whether the user finished — cutting full-duplex deployment cost and latency. Sophon-realtime is the actual full-duplex dialogue model: low latency, paralinguistic understanding/expression, calling backend models. Harness is the overall architecture that includes DuplexJev, Sophon-realtime, backend models, a persona “memory palace,” and cloud skill execution.

**Q5:** Which off-the-shelf models do you use, what must you build yourselves, and where does the hardest data come from?

**A:** We ship multiple open models with Qwen and Gemma as backbones. The hardest data to fill is paralinguistically labeled datasets. We partner with Zhejiang University and MagicData on that.

**Q6:** Why split judgment, reply, and expression across models? Did you try one model for everything?

**A:** Sophon-Realtime can in fact do most of the work alone, but still needs a backend model for sustained complex work. That mode is expensive; DuplexJev can cut cost by two orders of magnitude, so that is what we recommend.

**Q7:** You published “10 judgments in 92 ms, 21× faster than transcribe-then-ask-LLM.” Which slice of latency is that? On real devices, how much does user-finish-to-AI-speak improve?

**A:** Even a single judgment shows the speed gain of judging audio directly. Skipping transcription saves the autoregressive ASR latency — roughly an extra 300 ms. The interesting part is batch inference: multiple judgments on one audio segment, or multiple segments in one forward pass.

**Q8:** Between “hearing tone” and “speaking with the right tone,” which is harder?

**A:** Paralinguistic understanding and expression are both scarce industry-wide. Understanding is more important, because you can reinforce expression from understanding and grow the latter quickly — and that is what we do. We can imitate your stress and emotion, infer intent hidden behind the words, and switch speaking style by context. Sometimes paralinguistics change the decision itself — e.g. fear in a transfer request can cut the transfer.

**Q9 / Q10:** DuplexJev is fully open-sourced (pip, HF Space, online API). LiveKit and Pipecat integrate it as turn-taking and classification plugins. Their philosophy: first give the model paralinguistic understanding and expression as an information privilege; applications can wait for the right scenes to appear.

**Q11–Q20 (condensed with numbers preserved):** Buyers are AI dialogue hardware makers purchasing SDK/API (dipal, dimi, Xiaotie, etc.): 300+ SKUs, 100k+ terminals. ~20% of SKUs mass-produced; ~3,000+ concurrent. Brand customers pay and are easy to serve; many Shenzhen small hardware makers are hard to monetize. Trial-to-production ~1 month with almost no dedicated staff — docs and examples, automated onboarding preferred. Open source is meant to push the industry toward full duplex and pull back customers who cannot fine-tune alone. AU FY2025 revenue USD 300k; FY2026 expected USD 1M. Largest customer ~25% of revenue (NDA). dipal end users pay monthly with a 5-hour chat quota. First 10 payers mostly inbound; almost no spend on promotion. Churn described as very low; growth tracks customer mass-production and distribution.

**Q21:** Melbourne lab 4 on model R&D/training; China 16 on interfaces, backend, sales, ops. Constraint is still sustained model R&D and technical lead.

**Q22:** Hardware customers ~70% domestic / 30% overseas; end users roughly reverse — overseas end users more willing to pay.

**Q23:** They offer leading open models for secondary development and want an ecosystem. “Open source does not compete on price.”

**Q24:** Sophon series to open-source end of October — intended as the only commercially usable full-duplex open-source model and framework. Ambition stated as “opening a new era.”

_This article is industry reporting and observation only, and does not constitute investment advice. Company claims (scores, “only commercially usable,” churn) are attributed to the interviewee and should be read with the footnotes and caveats above._

---

Original publication: https://uniqueresearch.substack.com/p/ai-learned-to-talk-a-long-time-ago
On-site reading page: https://ffcap.cn/en/research/ai-learned-to-talk-a-long-time-ago