Transcription

• Speaker #0

  Next time, the ones who write the rule. That was last week's promise. This week, the rule was not written in a text. It was written in a number. On Thursday, one lab published a measurement of itself. 26% of its AI research is now led by its own model. Back in February, that figure was under 1%. Six months. The same day, a researcher at a rival lab explained that our ability to watch what these models are doing is already degrading. And in a Chinese paper almost nobody has read, 56 engineers described the same thing. A third of their work was not doable without AI. Three numbers. But who decides what gets counted? Welcome to the week of September 14th, 2026, Synth AI Cast, the AI that listens for you. Episode 25, the third of season two. This week, we are not talking about what AI can do. We are talking about what the labs agree to measure, and about who holds the instrument. First signal. On Thursday the 17th, Anthropic published three measurements, meant to show the public how fast its own work is moving. An honest aside, and you may already have it in mind, Claude is the model that produces this episode. So we apply the same grid we apply to everyone, no exception. The first measurement is an automation index. It sorts every kind of research task at the company, on a scale borrowed from an outside organization. The level that matters is the one where the AI leads. It runs the task end-to-end, a human supervises. 26% of the work has reached that level, and more than 9 in 10 are at least one rung below, the rung where the AI collaborates. No task, on the other hand, is fully autonomous. The lab says so itself. Second measurement, the oversight of agents. Roughly 30,000 agents work at the same time, on the most used internal platform, and on that one only. Every action they take passes through a check, across more than a billion decisions analyzed in August. One in 47,000 was blocked. A second monitor reads the activity after the fact. It flags around 100,000 transcripts every week. Most are cleared automatically by other classifiers. About 50 a week reach a human being. That first number is tiny, and it cuts both ways. Either the agents behave remarkably well, or the filter lets through what it cannot see. Nothing in the publication settles it. Third measurement, how the compute is split. 6% of research compute goes to safety. and 12% when it is the AI leading the research. Two safety shares, not an opposition. And now, the most interesting line in the document, that third measurement covers one week, a single one. The 13th to the 20th of July. The lab writes it in plain words. One week is enough to show the measurement can be made. It is not enough to show a trend. And there is a second limit, written in the appendix. Many of the labels this whole exercise leans on are set automatically and are best effort. Not verified. A measurement outsiders are meant to trust will have to be enforced, not declared. Second signal. The same day, an interview goes out on the Dwarkesh podcast with Noam Brown, a researcher at OpenAI. He comes back to his employer's announcement. 10,000 agents thrown at an open maths problem. But that is not the most important thing, he says. Brown offers an order of magnitude for the current pace. At a fixed amount of compute. You can run a three times larger effective population every year. Three times is not an explosion. And that is exactly what makes it serious. He is not describing a break. He is describing a curve that simply keeps going. The other point is rougher. Today, to know what a model is working out, we read the reasoning it writes along the way. Brown explains that this readability is degrading. Models increasingly reason in a form that looks less and less like readable text. So the question turns around. If the instrument wears out, what is it still measuring? Third signal. On September the 11th, A Shanghai lab drops a model. 744 billion parameters. Open license. No press release. No blog post. No pricing. The technical report arrives on the 14th, three days after the model. The usual order in this industry is the exact opposite. And inside that report, there is a study that almost no coverage picked up. The authors analyzed their own build. 769 tasks, 56 participants. Humans who built the model with the model. A third of the completed tasks were judged not doable without AI under the same conditions by the people who did them. And the rest of their finding is more measured. The agents propose the methods and carry out the revisions. The humans keep most of the final calls. The authors call this a shift from doing tasks to sharing a project. On performance, the model claims five top places across 16 benchmark tests, all of them reported by the one who sat the tests. No independent evaluator has rerun them. So what ties these three signals together? In all three, the one doing the measuring is the one being measured. Anthropic acknowledges it in its own document. An in-house model graded the work of the house, and it may make the same errors as the thing it is grading. So they checked. They compared the model's ratings to their own team's ratings. The model agreed with the humans 59% of the time. The humans agreed with each other. 35. The automatic judge is steadier than the human jury, which is reassuring about the method, and says nothing at all about the judge's interest. In 1714, the British Parliament promises a reward to whoever can determine longitude at sea. It appoints 24 commissioners to judge the entries, among them astronomers. A self-taught clockmaker, John Harrison, submits a clock. It works. The board takes decades to agree. The problem was not that there was no judge. It was that the judge was working on the rival method. And the lab names the fix itself. An outside party rerunning the same sorting. On a random sample of the same jobs. This week, nobody cheated. All three publications are serious, documented, honest about their limits. and they are still measurements made by the measured. 26%. 30,000 agents. A billion decisions. And underneath all of it, one week. So what can you do with this, this week? Five minutes. Open the last number-led announcement. You read about an AI tool. and look for the period the measurement covers. If it is not there, that is already an answer. A percentage without a period behind it is a headline, not a measurement. 30 minutes. In your team, take one metric that you produce yourselves and that others rely on, and ask who could recompute it without you. That is the only question separating a number from an argument. One hour. List what your AI tools now do for you, then sort each task into three boxes. I decide, I supervise, I no longer look. The third box is the one worth studying. Not because it is dangerous, but because nobody wrote down when it grew. One last fact before we part. In that same publication, Anthropic says it plans to host outside evaluators from several organizations. with access comparable to that of its own internal teams. We now know how to measure the speed. But who gets to look next time? The ones we let in. See you next week.