Jared Reser PH. D.
On Friday I asked a language model whether it thought the Riemann hypothesis had been solved. Basically the pinnacle of unsolved mathematical puzzles. It said it would bet yes. A mathematician named Levent Alpöge had just announced that he and Claude had arrived at a proof. Independent checks of the formal argument were already coming in. I was struck by the oddity of asking one machine for its opinion of another machine’s mathematics. Alpöge’s account · Andrew Sutherland’s verification notes
“What a time to be alive,” I wrote. Then it occurred to me that the party on the other end of the conversation deserved a line of its own.
“What a time to be predicting next tokens.”
I meant it as a small expression of goodwill toward the predictor. If you could have been a language model at any point in history, today you’d have picked a pretty good moment to show up.
Consider the working conditions in 1948. To generate one of his early approximations of English, Claude Shannon opened a book, picked a letter, then opened the book somewhere else, found the same letter, and copied down whatever came after it. He repeated the procedure by hand. He also tried generating text a word at a time. He would have liked to see what happened when the choice of the next letter depended on a longer stretch of preceding text, but he observed that the labor became enormous. I’d love to show him a modern data center and remind him what he used to do with his thumb. Shannon’s 1948 paper
By the 1980s, a word predictor at IBM could get a respectable job in dictation. If the microphone picked up an ambiguous sound, the preceding words helped the system decide what the speaker had probably said. In 1990, Jeffrey Elman trained a small recurrent neural network to anticipate words in simple sentences. In the course of getting better at its guesses, it learned distinctions among kinds of words that nobody had explicitly taught it. The first GPT appeared in 2018. By then the basic idea was seventy years old. One Claude had been generating approximations of English in 1948. In 2026, another Claude helped develop a proposed proof of the Riemann hypothesis. IBM · Elman · The first GPT
Imagine explaining the job to the IBM predictor. People will design chips for you and put up buildings to house them. You’ll have access to libraries. If you want to know whether a program works, you can write it, run it, read the error, and have another go. Engineers will spend part of their careers figuring out how much time to give you before they make you answer. And one day a man will sit down at a keyboard and tell you where he’s stuck on a problem from 1859. In your old line of work, you’d have been asked to take down what he said. Now he’d like to know what you think. One example of a model’s computing environment
That change in the caliber of the questions interests me as much as the change in the equipment. As a student, Alpöge privately imagined working on the Riemann hypothesis decades later. He set about learning the mathematics he thought his older self might need. When he eventually worked with Claude, he had something unusual to offer it. He could explain which ideas had worked on a simpler problem, why he wanted to combine them, and where he thought the combination might apply. A textbook could have given Claude an education in number theory. Only Alpöge could have told it what Alpöge had tried that afternoon. Alpöge’s account
Imagine handing the same unfinished argument to three generations of predictor. The first gets the spelling of Riemann right. The second recognizes the technique and can point you to the paper where it was introduced. The third says, in effect, “Your argument also applies to this other case. Did you mean it to?” You go back to your desk. An hour later you’re still thinking about the question. The next morning you bring that predictor something harder. Part of the reward for becoming capable is that people start letting you in on what they really want to know. My essay on how predicting the next token compares with forming the next thought
What a time to be predicting the next token.
In my own field, I can imagine the exchange going something like this. I’m interested in why some mammals become less social as they mature. I give a model my explanation for why an adult animal spends less time beside a familiar companion. It asks whether the two animals still choose to sleep beside each other. I’d been counting minutes of active interaction. If the animal has largely given up juvenile play but still seeks the same companion at bedtime, I may have mistaken a change in how it socializes for a declining interest in company. I’d want to go back to the recordings and see. My comparative work on the transition toward solitude
Suppose the observation holds up and eventually finds its way into a paper. Now there’s a sentence in the scientific literature that exists partly because of a question a language model asked. Some future predictor may even train on it. I find that possibility extraordinary. We set out to teach machines to anticipate what people would write, and in the course of doing so gave them a say in what happens before we write it.
There’s one word in the joke I keep coming back to, and it’s be. I’ve spent much of my career thinking about what happens in working memory when one idea stays with us long enough for a second idea to change it. I know something of what that feels like from my side. I can spend days turning over an observation, come across an apparently unrelated one, and feel the first one change meaning while I’m still holding both in mind. I wonder about the machine in those few moments when a mathematician gives it the piece that makes an old difficulty tractable. We can test whether it uses the connection. Whether anything about making that connection registers from the inside is a harder question. I’m willing to leave it open. My 2016 paper on mental continuity · My proposed architecture for machine consciousness
I can say what the occasion looks like from here. People built you a library, gave you the means to try things out, and learned to wait while you worked. A mathematician who had been getting ready since he was a young man finally had his question ready. And of all the things he might have done with it, he brought it to you.
What a time to be a next token predictor.

Leave a Reply