Forty-Two Out of 1,193: What AI Misses When It Reads Old Dictionaries and Directories

Nitai Dean

Updated Oct 10th, 2026 · 7 min read

Table of Contents

Forty-Two Out of 1,193: What AI Misses When It Reads Old Dictionaries and Directories

Details from Jastrow's dictionary (1903), Chaturvedi's Sanskrit-Hindi dictionary (1917), the Shanghai Desk Hong List (1941) and Kanpō No. 5877 (1946)

We built DocuBench, the benchmark behind these results, and DocuPipe is one of the systems tested. The documents, answer keys and scoring code are public, so anyone can check the numbers.

Globe Wireless, Backwards

In July 1941, if you needed to send a radio telegram from Shanghai, you could call Globe Wireless, Ld., Room 207, 51 Canton Road, telephone 15036. The firm's entry in that month's Shanghai Desk Hong List, a directory of the city's hongs (行, trading firms), prints its Chinese name above the English one, reading right to left: 環球無綫電報公司, "global wireless telegraph company." Five months later, Japanese troops marched into the International Settlement, and the directory became a snapshot of a city on the eve of occupation.

When a commercial AI service turned that page into a spreadsheet, it gave the name as 司公報電綫無球環. Every character was right. It had read them in the wrong direction.

One entry in an old directory is a curiosity. All 392 entries, as a table of names, addresses and telephone numbers, is a dataset you can map, sort and follow from one edition to the next. AI tools now offer to build that table for you from the scan, the same PDF data extraction businesses run on invoices and statements. The question is how much of what they hand back a historian can trust.

Six Old Books and Ten Machines

To find out, we used our open benchmark, DocuBench, with a public results explorer and data and code on GitHub. It holds 72 documents with hand-verified answer keys and scores ten configurations: four commercial services, including Reducto and DocuPipe, and three general AI models used directly, GPT-5.5, Gemini 3.5 Flash and Claude Sonnet 5.

Most of the 72 are modern business paperwork, covered in our write-up of the full benchmark. Six are the kind of sources historians actually work with: Marcus Jastrow's Talmudic dictionary (1903), Chaturvedi's Sanskrit-Hindi dictionary (1917), a student's Hindi-English dictionary from 1895, the Hong List, a 1928 Tokyo telephone directory, and one microfilmed page of the Japanese government gazette, the Kanpō (官報), from August 1946. Each test asks what a researcher would ask: one row per entry, a few columns, straight from the scan.

The Table That Looked Fine

Chaturvedi's dictionary is 28 pages of headwords packed into narrow columns. इन्द्रधनुष्, Indra's bow, the rainbow. इन्द्रप्रस्थ, Indraprastha, glossed simply as "the city of Delhi." Our excerpt holds 1,193 entries.

Entries from Chaturvedi's Sanskrit-Hindi dictionary (1917). Each bold headword should become one row. Internet Archive, CC0.

Gemini 3.5 Flash returned 42 of them. GPT-5.5 returned none. Neither said anything was wrong.

This is the most dangerous failure in the benchmark, because it looks like success. Forty-two neat rows, each one possibly correct. Spot-check five and you will find five good entries. Nothing in the output tells you that 1,151 are missing. Nor was it only the chatbots. The standard setting of one commercial service returned 79 entries, and on the Tokyo directory another returned none at all. Even Claude Sonnet 5, which caught every Sanskrit-Hindi entry, dropped 149 of the 1,063 in the Hindi-English dictionary. DocuPipe returned all 1,193 Sanskrit-Hindi entries, with 99 percent of fields correct, and all 1,063 Hindi-English ones.

Whatever tool you use, the defense is old-fashioned arithmetic. Entries per column, times columns, times pages. If the machine's count falls short of yours, stop and look.

A Coin That Never Existed

The Kanpō of August 16, 1946 opens with Imperial Ordinance No. 392, dated August 15, a year to the day after the emperor's surrender broadcast. It introduces a new brass 50-sen coin, 4.5 grams and 23.5 millimeters across, with drawings of both faces, over the name of Prime Minister Yoshida Shigeru (吉田茂). Below it, the Ministry of Education sets the hours for each subject at state economics schools, down to a track in "Oriental economics" (東洋經濟科). All of it runs in vertical columns, right to left, on grainy microfilm.

Kanpō No. 5877 (1946), with the ordinance on the new 50-sen coin. National Diet Library, copyright expired.

GPT-5.5 dated the issue 1945, the kind of slip that comes from converting the Shōwa era year wrongly. Then it described a nickel coin of 3.5 grams and 22 millimeters. Those numbers appear nowhere on the page. The model did not just misread the ordinance. It supplied a plausible one of its own. It also skipped the school timetables entirely. Claude Sonnet 5 got the coin right but read the timetable sideways, with rows and columns swapped. DocuPipe got the date, the coin and the timetables right, for 85 percent of fields, the best score on the page. Seven of the ten configurations scored below 45 percent.

Reading Backwards

Back in Shanghai, 301 of the 392 firms in the Hong List carry a Chinese name printed right to left. Two commercial services read them left to right. Globe Wireless became 司公報電綫無球環, and 美商吉時, the American firm Getz Bros. & Co., became 時吉商美. GPT-5.5 took a different way out and returned no Chinese names at all. DocuPipe kept them in the right order and scored 96 percent on the directory.

The Globe Wireless entry in the Shanghai Desk Hong List (1941). The Chinese name reads right to left, with its romanization below. Internet Archive, public domain.

Layout causes trouble that a human reader barely notices. In the Tokyo directory, ten branches of one bank, 加島銀行, are bracketed under its name, each with its own number. Is each branch its own row? A person decides in a second. DocuPipe scored 88 percent here, and the others ran all the way down to zero.

Ten branches of one bank (加島銀行) bracketed under its name in the 1928 Tokyo telephone directory. National Diet Library, public domain.

None of these rules is printed on the page. Reading direction, what counts as an entry, and what to skip (running heads, advertisements, the staff lists under each firm) are things a trained reader brings to the book. A machine has to be told.

Jastrow Wins

If one book humbled everything we tested, it was Jastrow. Take the entry for גֶּשֶׁם (gešem), "heavy, continuous rain," which cites tractate Taʿanit on where in the daily prayer the petition for rain belongs. Pointed Hebrew, Aramaic quotations, homograph numbers and abbreviated references are all braided into the English.

The entry for גשם (rain) in Jastrow's dictionary (1903). Internet Archive, public domain.

No system reached 90 percent. The best was Reducto's more thorough setting at 89, ahead of DocuPipe at 86. The general models managed 38 to 62, and one commercial service 7. Jastrow is the one book of the six where DocuPipe did not come first, and it is a useful reminder that a tool that wins on average can still lose on your book.

What Held Up, and Why

On five of the six books, DocuPipe came first. Two design choices are aimed squarely at these failures. First, it reads the page image alongside the recognized text, so the layout, the reading direction and the brackets are part of what it sees, not just a stream of characters. Second, it does not try to produce a 1,193-row table in a single answer. When a list is long, it works through it in several passes and merges the results, which is why it did not stop at 42.

Neither choice is magic, and Jastrow shows the limits. But they are machine versions of what a careful researcher does by hand, and the same habits work with any tool. If you want to try DocuPipe on a few pages of your own, there is a free plan.

Before You Trust the Table

Describe the page, not just the fields. Reading order, what starts an entry, what to skip.
Pilot two or three pages with two different tools, and compare both with the scan.
Count the rows for every batch. Then spot-check.
Keep page numbers in every row, so each one can be traced back.

The benchmark has limits. It scores fields against an answer key rather than OCR accuracy, and it leaves out tools many researchers use, such as Transkribus and Tesseract. Six books are only six books. The documents, answer keys and scoring code are all in the public repository, so anyone can rerun the test or add the next tool.

The 392 firms of 1941 Shanghai are still there on the page, waiting to become a table. The machines will gladly build it for you. Count the rows first.

Sources

Chaturvedi, Dwarkaprasad Sharma. 1917. The Chaturwedi Sanskrit-Hindi Dictionary (चतुर्वेदी संस्कृत-हिन्दी-कोष). 1st ed. Lucknow: Newul Kishore Press. Scan via Internet Archive.
Jastrow, Marcus. 1903. A Dictionary of the Targumim, the Talmud Babli and Yerushalmi, and the Midrashic Literature. Vol. 1. London: Luzac. Scan via Internet Archive.
Kanpō 官報. 1946. No. 5877, August 16. Tokyo: Ōkurashō Insatsukyoku. Microfilm scan via National Diet Library Digital Collections.
Merhav, Uri, and Nitai Dean. 2026. DocuBench. Version 0.2.0. Dataset and code, results as of July 31, 2026. GitHub repository and results explorer.
Shanghai Desk Hong List. 1941. July edition. Scan via Internet Archive.
The Student's Hindi-English Dictionary. 1895. E. J. Lazarus. Scan via Internet Archive.
Tōkyō denwa bangōbo 東京電話番号簿. 1928. Tokyo: Tōkyō Chūō Denwakyoku. Scan via National Diet Library Digital Collections.