Field Notes · Arabic breaks silently, part two

The Arabic fix everyone recommends is now the thing breaking your Arabic

I set out to prove that Arabic needs reshaping before it is drawn. The render proved the opposite, and the reversal turns out to be the useful part.

Arabic is cursive and bidirectional. Every letter changes shape depending on its neighbours, and the order the characters are stored in is not the order they appear on screen. For years the standard fix in Python has been two libraries — arabic_reshaper to join the letters, python-bidi to reorder them — applied before the text is drawn. Search for “render Arabic Pillow” and that is what you will find, everywhere.

I built a benchmark to demonstrate it. It demonstrated the opposite.

Four paths, one word

The word is مرحبا بكمwelcome. Same string, same font, four rendering paths:

Four renderings of the Arabic phrase marhaban bikum. Modern engine without preprocessing is correct; modern engine with arabic_reshaper and bidi is mangled; no shaping engine without preprocessing is mangled; no shaping engine with preprocessing is correct.

Rows 1 and 2 are the same renderer. The only difference is that row 2 applies the recommended recipe — and row 2 is the broken one.

Why

Pillow 12 links Raqm, which means HarfBuzz, which means it already performs shaping and the bidi algorithm itself. Feeding it text that has already been shaped and already been reordered makes it do both a second time. Two rights make a wrong.

So the question is not “does Arabic need reshaping”. It is:

Does this rendering path already do complex-text layout?
Yes → reshaping breaks it. No → reshaping is required.

The recipe was correct when it was written. Text stacks caught up; the advice did not.

Measured

Three Arabic fonts, five strings, four paths, scored against a verified-correct reference on shape similarity:

Rendering pathIdenticalRecognisableBroken
Modern engine, text as-is1500
Modern engine + reshaper + bidi0114
No shaping engine, text as-is0015
No shaping engine + reshaper + bidi096

The last row is the interesting one. On a renderer with no shaping, the recipe is a partial rescue — plain Arabic comes back legible, but digits, embedded Latin and diacritics still fail. Positioning combining marks and resolving bidirectional runs needs real shaping regardless of what you pre-process. Treating the recipe as a fix rather than a patch is how those cases slip through.

A separate trap, in the fonts

While testing I rendered في عام 2026 and the year came out as empty boxes. Checking the font tables rather than guessing: SF Arabic and Geeza Pro contain no Latin letters or digits at all.

The same Arabic sentence containing the year 2026 rendered in two fonts. IBM Plex Sans Arabic shows the digits correctly; SF Arabic renders them as empty notdef boxes.

This is the same failure mode as the numerals problem in the previous piece: the output looks entirely plausible to anyone not reading the part that broke. An Arabic caption with a date, a price or a brand name in it will ship looking fine.

What I got wrong, and how

Twice, and both times only rendering it caught the error.

First the premise: I was so confident reshaping was required that I made it the reference the others were scored against — so the correct renderings scored worst. Then the metric: my similarity measure compared position as well as shape, so a visually identical render scored zero because the two engines place glyphs a few pixels apart. Normalising to the ink bounding box fixed it, and reporting three bands instead of pass / fail is what exposed the partial-rescue result.

If there is a lesson beyond Arabic, it is that a benchmark inherits the assumptions of whoever wrote it, and the only reliable way to find them is to look at the output instead of the score.

The data

The rendering harness and the per-case results are published as ArShape on Hugging Face, alongside ArNum-TTS from part one. The images are not shipped — they regenerate in seconds from system fonts.

Part three: Your Arabic PDF is fine. What reads it is not.

The practical version

  • Check whether your renderer does complex-text layout. In Pillow: PIL.features.check("raqm").
  • If it does, pass Arabic through untouched. Do not reshape.
  • If it does not, reshape — and test digits, Latin and diacritics separately, because those will still be wrong.
  • Check your font actually contains every character you are about to draw.