What I have built, published and shipped

nmt_nfkc is the default normaliser in SentencePiece, which sits under a large share of language-model text pipelines. It mapped U+200E LEFT-TO-RIGHT MARK and U+200F RIGHT-TO-LEFT MARK to a space, and left U+061C ARABIC LETTER MARK in place. All three are zero-width invisible bidi controls, and ALM is the Arabic-script counterpart of the other two, differing only in its bidi class. The consequence is that two Arabic strings that render identically tokenise differently, and the word carrying the invisible mark loses its word-initial piece: on google/mt5-base, بالعالم went from one piece to three.

To be precise about who did what: I reported it, and I did not write the fix. Taku Kudo, the author of SentencePiece, wrote and committed it three days later, one mapping line in src/builder.cc, the precompiled tables regenerated, and a regression test covering all three marks.

The argument that landed was consistency, not frequency. ALM is genuinely uncommon and the report says so and measures it, so the case was never “this happens a lot”. It was that the normaliser already handled two characters of exactly this class and had missed the third.

And it is not in a release. The newest tag is v0.2.2 from July 2026, and PyPI serves that same version, so the normaliser anyone installs today still passes U+061C through. A merge is not the public moment; the release that ships it is, which is the same caveat the pypdf entry below carried until 6.19.0 went out.

There is a second limit, and it is the more interesting one. A trained SentencePiece model does not call the library's rule table at inference. The rules are compiled and embedded into the .model file, by design, so that training and inference normalise identically. Every model that already exists therefore keeps the table it was built with. This fix cannot reach them retroactively, even once it ships.

google/sentencepiece #1331 · closed as completed · fixed in c9e4719 · not yet in a release, newest is v0.2.2 from July 2026

pypdf extracted Arabic-Indic digits backwards. The codepoints U+0660–U+0669 sat inside the library's right-to-left character range, so each digit was prepended to the output instead of appended: ١٢٣٤ came out of a PDF as ٤٣٢١. The extended Arabic-Indic set, the Persian digits, did the same. Digits are not right-to-left even inside right-to-left text, so the fix is to move them into the neutral ranges and let the surrounding text decide the direction. Opened 13 September, reviewed, approved and merged by stefan6419846 the following day.

To be precise about the size: the library change is two lines. The other 81 are a new test, which builds its page out of a Type0/Identity-H font and a ToUnicode CMap rather than checking in a binary fixture — so it needs no producer library to run and has nothing in it to go stale. And it shipped on 16 September in pypdf 6.19.0, whose changelog names the pull request and its author.

What this is, and what it is not: it is my first merged code contribution. The matplotlib pull request further down this page is a thirty-line documentation change, and I would rather say that here than let a reader take the two for the same thing. What makes this one worth listing is not its size — it is that a maintainer of a library with over ten thousand stars read the diff, approved it and merged it. That is someone else's judgement, not my own measurement of my own claim.

The listing that followed: pypdf's CONTRIBUTORS.md now carries my name against “Arabic and RTL text handling”, with a link to this site. It is one line in a file. It is listed here only because it is a page on somebody else's project rather than a page of mine.

py-pdf/pypdf #4077 · merged · released in 6.19.0, 16 September 2026 · listed in CONTRIBUTORS.md

Two of the projects that the source check in arabic-lint flagged were reported on 8 September. On 12 September justncodes closed both as completed: “Confirmed, and thanks for the thorough report.”

To be precise about what that is: one maintainer, not two independent confirmations. The two repositories are siblings carrying the same helper on the same line, and he closed them two and a half minutes apart with a byte-identical comment.

What makes it worth listing is that he did not stop at agreeing. He rejected the fix I proposed: requiring matplotlib 3.11 would have left existing installs sitting on 3.10 with no shaping at all, because their updater installs missing packages but never bumps versions. He gated on the version instead, passing the logical string straight through on 3.11 and later and pre-shaping below it. He also found a second call site the report never mentioned, in their attendance table reports, and fixed that too. It ships in their next release.

Most of this page is my own measurement of my own claim. This one is not. Someone else read the report and acted on it in code I have never seen. Four reports went out on 8 September: two are fixed, two have had no reply at all.

whiteout-project/bot #110 · kingshot-project/Kingshot-Discord-Bot #28 · both closed as completed

The other two reports have had no reply and no comment of any kind: a ComfyUI node that draws Persian through Raqm-enabled Pillow, and the chart labels in an Arabic poetry dataset. Both are still open. Four filed, two fixed by one maintainer, two silent is the honest score, and a page that listed only the two that landed would not be worth checking.

ComfyUI-PersianText #3 · NoorBayan/Diwan #3 · open, no response

matplotlib 3.11 shapes Arabic itself. That silently reverses the arabic_reshaper + python-bidi workaround that every tutorial, Stack Overflow answer and blog post still recommends — no error, no warning, just backwards labels on a chart nobody who reads Arabic was asked to check. The upgrade guide now tells 3.11 upgraders to remove that workaround.

To be precise about what was merged: the pull request is a documentation change, thirty lines. The work behind it was establishing that the reversal happens at all, that it is unconditional, and that the fix everyone recommends had become the bug. Three maintainers, including the project lead, engaged with the finding.

matplotlib #32263 · merged

The same class of defect one layer down: under the Raqm layout engine, pre-shaped Arabic is corrupted rather than drawn. Compounded by a trap that costs people days — whether Raqm is active is a property of the runtime, not of the Pillow version, so the identical wheel behaves differently on two machines depending on the host libraries. The patch warns at run time and documents that the layout engine is a build-time decision.

Pillow #9925 · open, 8 of 8 checks passing, no maintainer response since 5 September

The canonical question on drawing right-to-left text in matplotlib has been read 15,237 times over thirteen years, and every answer on it predates 3.11 — so the most-read guidance on the internet now tells people to do the thing that reverses their charts. I answered it with the version boundary, the one code path that works on both sides of it, and the pixel measurement that separates “correct” from “reversed” (mean absolute difference 7.75 against 76.28). The asker accepted it.

To be precise about the scale: the answer scores 1. It is the reach of the question that matters here, not a vote count — and the correction now sits at the top of the page people actually land on.

Answer on “Matplotlib: Writing right-to-left text” · accepted

The findings above are about code. This asks whether the damage actually reaches the data models are trained on. 276 datasets readable, 26,318 rows, 119,517 text fields, scanned with the linter and published with the scanner attached so the numbers can be re-derived rather than trusted.

One dataset stores contextual glyph forms in every label sampled — 1,000 of 1,000, across ten windows spanning its whole 20,000-row split. It is an OCR set, so the affected field is the ground-truth label, which is the path from a rendering bug into model weights. Only the shaping half of the usual recipe ran there: word order is normal, so NFKC alone recovers the text. I originally reported reversed order as well, which was wrong, and the correction is recorded on the dataset card and in the thread on that repository. Twenty more carried isolated stray glyphs — not the same defect, but not harmless either: measured against four real vocabularies, one stray form in a four-letter word is enough for an Arabic-specific tokenizer to emit [UNK], discarding the word rather than splitting it.

To be precise about the scale: that dataset has 17 downloads. This shows the mechanism reaches training data. It is not evidence that widely-used Arabic corpora are affected, and the card says so in the same words.

arabic-corpus-audit · CC BY 4.0 · archived and citable at doi.org/10.5281/zenodo.22733934

The same Arabic sentence written with Western digits (2026) and Arabic-Indic digits (٢٠٢٦) produces wildly different speech. One engine reads the first at 73% intelligibility and the second at 7%; another handles both at 80%. The failure is invisible in evaluation because TTS demos contain no numbers — it surfaces in finished broadcast work, where a date or a price is spoken as noise.

Read the write-up · ArNum-TTS dataset · CC BY 4.0

arabic_reshaper + python-bidi is the advice everywhere for drawing Arabic. On any modern text stack it produces مكب ابحرم where مرحبا بكم should be, because the renderer already did that work and it now happens twice. Measured across three fonts and four rendering paths, with the partial-rescue case that a single pass/fail would have hidden.

Read the write-up · ArShape dataset · CC BY 4.0

Naive generators store Arabic in logical order and laid-out ones store it visually; extractors then either re-apply the bidirectional algorithm or do not. The pairing is what decides the outcome, not the tool: LibreOffice into pypdf reads 5 of 5, LibreOffice into pdfminer reads 0 of 5 — same file, same text, opposite results.

Read the write-up · ArPDF dataset · CC BY 4.0

This is the film half of the page, and it is measurement rather than a showreel. Generative video is handed a camera move and is under no obligation to deliver it; the usual check is one person watching sixty clips and forming an impression. shotdrift measures the camera path out of the pixels instead - pan, zoom and roll, frame by frame - by tracking a grid of tiles with phase correlation and fitting a single similarity transform to all of them. What that one-camera fit cannot explain is the other half of the product: if the tiles refuse to agree with any single camera, something in the frame is moving independently of one. Every figure is reported in fractions of the frame width rather than pixels, so a bound means the same thing on a 720p proxy and a 4K master.

The second question is asked separately, because it is a different question. --expect push-in compares the measured path against the move that was actually ordered, and checks it three ways, because the three failures need three different fixes: did the move happen at all, was it the right way round, and was it held or did part of the travel run backwards. A clip can be smooth, coherent, and entirely consistent with one physical camera, and still be a pan sold as a push-in. Both facts are reported, and neither is allowed to stand in for the other.

The governing requirement was silence on real footage, because the first thing anyone does with a noisy alarm is stop reading it. Across eleven shots of real single-camera material nothing fires except one soft finding on a stage camera that genuinely reframed and came back. The real maxima sit under the shipped bounds: incoherence 0.00117 against a soft bound of 0.004, closure 0.0017 against 0.02, jerk 1.78 against 1.9. That last margin is 1.07x, which is thin, and it is stated rather than smoothed over. Alongside it: 43 unit tests, and seven synthetic controls that each have to name their own fault, because “something is wrong” is not a finding.

Three checks were removed because they failed that validation, and they are the part worth reading. The intended headline feature was catching invented, melting geometry. Measured over a fixed warp budget, real footage reached a structural residual of 1.06 while a literal cross-dissolve between two entirely different worlds read 0.358. Real scenes contain people, LED walls and vision mixes, so they change structure more than melting geometry does; the ordering is not marginal, it is reversed by a factor of three, and no threshold separates the two in either direction. So that check is not shipped, and the dissolve is pinned as a test that must keep reading clean. Counting direction changes went the same way: an operator riding a zoom rocker produced 27 reversals on real footage against 3 in a deliberately broken control, so the count carries no signal on its own.

Editing is handled rather than assumed away. A clip containing cuts is segmented and each shot measured on its own, because run as one take an edited reel reported thirteen reversals and a broken closure - all true, and useless, since there were six shots and no single camera move was ever there to hold.

To be precise about what is not validated: no generated video is in the calibration set. Every clip in it is real camera footage or a synthetic control, so the bounds are validated for not flagging reality and for naming known faults, and are unvalidated against the particular failure modes of any one generator. Eleven real shots is a small sample, all from live-event cameras: no drone, no gimbal, no handheld documentary, no film scan. Two of the broken bounds have no positive control behind them and the calibration says which.

It needs numpy, pillow and ffmpeg. No OpenCV, no torch, no network, no GPU - a tool that needs a two-gigabyte wheel to measure a camera move does not get run.

github.com/Syamjith-NK/shotdrift · MIT · 43 tests, and the calibration published beside the code so the thresholds can be re-derived

pip install arabic-lint. Finds Arabic that was corrupted before it was stored — the reshape-then-bidi recipe written back into JSON, localisation files and exports, where it survives every later render. Zero dependencies, exits non-zero, drops straight into CI.

It refuses to repair stored text, and that asymmetry is the point. Undoing stored corruption round-trips exactly until the text contains a lam-alef ligature, and then السلام comes back as السالم — a real word, a different word, one that survives a human proofread. So there is no automatic repair for damage already written to disk. --fix rewrites source only, and only the form where the whole recipe sits inside one expression, because get_display(reshape(text)) can provably become text. Where the two halves are split across lines it refuses and says why: deleting just the second call strips the reordering and leaves the shaping applied, which is still wrong and now looks fixed.

It also carries a source check, which finds the code that will corrupt text at render time rather than the damage after the fact. The recipe appears in 3,168 indexed Python files, and reporting all of them would be worthless: whether it is a bug depends entirely on what draws the text. matplotlib 3.11 and Raqm-enabled Pillow shape and reorder for you, so pre-shaping reverses the string. ReportLab and non-Raqm Pillow do not, so pre-shaping is required. The check stays silent on the second group, on terminal output, and on a helper a script never calls.

I read six real projects to build it. Three were broken when I read them in September and were filed: a Discord bot whose chart helper pre-shaped unconditionally, on the pre-3.11 assumption, a ComfyUI node built to render Persian, and an Arabic poetry dataset. A fourth report went to a sibling project by the same maintainer carrying the same helper on the same line, found while reading the first. The other three of the six were correct and were left alone. Deciding which is which is the work.

The state today: the two bots are fixed. That helper now gates on the installed matplotlib version and documents it, so the description above is what was true when it was reported, not what is true now. The ComfyUI node and the poetry dataset are still open, with no reply.

pypi.org/project/arabic-lint · MIT · the write-up

A generative image product shipped end to end: application, authentication, billing, review and release. Approved and live.

A public interactive piece: visitors direct a single cinematic frame and watch it generated. Live on this site.