v1.0 · test set v0.1 · 2026-10-14
How we score Arabic support
No tool has a published score. Test set v0.1 is a protocol, the run has not started, and scores.json stores null on every row until a real run is added.
Two layers, no silent merge
The directory review produces a number out of 100 only when five raw criteria, each from 0 to 10, are present: the rounded sum of weight × raw × 10. If a criterion is missing, the result stays empty. We do not show a partial total as if it were final.
Test set v0.1 scores sections (MSA, Gulf, Egyptian, Levantine, direction, diacritics, OCR, speech, and a niche) with item scores from 0 to 5. The specs do not fix a formula that folds those sections into the 35% “Arabic quality” criterion. That formula will not be chosen in code before the first run, and the methodology version will move when it is fixed.
Directory-review weights
| Score breakdown | Weight |
|---|---|
| Arabic support quality | 35% |
| Accuracy and output quality | 20% |
| Price and value | 15% |
| Ease of use | 15% |
| Privacy and data handling | 15% |
Σ = 1
How a run works
- Prompts are copied verbatim. An edit gets a new version suffix.
- Every run is logged: tool, plan, model version, interface, UI language, Saudi time, item id, output, scorer, and scores.
- A fresh session, no custom memory, default settings, Arabic UI when the tool has one.
- Three attempts. The item score is the median. A refusal or error is 0 with a note.
- Two native-speaker scorers, one of them Gulf for Gulf items. A gap above one point brings in a third. A model may pre-score. A person decides.
- Transcripts and the date are published. A later weekly re-run waits until scores exist to re-run.
Item scale
0 fail, 1 mostly wrong, 2 poor, 3 usable with errors, 4 good with small notes, 5 native-like. Item score = mean of its criteria ÷ 5 × 100. Section score = mean of its items. The section composite is a weighted mean of the sections that apply, with weights re-normalised. Nominal weights: MSA 20, Gulf 15, Egyptian 10, Levantine 10, direction 10, diacritics 5, OCR 10, speech 10, niche 10.
What applies to which tool
Chat, writing, and agents: MSA, dialects, direction, and diacritics, plus a niche item for agents. Coding: a short MSA check, direction, and vibe-coding items. Image and video: text inside the image. Voice: synthesis and recognition. OCR: its own items. Games: player slang and UI strings.
Prompts you can already copy
Full prompts that do not need a sample file are in the prompt library, marked untested: VC-01, VC-02, AG-01, GM-02, MSA-02, and GLF-02. The other items (a news summary, parsing a sentence, arithmetic, idioms, OCR, speech) need sample files that are not in the repo yet, so they cannot be reproduced today. The missing set is a news text, a bidi mix, a vocalised poem, five OCR images with ground truth, seven audio clips, a synthetic invoice email, and a CSS direction bug.
What we will not do
- We will not turn a vendor’s published figure into our score.
- We will not put a temporary guess on the page.
- We will not synthesise a user rating. A visitor average is written only after visible votes with a real count.
Limits
Publishing the prompts, and the answer key for some items, makes memorisation possible. v0.1 accepts that because the goal is reproduction and citation. An item will be replaced with a new version if memorisation shows up. Audio and image samples are absent, so those items cannot be run yet. The Maghrebi cell on a tool page stays empty until items for it exist.
scores.json · scores.csv · CC BY 4.0