claude_fable_5 v2 Evaluation

Generated: 2026-09-20 11:19:23 UTC

Pairs100
Distinct lemmas100
Slope0.909
R^20.985
BLEU-454.5%
chrF++74.6%
METEOR78.2%
ROUGE-L79.3%
BERTScore96.8%
COMET80.6%
BLEURT77.6%

Profile version id: 2016

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Fable 5; copied from legacy_scholarly v2 (source profile_version_id=274); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.909
Intercept2.478
R^20.985
P-value2.18e-91
Pearson r0.993
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.590
Mean absolute length residual2.863
Mean absolute percent error6.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.410, R^2 = 0.168, p = 2.29e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.456, R^2 = 0.208, p = 1.88e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.465, R^2 = 0.217, p = 1.07e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.456, R^2 = 0.208, p = 1.84e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.612, R^2 = 0.374, p = 1.37e-11.
COMET vs passage length
COMET: significant negative; r = -0.484, R^2 = 0.234, p = 3.45e-07.
BLEURT vs passage length
BLEURT: significant negative; r = -0.719, R^2 = 0.516, p = 3.85e-17.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.334, R^2 = 0.111, p = 0.0007.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.359, R^2 = 0.129, p = 0.0002.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.347, R^2 = 0.121, p = 0.0004.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.345, R^2 = 0.119, p = 0.0004.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.410 0.168 2.29e-05 -0.00254 ok
chrF++ 100 significant negative -0.456 0.208 1.88e-06 -0.00167 ok
METEOR 100 significant negative -0.465 0.217 1.07e-06 -0.00191 ok
ROUGE-L 100 significant negative -0.456 0.208 1.84e-06 -0.00165 ok
BERTScore 100 significant negative -0.612 0.374 1.37e-11 -0.00043 ok
COMET 100 significant negative -0.484 0.234 3.45e-07 -0.00087 ok
BLEURT 100 significant negative -0.719 0.516 3.85e-17 -0.00238 ok
Trigram precision 100 significant negative -0.334 0.111 0.0007 -0.00233 ok
Trigram recall 100 significant negative -0.359 0.129 0.0002 -0.00248 ok
Trigram F1 100 significant negative -0.347 0.121 0.0004 -0.00240 ok
Trigram Jaccard 100 significant negative -0.345 0.119 0.0004 -0.00235 ok

Translation Similarity Metrics

Mean BLEU-454.5%
Mean chrF++74.6%
Mean METEOR78.2%
Mean ROUGE-L79.3%
Mean BERTScore96.8%
Mean COMET80.6%
Mean BLEURT77.6%
Legacy smoothed corpus BLEU49.8%
Unigram F179.8%
Bigram F156.7%
Trigram precision44.1%
Trigram recall42.5%
Trigram F143.3%
Trigram Jaccard27.6%
4-gram F133.8%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 107 -20.857 16.3% 43.9% 45.3% 50.6% 88.2% 69.4% 51.1% reviewed/approved 7007 claude-fable-5
2604 Καρχηδών 117 125 16.222 35.8% 65.8% 67.2% 66.4% 94.5% 79.7% 64.3% reviewed/approved 7015 claude-fable-5
2484 Καρία 270 233 -14.786 40.6% 64.2% 61.8% 68.0% 92.7% 80.8% 57.2% reviewed/approved 7010 claude-fable-5
2626 Κατάνη 95 99 10.210 36.8% 62.8% 68.7% 67.0% 94.3% 74.0% 67.5% reviewed/approved 7026 claude-fable-5
2603 Κάρυστος 175 171 9.526 46.5% 67.7% 69.5% 70.5% 95.0% 77.6% 62.2% reviewed/approved 7014 claude-fable-5
7266 Κώμη 79 83 8.747 23.2% 50.7% 45.0% 54.3% 93.7% 72.6% 55.5% reviewed/approved 7059 claude-fable-5
2116 Κάληρος 40 31 -7.820 25.8% 58.1% 62.4% 64.8% 94.3% 76.0% 71.0% reviewed/approved 6987 claude-fable-5
2328 Καλλίπολις 43 49 7.455 45.6% 66.2% 70.0% 69.6% 94.9% 75.6% 69.8% reviewed/approved 6994 claude-fable-5
2119 Κάλλατις 65 68 6.467 49.5% 71.6% 75.4% 72.2% 96.2% 73.5% 76.7% reviewed/approved 6990 claude-fable-5
2342 Κάμιρος 54 58 6.461 56.0% 78.5% 75.2% 82.1% 97.5% 80.8% 71.2% reviewed/approved 7002 claude-fable-5
2465 Κάνωπος 83 84 6.113 43.0% 67.6% 81.2% 77.8% 95.6% 77.6% 71.8% reviewed/approved 7006 claude-fable-5
2605 Κάσιον 56 59 5.644 50.0% 73.6% 81.3% 74.8% 96.5% 77.9% 72.3% reviewed/approved 7016 claude-fable-5
7255 Κυτέριον 22 28 5.534 52.4% 74.4% 70.6% 72.0% 96.5% 78.1% 78.7% reviewed/approved 7048 claude-fable-5
7260 Κυχρεῖος πάγος 61 63 5.101 49.2% 70.4% 76.6% 67.7% 96.1% 76.8% 65.7% reviewed/approved 7053 claude-fable-5
2609 Κάσπειρος 135 130 4.868 42.8% 65.3% 58.7% 65.7% 90.2% 73.1% 66.3% reviewed/approved 7018 claude-fable-5
3288 Κορώνεια 110 107 4.582 44.1% 67.7% 67.5% 74.3% 96.0% 76.0% 68.6% reviewed/approved 7031 claude-fable-5
7251 Κύρτος 66 67 4.558 38.1% 58.3% 71.5% 66.2% 92.4% 71.5% 62.8% reviewed/approved 7044 claude-fable-5
7259 Κύφος 54 56 4.461 66.3% 85.3% 91.0% 89.1% 98.1% 78.0% 81.9% reviewed/approved 7052 claude-fable-5
7249 Κύρρος 45 39 -4.362 42.1% 61.9% 66.2% 71.4% 95.7% 73.6% 67.6% reviewed/approved 7042 claude-fable-5
2330 Καλύβη 24 20 -4.283 50.7% 69.5% 65.0% 81.8% 97.3% 80.3% 77.1% reviewed/approved 6996 claude-fable-5
7257 Κυτώνιον 25 21 -4.191 72.4% 80.7% 79.6% 82.6% 98.5% 84.6% 82.2% reviewed/approved 7050 claude-fable-5
2079 Καισάρεια 39 34 -3.911 48.2% 70.4% 80.1% 84.9% 96.1% 78.9% 74.9% reviewed/approved 6976 claude-fable-5
7246 Κύρις 20 17 -3.649 67.0% 80.8% 85.2% 86.5% 97.4% 84.2% 77.9% reviewed/approved 7039 claude-fable-5
2331 Κάλυδνα 43 45 3.455 56.0% 78.4% 89.6% 90.9% 96.9% 82.3% 79.4% reviewed/approved 6997 claude-fable-5
7261 Κύψελα 34 30 -3.368 66.4% 78.6% 82.9% 84.4% 98.4% 82.0% 81.1% reviewed/approved 7054 claude-fable-5
2059 Καδμεία 20 24 3.351 41.7% 71.7% 74.9% 77.3% 97.1% 83.0% 79.3% reviewed/approved 6966 claude-fable-5
2802 Κάδρεμα 24 21 -3.283 52.0% 68.0% 70.8% 78.3% 97.6% 83.2% 83.0% reviewed/approved 6969 claude-fable-5
2056 Καβειρία 112 101 -3.235 34.8% 72.3% 67.8% 66.7% 96.1% 79.1% 64.3% reviewed/approved 6963 claude-fable-5
2597 Καρπασία 112 101 -3.235 45.6% 72.6% 75.4% 77.9% 96.3% 77.4% 66.1% reviewed/approved 7011 claude-fable-5
3530 Κριώα 26 23 -3.100 42.4% 70.3% 76.2% 77.6% 97.1% 78.0% 76.2% reviewed/approved 7034 claude-fable-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.115.

Positive termCoefficientDocsMean presentMean absent
απο 1.5412 27 2.700 -0.999
και 1.2692 66 0.214 -0.416
δε και 0.9591 8 2.283 -0.199
τε 0.8875 5 4.453 -0.234
οι 0.8645 11 2.998 -0.371
τον 0.7748 10 2.614 -0.290
εστι δε και 0.7662 3 7.086 -0.219
εν τη 0.7596 7 3.639 -0.274
πολις 0.7405 68 0.533 -1.132
της 0.7290 35 0.342 -0.184
και αλλη 0.7101 3 9.546 -0.295
ου 0.6895 14 2.137 -0.348
εκαλειτο δε 0.6346 3 3.654 -0.113
αφ 0.6265 6 3.086 -0.197
τη 0.6186 15 1.452 -0.256
πολιτης 0.6185 18 2.381 -0.523
παιδος 0.6136 4 4.161 -0.173
απο του 0.6020 9 3.057 -0.302
εστι δε 0.5957 5 3.480 -0.183
αλλη 0.5833 4 7.024 -0.293
Negative termCoefficientDocsMean presentMean absent
γαρ -1.5661 9 -1.703 0.168
δια -1.0534 16 -0.227 0.043
το -1.0450 70 -0.140 0.326
δια του -0.9403 10 -1.359 0.151
ηρωδιανος -0.6650 3 -12.006 0.371
φησιν -0.6575 10 -2.824 0.314
ζευς -0.6504 3 -11.360 0.351
θρακης -0.6279 5 -2.496 0.131
το εθνικον -0.5799 56 -0.541 0.689
εθνικον -0.5799 56 -0.541 0.689
πολις θρακης -0.5681 4 -2.623 0.109
οτι -0.5545 5 -0.255 0.013
μεταξυ -0.4856 5 -1.193 0.063
εξ -0.4652 3 -6.281 0.194
λυκιας -0.4578 3 -2.875 0.089
δε οτι -0.4565 3 -2.751 0.085
εθνος -0.4145 8 -2.714 0.236
εις -0.4127 7 -3.058 0.230
χωρα -0.3898 6 -3.117 0.199
τοπικα -0.3840 3 -2.503 0.077

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.218.

Positive termCoefficientDocsMean presentMean absent
γαρ 2.0274 9 7.336 2.421
δε 1.6218 46 3.932 1.952
δια 1.6007 16 6.271 2.214
δια του 1.5575 10 7.790 2.315
του 1.3401 37 4.467 1.921
εκαλειτο 1.1897 7 7.552 2.510
εν 1.1257 37 4.078 2.149
την 1.0312 13 6.575 2.308
εκαλειτο δε 1.0008 3 13.511 2.534
της 0.8594 35 4.264 2.109
δε και 0.8571 8 6.958 2.507
δε της 0.8533 6 9.059 2.467
απο 0.8489 27 4.966 2.085
ουτως 0.8231 5 8.759 2.553
και 0.8191 66 3.528 1.572
ος 0.8052 6 7.685 2.555
τω 0.7594 16 5.217 2.415
οτι 0.6571 5 5.221 2.739
εκαλειτο δε και 0.6532 3 13.511 2.534
ζευς 0.6313 3 12.403 2.568
Negative termCoefficientDocsMean presentMean absent
ως -0.9219 52 2.857 2.869
το εθνικον -0.7882 56 2.770 2.982
εθνικον -0.7882 56 2.770 2.982
προς τη -0.5416 4 1.681 2.912
πλησιον -0.4466 8 1.508 2.981
εθνος -0.4248 8 3.265 2.828
τους -0.4119 4 1.994 2.899
δευτερω -0.4071 3 1.216 2.914
στραβων -0.3971 15 3.157 2.811
καππαδοκιας -0.3690 3 2.234 2.882
μια -0.3681 3 1.156 2.916
παυσανιας -0.3648 4 1.545 2.918
τεταρτω -0.3645 3 1.250 2.913
νησος -0.3530 9 2.426 2.906
των -0.3426 17 3.981 2.634
ηροδοτος -0.3327 9 3.048 2.845
εκαταιος ασια -0.3311 5 1.215 2.950
ασια -0.3311 5 1.215 2.950
πορρω -0.3094 3 0.945 2.922
ου πορρω -0.3094 3 0.945 2.922

AI English terms predicting large absolute residuals

Features: 317; ridge alpha: 2.894; cross-validated R^2: -0.166.

Positive termCoefficientDocsMean presentMean absent
for 2.4162 12 6.619 2.351
the 2.3528 94 2.925 1.894
was 1.6033 17 5.392 2.345
with 1.4918 16 5.816 2.300
with the 1.4318 6 7.922 2.540
all 1.3391 4 9.104 2.603
called 1.3048 24 5.413 2.058
whose 1.2665 3 9.485 2.658
athens 1.2322 3 15.763 2.464
was called 1.1647 5 9.288 2.525
it 1.1109 36 4.124 2.154
diphthong 1.0746 3 12.960 2.551
at 1.0021 7 9.029 2.399
on 0.9826 16 4.664 2.520
the form 0.9033 3 12.960 2.551
so 0.8988 10 7.338 2.366
it with 0.8946 3 12.825 2.555
and 0.8828 58 3.688 1.723
has 0.8721 9 6.999 2.454
after 0.8165 18 5.837 2.210
Negative termCoefficientDocsMean presentMean absent
as in -0.9358 33 2.259 3.160
the ethnonym is -0.8176 52 2.717 3.021
ethnonym is -0.8176 52 2.717 3.021
ethnonym -0.7713 58 2.796 2.956
the ethnonym -0.7573 57 2.801 2.945
near -0.7218 16 1.957 3.036
been -0.6634 4 1.620 2.915
one of -0.5735 5 1.574 2.931
kappadokia -0.5622 4 1.861 2.905
a city -0.5481 64 2.868 2.855
as -0.4643 52 3.353 2.332
promontory -0.4544 5 2.138 2.901
says that -0.4498 3 1.265 2.912
having -0.4496 3 0.948 2.922
crete -0.4496 3 0.948 2.922
island -0.4416 14 2.620 2.903
about -0.4235 3 1.123 2.917
book -0.4004 39 2.550 3.063
herodotus -0.3996 9 3.048 2.845
and that -0.3994 3 0.893 2.924

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: -0.113.

Positive termCoefficientDocsMean presentMean absent
with 3.8841 16 5.932 2.278
the 2.3345 90 3.003 1.607
is with 1.7399 4 13.674 2.413
when 1.7128 7 9.304 2.378
was 1.5789 20 5.433 2.220
on 1.4685 16 4.839 2.487
sanctuary 1.4438 3 10.871 2.615
for 1.3730 11 5.926 2.484
on the 1.3062 10 5.011 2.624
long 1.2419 3 14.797 2.494
and 1.1987 52 3.938 1.698
this 1.1890 15 7.397 2.063
diphthong 1.1431 3 12.960 2.551
used to be 1.1369 6 8.766 2.486
forms 1.1338 4 9.905 2.570
called 1.0729 21 5.147 2.256
of 1.0662 66 3.549 1.531
with the 1.0118 5 9.516 2.513
because 1.0031 12 7.172 2.275
their 0.9158 3 13.511 2.534
Negative termCoefficientDocsMean presentMean absent
as in -1.4134 29 2.329 3.081
the ethnonym is -1.1155 52 2.700 3.040
ethnonym is -1.1155 52 2.700 3.040
ethnonym -0.9736 57 2.839 2.895
the ethnonym -0.9530 56 2.845 2.886
kappadokia -0.9379 4 1.861 2.905
near -0.8828 16 2.493 2.933
as -0.8022 60 3.000 2.657
but -0.7620 12 2.322 2.937
ethnonyms -0.7090 3 1.617 2.902
is a -0.6846 31 3.255 2.687
crete -0.6710 3 0.948 2.922
a city -0.6669 64 2.859 2.869
a city in -0.6291 40 2.891 2.844
it is -0.5792 24 4.115 2.468
or after -0.5718 3 2.178 2.884
city in -0.5700 41 2.959 2.797
island -0.5008 14 2.620 2.903
a people -0.4987 5 1.860 2.916
strabo -0.4813 15 3.157 2.811

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.