claude_fable_5 v2 Evaluation

Generated: 2026-07-25 12:52:09 UTC

Pairs100
Distinct lemmas100
Slope0.909
R^20.985
BLEU-454.3%
chrF++74.4%
METEOR78.1%
ROUGE-L79.1%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2016

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Fable 5; copied from legacy_scholarly v2 (source profile_version_id=274); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.909
Intercept2.460
R^20.985
P-value2.28e-91
Pearson r0.993
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.590
Mean absolute length residual2.869
Mean absolute percent error6.2%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.401, R^2 = 0.161, p = 3.55e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.446, R^2 = 0.199, p = 3.37e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.457, R^2 = 0.209, p = 1.73e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.442, R^2 = 0.195, p = 4.24e-06.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.327, R^2 = 0.107, p = 0.0009.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.349, R^2 = 0.122, p = 0.0004.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.339, R^2 = 0.115, p = 0.0006.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.340, R^2 = 0.115, p = 0.0005.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.401 0.161 3.55e-05 -0.00250 ok
chrF++ 100 significant negative -0.446 0.199 3.37e-06 -0.00165 ok
METEOR 100 significant negative -0.457 0.209 1.73e-06 -0.00187 ok
ROUGE-L 100 significant negative -0.442 0.195 4.24e-06 -0.00161 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.327 0.107 0.0009 -0.00229 ok
Trigram recall 100 significant negative -0.349 0.122 0.0004 -0.00243 ok
Trigram F1 100 significant negative -0.339 0.115 0.0006 -0.00236 ok
Trigram Jaccard 100 significant negative -0.340 0.115 0.0005 -0.00232 ok

Translation Similarity Metrics

Mean BLEU-454.3%
Mean chrF++74.4%
Mean METEOR78.1%
Mean ROUGE-L79.1%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU49.8%
Unigram F179.8%
Bigram F156.6%
Trigram precision44.1%
Trigram recall42.4%
Trigram F143.2%
Trigram Jaccard27.6%
4-gram F133.8%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 107 -20.866 16.3% 43.9% 45.3% 50.6% N/A N/A N/A reviewed/approved 7007 claude-fable-5
2604 Καρχηδών 117 125 16.217 35.8% 65.8% 67.2% 66.4% N/A N/A N/A reviewed/approved 7015 claude-fable-5
2484 Καρία 270 233 -14.820 40.6% 64.2% 61.8% 68.0% N/A N/A N/A reviewed/approved 7010 claude-fable-5
2626 Κατάνη 95 99 10.210 36.8% 62.8% 68.7% 67.0% N/A N/A N/A reviewed/approved 7026 claude-fable-5
2603 Κάρυστος 175 171 9.510 46.5% 67.7% 69.5% 70.5% N/A N/A N/A reviewed/approved 7014 claude-fable-5
7266 Κώμη 79 83 8.749 23.2% 50.7% 45.0% 54.3% N/A N/A N/A reviewed/approved 7059 claude-fable-5
2116 Κάληρος 40 31 -7.810 25.8% 58.1% 62.4% 64.8% N/A N/A N/A reviewed/approved 6987 claude-fable-5
2328 Καλλίπολις 43 49 7.464 45.6% 66.2% 70.0% 69.6% N/A N/A N/A reviewed/approved 6994 claude-fable-5
2119 Κάλλατις 65 68 6.472 49.5% 71.6% 75.4% 72.2% N/A N/A N/A reviewed/approved 6990 claude-fable-5
2342 Κάμιρος 54 58 6.468 56.0% 78.5% 75.2% 82.1% N/A N/A N/A reviewed/approved 7002 claude-fable-5
2465 Κάνωπος 83 84 6.115 43.0% 67.6% 81.2% 77.8% N/A N/A N/A reviewed/approved 7006 claude-fable-5
2605 Κάσιον 56 59 5.651 50.0% 73.6% 81.3% 74.8% N/A N/A N/A reviewed/approved 7016 claude-fable-5
7255 Κυτέριον 22 28 5.548 52.4% 74.4% 70.6% 72.0% N/A N/A N/A reviewed/approved 7048 claude-fable-5
7260 Κυχρεῖος πάγος 61 63 5.107 49.2% 70.4% 76.6% 67.7% N/A N/A N/A reviewed/approved 7053 claude-fable-5
2609 Κάσπειρος 135 130 4.860 42.8% 65.3% 58.7% 65.7% N/A N/A N/A reviewed/approved 7018 claude-fable-5
3288 Κορώνεια 110 107 4.579 44.1% 67.7% 67.5% 74.3% N/A N/A N/A reviewed/approved 7031 claude-fable-5
7251 Κύρτος 66 67 4.563 38.1% 58.3% 71.5% 66.2% N/A N/A N/A reviewed/approved 7044 claude-fable-5
7259 Κύφος 54 56 4.468 66.3% 85.3% 91.0% 89.1% N/A N/A N/A reviewed/approved 7052 claude-fable-5
7249 Κύρρος 45 39 -4.353 42.1% 61.9% 66.2% 71.4% N/A N/A N/A reviewed/approved 7042 claude-fable-5
2330 Καλύβη 24 20 -4.270 50.7% 69.5% 65.0% 81.8% N/A N/A N/A reviewed/approved 6996 claude-fable-5
7257 Κυτώνιον 25 21 -4.179 72.4% 80.7% 79.6% 82.6% N/A N/A N/A reviewed/approved 7050 claude-fable-5
2079 Καισάρεια 39 34 -3.901 44.7% 68.2% 77.4% 82.2% N/A N/A N/A reviewed/approved 6976 claude-fable-5
7246 Κύρις 20 17 -3.635 67.0% 80.8% 85.2% 86.5% N/A N/A N/A reviewed/approved 7039 claude-fable-5
2331 Κάλυδνα 43 45 3.464 56.0% 78.4% 89.6% 90.9% N/A N/A N/A reviewed/approved 6997 claude-fable-5
2059 Καδμεία 20 24 3.365 41.7% 71.7% 74.9% 77.3% N/A N/A N/A reviewed/approved 6966 claude-fable-5
7261 Κύψελα 34 30 -3.357 66.4% 78.6% 82.9% 84.4% N/A N/A N/A reviewed/approved 7054 claude-fable-5
2802 Κάδρεμα 24 21 -3.270 52.0% 68.0% 70.8% 78.3% N/A N/A N/A reviewed/approved 6969 claude-fable-5
2056 Καβειρία 112 101 -3.239 34.8% 72.3% 67.8% 66.7% N/A N/A N/A reviewed/approved 6963 claude-fable-5
2597 Καρπασία 112 101 -3.239 45.6% 72.6% 75.4% 77.9% N/A N/A N/A reviewed/approved 7011 claude-fable-5
3530 Κριώα 26 23 -3.087 42.4% 70.3% 76.2% 77.6% N/A N/A N/A reviewed/approved 7034 claude-fable-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.111.

Positive termCoefficientDocsMean presentMean absent
απο 1.5425 27 2.702 -0.999
και 1.2817 66 0.221 -0.429
δε και 0.9597 8 2.281 -0.198
τε 0.8876 5 4.457 -0.235
οι 0.8673 11 3.002 -0.371
τον 0.7776 10 2.619 -0.291
εστι δε και 0.7665 3 7.087 -0.219
εν τη 0.7610 7 3.648 -0.275
πολις 0.7524 68 0.541 -1.150
της 0.7347 35 0.347 -0.187
και αλλη 0.7103 3 9.544 -0.295
ου 0.6919 14 2.139 -0.348
εκαλειτο δε 0.6336 3 3.636 -0.112
αφ 0.6267 6 3.087 -0.197
πολιτης 0.6238 18 2.388 -0.524
τη 0.6208 15 1.456 -0.257
παιδος 0.6146 4 4.169 -0.174
απο του 0.6026 9 3.061 -0.303
εστι δε 0.5963 5 3.480 -0.183
αλλη 0.5834 4 7.025 -0.293
Negative termCoefficientDocsMean presentMean absent
γαρ -1.5682 9 -1.710 0.169
δια -1.0528 16 -0.226 0.043
το -1.0351 70 -0.132 0.308
δια του -0.9397 10 -1.357 0.151
ηρωδιανος -0.6651 3 -12.016 0.372
φησιν -0.6569 10 -2.825 0.314
ζευς -0.6508 3 -11.373 0.352
θρακης -0.6235 5 -2.485 0.131
το εθνικον -0.5693 56 -0.533 0.678
εθνικον -0.5693 56 -0.533 0.678
πολις θρακης -0.5635 4 -2.610 0.109
οτι -0.5530 5 -0.252 0.013
μεταξυ -0.4821 5 -1.186 0.062
εθνος -0.4769 8 -2.823 0.245
εξ -0.4647 3 -6.280 0.194
λυκιας -0.4561 3 -2.867 0.089
δε οτι -0.4553 3 -2.744 0.085
εις -0.4134 7 -3.065 0.231
χωρα -0.3875 6 -3.114 0.199
τοπικα -0.3829 3 -2.498 0.077

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.229.

Positive termCoefficientDocsMean presentMean absent
γαρ 2.0276 9 7.337 2.427
δε 1.6137 46 3.931 1.964
δια 1.5982 16 6.270 2.221
δια του 1.5562 10 7.792 2.322
του 1.3338 37 4.468 1.930
εκαλειτο 1.1867 7 7.556 2.516
εν 1.1124 37 4.079 2.159
την 1.0311 13 6.578 2.315
εκαλειτο δε 1.0002 3 13.516 2.540
δε και 0.8565 8 6.961 2.513
δε της 0.8522 6 9.060 2.474
απο 0.8501 27 4.970 2.092
της 0.8481 35 4.261 2.119
ουτως 0.8220 5 8.763 2.559
και 0.8045 66 3.527 1.592
ος 0.8038 6 7.687 2.562
τω 0.7552 16 5.218 2.422
οτι 0.6535 5 5.218 2.745
εκαλειτο δε και 0.6529 3 13.516 2.540
ζευς 0.6315 3 12.418 2.574
Negative termCoefficientDocsMean presentMean absent
ως -0.9313 52 2.856 2.883
το εθνικον -0.8008 56 2.767 2.998
εθνικον -0.8008 56 2.767 2.998
προς τη -0.5445 4 1.682 2.919
πλησιον -0.4538 8 1.505 2.988
δευτερω -0.4142 3 1.212 2.920
τους -0.4141 4 1.991 2.906
στραβων -0.4043 15 3.154 2.819
τεταρτω -0.3796 3 1.238 2.920
καππαδοκιας -0.3720 3 2.228 2.889
μια -0.3719 3 1.157 2.922
παυσανιας -0.3689 4 1.538 2.925
νησος -0.3580 9 2.422 2.913
εθνος -0.3460 8 3.374 2.825
εκαταιος ασια -0.3423 5 1.204 2.957
ασια -0.3423 5 1.204 2.957
ηροδοτος -0.3383 9 3.048 2.851
πορρω -0.3160 3 0.934 2.929
ου πορρω -0.3160 3 0.934 2.929
φασι -0.2866 8 3.154 2.844

AI English terms predicting large absolute residuals

Features: 317; ridge alpha: 2.894; cross-validated R^2: -0.170.

Positive termCoefficientDocsMean presentMean absent
for 2.4172 12 6.621 2.357
the 2.3488 94 2.932 1.879
was 1.5974 17 5.393 2.352
with 1.4924 16 5.816 2.308
with the 1.4331 6 7.927 2.546
all 1.3390 4 9.107 2.609
called 1.3027 24 5.417 2.065
whose 1.2662 3 9.486 2.664
athens 1.2316 3 15.764 2.470
was called 1.1640 5 9.293 2.531
it 1.1063 36 4.125 2.162
diphthong 1.0768 3 12.975 2.557
at 1.0023 7 9.032 2.405
on 0.9828 16 4.666 2.527
the form 0.9050 3 12.975 2.557
so 0.8973 10 7.336 2.373
it with 0.8934 3 12.835 2.561
has 0.8719 9 7.004 2.460
and 0.8664 58 3.688 1.738
zeus 0.8158 5 10.483 2.468
Negative termCoefficientDocsMean presentMean absent
as in -0.9395 33 2.258 3.170
the ethnonym is -0.8303 52 2.715 3.037
ethnonym is -0.8303 52 2.715 3.037
ethnonym -0.7826 58 2.794 2.973
the ethnonym -0.7686 57 2.799 2.962
near -0.7290 16 1.954 3.043
been -0.6648 4 1.618 2.921
kappadokia -0.5703 4 1.857 2.911
a city -0.5570 64 2.867 2.873
one of -0.4727 5 1.754 2.928
as -0.4710 52 3.353 2.345
promontory -0.4588 5 2.131 2.908
says that -0.4563 3 1.258 2.919
having -0.4511 3 0.948 2.928
crete -0.4511 3 0.948 2.928
island -0.4500 14 2.619 2.910
about -0.4245 3 1.126 2.923
book -0.4123 39 2.547 3.075
a city of -0.4062 49 2.822 2.914
herodotus -0.4035 9 3.048 2.851

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: -0.109.

Positive termCoefficientDocsMean presentMean absent
with 3.8820 16 5.934 2.285
the 2.3595 90 3.010 1.601
is with 1.7351 4 13.680 2.419
when 1.7043 7 9.305 2.385
was 1.5598 20 5.433 2.228
on 1.4683 16 4.842 2.493
sanctuary 1.4422 3 10.877 2.621
for 1.3689 11 5.929 2.491
on the 1.3018 10 5.009 2.631
long 1.2396 3 14.812 2.500
and 1.1780 52 3.938 1.711
this 1.1745 15 7.398 2.070
diphthong 1.1408 3 12.975 2.557
used to be 1.1361 6 8.769 2.493
forms 1.1356 4 9.918 2.575
called 1.0614 21 5.149 2.263
of 1.0551 66 3.549 1.550
with the 1.0138 5 9.524 2.519
because 0.9928 12 7.173 2.282
their 0.9150 3 13.516 2.540
Negative termCoefficientDocsMean presentMean absent
as in -1.4096 29 2.329 3.089
the ethnonym is -1.1339 52 2.697 3.056
ethnonym is -1.1339 52 2.697 3.056
kappadokia -1.0628 3 1.176 2.921
ethnonym -0.9886 57 2.837 2.912
the ethnonym -0.9698 56 2.842 2.903
near -0.9045 16 2.491 2.941
as -0.8100 60 2.999 2.675
but -0.7600 12 2.323 2.944
ethnonyms -0.7045 3 1.621 2.908
is a -0.6874 31 3.256 2.695
crete -0.6700 3 0.948 2.928
a city -0.6681 64 2.858 2.888
a city in -0.6330 40 2.889 2.856
it is -0.5762 24 4.116 2.475
city in -0.5740 41 2.956 2.808
or after -0.5713 3 2.183 2.890
island -0.5150 14 2.619 2.910
a people -0.5091 5 1.852 2.923
strabo -0.4939 15 3.154 2.819

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.