claude_sonnet_5 v2 Evaluation

Generated: 2026-09-20 11:19:48 UTC

Pairs100
Distinct lemmas100
Slope0.919
R^20.984
BLEU-447.2%
chrF++69.7%
METEOR75.5%
ROUGE-L76.8%
BERTScore96.1%
COMET79.5%
BLEURT74.7%

Profile version id: 2010

First translation using this version: 2026-07-03

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Sonnet 5; copied from legacy_scholarly v2 (source profile_version_id=274); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.919
Intercept1.783
R^20.984
P-value6.44e-90
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.370
Mean absolute length residual2.864
Mean absolute percent error6.2%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.335, R^2 = 0.112, p = 0.0007.
chrF++ vs passage length
chrF++: significant negative; r = -0.360, R^2 = 0.129, p = 0.0002.
METEOR vs passage length
METEOR: significant negative; r = -0.504, R^2 = 0.254, p = 8.82e-08.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.494, R^2 = 0.244, p = 1.78e-07.
BERTScore vs passage length
BERTScore: significant negative; r = -0.536, R^2 = 0.288, p = 8.82e-09.
COMET vs passage length
COMET: significant negative; r = -0.412, R^2 = 0.170, p = 2.03e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.722, R^2 = 0.521, p = 2.45e-17.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.360, R^2 = 0.129, p = 0.0002.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.364, R^2 = 0.133, p = 0.0002.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.363, R^2 = 0.132, p = 0.0002.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.363, R^2 = 0.132, p = 0.0002.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.335 0.112 0.0007 -0.00201 ok
chrF++ 100 significant negative -0.360 0.129 0.0002 -0.00127 ok
METEOR 100 significant negative -0.504 0.254 8.82e-08 -0.00221 ok
ROUGE-L 100 significant negative -0.494 0.244 1.78e-07 -0.00186 ok
BERTScore 100 significant negative -0.536 0.288 8.82e-09 -0.00040 ok
COMET 100 significant negative -0.412 0.170 2.03e-05 -0.00071 ok
BLEURT 100 significant negative -0.722 0.521 2.45e-17 -0.00222 ok
Trigram precision 100 significant negative -0.360 0.129 0.0002 -0.00252 ok
Trigram recall 100 significant negative -0.364 0.133 0.0002 -0.00251 ok
Trigram F1 100 significant negative -0.363 0.132 0.0002 -0.00251 ok
Trigram Jaccard 100 significant negative -0.363 0.132 0.0002 -0.00229 ok

Translation Similarity Metrics

Mean BLEU-447.2%
Mean chrF++69.7%
Mean METEOR75.5%
Mean ROUGE-L76.8%
Mean BERTScore96.1%
Mean COMET79.5%
Mean BLEURT74.7%
Legacy smoothed corpus BLEU46.2%
Unigram F177.7%
Bigram F153.6%
Trigram precision40.5%
Trigram recall38.7%
Trigram F139.6%
Trigram Jaccard24.7%
4-gram F130.1%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 132 22.659 28.1% 58.6% 59.8% 59.2% 93.0% 78.3% 62.2% reviewed/approved 6329 claude-sonnet-5
2468 Καπετώλιον 138 112 -16.647 18.0% 47.1% 45.9% 51.2% 88.8% 71.5% 53.7% reviewed/approved 6321 claude-sonnet-5
2484 Καρία 270 237 -12.995 30.5% 57.6% 54.7% 60.4% 91.1% 80.5% 53.2% reviewed/approved 6324 claude-sonnet-5
2056 Καβειρία 112 92 -12.745 33.1% 67.6% 61.0% 65.7% 95.5% 78.5% 63.4% reviewed/approved 6277 claude-sonnet-5
2342 Κάμιρος 54 62 10.575 44.1% 74.5% 72.2% 74.1% 96.6% 80.6% 70.4% reviewed/approved 6316 claude-sonnet-5
3288 Κορώνεια 110 113 10.094 41.9% 67.3% 63.5% 70.5% 94.8% 74.5% 62.4% reviewed/approved 6345 claude-sonnet-5
2609 Κάσπειρος 135 134 8.111 44.0% 66.8% 61.6% 69.1% 90.1% 73.2% 65.1% reviewed/approved 6332 claude-sonnet-5
2455 Κάναστρον 60 49 -7.941 27.9% 60.0% 57.3% 66.1% 94.7% 71.3% 57.8% reviewed/approved 6319 claude-sonnet-5
2603 Κάρυστος 175 170 7.339 38.8% 63.1% 63.6% 65.5% 95.0% 78.3% 64.4% reviewed/approved 6328 claude-sonnet-5
2055 Καβασσός 109 109 7.013 28.8% 61.3% 60.0% 64.2% 94.4% 76.7% 56.2% reviewed/approved 6276 claude-sonnet-5
7255 Κυτέριον 22 29 6.993 55.4% 77.6% 73.2% 74.5% 97.2% 79.6% 79.6% reviewed/approved 6362 claude-sonnet-5
2116 Κάληρος 40 32 -6.555 13.8% 49.5% 56.4% 61.1% 93.1% 77.3% 71.1% reviewed/approved 6301 claude-sonnet-5
2346 Κάναι 77 79 6.431 59.0% 75.4% 87.4% 85.9% 95.7% 78.2% 71.7% reviewed/approved 6318 claude-sonnet-5
7266 Κώμη 79 69 -5.408 20.7% 45.8% 39.6% 48.6% 92.9% 72.7% 53.1% reviewed/approved 6373 claude-sonnet-5
2607 Κάσος 57 49 -5.183 26.3% 58.1% 60.0% 75.5% 94.5% 76.0% 68.5% reviewed/approved 6331 claude-sonnet-5
7260 Κυχρεῖος πάγος 61 63 5.140 54.1% 73.1% 79.8% 74.2% 96.6% 76.6% 68.5% reviewed/approved 6367 claude-sonnet-5
7254 Κύτα 84 84 4.996 36.2% 65.8% 72.1% 72.6% 95.4% 78.6% 69.4% reviewed/approved 6361 claude-sonnet-5
2605 Κάσιον 56 58 4.736 60.9% 77.3% 81.3% 77.2% 97.0% 78.7% 74.1% reviewed/approved 6330 claude-sonnet-5
2470 Καππαδοκία 68 69 4.705 32.3% 65.0% 85.4% 80.3% 96.3% 76.4% 72.3% reviewed/approved 6322 claude-sonnet-5
2085 Καλάσιρις 13 18 4.266 20.8% 66.1% 77.7% 64.5% 94.7% 76.0% 74.8% reviewed/approved 6296 claude-sonnet-5
2326 Καλλίαρος 47 49 4.010 49.0% 73.5% 88.2% 85.4% 97.1% 75.4% 74.9% reviewed/approved 6306 claude-sonnet-5
2330 Καλύβη 24 20 -3.846 41.4% 62.7% 61.9% 72.7% 96.2% 79.9% 76.6% reviewed/approved 6310 claude-sonnet-5
2331 Κάλυδνα 43 45 3.687 48.0% 74.4% 85.8% 88.6% 96.1% 81.4% 76.6% reviewed/approved 6311 claude-sonnet-5
3530 Κριώα 26 22 -3.685 51.4% 70.6% 77.8% 83.3% 97.3% 79.7% 76.3% reviewed/approved 6348 claude-sonnet-5
7253 Κυρτώνιος 18 22 3.670 35.1% 65.1% 68.6% 65.0% 96.5% 76.7% 80.9% reviewed/approved 6360 claude-sonnet-5
2054 Καβαλίς 65 65 3.462 67.8% 77.0% 69.7% 81.5% 97.1% 77.3% 75.0% reviewed/approved 6275 claude-sonnet-5
3496 Κοτιάειον 71 64 -3.053 28.8% 52.8% 54.2% 60.7% 93.5% 75.1% 73.4% reviewed/approved 6346 claude-sonnet-5
2465 Κάνωπος 83 81 2.915 40.3% 66.1% 77.0% 75.6% 95.5% 76.7% 69.1% reviewed/approved 6320 claude-sonnet-5
7264 Κωλιάς 61 55 -2.860 28.7% 57.4% 54.5% 67.2% 95.1% 72.5% 57.3% reviewed/approved 6371 claude-sonnet-5
7257 Κυτώνιον 25 22 -2.765 75.1% 80.0% 84.7% 89.4% 97.6% 81.1% 77.6% reviewed/approved 6364 claude-sonnet-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 8.377; cross-validated R^2: -0.016.

Positive termCoefficientDocsMean presentMean absent
δε και 1.1361 8 4.485 -0.390
εστι δε και 0.8606 3 11.480 -0.355
απο 0.8030 27 1.583 -0.586
οι 0.7615 11 3.033 -0.375
εκαλειτο δε 0.6398 3 5.667 -0.175
πολις 0.6184 68 0.555 -1.180
εστι δε 0.6113 5 4.415 -0.232
δυναται 0.5949 3 8.033 -0.248
ου 0.5943 14 1.978 -0.322
ος 0.5801 6 2.966 -0.189
πολιτης 0.5719 18 3.194 -0.701
τη 0.5247 15 1.705 -0.301
ομηρος 0.4701 3 6.425 -0.199
τε 0.4237 5 1.956 -0.103
παιδος 0.4126 4 3.869 -0.161
στραβων 0.4052 15 1.820 -0.321
εν τη 0.3979 7 2.531 -0.190
και αλλη 0.3970 3 8.846 -0.274
αλλη 0.3805 4 6.778 -0.282
εκαλειτο 0.3737 7 1.702 -0.128
Negative termCoefficientDocsMean presentMean absent
το -0.9879 70 -0.206 0.481
γαρ -0.8697 9 -1.119 0.111
δια του -0.6330 10 -1.283 0.143
δια -0.6073 16 0.119 -0.023
περι -0.5447 11 -2.967 0.367
θρακης -0.5423 5 -3.152 0.166
το εθνικον -0.5080 56 -0.715 0.910
εθνικον -0.5080 56 -0.715 0.910
ηρωδιανος -0.4814 3 -10.523 0.325
οτι -0.4755 5 -1.363 0.072
ει -0.4196 4 -5.970 0.249
ης -0.4042 3 -4.355 0.135
τα -0.4013 12 -1.186 0.162
του -0.3840 37 1.027 -0.603
εθνος -0.3503 8 -3.116 0.271
οικητωρ -0.3377 6 -3.626 0.231
μεν -0.3352 4 -6.655 0.277
πολις θρακης -0.3322 4 -1.955 0.081
τους -0.2885 4 -1.881 0.078
ζευς -0.2832 3 -7.737 0.239

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.099.

Positive termCoefficientDocsMean presentMean absent
και 2.0656 66 3.708 1.225
δε 1.9503 46 4.271 1.665
εκαλειτο 1.7442 7 8.872 2.412
την 1.7065 13 8.077 2.085
γαρ 1.6856 9 7.306 2.425
ος 1.2950 6 9.533 2.438
εκαλειτο δε 1.2645 3 14.331 2.509
δε και 1.2325 8 7.733 2.441
εστι δε και 1.0967 3 11.480 2.598
δια 1.0141 16 5.727 2.319
εστι δε 1.0124 5 9.513 2.514
τα 0.9898 12 6.791 2.329
τε 0.9492 5 6.193 2.689
δια του 0.8971 10 6.792 2.428
της 0.8906 35 4.653 1.901
απο 0.7877 27 5.333 1.951
δε της 0.7811 6 9.196 2.460
εκαλειτο δε και 0.7390 3 14.331 2.509
εις 0.7088 7 8.476 2.442
του 0.6668 37 4.257 2.046
Negative termCoefficientDocsMean presentMean absent
ως -0.6944 52 3.081 2.629
προς -0.6937 15 2.644 2.903
προς τη -0.6301 4 0.803 2.950
το εθνικον -0.6070 56 3.006 2.683
εθνικον -0.6070 56 3.006 2.683
τον -0.5680 10 2.327 2.924
στραβων -0.5267 15 2.874 2.862
τεταρτω -0.5045 3 0.778 2.929
εκ -0.5000 6 2.221 2.905
πλησιον -0.4971 8 2.105 2.930
νησος -0.4330 9 1.974 2.952
τους -0.4315 4 1.999 2.900
οικητωρ -0.4122 6 3.731 2.809
αυτην -0.3735 9 2.570 2.893
και το -0.3554 9 3.146 2.836
πολιτης -0.3479 18 3.859 2.646
δυο -0.3379 4 3.096 2.854
εκαταιος -0.3343 14 2.686 2.893
πορρω -0.3292 3 0.636 2.933
ου πορρω -0.3292 3 0.636 2.933

AI English terms predicting large absolute residuals

Features: 295; ridge alpha: 5.878; cross-validated R^2: -0.156.

Positive termCoefficientDocsMean presentMean absent
and 1.7110 56 4.015 1.399
was 1.1617 17 6.191 2.183
for 1.1592 13 5.656 2.447
with 0.9692 17 6.300 2.160
the 0.9514 92 3.001 1.286
called 0.9484 26 5.121 2.071
of 0.8473 84 3.000 2.153
by 0.8354 12 7.155 2.279
also 0.8129 45 4.071 1.877
who 0.7322 9 6.948 2.460
as a 0.6445 3 7.210 2.730
says 0.6400 19 6.274 2.064
was called 0.6241 5 9.310 2.525
but 0.6222 23 5.673 2.025
form 0.6201 8 6.552 2.543
there 0.6118 32 4.710 1.995
athens 0.5477 3 13.474 2.536
is also 0.5346 30 4.174 2.303
after 0.5319 19 6.024 2.123
has 0.5262 5 9.842 2.497
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.5061 53 2.828 2.905
near -0.5027 19 2.212 3.017
as in -0.4914 32 2.481 3.044
the ethnonym -0.4330 57 2.878 2.845
ethnonym is -0.4200 54 3.011 2.691
ethnonym -0.3514 58 3.048 2.609
one -0.3334 10 2.945 2.855
hecataeus -0.3144 5 0.759 2.975
city of -0.3075 52 3.116 2.591
a city -0.3067 63 3.064 2.524
which the -0.3054 5 1.385 2.942
a city of -0.2894 50 3.129 2.599
and the -0.2810 10 2.872 2.863
strabo -0.2736 15 2.874 2.862
book -0.2695 40 2.744 2.944
city -0.2378 68 3.031 2.509
an island -0.2336 10 1.945 2.966
island -0.2310 14 2.491 2.925
those -0.2284 3 1.682 2.901
sea -0.2278 10 2.535 2.901

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: -0.109.

Positive termCoefficientDocsMean presentMean absent
and 2.0812 52 4.174 1.445
with 1.7924 16 5.815 2.302
the 1.6588 90 3.010 1.547
to 1.0735 31 4.791 1.998
this 1.0382 15 7.851 1.984
is with 1.0125 4 13.658 2.414
who 0.9446 13 5.910 2.409
also 0.9184 43 3.936 2.055
was 0.8696 20 5.299 2.255
used to be 0.8575 6 10.291 2.390
too 0.8368 3 12.307 2.572
used 0.8109 8 9.356 2.299
used to 0.8109 8 9.356 2.299
forms 0.7767 4 11.773 2.493
to be called 0.7748 5 10.881 2.442
called 0.7311 21 5.114 2.266
be 0.7308 15 6.966 2.140
the form 0.7182 9 7.263 2.429
from 0.6827 37 3.841 2.290
form 0.6585 11 6.205 2.451
Negative termCoefficientDocsMean presentMean absent
as in -1.0488 29 1.968 3.230
the ethnonym is -0.6335 52 2.691 3.052
ethnonym is -0.6335 52 2.691 3.052
an -0.5670 22 2.364 3.005
near -0.5525 16 2.357 2.961
a city in -0.5216 40 3.032 2.752
city in -0.4932 41 3.074 2.718
the ethnonym -0.4551 56 2.999 2.692
was from -0.4542 4 0.797 2.950
ethnonym -0.4247 57 3.070 2.591
a city -0.3506 64 3.029 2.571
strabo -0.3450 15 2.874 2.862
city -0.3348 70 2.985 2.582
one of -0.3105 5 1.635 2.929
book -0.3077 39 2.789 2.912
sicily -0.3024 3 0.937 2.924
of his -0.3007 18 2.609 2.920
island -0.2864 14 2.491 2.925
his -0.2848 34 2.707 2.945
name -0.2758 7 3.439 2.821

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.