claude_sonnet_5 v2 Evaluation

Generated: 2026-07-25 12:52:22 UTC

Pairs100
Distinct lemmas100
Slope0.919
R^20.984
BLEU-447.2%
chrF++69.6%
METEOR75.3%
ROUGE-L76.6%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2010

First translation using this version: 2026-07-03

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Sonnet 5; copied from legacy_scholarly v2 (source profile_version_id=274); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.919
Intercept1.765
R^20.984
P-value6.85e-90
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.370
Mean absolute length residual2.870
Mean absolute percent error6.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.334, R^2 = 0.112, p = 0.0007.
chrF++ vs passage length
chrF++: significant negative; r = -0.355, R^2 = 0.126, p = 0.0003.
METEOR vs passage length
METEOR: significant negative; r = -0.493, R^2 = 0.243, p = 1.84e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.482, R^2 = 0.232, p = 3.89e-07.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.354, R^2 = 0.125, p = 0.0003.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.356, R^2 = 0.127, p = 0.0003.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.356, R^2 = 0.127, p = 0.0003.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.357, R^2 = 0.127, p = 0.0003.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.334 0.112 0.0007 -0.00201 ok
chrF++ 100 significant negative -0.355 0.126 0.0003 -0.00126 ok
METEOR 100 significant negative -0.493 0.243 1.84e-07 -0.00217 ok
ROUGE-L 100 significant negative -0.482 0.232 3.89e-07 -0.00182 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.354 0.125 0.0003 -0.00248 ok
Trigram recall 100 significant negative -0.356 0.127 0.0003 -0.00246 ok
Trigram F1 100 significant negative -0.356 0.127 0.0003 -0.00247 ok
Trigram Jaccard 100 significant negative -0.357 0.127 0.0003 -0.00225 ok

Translation Similarity Metrics

Mean BLEU-447.2%
Mean chrF++69.6%
Mean METEOR75.3%
Mean ROUGE-L76.6%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU46.2%
Unigram F177.6%
Bigram F153.5%
Trigram precision40.4%
Trigram recall38.7%
Trigram F139.5%
Trigram Jaccard24.6%
4-gram F130.0%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 132 22.654 28.1% 58.6% 59.8% 59.2% N/A N/A N/A reviewed/approved 6329 claude-sonnet-5
2468 Καπετώλιον 138 112 -16.655 18.0% 47.1% 45.9% 51.2% N/A N/A N/A reviewed/approved 6321 claude-sonnet-5
2484 Καρία 270 237 -13.029 30.5% 57.6% 54.7% 60.4% N/A N/A N/A reviewed/approved 6324 claude-sonnet-5
2056 Καβειρία 112 92 -12.749 33.1% 67.6% 61.0% 65.7% N/A N/A N/A reviewed/approved 6277 claude-sonnet-5
2342 Κάμιρος 54 62 10.582 44.1% 74.5% 72.2% 74.1% N/A N/A N/A reviewed/approved 6316 claude-sonnet-5
3288 Κορώνεια 110 113 10.090 41.9% 67.3% 63.5% 70.5% N/A N/A N/A reviewed/approved 6345 claude-sonnet-5
2609 Κάσπειρος 135 134 8.103 44.0% 66.8% 61.6% 69.1% N/A N/A N/A reviewed/approved 6332 claude-sonnet-5
2455 Κάναστρον 60 49 -7.935 27.9% 60.0% 57.3% 66.1% N/A N/A N/A reviewed/approved 6319 claude-sonnet-5
2603 Κάρυστος 175 170 7.323 38.8% 63.1% 63.6% 65.5% N/A N/A N/A reviewed/approved 6328 claude-sonnet-5
2055 Καβασσός 109 109 7.010 28.8% 61.3% 60.0% 64.2% N/A N/A N/A reviewed/approved 6276 claude-sonnet-5
7255 Κυτέριον 22 29 7.006 55.4% 77.6% 73.2% 74.5% N/A N/A N/A reviewed/approved 6362 claude-sonnet-5
2116 Κάληρος 40 32 -6.545 13.8% 49.5% 56.4% 61.1% N/A N/A N/A reviewed/approved 6301 claude-sonnet-5
2346 Κάναι 77 79 6.434 59.0% 75.4% 87.4% 85.9% N/A N/A N/A reviewed/approved 6318 claude-sonnet-5
7266 Κώμη 79 69 -5.405 20.7% 45.8% 39.6% 48.6% N/A N/A N/A reviewed/approved 6373 claude-sonnet-5
2607 Κάσος 57 49 -5.176 26.3% 58.1% 60.0% 75.5% N/A N/A N/A reviewed/approved 6331 claude-sonnet-5
7260 Κυχρεῖος πάγος 61 63 5.146 54.1% 73.1% 79.8% 74.2% N/A N/A N/A reviewed/approved 6367 claude-sonnet-5
7254 Κύτα 84 84 4.997 36.2% 65.8% 68.5% 71.4% N/A N/A N/A reviewed/approved 6361 claude-sonnet-5
2605 Κάσιον 56 58 4.743 60.9% 77.3% 81.3% 77.2% N/A N/A N/A reviewed/approved 6330 claude-sonnet-5
2470 Καππαδοκία 68 69 4.709 32.3% 65.0% 85.4% 80.3% N/A N/A N/A reviewed/approved 6322 claude-sonnet-5
2085 Καλάσιρις 13 18 4.281 20.8% 66.1% 77.7% 64.5% N/A N/A N/A reviewed/approved 6296 claude-sonnet-5
2326 Καλλίαρος 47 49 4.019 49.0% 73.5% 88.2% 85.4% N/A N/A N/A reviewed/approved 6306 claude-sonnet-5
2330 Καλύβη 24 20 -3.833 41.4% 62.7% 61.9% 72.7% N/A N/A N/A reviewed/approved 6310 claude-sonnet-5
2331 Κάλυδνα 43 45 3.697 48.0% 74.4% 85.8% 88.6% N/A N/A N/A reviewed/approved 6311 claude-sonnet-5
7253 Κυρτώνιος 18 22 3.684 35.1% 65.1% 68.6% 65.0% N/A N/A N/A reviewed/approved 6360 claude-sonnet-5
3530 Κριώα 26 22 -3.672 51.4% 70.6% 77.8% 83.3% N/A N/A N/A reviewed/approved 6348 claude-sonnet-5
2054 Καβαλίς 65 65 3.468 67.8% 77.0% 69.7% 81.5% N/A N/A N/A reviewed/approved 6275 claude-sonnet-5
3496 Κοτιάειον 71 64 -3.049 28.8% 52.8% 54.2% 60.7% N/A N/A N/A reviewed/approved 6346 claude-sonnet-5
2465 Κάνωπος 83 81 2.917 40.3% 66.1% 77.0% 75.6% N/A N/A N/A reviewed/approved 6320 claude-sonnet-5
7264 Κωλιάς 61 55 -2.854 28.7% 57.4% 54.5% 67.2% N/A N/A N/A reviewed/approved 6371 claude-sonnet-5
7257 Κυτώνιον 25 22 -2.752 75.1% 80.0% 84.7% 89.4% N/A N/A N/A reviewed/approved 6364 claude-sonnet-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 8.377; cross-validated R^2: -0.014.

Positive termCoefficientDocsMean presentMean absent
δε και 1.1366 8 4.482 -0.390
εστι δε και 0.8609 3 11.480 -0.355
απο 0.8042 27 1.586 -0.587
οι 0.7635 11 3.037 -0.375
εκαλειτο δε 0.6391 3 5.649 -0.175
πολις 0.6279 68 0.564 -1.198
εστι δε 0.6117 5 4.416 -0.232
ου 0.5960 14 1.981 -0.322
δυναται 0.5951 3 8.035 -0.249
ος 0.5803 6 2.967 -0.189
πολιτης 0.5758 18 3.201 -0.703
τη 0.5265 15 1.709 -0.302
ομηρος 0.4706 3 6.430 -0.199
τε 0.4239 5 1.960 -0.103
παιδος 0.4134 4 3.878 -0.162
στραβων 0.4082 15 1.827 -0.322
εν τη 0.3991 7 2.539 -0.191
και αλλη 0.3971 3 8.845 -0.274
αλλη 0.3806 4 6.779 -0.282
εκαλειτο 0.3746 7 1.698 -0.128
Negative termCoefficientDocsMean presentMean absent
το -0.9798 70 -0.198 0.463
γαρ -0.8711 9 -1.125 0.111
δια του -0.6324 10 -1.282 0.142
δια -0.6067 16 0.120 -0.023
περι -0.5435 11 -2.963 0.366
θρακης -0.5393 5 -3.141 0.165
το εθνικον -0.4994 56 -0.706 0.899
εθνικον -0.4994 56 -0.706 0.899
ηρωδιανος -0.4814 3 -10.533 0.326
οτι -0.4745 5 -1.359 0.072
ει -0.4198 4 -5.975 0.249
ης -0.4035 3 -4.349 0.134
τα -0.4012 12 -1.189 0.162
εθνος -0.3980 8 -3.226 0.281
του -0.3791 37 1.032 -0.606
οικητωρ -0.3357 6 -3.622 0.231
μεν -0.3353 4 -6.662 0.278
πολις θρακης -0.3291 4 -1.942 0.081
τους -0.2868 4 -1.872 0.078
ζευς -0.2835 3 -7.750 0.240

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.105.

Positive termCoefficientDocsMean presentMean absent
και 2.0504 66 3.707 1.246
δε 1.9442 46 4.272 1.676
εκαλειτο 1.7387 7 8.870 2.419
την 1.7061 13 8.080 2.092
γαρ 1.6857 9 7.307 2.431
ος 1.2937 6 9.535 2.445
εκαλειτο δε 1.2639 3 14.335 2.516
δε και 1.2330 8 7.740 2.447
εστι δε και 1.0957 3 11.480 2.604
δια 1.0130 16 5.729 2.326
εστι δε 1.0111 5 9.515 2.520
τα 0.9858 12 6.791 2.336
τε 0.9466 5 6.193 2.695
δια του 0.8961 10 6.794 2.434
της 0.8823 35 4.653 1.910
απο 0.7879 27 5.336 1.958
δε της 0.7791 6 9.193 2.467
εκαλειτο δε και 0.7388 3 14.335 2.516
εις 0.7084 7 8.478 2.448
του 0.6603 37 4.257 2.056
Negative termCoefficientDocsMean presentMean absent
ως -0.7043 52 3.080 2.643
προς -0.7023 15 2.640 2.911
προς τη -0.6338 4 0.802 2.956
το εθνικον -0.6204 56 3.004 2.700
εθνικον -0.6204 56 3.004 2.700
τον -0.5683 10 2.329 2.930
στραβων -0.5311 15 2.874 2.870
τεταρτω -0.5193 3 0.769 2.935
πλησιον -0.5056 8 2.100 2.937
εκ -0.5019 6 2.223 2.911
νησος -0.4372 9 1.971 2.959
τους -0.4338 4 1.996 2.907
οικητωρ -0.4165 6 3.731 2.815
αυτην -0.3770 9 2.566 2.900
και το -0.3560 9 3.147 2.843
πολιτης -0.3552 18 3.858 2.653
εκαταιος -0.3455 14 2.679 2.901
δυο -0.3393 4 3.100 2.861
πορρω -0.3341 3 0.634 2.939
ου πορρω -0.3341 3 0.634 2.939

AI English terms predicting large absolute residuals

Features: 295; ridge alpha: 5.878; cross-validated R^2: -0.162.

Positive termCoefficientDocsMean presentMean absent
and 1.7017 56 4.014 1.414
was 1.1590 17 6.192 2.190
for 1.1583 13 5.658 2.454
with 0.9665 17 6.299 2.168
the 0.9468 92 3.009 1.272
called 0.9456 26 5.121 2.079
of 0.8373 84 2.998 2.200
by 0.8325 12 7.154 2.286
also 0.8087 45 4.071 1.887
who 0.7311 9 6.947 2.467
as a 0.6447 3 7.214 2.736
says 0.6385 19 6.277 2.071
was called 0.6226 5 9.306 2.531
form 0.6203 8 6.558 2.549
but 0.6201 23 5.675 2.033
there 0.6086 32 4.709 2.005
athens 0.5472 3 13.476 2.542
is also 0.5319 30 4.175 2.311
after 0.5314 19 6.026 2.130
has 0.5268 5 9.850 2.503
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.5141 53 2.825 2.921
near -0.5089 19 2.208 3.026
as in -0.4954 32 2.480 3.054
the ethnonym -0.4406 57 2.876 2.863
ethnonym is -0.4280 54 3.008 2.708
ethnonym -0.3589 58 3.046 2.627
one -0.3340 10 2.945 2.862
hecataeus -0.3187 5 0.748 2.982
a city -0.3135 63 3.062 2.544
city of -0.3133 52 3.115 2.605
which the -0.3069 5 1.384 2.948
a city of -0.2951 50 3.128 2.613
and the -0.2833 10 2.869 2.870
strabo -0.2766 15 2.874 2.870
book -0.2764 40 2.743 2.955
city -0.2451 68 3.030 2.531
an island -0.2375 10 1.941 2.973
island -0.2351 14 2.488 2.932
sea -0.2341 10 2.528 2.908
his -0.2326 34 2.705 2.955

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 4.125; cross-validated R^2: -0.117.

Positive termCoefficientDocsMean presentMean absent
and 2.0642 52 4.174 1.458
with 1.7900 16 5.815 2.309
the 1.6490 90 3.018 1.538
to 1.0671 31 4.790 2.008
this 1.0328 15 7.851 1.991
is with 1.0107 4 13.664 2.420
who 0.9417 13 5.908 2.416
also 0.9085 43 3.936 2.066
was 0.8556 20 5.298 2.263
used to be 0.8545 6 10.290 2.397
too 0.8379 3 12.311 2.578
used 0.8072 8 9.354 2.306
used to 0.8072 8 9.354 2.306
forms 0.7751 4 11.786 2.499
to be called 0.7725 5 10.883 2.448
be 0.7291 15 6.967 2.147
called 0.7243 21 5.114 2.274
the form 0.7177 9 7.269 2.435
from 0.6777 37 3.840 2.300
form 0.6575 11 6.210 2.457
Negative termCoefficientDocsMean presentMean absent
as in -1.0519 29 1.969 3.238
the ethnonym is -0.6457 52 2.688 3.068
ethnonym is -0.6457 52 2.688 3.068
near -0.5632 16 2.353 2.969
a city in -0.5289 40 3.030 2.764
an -0.5271 22 2.404 3.002
city in -0.5003 41 3.071 2.730
the ethnonym -0.4663 56 2.997 2.709
was from -0.4542 4 0.795 2.957
ethnonym -0.4363 57 3.067 2.609
a city -0.3575 64 3.027 2.592
strabo -0.3534 15 2.874 2.870
city -0.3429 70 2.984 2.605
book -0.3169 39 2.788 2.923
one of -0.3121 5 1.627 2.936
of his -0.3081 18 2.609 2.927
sicily -0.3042 3 0.936 2.930
his -0.2982 34 2.705 2.955
island -0.2948 14 2.488 2.932
name -0.2795 7 3.441 2.827

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.