claude_opus_4_8 v2 Evaluation

Generated: 2026-09-20 11:19:36 UTC

Pairs100
Distinct lemmas100
Slope0.966
R^20.989
BLEU-450.9%
chrF++72.5%
METEOR78.0%
ROUGE-L78.5%
BERTScore96.4%
COMET80.0%
BLEURT75.8%

Profile version id: 2013

First translation using this version: 2026-08-22

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Opus 4.8; copied from legacy_scholarly v2 (source profile_version_id=274); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.966
Intercept1.173
R^20.989
P-value8.04e-98
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words43.830
Mean absolute length residual2.500
Mean absolute percent error5.5%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.454, R^2 = 0.206, p = 2.05e-06.
chrF++ vs passage length
chrF++: significant negative; r = -0.475, R^2 = 0.226, p = 5.84e-07.
METEOR vs passage length
METEOR: significant negative; r = -0.514, R^2 = 0.264, p = 4.60e-08.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.530, R^2 = 0.281, p = 1.45e-08.
BERTScore vs passage length
BERTScore: significant negative; r = -0.628, R^2 = 0.394, p = 2.66e-12.
COMET vs passage length
COMET: significant negative; r = -0.467, R^2 = 0.218, p = 9.49e-07.
BLEURT vs passage length
BLEURT: significant negative; r = -0.727, R^2 = 0.529, p = 1.07e-17.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.398, R^2 = 0.158, p = 4.18e-05.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.406, R^2 = 0.165, p = 2.76e-05.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.403, R^2 = 0.163, p = 3.18e-05.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.384, R^2 = 0.147, p = 8.07e-05.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.454 0.206 2.05e-06 -0.00281 ok
chrF++ 100 significant negative -0.475 0.226 5.84e-07 -0.00182 ok
METEOR 100 significant negative -0.514 0.264 4.60e-08 -0.00213 ok
ROUGE-L 100 significant negative -0.530 0.281 1.45e-08 -0.00202 ok
BERTScore 100 significant negative -0.628 0.394 2.66e-12 -0.00047 ok
COMET 100 significant negative -0.467 0.218 9.49e-07 -0.00089 ok
BLEURT 100 significant negative -0.727 0.529 1.07e-17 -0.00248 ok
Trigram precision 100 significant negative -0.398 0.158 4.18e-05 -0.00285 ok
Trigram recall 100 significant negative -0.406 0.165 2.76e-05 -0.00287 ok
Trigram F1 100 significant negative -0.403 0.163 3.18e-05 -0.00286 ok
Trigram Jaccard 100 significant negative -0.384 0.147 8.07e-05 -0.00270 ok

Translation Similarity Metrics

Mean BLEU-450.9%
Mean chrF++72.5%
Mean METEOR78.0%
Mean ROUGE-L78.5%
Mean BERTScore96.4%
Mean COMET80.0%
Mean BLEURT75.8%
Legacy smoothed corpus BLEU48.5%
Unigram F178.6%
Bigram F154.7%
Trigram precision41.1%
Trigram recall40.8%
Trigram F140.9%
Trigram Jaccard25.7%
4-gram F131.8%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 138 23.784 29.1% 60.4% 68.7% 62.8% 92.5% 77.3% 59.5% reviewed/approved 20456 claude-opus-4.8
2342 Κάμιρος 54 64 10.653 38.0% 71.2% 72.0% 71.2% 95.7% 78.9% 71.5% reviewed/approved 20443 claude-opus-4.8
2484 Καρία 270 252 -10.042 36.5% 61.3% 60.4% 63.5% 91.8% 80.8% 58.0% reviewed/approved 20451 claude-opus-4.8
2597 Καρπασία 112 100 -9.385 40.3% 69.4% 70.0% 72.6% 95.9% 75.1% 59.4% reviewed/approved 20452 claude-opus-4.8
2056 Καβειρία 112 101 -8.385 30.0% 66.7% 67.4% 66.7% 95.3% 78.1% 60.9% reviewed/approved 20404 claude-opus-4.8
7259 Κύφος 54 61 7.653 48.7% 74.5% 81.6% 76.5% 96.3% 75.4% 71.2% reviewed/approved 20493 claude-opus-4.8
3288 Κορώνεια 110 115 7.547 38.2% 64.8% 59.6% 69.0% 94.7% 75.3% 63.1% reviewed/approved 20472 claude-opus-4.8
2468 Καπετώλιον 138 128 -6.506 16.6% 47.4% 50.7% 53.4% 87.9% 69.1% 51.4% reviewed/approved 20448 claude-opus-4.8
7260 Κυχρεῖος πάγος 61 66 5.890 43.3% 67.2% 71.4% 70.9% 95.7% 76.5% 68.0% reviewed/approved 20494 claude-opus-4.8
2116 Κάληρος 40 34 -5.820 15.2% 51.8% 57.9% 56.8% 93.3% 77.5% 73.9% reviewed/approved 20428 claude-opus-4.8
7255 Κυτέριον 22 28 5.571 48.9% 74.3% 70.6% 72.0% 96.3% 78.0% 77.0% reviewed/approved 20489 claude-opus-4.8
2609 Κάσπειρος 135 137 5.392 40.6% 63.2% 60.6% 64.7% 90.0% 72.4% 63.3% reviewed/approved 20459 claude-opus-4.8
2605 Κάσιον 56 60 4.721 54.7% 75.7% 82.4% 79.3% 96.4% 78.3% 73.2% reviewed/approved 20457 claude-opus-4.8
2465 Κάνωπος 83 86 4.634 46.6% 72.1% 84.1% 82.8% 96.1% 78.5% 75.1% reviewed/approved 20447 claude-opus-4.8
2625 Κατακεκαυμένη 46 50 4.383 64.9% 83.0% 89.2% 85.4% 97.6% 80.4% 80.2% reviewed/approved 20465 claude-opus-4.8
2330 Καλύβη 24 20 -4.361 52.0% 72.5% 65.0% 81.8% 97.7% 80.7% 77.1% reviewed/approved 20437 claude-opus-4.8
7257 Κυτώνιον 25 21 -4.328 66.3% 76.9% 79.6% 82.6% 98.2% 84.0% 80.2% reviewed/approved 20491 claude-opus-4.8
7261 Κύψελα 34 30 -4.023 67.1% 80.0% 85.7% 87.5% 98.2% 81.4% 80.2% reviewed/approved 20495 claude-opus-4.8
7267 Κωνώπη 64 59 -4.009 54.5% 73.2% 73.2% 79.7% 97.0% 78.0% 72.3% reviewed/approved 20501 claude-opus-4.8
2079 Καισάρεια 39 35 -3.854 48.2% 70.3% 79.8% 83.8% 96.0% 78.6% 74.8% reviewed/approved 20417 claude-opus-4.8
7250 Κυρταία 40 36 -3.820 55.9% 70.5% 71.0% 73.7% 96.7% 68.8% 70.4% reviewed/approved 20484 claude-opus-4.8
7254 Κύτα 84 86 3.668 35.9% 66.0% 69.7% 70.6% 94.8% 77.7% 64.9% reviewed/approved 20488 claude-opus-4.8
2081 Καλάθη 49 45 -3.516 72.4% 83.3% 87.5% 91.5% 98.4% 79.2% 80.4% reviewed/approved 20419 claude-opus-4.8
2059 Καδμεία 20 24 3.503 28.6% 55.7% 74.9% 77.3% 94.5% 79.2% 77.2% reviewed/approved 20407 claude-opus-4.8
2624 Κατάβαθμος 20 17 -3.497 63.7% 79.4% 84.4% 86.5% 96.0% 81.4% 76.8% reviewed/approved 20464 claude-opus-4.8
7253 Κυρτώνιος 18 22 3.436 40.6% 69.3% 68.6% 65.0% 96.8% 77.2% 81.6% reviewed/approved 20487 claude-opus-4.8
2326 Καλλίαρος 47 50 3.416 47.8% 74.3% 88.2% 84.5% 97.2% 75.8% 75.8% reviewed/approved 20433 claude-opus-4.8
3530 Κριώα 26 23 -3.294 54.8% 73.8% 77.5% 81.6% 97.4% 78.0% 76.5% reviewed/approved 20475 claude-opus-4.8
2470 Καππαδοκία 68 70 3.127 31.0% 63.9% 85.8% 79.7% 95.8% 76.1% 72.2% reviewed/approved 20449 claude-opus-4.8
2119 Κάλλατις 65 67 3.025 39.2% 65.4% 75.9% 72.7% 95.8% 70.3% 71.7% reviewed/approved 20431 claude-opus-4.8

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 49.239; cross-validated R^2: -0.110.

Positive termCoefficientDocsMean presentMean absent
δε και 0.1900 8 3.594 -0.313
απο 0.1660 27 1.754 -0.649
εστι δε και 0.1577 3 10.871 -0.336
ος 0.1485 6 5.032 -0.321
παιδος 0.1329 4 5.607 -0.234
και 0.1274 66 0.139 -0.270
τε 0.1236 5 3.516 -0.185
δε 0.1190 46 0.518 -0.441
οι 0.1184 11 2.461 -0.304
εκαλειτο δε 0.1171 3 5.496 -0.170
πολιτης 0.1159 18 2.615 -0.574
εστι δε 0.1157 5 4.644 -0.244
πολις 0.0970 68 0.302 -0.642
εκαλειτο 0.0882 7 1.539 -0.116
και αλλη 0.0777 3 9.164 -0.283
εν τη 0.0664 7 2.026 -0.153
εκληθη 0.0637 4 3.651 -0.152
δυο 0.0610 4 2.436 -0.101
πολις και 0.0590 6 4.215 -0.269
δυναται 0.0585 3 3.877 -0.120
Negative termCoefficientDocsMean presentMean absent
το -0.1282 70 -0.253 0.590
θρακης -0.1042 5 -2.471 0.130
πολις θρακης -0.1031 4 -2.802 0.117
θεοπομπος -0.1015 5 -3.570 0.188
εστι και -0.0942 20 -0.961 0.240
γαρ -0.0860 9 -1.094 0.108
οικητωρ -0.0760 6 -3.066 0.196
οτι -0.0756 5 -2.050 0.108
δεκατω -0.0668 3 -4.268 0.132
το εθνικον -0.0655 56 -0.333 0.424
εθνικον -0.0655 56 -0.333 0.424
μεταξυ -0.0647 5 -0.493 0.026
ως -0.0635 52 -0.040 0.044
φησιν -0.0630 10 -0.425 0.047
πλησιον -0.0611 8 -1.083 0.094
δια -0.0599 16 0.131 -0.025
και του -0.0591 4 -2.300 0.096
εθνος -0.0563 8 -2.403 0.209
αυτην -0.0477 9 -0.813 0.080
τα -0.0473 12 -1.272 0.173

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.109.

Positive termCoefficientDocsMean presentMean absent
την 2.0511 13 6.670 1.877
δε και 1.9378 8 7.068 2.103
εκαλειτο 1.8609 7 6.542 2.196
δε 1.7846 46 3.691 1.486
εκαλειτο δε 1.6519 3 12.190 2.201
και 1.5897 66 3.226 1.092
εστι δε και 1.4697 3 10.871 2.241
ος 1.3827 6 7.200 2.200
εστι δε 1.3008 5 8.401 2.190
εστι 1.0390 24 4.376 1.908
απο 0.9880 27 4.728 1.676
εν 0.9570 37 3.293 2.035
εκαλειτο δε και 0.8758 3 12.190 2.201
φησιν 0.8284 10 6.063 2.104
παιδος 0.8024 4 5.607 2.371
αφ 0.7994 6 5.294 2.322
δια 0.7380 16 4.676 2.086
μεν 0.7051 4 7.005 2.313
μητροπολις 0.6799 3 9.712 2.277
της 0.6450 35 3.552 1.934
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.7957 56 2.279 2.781
εθνικον -0.7957 56 2.279 2.781
προς -0.7707 15 2.434 2.512
τον -0.6918 10 2.071 2.548
κωμη -0.6639 3 1.264 2.539
στραβων -0.6618 15 2.262 2.542
και πολις -0.5831 5 1.951 2.529
τεταρτω -0.5820 3 0.893 2.550
εκαταιος -0.5769 14 1.872 2.603
ορος -0.5632 6 1.750 2.548
προς τη -0.5488 4 1.554 2.540
το -0.5474 70 2.402 2.729
πλησιον -0.5469 8 1.410 2.595
των -0.5242 17 3.454 2.305
λυκιας -0.5242 3 0.897 2.550
αλεξανδρος -0.5139 3 0.178 2.572
ως -0.5114 52 2.812 2.163
δευτερω -0.5110 3 0.832 2.552
καππαδοκιας -0.4672 3 1.901 2.519
πολις προς -0.4660 3 1.685 2.526

AI English terms predicting large absolute residuals

Features: 330; ridge alpha: 4.125; cross-validated R^2: -0.153.

Positive termCoefficientDocsMean presentMean absent
and 1.8507 58 3.348 1.330
was called 1.1641 5 9.405 2.137
called 1.1228 26 4.534 1.786
there 1.1196 32 4.521 1.549
was 0.9517 17 4.558 2.079
the 0.9380 94 2.583 1.206
there is 0.9332 26 4.584 1.768
there is also 0.8029 24 4.653 1.821
and there 0.7336 5 9.687 2.122
while 0.6778 5 9.514 2.131
after 0.6624 19 5.146 1.880
who 0.6497 12 5.168 2.137
and there is 0.6228 5 9.687 2.122
too was 0.6188 3 13.239 2.168
too was called 0.6188 3 13.239 2.168
most 0.6063 6 8.629 2.109
come 0.5857 4 10.139 2.182
is also 0.5797 32 4.150 1.724
their 0.5682 4 8.714 2.241
distinguished 0.5650 4 9.104 2.225
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.5956 53 2.150 2.896
ethnonym is -0.5423 54 2.265 2.777
the ethnonym -0.5360 56 2.164 2.928
ethnonym -0.4984 58 2.260 2.832
a city -0.4196 63 2.821 1.954
and in the -0.3599 12 2.126 2.551
as in -0.3555 31 2.859 2.339
a city and -0.3432 3 0.386 2.566
strabo -0.3347 15 2.262 2.542
near -0.3221 17 2.243 2.553
call -0.2959 4 1.149 2.557
in his asia -0.2916 5 0.751 2.592
his asia -0.2916 5 0.751 2.592
here -0.2849 3 0.866 2.551
hekataios -0.2835 14 1.872 2.603
lycia -0.2782 3 0.897 2.550
came -0.2715 7 2.660 2.488
strabo book -0.2709 9 1.864 2.563
a small -0.2694 4 1.008 2.563
a city of -0.2582 48 2.882 2.148

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: -0.156.

Positive termCoefficientDocsMean presentMean absent
and 1.7880 52 3.578 1.333
was 0.9436 20 4.130 2.093
who 0.8378 13 4.962 2.133
with 0.7952 16 4.524 2.115
called 0.7874 21 4.187 2.052
too 0.7730 3 11.906 2.209
after 0.7644 22 5.164 1.749
them 0.7621 5 9.373 2.139
is with 0.7465 4 10.224 2.179
there is also 0.7338 22 4.497 1.937
also 0.7250 43 3.669 1.619
came 0.7175 4 10.055 2.186
the 0.7008 90 2.661 1.059
son 0.6964 17 5.579 1.870
son of 0.6964 17 5.579 1.870
there is 0.6511 24 4.238 1.952
used to be 0.6233 6 7.605 2.175
there 0.6015 34 4.038 1.708
local 0.5993 6 6.497 2.245
their 0.5892 3 12.190 2.201
Negative termCoefficientDocsMean presentMean absent
as in -0.7620 29 2.165 2.637
an -0.6240 22 1.925 2.663
the ethnonym is -0.6107 52 2.161 2.868
ethnonym is -0.6107 52 2.161 2.868
ethnonym -0.5329 57 2.291 2.778
the ethnonym -0.5198 56 2.305 2.749
ethnonyms -0.3491 3 1.045 2.545
near -0.3491 16 2.003 2.595
strabo -0.3343 15 2.262 2.542
a city -0.3312 64 2.814 1.943
city and -0.3271 4 0.597 2.580
a city and -0.3271 4 0.597 2.580
was from -0.3012 4 1.088 2.559
kappadokia -0.3011 4 2.207 2.513
in his asia -0.2935 5 0.751 2.592
his asia -0.2935 5 0.751 2.592
a city in -0.2915 40 2.937 2.209
they are -0.2887 7 1.900 2.546
in the -0.2849 25 2.613 2.463
alexander -0.2800 3 0.178 2.572

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.