gpt-4o-2024-08-06 v2 Evaluation

Generated: 2026-09-20 11:20:26 UTC

Pairs100
Distinct lemmas100
Slope0.922
R^20.983
BLEU-438.8%
chrF++64.8%
METEOR70.8%
ROUGE-L72.4%
BERTScore95.3%
COMET77.7%
BLEURT71.1%

Profile version id: 2411

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-08-06) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.922
Intercept2.258
R^20.983
P-value5.91e-89
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.960
Mean absolute length residual3.176
Mean absolute percent error6.9%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.339, R^2 = 0.115, p = 0.0006.
chrF++ vs passage length
chrF++: significant negative; r = -0.367, R^2 = 0.134, p = 0.0002.
METEOR vs passage length
METEOR: significant negative; r = -0.461, R^2 = 0.213, p = 1.39e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.447, R^2 = 0.200, p = 3.06e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.627, R^2 = 0.393, p = 3.10e-12.
COMET vs passage length
COMET: significant negative; r = -0.418, R^2 = 0.175, p = 1.52e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.761, R^2 = 0.579, p = 4.14e-20.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.324, R^2 = 0.105, p = 0.0010.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.327, R^2 = 0.107, p = 0.0009.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.326, R^2 = 0.107, p = 0.0009.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.322, R^2 = 0.104, p = 0.0011.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.339 0.115 0.0006 -0.00193 ok
chrF++ 100 significant negative -0.367 0.134 0.0002 -0.00138 ok
METEOR 100 significant negative -0.461 0.213 1.39e-06 -0.00227 ok
ROUGE-L 100 significant negative -0.447 0.200 3.06e-06 -0.00194 ok
BERTScore 100 significant negative -0.627 0.393 3.10e-12 -0.00052 ok
COMET 100 significant negative -0.418 0.175 1.52e-05 -0.00078 ok
BLEURT 100 significant negative -0.761 0.579 4.14e-20 -0.00271 ok
Trigram precision 100 significant negative -0.324 0.105 0.0010 -0.00223 ok
Trigram recall 100 significant negative -0.327 0.107 0.0009 -0.00224 ok
Trigram F1 100 significant negative -0.326 0.107 0.0009 -0.00223 ok
Trigram Jaccard 100 significant negative -0.322 0.104 0.0011 -0.00190 ok

Translation Similarity Metrics

Mean BLEU-438.8%
Mean chrF++64.8%
Mean METEOR70.8%
Mean ROUGE-L72.4%
Mean BERTScore95.3%
Mean COMET77.7%
Mean BLEURT71.1%
Legacy smoothed corpus BLEU38.4%
Unigram F173.0%
Bigram F145.7%
Trigram precision31.7%
Trigram recall30.8%
Trigram F131.2%
Trigram Jaccard18.5%
4-gram F122.1%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
3288 Κορώνεια 110 120 16.332 25.4% 57.4% 61.2% 61.5% 90.7% 74.8% 57.6% reviewed/approved 12745 gpt-4o-2024-08-06
2604 Καρχηδών 117 125 14.879 36.6% 66.0% 66.1% 66.4% 94.4% 78.2% 63.3% reviewed/approved 12697 gpt-4o-2024-08-06
2055 Καβασσός 109 88 -14.746 21.0% 51.3% 47.8% 52.8% 92.8% 73.7% 45.9% reviewed/approved 12538 gpt-4o-2024-08-06
2056 Καβειρία 112 91 -14.512 15.4% 48.8% 45.9% 59.1% 92.1% 70.4% 56.2% reviewed/approved 12541 gpt-4o-2024-08-06
2465 Κάνωπος 83 91 12.224 37.5% 67.8% 83.6% 75.9% 94.8% 75.1% 65.4% reviewed/approved 12670 gpt-4o-2024-08-06
2484 Καρία 270 239 -12.174 23.1% 52.6% 43.1% 56.6% 89.1% 75.9% 40.0% reviewed/approved 12682 gpt-4o-2024-08-06
2328 Καλλίπολις 43 54 12.100 37.9% 63.8% 72.3% 66.0% 93.6% 75.5% 64.9% reviewed/approved 12634 gpt-4o-2024-08-06
2603 Κάρυστος 175 173 9.408 31.3% 60.4% 65.0% 63.2% 93.3% 77.3% 58.5% reviewed/approved 12694 gpt-4o-2024-08-06
7248 Κύρου πόλις 32 40 8.241 24.4% 65.9% 78.1% 72.2% 94.5% 77.0% 78.9% reviewed/approved 12775 gpt-4o-2024-08-06
2342 Κάμιρος 54 60 7.959 34.9% 64.5% 70.1% 68.4% 94.9% 77.4% 63.9% reviewed/approved 12658 gpt-4o-2024-08-06
2609 Κάσπειρος 135 119 -7.716 34.9% 56.8% 48.2% 63.0% 89.3% 70.0% 55.4% reviewed/approved 12706 gpt-4o-2024-08-06
7255 Κυτέριον 22 29 6.460 39.1% 68.2% 67.5% 70.6% 96.1% 77.8% 75.3% reviewed/approved 12796 gpt-4o-2024-08-06
7247 Κύρνος 50 42 -6.353 28.1% 58.1% 67.1% 67.4% 95.3% 77.8% 72.9% reviewed/approved 12772 gpt-4o-2024-08-06
2605 Κάσιον 56 60 6.115 46.9% 69.6% 73.5% 70.7% 95.8% 76.1% 69.8% reviewed/approved 12700 gpt-4o-2024-08-06
7259 Κύφος 54 58 5.959 43.0% 70.2% 73.8% 71.4% 95.3% 74.6% 63.4% reviewed/approved 12808 gpt-4o-2024-08-06
2116 Κάληρος 40 34 -5.134 17.5% 47.6% 50.3% 48.6% 93.6% 74.3% 70.9% reviewed/approved 12613 gpt-4o-2024-08-06
2470 Καππαδοκία 68 70 5.052 24.9% 59.6% 76.4% 73.9% 95.8% 75.9% 70.9% reviewed/approved 12676 gpt-4o-2024-08-06
2329 Κάλπη 57 50 -4.807 42.5% 65.2% 67.6% 77.8% 96.0% 73.9% 72.9% reviewed/approved 12637 gpt-4o-2024-08-06
2326 Καλλίαρος 47 50 4.413 41.0% 66.5% 77.9% 74.2% 96.3% 72.1% 61.0% reviewed/approved 12628 gpt-4o-2024-08-06
3530 Κριώα 26 22 -4.227 37.5% 63.4% 69.6% 75.0% 95.4% 76.0% 71.8% reviewed/approved 12754 gpt-4o-2024-08-06
7251 Κύρτος 66 59 -4.104 24.4% 49.8% 74.1% 70.4% 91.7% 68.6% 55.8% reviewed/approved 12784 gpt-4o-2024-08-06
2331 Κάλυδνα 43 46 4.100 37.5% 66.0% 80.5% 83.1% 95.2% 78.7% 73.6% reviewed/approved 12643 gpt-4o-2024-08-06
2119 Κάλλατις 65 66 3.818 34.8% 61.1% 70.3% 64.1% 93.9% 70.8% 68.1% reviewed/approved 12622 gpt-4o-2024-08-06
7267 Κωνώπη 64 65 3.740 44.8% 68.7% 68.5% 72.9% 96.1% 77.1% 73.1% reviewed/approved 12832 gpt-4o-2024-08-06
7261 Κύψελα 34 30 -3.603 51.9% 71.4% 78.9% 78.1% 97.6% 80.6% 75.0% reviewed/approved 12814 gpt-4o-2024-08-06
2597 Καρπασία 112 102 -3.512 30.5% 61.7% 53.5% 65.4% 94.9% 76.1% 58.4% reviewed/approved 12685 gpt-4o-2024-08-06
2057 Καβελλιών 34 37 3.397 33.7% 66.8% 68.9% 70.4% 96.3% 77.5% 67.9% reviewed/approved 12544 gpt-4o-2024-08-06
2325 Καλλίαι 24 21 -3.383 37.0% 65.6% 75.9% 80.0% 96.8% 79.7% 74.9% reviewed/approved 12625 gpt-4o-2024-08-06
2330 Καλύβη 24 21 -3.383 41.4% 62.9% 64.1% 80.0% 96.4% 79.8% 77.2% reviewed/approved 12640 gpt-4o-2024-08-06
2625 Κατακεκαυμένη 46 48 3.334 44.4% 67.3% 72.8% 70.2% 95.9% 75.2% 72.8% reviewed/approved 12724 gpt-4o-2024-08-06

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.063.

Positive termCoefficientDocsMean presentMean absent
πολις 1.6084 68 0.438 -0.932
απο 1.5751 27 2.162 -0.800
πολιτης 1.4622 18 3.628 -0.796
δε και 1.3892 8 3.863 -0.336
εν τη 1.3860 7 4.502 -0.339
εστι δε και 1.3377 3 10.437 -0.323
και 1.2731 66 0.258 -0.500
δευτερα 1.1446 3 9.068 -0.280
τριτη 1.0545 3 8.740 -0.270
παιδος 1.0496 4 5.026 -0.209
πολις προς 0.9609 3 6.573 -0.203
ος 0.9568 6 4.091 -0.261
και το 0.9555 9 2.670 -0.264
πολιχνιον 0.9290 3 6.224 -0.193
εστι δε 0.9220 5 3.908 -0.206
τη 0.8931 15 1.575 -0.278
στραβων 0.8516 15 2.528 -0.446
κατα 0.8511 10 2.188 -0.243
εις 0.8317 7 2.758 -0.208
εκαλειτο δε 0.7738 3 4.037 -0.125
Negative termCoefficientDocsMean presentMean absent
το -1.4595 70 -0.257 0.599
καππαδοκιας -1.4306 3 -6.114 0.189
ως το -1.2492 3 -11.121 0.344
τοπικα -0.9570 3 -6.156 0.190
τα τοπικα -0.9570 3 -6.156 0.190
εθνος -0.9316 8 -4.141 0.360
γαρ -0.9242 9 -2.472 0.245
παυσανιας -0.9206 4 -5.150 0.215
οικητωρ -0.8801 6 -3.789 0.242
λυκιας -0.8759 3 -5.994 0.185
και του -0.8416 4 -4.759 0.198
τε -0.7573 5 -0.510 0.027
οτι -0.7497 5 -1.432 0.075
ης -0.6997 3 -3.650 0.113
ειναι -0.6939 8 -1.090 0.095
φησιν -0.6672 10 -1.427 0.159
θρακης -0.6589 5 -2.312 0.122
πλησιον -0.6347 8 -0.663 0.058
ελλανικος -0.6259 4 -6.505 0.271
το θηλυκον -0.6238 6 -3.282 0.210

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.034.

Positive termCoefficientDocsMean presentMean absent
την 2.2597 13 7.143 2.584
και 1.9154 66 3.800 1.965
δε 1.8012 46 4.483 2.064
δυναται 1.4208 3 12.513 2.888
εκαλειτο 1.4155 7 8.088 2.807
παρα 1.3562 7 7.716 2.835
εν 1.2660 37 4.511 2.392
απο 1.1965 27 5.416 2.348
κατα 1.1625 10 6.478 2.810
βοιωτιας 1.1500 3 10.664 2.945
εκαλειτο δε 1.1384 3 12.153 2.899
εστι δε 1.1365 5 9.713 2.832
τριτη 1.0364 3 10.215 2.959
ως το 1.0230 3 11.121 2.931
εστι δε και 0.9824 3 10.437 2.952
πολις εν 0.9821 5 7.608 2.943
δευτερα 0.9668 3 9.886 2.969
δε και 0.9662 8 7.460 2.804
εστι 0.9568 24 4.877 2.639
της 0.9335 35 4.593 2.414
Negative termCoefficientDocsMean presentMean absent
το εθνικον -1.3854 56 2.820 3.630
εθνικον -1.3854 56 2.820 3.630
δευτερω -0.9747 3 0.435 3.261
προς τη -0.7795 4 1.691 3.238
αυτην -0.7533 9 2.573 3.236
στραβων -0.7316 15 3.371 3.142
πλησιον -0.6774 8 1.931 3.285
των -0.6265 17 4.085 2.990
αυτην φησιν -0.6034 3 1.348 3.233
του -0.5978 37 4.259 2.541
ηροδοτος -0.5916 9 3.296 3.165
ην -0.5703 13 3.199 3.173
εξ -0.5657 3 1.375 3.232
τεταρτω -0.5287 3 1.898 3.216
κωμη -0.5179 3 1.976 3.214
μια -0.4858 3 1.273 3.235
ορος -0.4823 6 3.323 3.167
ως στραβων -0.4587 4 2.061 3.223
πορρω -0.4399 3 0.836 3.249
ου πορρω -0.4399 3 0.836 3.249

AI English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: 0.027.

Positive termCoefficientDocsMean presentMean absent
and 2.1526 55 4.160 1.975
is a city 2.0901 3 13.023 2.872
of 1.3254 85 3.365 2.108
the 1.0588 94 3.240 2.185
according 1.0319 7 7.276 2.868
according to 1.0319 7 7.276 2.868
is 0.9863 90 3.330 1.791
is also a 0.9637 16 6.103 2.619
but 0.9572 10 6.586 2.798
also a 0.9393 17 5.985 2.601
is a 0.9289 30 4.308 2.692
from 0.8774 37 4.481 2.410
whom 0.8770 5 7.272 2.961
claims 0.8652 3 6.420 3.076
written 0.8394 3 9.724 2.974
written with 0.8394 3 9.724 2.974
was called 0.8271 4 9.316 2.921
places 0.8105 5 8.235 2.910
he 0.7840 3 9.853 2.970
also 0.7790 43 4.451 2.215
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.2467 53 2.501 3.938
the ethnonym -1.1440 56 2.634 3.867
ethnonym is -1.0735 54 2.723 3.708
as in -1.0219 37 2.632 3.496
near -0.7802 18 2.280 3.373
ethnonym -0.7774 58 3.047 3.355
there was -0.6061 4 1.012 3.267
because the -0.5980 3 0.789 3.250
it is -0.5659 27 3.766 2.958
with an -0.4727 4 1.725 3.237
strabo book -0.4478 4 1.371 3.252
the second book -0.4430 3 0.435 3.261
the second -0.4430 3 0.435 3.261
second book -0.4430 3 0.435 3.261
in the second -0.4430 3 0.435 3.261
another in -0.4401 3 1.485 3.229
from which -0.4118 8 3.443 3.153
strabo -0.4024 15 3.371 3.142
of his -0.3989 15 2.743 3.253
city and -0.3987 3 1.335 3.233

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: 0.031.

Positive termCoefficientDocsMean presentMean absent
the 1.5494 90 3.380 1.344
and 1.4044 52 4.267 1.995
of 1.3702 66 3.894 1.784
for 1.1902 11 7.887 2.594
also 1.0140 43 4.511 2.170
this 0.8599 15 7.701 2.378
all 0.8076 3 12.049 2.902
form 0.7912 11 5.474 2.892
there 0.7855 34 4.787 2.347
boiotia 0.7467 3 10.664 2.945
who 0.7398 13 6.168 2.729
from 0.6997 37 4.241 2.551
used to be 0.6854 6 9.422 2.778
is from 0.6726 12 5.293 2.888
used 0.6687 8 8.891 2.680
used to 0.6687 8 8.891 2.680
is a city 0.6662 3 9.054 2.995
kappadokia 0.6225 4 5.849 3.065
a 0.6146 91 3.277 2.159
is also 0.6123 31 5.086 2.318
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.0078 52 2.453 3.961
ethnonym is -1.0078 52 2.453 3.961
the ethnonym -0.7666 56 2.881 3.553
ethnonym -0.6580 57 3.089 3.293
as in -0.6328 29 2.931 3.277
name -0.5667 7 1.322 3.316
one of -0.5159 5 0.882 3.297
herodotos -0.4446 9 3.296 3.165
which -0.4187 20 3.241 3.160
one of the -0.4168 4 0.995 3.267
near -0.4149 16 2.640 3.279
herodotos book -0.3884 5 2.190 3.228
book -0.3782 39 3.254 3.127
city and -0.3772 4 0.877 3.272
a city and -0.3772 4 0.877 3.272
says that it -0.3288 8 3.491 3.149
an -0.3195 22 3.590 3.060
either -0.3132 6 4.153 3.114
with a -0.3026 4 1.801 3.234
was from -0.2963 4 1.655 3.240

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.