gpt-5 v2 Evaluation

Generated: 2026-09-20 11:21:04 UTC

Pairs100
Distinct lemmas100
Slope0.821
R^20.968
BLEU-447.8%
chrF++70.1%
METEOR74.7%
ROUGE-L76.4%
BERTScore96.1%
COMET78.8%
BLEURT73.6%

Profile version id: 2344

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.821
Intercept4.483
R^20.968
P-value2.31e-75
Pearson r0.984
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words40.740
Mean absolute length residual3.636
Mean absolute percent error7.9%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.421, R^2 = 0.177, p = 1.28e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.465, R^2 = 0.217, p = 1.07e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.489, R^2 = 0.239, p = 2.46e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.436, R^2 = 0.190, p = 5.84e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.680, R^2 = 0.462, p = 7.56e-15.
COMET vs passage length
COMET: significant negative; r = -0.522, R^2 = 0.273, p = 2.55e-08.
BLEURT vs passage length
BLEURT: significant negative; r = -0.809, R^2 = 0.655, p = 2.27e-24.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.302, R^2 = 0.091, p = 0.0023.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.338, R^2 = 0.115, p = 0.0006.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.323, R^2 = 0.104, p = 0.0010.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.319, R^2 = 0.102, p = 0.0012.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.421 0.177 1.28e-05 -0.00269 ok
chrF++ 100 significant negative -0.465 0.217 1.07e-06 -0.00183 ok
METEOR 100 significant negative -0.489 0.239 2.46e-07 -0.00229 ok
ROUGE-L 100 significant negative -0.436 0.190 5.84e-06 -0.00172 ok
BERTScore 100 significant negative -0.680 0.462 7.56e-15 -0.00052 ok
COMET 100 significant negative -0.522 0.273 2.55e-08 -0.00100 ok
BLEURT 100 significant negative -0.809 0.655 2.27e-24 -0.00323 ok
Trigram precision 100 significant negative -0.302 0.091 0.0023 -0.00219 ok
Trigram recall 100 significant negative -0.338 0.115 0.0006 -0.00246 ok
Trigram F1 100 significant negative -0.323 0.104 0.0010 -0.00234 ok
Trigram Jaccard 100 significant negative -0.319 0.102 0.0012 -0.00224 ok

Translation Similarity Metrics

Mean BLEU-447.8%
Mean chrF++70.1%
Mean METEOR74.7%
Mean ROUGE-L76.4%
Mean BERTScore96.1%
Mean COMET78.8%
Mean BLEURT73.6%
Legacy smoothed corpus BLEU44.6%
Unigram F177.3%
Bigram F153.0%
Trigram precision40.5%
Trigram recall37.2%
Trigram F138.8%
Trigram Jaccard24.0%
4-gram F129.5%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 189 -37.212 27.0% 51.6% 47.4% 61.4% 87.9% 72.8% 21.6% reviewed/approved 11781 gpt-5-2025-08-07
2603 Κάρυστος 175 169 20.804 37.5% 63.7% 64.1% 68.6% 94.8% 77.0% 62.0% reviewed/approved 11793 gpt-5-2025-08-07
2609 Κάσπειρος 135 127 11.652 42.8% 66.0% 68.5% 66.4% 90.0% 70.7% 58.4% reviewed/approved 11805 gpt-5-2025-08-07
2604 Καρχηδών 117 112 11.434 32.3% 61.4% 62.5% 66.7% 93.5% 77.5% 57.9% reviewed/approved 11796 gpt-5-2025-08-07
2465 Κάνωπος 83 83 10.356 39.0% 65.4% 82.0% 78.3% 95.3% 78.1% 68.8% reviewed/approved 11769 gpt-5-2025-08-07
2056 Καβειρία 112 87 -9.460 28.3% 63.0% 52.4% 62.3% 94.8% 76.3% 56.2% reviewed/approved 11640 gpt-5-2025-08-07
7254 Κύτα 84 82 8.535 45.1% 66.0% 61.6% 73.5% 94.3% 77.4% 68.4% reviewed/approved 11892 gpt-5-2025-08-07
2605 Κάσιον 56 59 8.529 42.1% 68.1% 73.9% 69.6% 95.5% 75.1% 66.6% reviewed/approved 11799 gpt-5-2025-08-07
2470 Καππαδοκία 68 68 7.674 28.7% 61.1% 77.1% 76.5% 95.6% 74.0% 65.0% reviewed/approved 11775 gpt-5-2025-08-07
7266 Κώμη 79 77 7.641 20.4% 48.6% 48.6% 48.7% 92.6% 70.7% 51.4% reviewed/approved 11928 gpt-5-2025-08-07
2079 Καισάρεια 39 29 -7.511 29.1% 51.4% 55.2% 64.7% 92.8% 68.6% 61.7% reviewed/approved 11679 gpt-5-2025-08-07
2083 Καλαμένθη 24 17 -7.192 32.1% 54.9% 49.1% 63.4% 94.9% 75.3% 72.0% reviewed/approved 11691 gpt-5-2025-08-07
2330 Καλύβη 24 17 -7.192 38.6% 58.3% 57.5% 68.3% 96.0% 77.4% 76.6% reviewed/approved 11739 gpt-5-2025-08-07
2119 Κάλλατις 65 65 7.138 48.0% 69.0% 73.2% 70.8% 95.6% 69.5% 66.3% reviewed/approved 11721 gpt-5-2025-08-07
2342 Κάμιρος 54 55 6.171 55.5% 74.7% 79.7% 80.7% 96.9% 79.5% 70.8% reviewed/approved 11757 gpt-5-2025-08-07
7255 Κυτέριον 22 28 5.450 46.7% 74.2% 64.6% 72.0% 97.0% 78.2% 79.5% reviewed/approved 11895 gpt-5-2025-08-07
7260 Κυχρεῖος πάγος 61 60 5.422 42.6% 65.5% 72.7% 67.8% 95.9% 76.0% 60.8% reviewed/approved 11910 gpt-5-2025-08-07
2116 Κάληρος 40 32 -5.332 20.4% 51.9% 55.1% 58.3% 93.8% 75.2% 70.8% reviewed/approved 11712 gpt-5-2025-08-07
2346 Κάναι 77 73 5.283 58.3% 74.8% 85.7% 86.7% 96.0% 78.7% 75.8% reviewed/approved 11763 gpt-5-2025-08-07
2328 Καλλίπολις 43 45 5.204 50.0% 68.7% 73.8% 68.2% 95.2% 76.8% 70.4% reviewed/approved 11733 gpt-5-2025-08-07
3288 Κορώνεια 110 100 5.183 32.1% 62.8% 62.3% 71.1% 95.1% 75.8% 63.1% reviewed/approved 11844 gpt-5-2025-08-07
7259 Κύφος 54 54 5.171 42.5% 69.1% 77.1% 70.4% 95.5% 75.1% 68.7% reviewed/approved 11907 gpt-5-2025-08-07
2599 Καρπήσιοι 10 8 -4.695 44.3% 74.2% 70.6% 77.8% 97.6% 88.7% 79.8% reviewed/approved 11787 gpt-5-2025-08-07
2114 Κάλβιος 18 15 -4.265 30.9% 58.2% 65.3% 66.7% 94.9% 74.3% 72.3% reviewed/approved 11706 gpt-5-2025-08-07
2628 Καταονία 24 20 -4.192 42.8% 65.4% 61.5% 68.2% 95.4% 78.5% 74.7% reviewed/approved 11835 gpt-5-2025-08-07
2054 Καβαλίς 65 62 4.138 61.7% 75.0% 75.8% 85.0% 97.3% 75.5% 74.1% reviewed/approved 11634 gpt-5-2025-08-07
2059 Καδμεία 20 25 4.092 38.1% 70.5% 71.7% 75.6% 96.2% 78.7% 75.1% reviewed/approved 11649 gpt-5-2025-08-07
2622 Καστωλοῦ πεδίον 42 43 4.026 44.2% 68.9% 82.2% 77.6% 95.8% 75.5% 70.5% reviewed/approved 11814 gpt-5-2025-08-07
7246 Κύρις 20 17 -3.908 67.0% 80.8% 85.2% 86.5% 97.4% 84.2% 77.9% reviewed/approved 11868 gpt-5-2025-08-07
3496 Κοτιάειον 71 59 -3.790 29.1% 51.1% 55.8% 58.5% 93.6% 71.8% 66.5% reviewed/approved 11847 gpt-5-2025-08-07

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.123.

Positive termCoefficientDocsMean presentMean absent
οι 2.6973 11 5.817 -0.719
απο 2.4799 27 2.547 -0.942
και 2.4420 66 0.744 -1.444
τε 2.3687 5 6.075 -0.320
ου 2.1516 14 4.062 -0.661
και το 2.1024 9 5.849 -0.578
αφ ου 1.6422 5 5.510 -0.290
εκληθη 1.5962 4 9.695 -0.404
ος 1.5544 6 4.577 -0.292
εν τη 1.5410 7 4.552 -0.343
εν 1.4643 37 1.791 -1.052
αφ 1.4624 6 5.274 -0.337
τον 1.3441 10 3.017 -0.335
τη 1.3125 15 1.680 -0.296
εστι δε και 1.2287 3 6.274 -0.194
εκληθη δε 1.2085 3 11.119 -0.344
υπο 1.2048 4 6.318 -0.263
δυο 1.1848 4 5.237 -0.218
πολιτης 1.1420 18 3.058 -0.671
απο του 1.1060 9 4.060 -0.402
Negative termCoefficientDocsMean presentMean absent
δια -1.8973 16 0.481 -0.092
εθνος -1.7944 8 -6.232 0.542
το -1.6821 70 0.297 -0.694
περι -1.6361 11 -3.084 0.381
οικητωρ -1.4057 6 -6.248 0.399
το θηλυκον -1.3991 6 -5.272 0.336
τω περι -1.3174 4 -8.034 0.335
εν τω περι -1.3174 4 -8.034 0.335
θρακης -1.2560 5 -3.055 0.161
καππαδοκιας -1.1106 3 -3.233 0.100
ηρωδιανος -1.0856 3 -13.798 0.427
το εθνικον -1.0493 56 -0.431 0.549
εθνικον -1.0493 56 -0.431 0.549
πολις θρακης -0.9486 4 -2.880 0.120
εστι και -0.9410 20 0.578 -0.144
τινες -0.9367 6 -2.462 0.157
ως ελλανικος -0.9238 3 -12.251 0.379
προς τη -0.9208 4 -2.398 0.100
κατα -0.9204 10 -1.713 0.190
φησιν -0.9168 10 -2.484 0.276

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.232.

Positive termCoefficientDocsMean presentMean absent
και 1.6152 66 4.250 2.444
δια 1.4071 16 7.518 2.897
την 1.2691 13 8.487 2.911
τη 1.2595 15 8.172 2.836
απο 1.2265 27 6.694 2.505
απο της 1.0980 6 13.547 3.004
δε 1.0845 46 4.835 2.615
γαρ 1.0662 9 10.449 2.962
εστι 1.0240 24 5.849 2.937
εκαλειτο δε 0.9784 3 23.150 3.033
αφ 0.8555 6 8.427 3.330
τω 0.8530 16 6.468 3.097
εκαλειτο δε και 0.8397 3 23.150 3.033
αφ ου 0.8044 5 9.294 3.338
εν 0.8021 37 4.803 2.951
περι 0.7912 11 6.735 3.253
δε και 0.7825 8 11.291 2.970
εις 0.7647 7 11.793 3.022
θηλυκον 0.7399 12 9.265 2.869
εκαλειτο 0.7195 7 11.373 3.054
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.6087 56 3.850 3.364
εθνικον -0.6087 56 3.850 3.364
τον -0.4907 10 3.658 3.634
δευτερω -0.4861 3 1.303 3.708
νησος -0.4687 9 2.705 3.728
πλησιον -0.4416 8 2.579 3.728
πολις -0.4363 68 3.917 3.039
πολιτης -0.4193 18 3.860 3.587
του δε -0.4068 3 1.318 3.708
τους -0.3925 4 2.379 3.689
προς τη -0.3844 4 2.676 3.676
μια -0.3792 3 1.713 3.696
ειναι -0.3475 8 2.654 3.722
ουδετερως -0.3387 3 0.770 3.725
ην -0.3340 13 3.176 3.705
εκ -0.3047 6 3.134 3.668
προς -0.3037 15 4.144 3.546
πολυβιος -0.3000 5 1.718 3.737
πορρω -0.2874 3 1.215 3.711
ου πορρω -0.2874 3 1.215 3.711

AI English terms predicting large absolute residuals

Features: 272; ridge alpha: 11.938; cross-validated R^2: -0.321.

Positive termCoefficientDocsMean presentMean absent
and 0.7919 56 4.549 2.474
the 0.7609 92 3.679 3.145
also 0.7280 40 5.076 2.676
with 0.6851 16 6.841 3.026
there 0.6235 33 5.689 2.625
by 0.5758 14 7.853 2.950
was 0.5417 18 6.786 2.945
on 0.5216 16 6.148 3.158
says with 0.5075 3 13.487 3.331
was called 0.4393 5 12.281 3.181
and the 0.4336 16 7.823 2.839
for 0.4224 14 8.279 2.880
when 0.4156 4 16.539 3.098
is also 0.4026 26 5.185 3.092
there is 0.3975 23 5.739 3.008
there is also 0.3951 22 5.908 2.995
his on 0.3829 3 14.911 3.287
usage 0.3816 3 15.691 3.263
is 0.3803 89 3.623 3.741
zeus 0.3611 4 17.957 3.039
Negative termCoefficientDocsMean presentMean absent
the ethnonym -0.3022 59 3.758 3.460
ethnonym -0.3022 59 3.758 3.460
the ethnonym is -0.2787 56 3.851 3.362
ethnonym is -0.2787 56 3.851 3.362
an -0.2476 23 3.098 3.797
a city -0.2180 65 3.987 2.984
in book -0.2140 17 4.251 3.510
city of -0.2137 47 3.263 3.967
a city of -0.2137 47 3.263 3.967
book -0.1879 39 4.054 3.369
an island -0.1865 10 2.718 3.738
one -0.1764 7 2.320 3.735
in book of -0.1664 16 4.427 3.485
city -0.1660 68 3.917 3.039
book says -0.1657 3 0.310 3.739
according to -0.1525 5 2.294 3.707
according -0.1525 5 2.294 3.707
neuter -0.1503 4 1.183 3.738
as in -0.1428 34 3.733 3.586
book of his -0.1404 15 4.682 3.451

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 5.878; cross-validated R^2: -0.216.

Positive termCoefficientDocsMean presentMean absent
the 1.5508 90 3.739 2.712
for 1.1765 11 9.407 2.923
also 1.1552 43 5.013 2.598
is with 1.0757 4 13.681 3.218
there 1.0174 34 5.429 2.713
with 0.9196 16 6.611 3.069
is also 0.8603 31 5.653 2.730
is 0.7908 87 3.708 3.156
usage 0.7679 4 11.682 3.301
when 0.7662 7 10.186 3.143
there is 0.7555 24 5.729 2.975
and 0.7343 52 4.676 2.510
to 0.7216 31 5.375 2.855
used to be 0.6964 6 13.249 3.023
according 0.6922 5 12.685 3.160
according to 0.6922 5 12.685 3.160
with the 0.6917 5 10.999 3.249
feminine is 0.6905 12 9.738 2.804
the feminine is 0.6905 12 9.738 2.804
after 0.6801 22 6.567 2.810
Negative termCoefficientDocsMean presentMean absent
as in -0.6898 29 3.643 3.633
the ethnonym is -0.5800 52 3.471 3.815
ethnonym is -0.5800 52 3.471 3.815
ethnonym -0.4630 57 3.839 3.367
a city -0.4493 64 3.913 3.144
the ethnonym -0.4476 56 3.872 3.336
a -0.4149 91 3.717 2.819
in book -0.3861 16 4.324 3.505
in book of -0.3723 14 2.261 3.860
one of -0.3485 5 1.472 3.750
a city in -0.3268 40 3.974 3.411
near -0.3262 16 4.087 3.550
territory -0.3213 5 1.949 3.725
one of the -0.3122 4 1.410 3.729
book -0.3020 39 4.054 3.369
an -0.2859 22 4.294 3.450
island -0.2851 14 3.140 3.717
is a -0.2849 31 5.077 2.989
of his -0.2806 18 4.150 3.523
city in -0.2721 41 4.085 3.324

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.