gpt-4o-2024-11-20 v2 Evaluation

Generated: 2026-09-20 11:20:39 UTC

Pairs100
Distinct lemmas100
Slope0.900
R^20.975
BLEU-440.3%
chrF++65.9%
METEOR71.4%
ROUGE-L72.5%
BERTScore95.4%
COMET78.2%
BLEURT72.0%

Profile version id: 2414

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-11-20) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.900
Intercept3.594
R^20.975
P-value1.30e-80
Pearson r0.988
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words43.310
Mean absolute length residual3.644
Mean absolute percent error7.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.330, R^2 = 0.109, p = 0.0008.
chrF++ vs passage length
chrF++: significant negative; r = -0.372, R^2 = 0.138, p = 0.0001.
METEOR vs passage length
METEOR: significant negative; r = -0.464, R^2 = 0.215, p = 1.17e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.418, R^2 = 0.175, p = 1.53e-05.
BERTScore vs passage length
BERTScore: significant negative; r = -0.600, R^2 = 0.360, p = 4.21e-11.
COMET vs passage length
COMET: significant negative; r = -0.406, R^2 = 0.165, p = 2.74e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.743, R^2 = 0.553, p = 8.16e-19.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.305, R^2 = 0.093, p = 0.0020.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.313, R^2 = 0.098, p = 0.0015.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.310, R^2 = 0.096, p = 0.0017.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.309, R^2 = 0.095, p = 0.0018.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.330 0.109 0.0008 -0.00203 ok
chrF++ 100 significant negative -0.372 0.138 0.0001 -0.00143 ok
METEOR 100 significant negative -0.464 0.215 1.17e-06 -0.00233 ok
ROUGE-L 100 significant negative -0.418 0.175 1.53e-05 -0.00184 ok
BERTScore 100 significant negative -0.600 0.360 4.21e-11 -0.00048 ok
COMET 100 significant negative -0.406 0.165 2.74e-05 -0.00077 ok
BLEURT 100 significant negative -0.743 0.553 8.16e-19 -0.00249 ok
Trigram precision 100 significant negative -0.305 0.093 0.0020 -0.00215 ok
Trigram recall 100 significant negative -0.313 0.098 0.0015 -0.00222 ok
Trigram F1 100 significant negative -0.310 0.096 0.0017 -0.00218 ok
Trigram Jaccard 100 significant negative -0.309 0.095 0.0018 -0.00195 ok

Translation Similarity Metrics

Mean BLEU-440.3%
Mean chrF++65.9%
Mean METEOR71.4%
Mean ROUGE-L72.5%
Mean BERTScore95.4%
Mean COMET78.2%
Mean BLEURT72.0%
Legacy smoothed corpus BLEU40.1%
Unigram F173.6%
Bigram F146.8%
Trigram precision33.1%
Trigram recall32.4%
Trigram F132.8%
Trigram Jaccard19.6%
4-gram F123.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 216 -30.478 26.8% 53.8% 47.7% 58.0% 90.2% 78.2% 43.2% reviewed/approved 12982 gpt-4o-2024-11-20
2604 Καρχηδών 117 132 23.156 30.6% 61.8% 58.1% 61.6% 93.3% 78.0% 63.2% reviewed/approved 12997 gpt-4o-2024-11-20
3288 Κορώνεια 110 115 12.453 17.3% 55.4% 54.5% 60.2% 93.2% 75.0% 59.1% reviewed/approved 13045 gpt-4o-2024-11-20
2346 Κάναι 77 85 12.139 45.0% 66.0% 71.5% 77.8% 94.9% 77.3% 63.2% reviewed/approved 12964 gpt-4o-2024-11-20
2342 Κάμιρος 54 63 10.829 40.4% 72.0% 69.7% 71.8% 95.4% 78.0% 64.0% reviewed/approved 12958 gpt-4o-2024-11-20
7259 Κύφος 54 62 9.829 38.4% 71.8% 74.2% 74.1% 94.6% 71.1% 62.0% reviewed/approved 13108 gpt-4o-2024-11-20
2597 Καρπασία 112 114 9.654 34.1% 64.5% 60.3% 65.5% 95.1% 76.2% 61.3% reviewed/approved 12985 gpt-4o-2024-11-20
2055 Καβασσός 109 92 -9.647 25.5% 55.9% 55.4% 56.7% 93.3% 75.5% 49.9% reviewed/approved 12838 gpt-4o-2024-11-20
2603 Κάρυστος 175 170 8.981 25.9% 56.0% 57.5% 60.9% 93.6% 77.1% 62.3% reviewed/approved 12994 gpt-4o-2024-11-20
2465 Κάνωπος 83 86 7.742 31.7% 61.7% 80.1% 73.4% 93.9% 74.6% 64.4% reviewed/approved 12970 gpt-4o-2024-11-20
2328 Καλλίπολις 43 50 7.725 34.0% 61.6% 63.7% 62.4% 94.8% 74.0% 69.0% reviewed/approved 12934 gpt-4o-2024-11-20
2470 Καππαδοκία 68 72 7.235 22.7% 62.1% 61.7% 68.6% 95.4% 78.4% 69.7% reviewed/approved 12976 gpt-4o-2024-11-20
7255 Κυτέριον 22 30 6.615 10.3% 48.2% 42.1% 50.0% 92.7% 73.7% 67.3% reviewed/approved 13096 gpt-4o-2024-11-20
2116 Κάληρος 40 33 -6.577 14.8% 44.0% 42.3% 46.6% 91.5% 72.7% 67.2% reviewed/approved 12913 gpt-4o-2024-11-20
2054 Καβαλίς 65 68 5.934 50.4% 69.0% 64.6% 73.7% 95.9% 75.8% 71.9% reviewed/approved 12835 gpt-4o-2024-11-20
2119 Κάλλατις 65 68 5.934 31.6% 60.1% 70.1% 66.2% 94.3% 71.9% 65.1% reviewed/approved 12922 gpt-4o-2024-11-20
7254 Κύτα 84 85 5.842 37.1% 63.0% 52.4% 66.3% 93.0% 76.4% 60.7% reviewed/approved 13093 gpt-4o-2024-11-20
7260 Κυχρεῖος πάγος 61 64 5.532 38.6% 64.7% 67.8% 59.2% 95.1% 75.0% 60.7% reviewed/approved 13111 gpt-4o-2024-11-20
3496 Κοτιάειον 71 62 -5.463 21.7% 43.4% 44.1% 46.6% 91.3% 71.3% 62.5% reviewed/approved 13048 gpt-4o-2024-11-20
2624 Κατάβαθμος 20 17 -4.585 51.7% 76.0% 77.8% 81.1% 95.7% 82.2% 77.2% reviewed/approved 13021 gpt-4o-2024-11-20
2078 Καινύς 22 19 -4.385 33.3% 58.2% 63.0% 63.4% 93.9% 81.2% 68.9% reviewed/approved 12874 gpt-4o-2024-11-20
7262 Κῶβρυς 22 19 -4.385 40.9% 60.2% 58.9% 68.3% 95.1% 71.4% 70.6% reviewed/approved 13117 gpt-4o-2024-11-20
7266 Κώμη 79 79 4.340 21.9% 50.4% 49.7% 50.6% 92.7% 73.0% 56.2% reviewed/approved 13129 gpt-4o-2024-11-20
2330 Καλύβη 24 21 -4.184 41.4% 62.9% 64.1% 80.0% 96.4% 79.8% 77.2% reviewed/approved 12940 gpt-4o-2024-11-20
7261 Κύψελα 34 30 -4.179 39.8% 62.8% 66.4% 68.8% 95.7% 77.9% 71.3% reviewed/approved 13114 gpt-4o-2024-11-20
2326 Καλλίαρος 47 50 4.126 43.5% 67.8% 81.3% 78.4% 96.1% 72.6% 68.4% reviewed/approved 12928 gpt-4o-2024-11-20
2605 Κάσιον 56 58 4.030 37.8% 63.8% 69.9% 70.2% 94.3% 73.1% 60.3% reviewed/approved 13000 gpt-4o-2024-11-20
3258 Κεκρυφάλεια 26 23 -3.983 53.5% 76.1% 85.1% 89.8% 98.3% 79.4% 81.1% reviewed/approved 13042 gpt-4o-2024-11-20
3530 Κριώα 26 23 -3.983 39.6% 69.1% 73.6% 77.6% 95.7% 76.5% 70.6% reviewed/approved 13054 gpt-4o-2024-11-20
2077 Καιρή 17 15 -3.887 49.7% 71.4% 76.3% 75.0% 97.4% 87.0% 80.6% reviewed/approved 12877 gpt-4o-2024-11-20

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.177.

Positive termCoefficientDocsMean presentMean absent
και 2.5045 66 0.618 -1.200
πολιτης 1.7977 18 5.047 -1.108
απο 1.6719 27 3.033 -1.122
δε και 1.4985 8 1.851 -0.161
εστι δε και 1.4676 3 11.111 -0.344
ος 1.4392 6 5.928 -0.378
ου 1.4156 14 3.733 -0.608
δυο 1.3643 4 7.911 -0.330
εστι δε 1.3590 5 6.764 -0.356
πολις 1.2295 68 0.487 -1.036
παιδος 1.2202 4 6.200 -0.258
πολιχνιον 1.1714 3 8.599 -0.266
την 1.1522 13 2.953 -0.441
εν τη 1.0961 7 3.670 -0.276
απο του 1.0092 9 4.604 -0.455
δευτερα 0.9452 3 6.732 -0.208
εκαλειτο δε 0.9176 3 0.553 -0.017
αφ ου 0.9023 5 5.051 -0.266
αυτην 0.8061 9 2.755 -0.273
βοιωτιας 0.8013 3 3.408 -0.105
Negative termCoefficientDocsMean presentMean absent
το -1.6678 70 -0.156 0.365
καππαδοκιας -1.4154 3 -5.503 0.170
περι -1.3617 11 -2.925 0.362
γαρ -1.2684 9 -2.485 0.246
οικητωρ -1.2288 6 -6.180 0.394
το εθνικον -1.1128 56 -0.838 1.066
εθνικον -1.1128 56 -0.838 1.066
εθνος -1.0676 8 -4.978 0.433
ουτως -0.8821 5 -6.880 0.362
τοπικα -0.8774 3 -5.033 0.156
τα τοπικα -0.8774 3 -5.033 0.156
πολις θρακης -0.8648 4 -3.258 0.136
τινες -0.8448 6 -2.609 0.167
μετα -0.8384 3 -11.856 0.367
τω περι -0.8188 4 -6.901 0.288
εν τω περι -0.8188 4 -6.901 0.288
ηρωδιανος -0.7934 3 -11.499 0.356
θρακης -0.7586 5 -2.320 0.122
και του -0.7545 4 -3.273 0.136
το θηλυκον -0.6942 6 -3.384 0.216

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.025.

Positive termCoefficientDocsMean presentMean absent
την 2.2661 13 8.665 2.894
δε 2.0899 46 4.809 2.652
εκαλειτο 1.7027 7 11.470 3.055
εστι 1.6991 24 6.174 2.845
απο 1.6598 27 7.039 2.389
εκαλειτο δε 1.5654 3 20.872 3.111
δε και 1.4714 8 10.958 3.008
και 1.1484 66 4.248 2.473
δια 1.1007 16 7.101 2.986
εστι δε και 1.0975 3 12.628 3.366
εκαλειτο δε και 1.0542 3 20.872 3.111
περι 1.0241 11 6.414 3.302
εν 1.0013 37 4.946 2.880
απο της 0.9700 6 10.421 3.212
ος 0.9319 6 7.994 3.367
εστι δε 0.9039 5 9.013 3.362
φησιν 0.8862 10 9.112 3.037
εστι και 0.8609 20 5.778 3.111
εν τω περι 0.8533 4 10.764 3.348
τω περι 0.8533 4 10.764 3.348
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.9683 56 3.287 4.099
εθνικον -0.9683 56 3.287 4.099
μια -0.7789 3 0.689 3.736
προς τη -0.6739 4 1.965 3.714
τους -0.6708 4 1.675 3.726
δευτερω -0.6678 3 1.195 3.720
ης -0.5534 3 1.562 3.709
και το -0.5395 9 3.839 3.625
του δε -0.5164 3 1.405 3.713
ως -0.4746 52 4.013 3.244
παυσανιας -0.4677 4 1.992 3.713
πολυβιος -0.4648 5 1.589 3.752
το -0.4641 70 3.521 3.932
εξ -0.4482 3 2.220 3.688
τον -0.4029 10 4.443 3.555
πολις προς -0.4028 3 3.825 3.639
ουδετερως -0.3672 3 2.221 3.688
φασι -0.3348 8 4.704 3.552
νησος -0.3322 9 3.560 3.652
αιγυπτου -0.3132 3 2.236 3.688

AI English terms predicting large absolute residuals

Features: 318; ridge alpha: 4.125; cross-validated R^2: -0.058.

Positive termCoefficientDocsMean presentMean absent
was 1.8496 17 7.232 2.909
the 1.5398 95 3.700 2.592
called 1.3801 27 6.224 2.690
there 1.3660 33 5.964 2.501
there is 1.3348 24 6.344 2.792
its 1.2971 6 8.514 3.333
diphthong 1.2866 3 13.622 3.336
there is also 1.2129 22 6.447 2.854
was called 1.1698 4 11.228 3.328
usage 1.0965 3 11.562 3.399
is also a 1.0913 11 10.071 2.850
by 1.0528 15 7.663 2.935
on 1.0514 13 6.346 3.241
is also 1.0060 31 5.801 2.675
also a 0.9678 12 9.375 2.863
at 0.9651 6 7.147 3.421
after 0.9547 19 6.697 2.928
of 0.9354 87 3.687 3.359
most 0.8725 4 16.345 3.115
among 0.8703 4 11.433 3.320
Negative termCoefficientDocsMean presentMean absent
ethnonym is -0.9214 54 3.381 3.954
the ethnonym is -0.9143 53 3.381 3.941
the ethnonym -0.8323 58 3.514 3.824
they -0.8255 7 1.675 3.792
ethnonym -0.6886 59 3.511 3.836
one of the -0.5301 3 0.689 3.736
one of -0.5301 3 0.689 3.736
from which -0.4794 3 1.509 3.710
as in -0.4670 34 4.399 3.255
island -0.4261 14 3.360 3.690
syria -0.4136 3 0.604 3.738
herodotus -0.4108 8 2.367 3.755
name of the -0.3985 3 0.722 3.735
name of -0.3985 3 0.722 3.735
either -0.3888 3 1.319 3.716
herodotus in -0.3646 4 1.051 3.752
states in -0.3432 3 1.576 3.708
and the -0.3400 8 3.095 3.692
book of -0.3323 15 4.266 3.535
polybios in book -0.3322 4 1.182 3.747

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: -0.022.

Positive termCoefficientDocsMean presentMean absent
there is 1.4261 24 6.515 2.738
there 1.3728 34 6.027 2.417
there is also 1.3593 22 6.894 2.728
the 1.3236 90 3.852 1.772
is with 1.2551 4 13.646 3.227
says 1.2281 17 7.363 2.882
and 1.1802 52 4.790 2.403
also 1.1321 43 5.155 2.505
is also 1.1297 31 5.942 2.612
used to be 1.0691 6 13.285 3.029
is also a 1.0307 14 8.862 2.795
to be called 0.9945 5 14.145 3.092
usage 0.9862 4 10.721 3.349
who 0.9827 13 7.259 3.104
to be 0.9804 9 10.789 2.938
also a 0.9272 15 8.386 2.807
diphthong 0.8926 3 13.622 3.336
on 0.8708 16 5.827 3.228
of 0.8514 66 4.228 2.512
their 0.8295 3 20.872 3.111
Negative termCoefficientDocsMean presentMean absent
as in -1.0458 29 3.480 3.711
the ethnonym is -0.8477 52 3.141 4.189
ethnonym is -0.8477 52 3.141 4.189
the ethnonym -0.7899 56 3.258 4.135
ethnonym -0.7380 57 3.370 4.007
one of -0.5979 5 1.489 3.758
and the -0.5951 15 2.607 3.827
one of the -0.5848 4 1.030 3.753
they -0.5381 12 4.111 3.581
syria -0.4956 3 0.604 3.738
in syria -0.4956 3 0.604 3.738
which -0.4938 20 3.693 3.632
was from -0.4936 4 1.342 3.740
in book of -0.4799 14 2.433 3.841
the same -0.4388 4 0.857 3.760
same -0.4388 4 0.857 3.760
herodotos book -0.4363 5 1.448 3.760
name -0.3923 7 2.170 3.755
book -0.3847 39 3.698 3.610
that of the -0.3594 3 0.722 3.735

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.