gpt-4o-2024-11-20 v2 Evaluation

Generated: 2026-07-25 12:52:53 UTC

Pairs100
Distinct lemmas100
Slope0.900
R^20.975
BLEU-440.2%
chrF++65.8%
METEOR71.4%
ROUGE-L72.4%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2414

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-11-20) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.900
Intercept3.578
R^20.975
P-value1.46e-80
Pearson r0.988
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words43.310
Mean absolute length residual3.650
Mean absolute percent error7.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.328, R^2 = 0.108, p = 0.0008.
chrF++ vs passage length
chrF++: significant negative; r = -0.363, R^2 = 0.132, p = 0.0002.
METEOR vs passage length
METEOR: significant negative; r = -0.449, R^2 = 0.201, p = 2.84e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.401, R^2 = 0.160, p = 3.63e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.305, R^2 = 0.093, p = 0.0020.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.312, R^2 = 0.097, p = 0.0016.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.309, R^2 = 0.096, p = 0.0017.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.308, R^2 = 0.095, p = 0.0018.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.328 0.108 0.0008 -0.00202 ok
chrF++ 100 significant negative -0.363 0.132 0.0002 -0.00141 ok
METEOR 100 significant negative -0.449 0.201 2.84e-06 -0.00228 ok
ROUGE-L 100 significant negative -0.401 0.160 3.63e-05 -0.00180 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.305 0.093 0.0020 -0.00215 ok
Trigram recall 100 significant negative -0.312 0.097 0.0016 -0.00222 ok
Trigram F1 100 significant negative -0.309 0.096 0.0017 -0.00218 ok
Trigram Jaccard 100 significant negative -0.308 0.095 0.0018 -0.00195 ok

Translation Similarity Metrics

Mean BLEU-440.2%
Mean chrF++65.8%
Mean METEOR71.4%
Mean ROUGE-L72.4%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU40.1%
Unigram F173.5%
Bigram F146.8%
Trigram precision33.1%
Trigram recall32.4%
Trigram F132.8%
Trigram Jaccard19.6%
4-gram F123.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 216 -30.507 26.8% 53.8% 47.7% 58.0% N/A N/A N/A reviewed/approved 12982 gpt-4o-2024-11-20
2604 Καρχηδών 117 132 23.153 30.6% 61.8% 58.1% 61.6% N/A N/A N/A reviewed/approved 12997 gpt-4o-2024-11-20
3288 Κορώνεια 110 115 12.451 17.3% 55.4% 54.5% 60.2% N/A N/A N/A reviewed/approved 13045 gpt-4o-2024-11-20
2346 Κάναι 77 85 12.143 45.0% 66.0% 71.5% 77.8% N/A N/A N/A reviewed/approved 12964 gpt-4o-2024-11-20
2342 Κάμιρος 54 63 10.837 40.4% 72.0% 69.7% 71.8% N/A N/A N/A reviewed/approved 12958 gpt-4o-2024-11-20
7259 Κύφος 54 62 9.837 38.4% 71.8% 74.2% 74.1% N/A N/A N/A reviewed/approved 13108 gpt-4o-2024-11-20
2597 Καρπασία 112 114 9.652 34.1% 64.5% 60.3% 65.5% N/A N/A N/A reviewed/approved 12985 gpt-4o-2024-11-20
2055 Καβασσός 109 92 -9.649 25.5% 55.9% 55.4% 56.7% N/A N/A N/A reviewed/approved 12838 gpt-4o-2024-11-20
2603 Κάρυστος 175 170 8.968 25.9% 56.0% 57.5% 60.9% N/A N/A N/A reviewed/approved 12994 gpt-4o-2024-11-20
2465 Κάνωπος 83 86 7.744 31.7% 61.7% 80.1% 73.4% N/A N/A N/A reviewed/approved 12970 gpt-4o-2024-11-20
2328 Καλλίπολις 43 50 7.734 34.0% 61.6% 63.7% 62.4% N/A N/A N/A reviewed/approved 12934 gpt-4o-2024-11-20
2470 Καππαδοκία 68 72 7.240 22.7% 62.1% 61.7% 68.6% N/A N/A N/A reviewed/approved 12976 gpt-4o-2024-11-20
7255 Κυτέριον 22 30 6.628 10.3% 48.2% 42.1% 50.0% N/A N/A N/A reviewed/approved 13096 gpt-4o-2024-11-20
2116 Κάληρος 40 33 -6.567 14.8% 44.0% 42.3% 46.6% N/A N/A N/A reviewed/approved 12913 gpt-4o-2024-11-20
2054 Καβαλίς 65 68 5.939 50.4% 69.0% 64.6% 73.7% N/A N/A N/A reviewed/approved 12835 gpt-4o-2024-11-20
2119 Κάλλατις 65 68 5.939 31.6% 60.1% 70.1% 66.2% N/A N/A N/A reviewed/approved 12922 gpt-4o-2024-11-20
7254 Κύτα 84 85 5.844 37.1% 63.0% 55.6% 65.1% N/A N/A N/A reviewed/approved 13093 gpt-4o-2024-11-20
7260 Κυχρεῖος πάγος 61 64 5.538 38.6% 64.7% 67.8% 59.2% N/A N/A N/A reviewed/approved 13111 gpt-4o-2024-11-20
3496 Κοτιάειον 71 62 -5.459 21.7% 43.4% 44.1% 46.6% N/A N/A N/A reviewed/approved 13048 gpt-4o-2024-11-20
2624 Κατάβαθμος 20 17 -4.572 51.7% 76.0% 77.8% 81.1% N/A N/A N/A reviewed/approved 13021 gpt-4o-2024-11-20
2599 Καρπήσιοι 11 9 -4.475 15.1% 47.9% 34.7% 40.0% N/A N/A N/A reviewed/approved 12988 gpt-4o-2024-11-20
2078 Καινύς 22 19 -4.372 33.3% 58.2% 63.0% 63.4% N/A N/A N/A reviewed/approved 12874 gpt-4o-2024-11-20
7262 Κῶβρυς 22 19 -4.372 40.9% 60.2% 58.9% 68.3% N/A N/A N/A reviewed/approved 13117 gpt-4o-2024-11-20
7266 Κώμη 79 79 4.343 21.9% 50.4% 49.7% 50.6% N/A N/A N/A reviewed/approved 13129 gpt-4o-2024-11-20
2330 Καλύβη 24 21 -4.171 41.4% 62.9% 64.1% 80.0% N/A N/A N/A reviewed/approved 12940 gpt-4o-2024-11-20
7261 Κύψελα 34 30 -4.169 39.8% 62.8% 66.4% 68.8% N/A N/A N/A reviewed/approved 13114 gpt-4o-2024-11-20
2326 Καλλίαρος 47 50 4.135 43.5% 67.8% 81.3% 78.4% N/A N/A N/A reviewed/approved 12928 gpt-4o-2024-11-20
2605 Κάσιον 56 58 4.037 37.8% 63.8% 69.9% 70.2% N/A N/A N/A reviewed/approved 13000 gpt-4o-2024-11-20
3258 Κεκρυφάλεια 26 23 -3.971 53.5% 76.1% 85.1% 89.8% N/A N/A N/A reviewed/approved 13042 gpt-4o-2024-11-20
3530 Κριώα 26 23 -3.971 39.6% 69.1% 73.6% 77.6% N/A N/A N/A reviewed/approved 13054 gpt-4o-2024-11-20

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.175.

Positive termCoefficientDocsMean presentMean absent
και 2.5203 66 0.625 -1.213
πολιτης 1.8046 18 5.054 -1.109
απο 1.6733 27 3.036 -1.123
δε και 1.4994 8 1.850 -0.161
εστι δε και 1.4682 3 11.113 -0.344
ος 1.4399 6 5.929 -0.378
ου 1.4192 14 3.736 -0.608
δυο 1.3663 4 7.916 -0.330
εστι δε 1.3602 5 6.766 -0.356
πολις 1.2438 68 0.496 -1.053
παιδος 1.2212 4 6.209 -0.259
πολιχνιον 1.1727 3 8.605 -0.266
την 1.1507 13 2.951 -0.441
εν τη 1.0976 7 3.678 -0.277
απο του 1.0101 9 4.608 -0.456
δευτερα 0.9464 3 6.738 -0.208
εκαλειτο δε 0.9165 3 0.538 -0.017
αφ ου 0.9016 5 5.050 -0.266
αυτην 0.8089 9 2.761 -0.273
βοιωτιας 0.8014 3 3.411 -0.105
Negative termCoefficientDocsMean presentMean absent
το -1.6558 70 -0.149 0.347
καππαδοκιας -1.4138 3 -5.496 0.170
περι -1.3596 11 -2.921 0.361
γαρ -1.2709 9 -2.490 0.246
οικητωρ -1.2254 6 -6.175 0.394
εθνος -1.1481 8 -5.085 0.442
το εθνικον -1.1000 56 -0.829 1.056
εθνικον -1.1000 56 -0.829 1.056
ουτως -0.8824 5 -6.885 0.362
τοπικα -0.8758 3 -5.027 0.155
τα τοπικα -0.8758 3 -5.027 0.155
πολις θρακης -0.8582 4 -3.246 0.135
τινες -0.8425 6 -2.600 0.166
μετα -0.8395 3 -11.867 0.367
τω περι -0.8182 4 -6.902 0.288
εν τω περι -0.8182 4 -6.902 0.288
ηρωδιανος -0.7936 3 -11.507 0.356
θρακης -0.7523 5 -2.309 0.122
και του -0.7491 4 -3.265 0.136
το θηλυκον -0.6935 6 -3.385 0.216

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.029.

Positive termCoefficientDocsMean presentMean absent
την 2.2660 13 8.668 2.900
δε 2.0825 46 4.809 2.662
εκαλειτο 1.6979 7 11.468 3.061
εστι 1.6921 24 6.172 2.853
απο 1.6576 27 7.040 2.396
εκαλειτο δε 1.5650 3 20.876 3.117
δε και 1.4704 8 10.960 3.014
και 1.1347 66 4.247 2.491
δια 1.1017 16 7.105 2.992
εστι δε και 1.0957 3 12.624 3.372
εκαλειτο δε και 1.0540 3 20.876 3.117
περι 1.0210 11 6.414 3.308
εν 0.9875 37 4.945 2.889
απο της 0.9701 6 10.426 3.217
ος 0.9307 6 7.996 3.372
εστι δε 0.9018 5 9.012 3.368
φησιν 0.8842 10 9.115 3.042
εστι και 0.8544 20 5.777 3.118
εν τω περι 0.8527 4 10.768 3.353
τω περι 0.8527 4 10.768 3.353
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.9822 56 3.283 4.116
εθνικον -0.9822 56 3.283 4.116
μια -0.7827 3 0.687 3.741
προς τη -0.6784 4 1.960 3.720
δευτερω -0.6746 3 1.191 3.726
τους -0.6721 4 1.675 3.732
ης -0.5528 3 1.570 3.714
και το -0.5401 9 3.839 3.631
του δε -0.5150 3 1.409 3.719
ως -0.4850 52 4.011 3.258
το -0.4768 70 3.518 3.956
παυσανιας -0.4680 4 1.995 3.719
πολυβιος -0.4676 5 1.589 3.758
εξ -0.4494 3 2.219 3.694
τον -0.4057 10 4.442 3.562
πολις προς -0.4028 3 3.821 3.644
ουδετερως -0.3697 3 2.220 3.694
νησος -0.3369 9 3.556 3.659
φασι -0.3359 8 4.702 3.558
αιγυπτου -0.3125 3 2.241 3.693

AI English terms predicting large absolute residuals

Features: 318; ridge alpha: 4.125; cross-validated R^2: -0.062.

Positive termCoefficientDocsMean presentMean absent
was 1.8444 17 7.232 2.916
the 1.5305 95 3.706 2.584
called 1.3768 27 6.226 2.697
there 1.3604 33 5.963 2.510
there is 1.3309 24 6.343 2.799
its 1.2985 6 8.521 3.339
diphthong 1.2882 3 13.633 3.341
there is also 1.2090 22 6.446 2.861
was called 1.1674 4 11.227 3.334
usage 1.0976 3 11.571 3.405
is also a 1.0891 11 10.073 2.856
on 1.0505 13 6.347 3.247
by 1.0500 15 7.663 2.942
is also 1.0030 31 5.802 2.683
also a 0.9661 12 9.378 2.869
at 0.9617 6 7.142 3.427
after 0.9538 19 6.699 2.935
of 0.9388 87 3.694 3.353
most 0.8719 4 16.356 3.120
among 0.8711 4 11.441 3.325
Negative termCoefficientDocsMean presentMean absent
ethnonym is -0.9317 54 3.377 3.970
the ethnonym is -0.9246 53 3.377 3.957
the ethnonym -0.8422 58 3.510 3.843
they -0.8277 7 1.682 3.798
ethnonym -0.6983 59 3.507 3.855
one of the -0.5306 3 0.687 3.741
one of -0.5306 3 0.687 3.741
from which -0.4800 3 1.512 3.716
as in -0.4776 34 4.395 3.266
island -0.4298 14 3.356 3.698
herodotus -0.4135 8 2.365 3.761
syria -0.4133 3 0.609 3.744
name of the -0.3990 3 0.720 3.740
name of -0.3990 3 0.720 3.740
either -0.3902 3 1.319 3.722
herodotus in -0.3664 4 1.051 3.758
states in -0.3429 3 1.579 3.714
and the -0.3425 8 3.097 3.698
book of -0.3359 15 4.265 3.541
polybios in book -0.3336 4 1.179 3.753

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 4.125; cross-validated R^2: -0.024.

Positive termCoefficientDocsMean presentMean absent
there is 1.4202 24 6.514 2.745
there 1.3676 34 6.027 2.425
there is also 1.3539 22 6.893 2.735
the 1.3132 90 3.859 1.764
is with 1.2499 4 13.657 3.233
says 1.2225 17 7.365 2.889
and 1.1618 52 4.789 2.416
is also 1.1239 31 5.942 2.620
also 1.1209 43 5.156 2.514
used to be 1.0648 6 13.284 3.035
is also a 1.0299 14 8.862 2.801
to be called 0.9906 5 14.148 3.097
usage 0.9824 4 10.732 3.355
to be 0.9766 9 10.787 2.944
who 0.9761 13 7.258 3.111
also a 0.9268 15 8.387 2.814
diphthong 0.8882 3 13.633 3.341
on 0.8670 16 5.829 3.235
of 0.8378 66 4.226 2.532
their 0.8284 3 20.876 3.117
Negative termCoefficientDocsMean presentMean absent
as in -1.0504 29 3.477 3.721
the ethnonym is -0.8597 52 3.137 4.205
ethnonym is -0.8597 52 3.137 4.205
the ethnonym -0.8013 56 3.255 4.153
ethnonym -0.7495 57 3.367 4.025
one of -0.5975 5 1.490 3.763
and the -0.5951 15 2.608 3.834
one of the -0.5851 4 1.028 3.759
they -0.5404 12 4.113 3.587
which -0.4967 20 3.692 3.639
syria -0.4951 3 0.609 3.744
in syria -0.4951 3 0.609 3.744
was from -0.4937 4 1.336 3.746
in book of -0.4871 14 2.428 3.849
the same -0.4390 4 0.855 3.766
same -0.4390 4 0.855 3.766
herodotos book -0.4383 5 1.449 3.766
book -0.3947 39 3.696 3.620
name -0.3911 7 2.168 3.761
that of the -0.3594 3 0.720 3.740

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.