gpt-5.2 v2 Evaluation

Generated: 2026-07-25 12:53:25 UTC

Pairs100
Distinct lemmas100
Slope0.922
R^20.988
BLEU-448.1%
chrF++70.4%
METEOR75.6%
ROUGE-L76.0%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2332

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 12:28:27.643960+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.2 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.922
Intercept1.802
R^20.988
P-value1.29e-95
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.500
Mean absolute length residual2.701
Mean absolute percent error6.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.394, R^2 = 0.155, p = 4.94e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.462, R^2 = 0.213, p = 1.31e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.465, R^2 = 0.216, p = 1.08e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.432, R^2 = 0.187, p = 7.10e-06.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.302, R^2 = 0.091, p = 0.0023.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.322, R^2 = 0.104, p = 0.0011.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.312, R^2 = 0.098, p = 0.0015.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.321, R^2 = 0.103, p = 0.0011.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.394 0.155 4.94e-05 -0.00270 ok
chrF++ 100 significant negative -0.462 0.213 1.31e-06 -0.00197 ok
METEOR 100 significant negative -0.465 0.216 1.08e-06 -0.00217 ok
ROUGE-L 100 significant negative -0.432 0.187 7.10e-06 -0.00170 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.302 0.091 0.0023 -0.00228 ok
Trigram recall 100 significant negative -0.322 0.104 0.0011 -0.00242 ok
Trigram F1 100 significant negative -0.312 0.098 0.0015 -0.00234 ok
Trigram Jaccard 100 significant negative -0.321 0.103 0.0011 -0.00233 ok

Translation Similarity Metrics

Mean BLEU-448.1%
Mean chrF++70.4%
Mean METEOR75.6%
Mean ROUGE-L76.0%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU45.9%
Unigram F177.4%
Bigram F152.7%
Trigram precision40.0%
Trigram recall38.4%
Trigram F139.2%
Trigram Jaccard24.4%
4-gram F129.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 126 16.371 30.2% 62.4% 72.8% 66.9% N/A N/A N/A reviewed/approved 11498 gpt-5.2-2025-12-11
2056 Καβειρία 112 91 -14.021 33.2% 60.7% 60.3% 68.0% N/A N/A N/A reviewed/approved 11343 gpt-5.2-2025-12-11
2484 Καρία 270 237 -13.633 32.8% 58.5% 52.5% 63.5% N/A N/A N/A reviewed/approved 11483 gpt-5.2-2025-12-11
2603 Κάρυστος 175 174 10.919 34.7% 61.9% 61.8% 65.9% N/A N/A N/A reviewed/approved 11495 gpt-5.2-2025-12-11
3288 Κορώνεια 110 114 10.822 33.3% 63.0% 67.1% 66.7% N/A N/A N/A reviewed/approved 11546 gpt-5.2-2025-12-11
2328 Καλλίπολις 43 50 8.569 40.8% 63.1% 68.6% 66.7% N/A N/A N/A reviewed/approved 11435 gpt-5.2-2025-12-11
2342 Κάμιρος 54 59 7.432 41.5% 68.3% 76.8% 70.8% N/A N/A N/A reviewed/approved 11459 gpt-5.2-2025-12-11
7251 Κύρτος 66 56 -6.628 22.7% 52.3% 60.7% 66.7% N/A N/A N/A reviewed/approved 11583 gpt-5.2-2025-12-11
2346 Κάναι 77 79 6.235 49.6% 66.8% 74.0% 79.5% N/A N/A N/A reviewed/approved 11465 gpt-5.2-2025-12-11
7255 Κυτέριον 22 28 5.923 29.0% 63.6% 65.0% 56.0% N/A N/A N/A reviewed/approved 11595 gpt-5.2-2025-12-11
2609 Κάσπειρος 135 132 5.782 53.0% 70.5% 70.9% 71.2% N/A N/A N/A reviewed/approved 11507 gpt-5.2-2025-12-11
7254 Κύτα 84 74 -5.216 43.9% 64.2% 63.1% 72.2% N/A N/A N/A reviewed/approved 11592 gpt-5.2-2025-12-11
7264 Κωλιάς 61 53 -5.020 33.9% 59.0% 56.1% 66.1% N/A N/A N/A reviewed/approved 11622 gpt-5.2-2025-12-11
2332 Καλυδών 20 25 4.766 26.9% 67.9% 68.3% 71.1% N/A N/A N/A reviewed/approved 11447 gpt-5.2-2025-12-11
2116 Κάληρος 40 34 -4.666 32.7% 63.0% 71.0% 67.6% N/A N/A N/A reviewed/approved 11414 gpt-5.2-2025-12-11
2605 Κάσιον 56 58 4.588 51.5% 72.7% 70.4% 73.7% N/A N/A N/A reviewed/approved 11501 gpt-5.2-2025-12-11
7249 Κύρρος 45 39 -4.274 30.5% 55.3% 61.8% 69.0% N/A N/A N/A reviewed/approved 11578 gpt-5.2-2025-12-11
7260 Κυχρεῖος πάγος 61 62 3.980 44.8% 70.3% 73.6% 66.7% N/A N/A N/A reviewed/approved 11610 gpt-5.2-2025-12-11
2083 Καλαμένθη 24 20 -3.921 12.8% 46.4% 48.5% 54.5% N/A N/A N/A reviewed/approved 11393 gpt-5.2-2025-12-11
2326 Καλλίαρος 47 49 3.883 53.5% 73.3% 86.5% 83.3% N/A N/A N/A reviewed/approved 11429 gpt-5.2-2025-12-11
7247 Κύρνος 50 44 -3.882 34.6% 62.4% 68.3% 72.3% N/A N/A N/A reviewed/approved 11572 gpt-5.2-2025-12-11
2059 Καδμεία 20 24 3.766 36.3% 73.2% 82.7% 81.8% N/A N/A N/A reviewed/approved 11352 gpt-5.2-2025-12-11
2079 Καισάρεια 39 34 -3.745 47.9% 70.2% 79.8% 84.9% N/A N/A N/A reviewed/approved 11381 gpt-5.2-2025-12-11
7253 Κυρτώνιος 18 22 3.609 41.4% 71.4% 77.4% 70.0% N/A N/A N/A reviewed/approved 11589 gpt-5.2-2025-12-11
2345 Κάναθα 17 21 3.531 58.3% 80.9% 89.5% 78.9% N/A N/A N/A reviewed/approved 11462 gpt-5.2-2025-12-11
2470 Καππαδοκία 68 68 3.529 16.3% 53.5% 82.3% 76.5% N/A N/A N/A reviewed/approved 11477 gpt-5.2-2025-12-11
2054 Καβαλίς 65 65 3.294 55.0% 70.7% 73.1% 75.4% N/A N/A N/A reviewed/approved 11337 gpt-5.2-2025-12-11
3496 Κοτιάειον 71 64 -3.236 28.9% 48.1% 49.9% 54.8% N/A N/A N/A reviewed/approved 11549 gpt-5.2-2025-12-11
2455 Κάναστρον 60 54 -3.098 19.6% 52.5% 54.1% 63.2% N/A N/A N/A reviewed/approved 11468 gpt-5.2-2025-12-11
2599 Καρπήσιοι 11 9 -2.940 31.7% 61.6% 62.3% 70.0% N/A N/A N/A reviewed/approved 11489 gpt-5.2-2025-12-11

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.088.

Positive termCoefficientDocsMean presentMean absent
δε και 0.0008 8 3.727 -0.324
πολις 0.0008 68 0.527 -1.119
εστι δε και 0.0007 3 9.587 -0.297
εν 0.0006 37 0.954 -0.560
απο 0.0006 27 0.938 -0.347
τη 0.0006 15 1.587 -0.280
οι 0.0005 11 3.217 -0.398
εν τη 0.0005 7 2.829 -0.213
εκαλειτο δε 0.0005 3 4.552 -0.141
πολιτης 0.0005 18 2.268 -0.498
ου 0.0004 14 1.134 -0.185
εστι δε 0.0004 5 3.191 -0.168
τε 0.0004 5 2.168 -0.114
παιδος 0.0004 4 3.495 -0.146
πολις προς 0.0004 3 5.719 -0.177
δευτερα 0.0004 3 6.209 -0.192
εκληθη 0.0004 4 5.494 -0.229
πολιχνιον 0.0004 3 5.366 -0.166
ος 0.0004 6 2.408 -0.154
δυναται 0.0004 3 5.163 -0.160
Negative termCoefficientDocsMean presentMean absent
εθνος -0.0007 8 -5.030 0.437
το -0.0006 70 -0.100 0.232
θρακης -0.0004 5 -2.046 0.108
ης -0.0004 3 -4.843 0.150
τινες -0.0004 6 -2.264 0.145
μεν -0.0004 4 -8.213 0.342
εστι και -0.0004 20 -0.446 0.111
ως το -0.0004 3 -7.302 0.226
οικητωρ -0.0004 6 -3.477 0.222
πολις θρακης -0.0004 4 -1.783 0.074
το θηλυκον -0.0003 6 -4.629 0.295
γαρ -0.0003 9 -1.089 0.108
χωρα -0.0003 6 -3.134 0.200
καππαδοκιας -0.0003 3 -2.640 0.082
τοπικα -0.0003 3 -3.013 0.093
τα τοπικα -0.0003 3 -3.013 0.093
οτι -0.0003 5 -1.553 0.082
δια -0.0003 16 0.717 -0.137
περι -0.0003 11 -1.523 0.188
δια του -0.0003 10 -0.599 0.067

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 1.425; cross-validated R^2: 0.121.

Positive termCoefficientDocsMean presentMean absent
την 2.5423 13 6.588 2.120
δε και 2.5105 8 8.071 2.234
εκαλειτο δε 2.3783 3 13.641 2.362
εκαλειτο 2.2653 7 7.233 2.360
δε 2.2152 46 3.806 1.759
και 2.0227 66 3.308 1.521
απο 1.9787 27 5.087 1.818
εστι δε 1.8372 5 8.800 2.380
εστι δε και 1.7493 3 9.587 2.488
ης 1.5668 3 7.354 2.557
εκαλειτο δε και 1.5531 3 13.641 2.362
τω 1.5160 16 4.423 2.373
τη 1.4971 15 4.737 2.341
της 1.4367 35 4.072 1.962
παρα 1.3310 7 6.480 2.416
αφ 1.3205 6 5.928 2.495
εν τω 1.2814 6 7.034 2.424
βοιωτιας 1.2721 3 9.053 2.504
παρα το 1.2404 3 6.373 2.587
οι 1.1979 11 5.121 2.401
Negative termCoefficientDocsMean presentMean absent
το εθνικον -1.1770 56 2.650 2.766
εθνικον -1.1770 56 2.650 2.766
κωμη -1.1015 3 1.464 2.739
τους -1.0525 4 1.687 2.743
νησος -0.9918 9 1.464 2.823
ειρηται -0.9372 3 1.889 2.726
πλησιον -0.9329 8 2.081 2.755
τον -0.9085 10 2.706 2.700
ειναι -0.8748 8 2.042 2.758
μια -0.8645 3 1.040 2.752
τεταρτω -0.8634 3 1.084 2.751
ποσειδωνος -0.8315 3 2.242 2.715
αυτην φησιν -0.8238 3 0.863 2.757
θεοπομπος -0.8201 5 1.078 2.786
του ποντου -0.7888 3 0.679 2.763
ποντου -0.7888 3 0.679 2.763
πορρω -0.7825 3 0.039 2.783
ου πορρω -0.7825 3 0.039 2.783
και το -0.7247 9 3.573 2.614
και νησος -0.6814 3 1.156 2.748

AI English terms predicting large absolute residuals

Features: 305; ridge alpha: 4.125; cross-validated R^2: 0.012.

Positive termCoefficientDocsMean presentMean absent
also 1.4521 43 3.608 2.016
and 1.2512 55 3.501 1.723
the 1.1233 91 2.825 1.441
of 1.0264 84 2.832 2.009
was 1.0063 18 4.891 2.220
called 0.8009 25 4.233 2.190
was called 0.7977 6 7.040 2.424
are 0.7889 20 4.985 2.130
from 0.7430 37 3.837 2.033
by 0.7239 15 4.782 2.333
whom 0.6820 5 6.360 2.508
with 0.6659 18 4.415 2.324
boiotia 0.6563 3 9.053 2.504
who 0.6012 9 5.825 2.392
is 0.5958 88 2.788 2.056
after 0.5954 15 4.934 2.307
usage 0.5636 4 6.713 2.533
son 0.5508 17 5.577 2.112
son of 0.5508 17 5.577 2.112
this 0.5362 7 6.514 2.414
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.6402 53 2.525 2.898
the ethnonym -0.6185 57 2.495 2.973
as in -0.5929 35 2.271 2.932
ethnonym is -0.5011 54 2.738 2.657
ethnonym -0.4856 58 2.694 2.710
one -0.4335 14 3.199 2.619
island -0.4092 14 1.949 2.823
one of -0.4050 4 1.138 2.766
an island -0.3688 10 1.494 2.835
book of his -0.3536 14 1.701 2.863
been -0.3522 4 1.231 2.762
of his -0.3332 15 1.853 2.850
one of the -0.3317 3 1.040 2.752
an -0.3232 26 2.324 2.833
not far from -0.3171 3 0.039 2.783
not far -0.3171 3 0.039 2.783
far from -0.3171 3 0.039 2.783
far -0.3171 3 0.039 2.783
herodotus book -0.3158 5 1.650 2.756
book -0.3038 39 2.598 2.767

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: 0.042.

Positive termCoefficientDocsMean presentMean absent
the 1.5347 90 2.855 1.313
also 1.2680 43 3.653 1.982
and 1.2224 52 3.583 1.745
of 0.9625 66 3.234 1.665
are 0.9269 25 4.272 2.177
after 0.9097 22 4.886 2.084
this 0.8840 15 6.334 2.059
from the form 0.8800 5 5.922 2.531
used 0.8224 8 8.081 2.233
used to 0.8224 8 8.081 2.233
is with 0.8213 4 8.094 2.476
son 0.8172 17 6.264 1.971
son of 0.8172 17 6.264 1.971
his on 0.7941 3 9.610 2.487
boiotia 0.7891 3 9.053 2.504
usage 0.7775 4 6.458 2.544
their 0.7741 3 13.641 2.362
are also 0.7512 4 6.937 2.524
called 0.7417 21 4.337 2.266
to 0.7359 31 3.698 2.252
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9473 52 2.279 3.157
ethnonym is -0.9473 52 2.279 3.157
as in -0.8443 29 2.244 2.887
an -0.7780 22 1.898 2.927
ethnonym -0.7076 57 2.658 2.757
the ethnonym -0.6770 56 2.683 2.723
and the -0.5855 15 2.788 2.685
one of -0.5831 5 0.903 2.795
it is -0.5798 24 2.615 2.728
one -0.5476 12 2.134 2.778
book -0.5358 39 2.598 2.767
is a -0.4908 31 3.380 2.396
one of the -0.4779 4 1.118 2.767
they -0.4762 12 3.536 2.587
herodotos book -0.4524 5 1.650 2.756
island -0.4519 14 1.949 2.823
in his asia -0.3924 5 1.591 2.759
his asia -0.3924 5 1.591 2.759
an island -0.3742 9 1.551 2.814
not far from -0.3729 3 0.039 2.783

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.