gpt-5.2 v2 Evaluation

Generated: 2026-09-20 11:21:30 UTC

Pairs100
Distinct lemmas100
Slope0.921
R^20.988
BLEU-448.2%
chrF++70.5%
METEOR75.7%
ROUGE-L76.1%
BERTScore96.0%
COMET79.0%
BLEURT74.3%

Profile version id: 2332

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 12:28:27.643960+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.2 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.921
Intercept1.820
R^20.988
P-value1.12e-95
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.500
Mean absolute length residual2.694
Mean absolute percent error6.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.401, R^2 = 0.161, p = 3.53e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.470, R^2 = 0.221, p = 8.23e-07.
METEOR vs passage length
METEOR: significant negative; r = -0.479, R^2 = 0.229, p = 4.72e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.442, R^2 = 0.195, p = 4.16e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.609, R^2 = 0.371, p = 1.83e-11.
COMET vs passage length
COMET: significant negative; r = -0.491, R^2 = 0.241, p = 2.16e-07.
BLEURT vs passage length
BLEURT: significant negative; r = -0.737, R^2 = 0.544, p = 2.22e-18.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.309, R^2 = 0.096, p = 0.0018.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.330, R^2 = 0.109, p = 0.0008.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.320, R^2 = 0.103, p = 0.0012.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.327, R^2 = 0.107, p = 0.0009.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.401 0.161 3.53e-05 -0.00274 ok
chrF++ 100 significant negative -0.470 0.221 8.23e-07 -0.00200 ok
METEOR 100 significant negative -0.479 0.229 4.72e-07 -0.00222 ok
ROUGE-L 100 significant negative -0.442 0.195 4.16e-06 -0.00174 ok
BERTScore 100 significant negative -0.609 0.371 1.83e-11 -0.00049 ok
COMET 100 significant negative -0.491 0.241 2.16e-07 -0.00103 ok
BLEURT 100 significant negative -0.737 0.544 2.22e-18 -0.00291 ok
Trigram precision 100 significant negative -0.309 0.096 0.0018 -0.00232 ok
Trigram recall 100 significant negative -0.330 0.109 0.0008 -0.00246 ok
Trigram F1 100 significant negative -0.320 0.103 0.0012 -0.00239 ok
Trigram Jaccard 100 significant negative -0.327 0.107 0.0009 -0.00236 ok

Translation Similarity Metrics

Mean BLEU-448.2%
Mean chrF++70.5%
Mean METEOR75.7%
Mean ROUGE-L76.1%
Mean BERTScore96.0%
Mean COMET79.0%
Mean BLEURT74.3%
Legacy smoothed corpus BLEU45.9%
Unigram F177.4%
Bigram F152.7%
Trigram precision40.0%
Trigram recall38.5%
Trigram F139.2%
Trigram Jaccard24.4%
4-gram F129.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 126 16.375 30.2% 62.4% 72.8% 66.9% 94.0% 78.0% 62.4% reviewed/approved 11498 gpt-5.2-2025-12-11
2056 Καβειρία 112 91 -14.018 33.2% 60.7% 60.3% 68.0% 94.3% 73.2% 55.5% reviewed/approved 11343 gpt-5.2-2025-12-11
2484 Καρία 270 237 -13.601 32.8% 58.5% 52.5% 63.5% 90.6% 76.3% 48.7% reviewed/approved 11483 gpt-5.2-2025-12-11
2603 Κάρυστος 175 174 10.933 34.7% 61.9% 61.8% 65.9% 94.6% 76.1% 58.4% reviewed/approved 11495 gpt-5.2-2025-12-11
3288 Κορώνεια 110 114 10.825 33.3% 63.0% 67.1% 66.7% 94.2% 74.5% 54.2% reviewed/approved 11546 gpt-5.2-2025-12-11
2328 Καλλίπολις 43 50 8.560 40.8% 63.1% 68.6% 66.7% 94.1% 76.1% 67.8% reviewed/approved 11435 gpt-5.2-2025-12-11
2342 Κάμιρος 54 59 7.424 41.5% 68.3% 76.8% 70.8% 95.2% 79.0% 67.3% reviewed/approved 11459 gpt-5.2-2025-12-11
7251 Κύρτος 66 56 -6.633 22.7% 52.3% 60.7% 66.7% 92.2% 69.5% 54.8% reviewed/approved 11583 gpt-5.2-2025-12-11
2346 Κάναι 77 79 6.232 49.6% 66.8% 74.0% 79.5% 94.5% 76.8% 61.9% reviewed/approved 11465 gpt-5.2-2025-12-11
7255 Κυτέριον 22 28 5.909 29.0% 63.6% 65.0% 56.0% 95.2% 76.2% 73.5% reviewed/approved 11595 gpt-5.2-2025-12-11
2609 Κάσπειρος 135 132 5.790 53.0% 70.5% 70.9% 71.2% 91.4% 72.6% 65.3% reviewed/approved 11507 gpt-5.2-2025-12-11
7254 Κύτα 84 74 -5.218 43.9% 64.2% 63.1% 73.4% 94.1% 74.0% 60.3% reviewed/approved 11592 gpt-5.2-2025-12-11
7264 Κωλιάς 61 53 -5.026 33.9% 59.0% 56.1% 66.1% 94.5% 72.0% 58.2% reviewed/approved 11622 gpt-5.2-2025-12-11
2332 Καλυδών 20 25 4.752 26.9% 67.9% 68.3% 71.1% 96.1% 84.1% 76.6% reviewed/approved 11447 gpt-5.2-2025-12-11
2116 Κάληρος 40 34 -4.676 32.7% 63.0% 71.0% 67.6% 94.5% 77.9% 73.8% reviewed/approved 11414 gpt-5.2-2025-12-11
2605 Κάσιον 56 58 4.581 51.5% 72.7% 70.4% 73.7% 95.6% 75.4% 70.0% reviewed/approved 11501 gpt-5.2-2025-12-11
7249 Κύρρος 45 39 -4.283 30.5% 55.3% 61.8% 69.0% 94.0% 69.8% 58.9% reviewed/approved 11578 gpt-5.2-2025-12-11
7260 Κυχρεῖος πάγος 61 62 3.974 44.8% 70.3% 73.6% 66.7% 96.3% 78.5% 64.4% reviewed/approved 11610 gpt-5.2-2025-12-11
2083 Καλαμένθη 24 20 -3.934 12.8% 46.4% 48.5% 54.5% 92.2% 75.1% 68.3% reviewed/approved 11393 gpt-5.2-2025-12-11
7247 Κύρνος 50 44 -3.890 34.6% 62.4% 68.3% 72.3% 96.4% 76.1% 79.2% reviewed/approved 11572 gpt-5.2-2025-12-11
2326 Καλλίαρος 47 49 3.874 53.5% 73.3% 86.5% 83.3% 97.1% 74.5% 74.1% reviewed/approved 11429 gpt-5.2-2025-12-11
2079 Καισάρεια 39 34 -3.755 47.9% 70.2% 79.8% 84.9% 95.4% 78.9% 75.2% reviewed/approved 11381 gpt-5.2-2025-12-11
2059 Καδμεία 20 24 3.752 36.3% 73.2% 82.7% 81.8% 96.6% 84.2% 80.9% reviewed/approved 11352 gpt-5.2-2025-12-11
7253 Κυρτώνιος 18 22 3.595 41.4% 71.4% 77.4% 70.0% 97.1% 82.7% 81.0% reviewed/approved 11589 gpt-5.2-2025-12-11
2470 Καππαδοκία 68 68 3.524 16.3% 53.5% 82.3% 76.5% 94.8% 75.4% 67.2% reviewed/approved 11477 gpt-5.2-2025-12-11
2345 Κάναθα 17 21 3.516 58.3% 80.9% 89.5% 78.9% 96.8% 86.3% 67.0% reviewed/approved 11462 gpt-5.2-2025-12-11
2054 Καβαλίς 65 65 3.289 55.0% 70.7% 73.1% 75.4% 95.6% 76.2% 69.9% reviewed/approved 11337 gpt-5.2-2025-12-11
3496 Κοτιάειον 71 64 -3.240 28.9% 48.1% 49.9% 54.8% 91.9% 70.6% 59.4% reviewed/approved 11549 gpt-5.2-2025-12-11
2455 Κάναστρον 60 54 -3.104 19.6% 52.5% 54.1% 63.2% 93.8% 68.8% 55.0% reviewed/approved 11468 gpt-5.2-2025-12-11
2330 Καλύβη 24 21 -2.934 46.8% 65.9% 64.1% 80.0% 96.8% 80.4% 79.5% reviewed/approved 11441 gpt-5.2-2025-12-11

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.091.

Positive termCoefficientDocsMean presentMean absent
δε και 0.0008 8 3.729 -0.324
πολις 0.0007 68 0.518 -1.102
εστι δε και 0.0007 3 9.587 -0.296
εν 0.0006 37 0.948 -0.557
απο 0.0006 27 0.936 -0.346
τη 0.0006 15 1.583 -0.279
οι 0.0005 11 3.213 -0.397
εκαλειτο δε 0.0005 3 4.569 -0.141
εν τη 0.0005 7 2.821 -0.212
πολιτης 0.0004 18 2.261 -0.496
ου 0.0004 14 1.131 -0.184
εστι δε 0.0004 5 3.190 -0.168
τε 0.0004 5 2.164 -0.114
παιδος 0.0004 4 3.486 -0.145
πολις προς 0.0004 3 5.715 -0.177
δευτερα 0.0004 3 6.203 -0.192
εκληθη 0.0004 4 5.492 -0.229
πολιχνιον 0.0004 3 5.360 -0.166
ος 0.0004 6 2.407 -0.154
δυναται 0.0004 3 5.160 -0.160
Negative termCoefficientDocsMean presentMean absent
το -0.0006 70 -0.107 0.251
εθνος -0.0006 8 -4.919 0.428
θρακης -0.0004 5 -2.058 0.108
ης -0.0004 3 -4.850 0.150
τινες -0.0004 6 -2.274 0.145
μεν -0.0004 4 -8.207 0.342
εστι και -0.0004 20 -0.449 0.112
ως το -0.0004 3 -7.301 0.226
οικητωρ -0.0004 6 -3.482 0.222
πολις θρακης -0.0004 4 -1.796 0.075
το θηλυκον -0.0003 6 -4.627 0.295
χωρα -0.0003 6 -3.137 0.200
γαρ -0.0003 9 -1.083 0.107
καππαδοκιας -0.0003 3 -2.647 0.082
τοπικα -0.0003 3 -3.019 0.093
τα τοπικα -0.0003 3 -3.019 0.093
οτι -0.0003 5 -1.557 0.082
δια -0.0003 16 0.716 -0.136
περι -0.0003 11 -1.526 0.189
δια του -0.0003 10 -0.600 0.067

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 1.425; cross-validated R^2: 0.126.

Positive termCoefficientDocsMean presentMean absent
την 2.5433 13 6.587 2.112
δε και 2.5105 8 8.067 2.227
εκαλειτο δε 2.3796 3 13.636 2.355
εκαλειτο 2.2781 7 7.234 2.352
δε 2.2249 46 3.806 1.747
και 2.0464 66 3.308 1.501
απο 1.9764 27 5.086 1.809
εστι δε 1.8417 5 8.798 2.373
εστι δε και 1.7524 3 9.587 2.481
ης 1.5689 3 7.351 2.550
εκαλειτο δε και 1.5535 3 13.636 2.355
τω 1.5187 16 4.420 2.365
τη 1.5023 15 4.736 2.333
της 1.4525 35 4.074 1.951
παρα 1.3313 7 6.471 2.410
αφ 1.3202 6 5.926 2.488
εν τω 1.2773 6 7.025 2.417
βοιωτιας 1.2703 3 9.048 2.497
παρα το 1.2415 3 6.367 2.580
οι 1.2034 11 5.120 2.394
Negative termCoefficientDocsMean presentMean absent
το εθνικον -1.1592 56 2.651 2.749
εθνικον -1.1592 56 2.651 2.749
κωμη -1.0986 3 1.460 2.732
τους -1.0503 4 1.682 2.736
νησος -0.9865 9 1.466 2.815
ειρηται -0.9389 3 1.884 2.719
πλησιον -0.9162 8 2.080 2.747
τον -0.9025 10 2.707 2.692
ειναι -0.8736 8 2.037 2.751
μια -0.8542 3 1.034 2.745
ποσειδωνος -0.8253 3 2.244 2.708
αυτην φησιν -0.8230 3 0.858 2.751
τεταρτω -0.8146 3 1.093 2.743
θεοπομπος -0.8112 5 1.087 2.778
του ποντου -0.7747 3 0.680 2.756
ποντου -0.7747 3 0.680 2.756
πορρω -0.7647 3 0.045 2.776
ου πορρω -0.7647 3 0.045 2.776
και το -0.7238 9 3.574 2.607
και νησος -0.6845 3 1.161 2.741

AI English terms predicting large absolute residuals

Features: 305; ridge alpha: 4.125; cross-validated R^2: 0.017.

Positive termCoefficientDocsMean presentMean absent
also 1.4592 43 3.606 2.006
and 1.2637 55 3.501 1.707
the 1.1202 91 2.817 1.452
of 1.0398 84 2.834 1.957
was 1.0098 18 4.890 2.212
called 0.8040 25 4.232 2.181
was called 0.7996 6 7.037 2.417
are 0.7934 20 4.983 2.122
from 0.7500 37 3.837 2.023
by 0.7303 15 4.787 2.324
whom 0.6827 5 6.361 2.501
with 0.6667 18 4.414 2.316
boiotia 0.6558 3 9.048 2.497
is 0.6074 88 2.790 1.990
who 0.6034 9 5.828 2.384
after 0.5954 15 4.929 2.299
usage 0.5609 4 6.697 2.527
son 0.5514 17 5.574 2.104
son of 0.5514 17 5.574 2.104
this 0.5407 7 6.524 2.406
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.6316 53 2.527 2.882
the ethnonym -0.6107 57 2.497 2.955
as in -0.5899 35 2.270 2.922
ethnonym is -0.4926 54 2.740 2.640
ethnonym -0.4779 58 2.695 2.692
one -0.4307 14 3.199 2.612
one of -0.4047 4 1.131 2.759
island -0.4028 14 1.952 2.815
an -0.3657 26 2.295 2.834
an island -0.3633 10 1.497 2.827
been -0.3503 4 1.232 2.755
book of his -0.3458 14 1.705 2.855
one of the -0.3320 3 1.034 2.745
of his -0.3256 15 1.856 2.842
not far from -0.3144 3 0.045 2.776
not far -0.3144 3 0.045 2.776
far from -0.3144 3 0.045 2.776
far -0.3144 3 0.045 2.776
herodotus book -0.3122 5 1.657 2.748
came -0.2978 4 1.255 2.754

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: 0.047.

Positive termCoefficientDocsMean presentMean absent
the 1.5385 90 2.846 1.324
also 1.2780 43 3.653 1.970
and 1.2383 52 3.582 1.731
of 0.9750 66 3.235 1.643
are 0.9298 25 4.270 2.168
after 0.9131 22 4.884 2.076
this 0.8913 15 6.335 2.051
from the form 0.8803 5 5.922 2.524
used 0.8257 8 8.082 2.225
used to 0.8257 8 8.082 2.225
is with 0.8249 4 8.089 2.469
son 0.8198 17 6.261 1.963
son of 0.8198 17 6.261 1.963
his on 0.7947 3 9.598 2.480
boiotia 0.7880 3 9.048 2.497
usage 0.7781 4 6.445 2.538
their 0.7746 3 13.636 2.355
are also 0.7553 4 6.934 2.517
called 0.7468 21 4.336 2.257
to 0.7420 31 3.699 2.242
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9354 52 2.281 3.141
ethnonym is -0.9354 52 2.281 3.141
as in -0.8466 29 2.242 2.878
an -0.8298 22 1.861 2.929
ethnonym -0.6976 57 2.659 2.740
the ethnonym -0.6663 56 2.684 2.706
one of -0.5846 5 0.897 2.788
and the -0.5836 15 2.788 2.677
it is -0.5809 24 2.613 2.719
one -0.5438 12 2.131 2.771
book -0.5203 39 2.601 2.753
is a -0.4883 31 3.380 2.386
one of the -0.4785 4 1.114 2.760
they -0.4714 12 3.535 2.579
herodotos book -0.4477 5 1.657 2.748
island -0.4418 14 1.952 2.815
in his asia -0.3803 5 1.602 2.751
his asia -0.3803 5 1.602 2.751
not far from -0.3692 3 0.045 2.776
not far -0.3692 3 0.045 2.776

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.