gpt-5.5 v2 Evaluation

Generated: 2026-09-20 11:22:07 UTC

Pairs100
Distinct lemmas100
Slope0.888
R^20.981
BLEU-452.0%
chrF++72.5%
METEOR77.4%
ROUGE-L78.1%
BERTScore96.4%
COMET80.0%
BLEURT76.1%

Profile version id: 274

First translation using this version: 2026-06-07

Total usable AI runs for this version: 102 runs across 102 lemmas

Created: 2026-03-06 21:55:13.641826+11:00

Active:

Notes: Reviewed house-style prompt based on 20 human-reviewed translations, 20 AI comparison translations, and reviewer guidance from Brady, Greta, and Gabriel. Activated 2026-03-06.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.888
Intercept2.974
R^20.981
P-value3.78e-86
Pearson r0.990
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.190
Mean absolute length residual3.079
Mean absolute percent error6.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.438, R^2 = 0.192, p = 5.28e-06.
chrF++ vs passage length
chrF++: significant negative; r = -0.456, R^2 = 0.208, p = 1.84e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.457, R^2 = 0.209, p = 1.71e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.456, R^2 = 0.208, p = 1.91e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.633, R^2 = 0.400, p = 1.64e-12.
COMET vs passage length
COMET: significant negative; r = -0.473, R^2 = 0.224, p = 6.81e-07.
BLEURT vs passage length
BLEURT: significant negative; r = -0.717, R^2 = 0.514, p = 4.68e-17.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.297, R^2 = 0.088, p = 0.0027.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.326, R^2 = 0.106, p = 0.0009.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.313, R^2 = 0.098, p = 0.0015.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.319, R^2 = 0.102, p = 0.0012.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.438 0.192 5.28e-06 -0.00293 ok
chrF++ 100 significant negative -0.456 0.208 1.84e-06 -0.00193 ok
METEOR 100 significant negative -0.457 0.209 1.71e-06 -0.00199 ok
ROUGE-L 100 significant negative -0.456 0.208 1.91e-06 -0.00175 ok
BERTScore 100 significant negative -0.633 0.400 1.64e-12 -0.00048 ok
COMET 100 significant negative -0.473 0.224 6.81e-07 -0.00093 ok
BLEURT 100 significant negative -0.717 0.514 4.68e-17 -0.00262 ok
Trigram precision 100 significant negative -0.297 0.088 0.0027 -0.00215 ok
Trigram recall 100 significant negative -0.326 0.106 0.0009 -0.00235 ok
Trigram F1 100 significant negative -0.313 0.098 0.0015 -0.00225 ok
Trigram Jaccard 100 significant negative -0.319 0.102 0.0012 -0.00223 ok

Translation Similarity Metrics

Mean BLEU-452.0%
Mean chrF++72.5%
Mean METEOR77.4%
Mean ROUGE-L78.1%
Mean BERTScore96.4%
Mean COMET80.0%
Mean BLEURT76.1%
Legacy smoothed corpus BLEU48.6%
Unigram F178.5%
Bigram F155.7%
Trigram precision43.3%
Trigram recall41.3%
Trigram F142.3%
Trigram Jaccard26.8%
4-gram F132.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 223 -19.801 33.1% 60.6% 56.7% 64.9% 91.2% 80.1% 55.0% reviewed/approved 3593 gpt-5.5
2604 Καρχηδών 117 125 18.101 35.8% 65.1% 78.3% 72.1% 94.2% 78.5% 63.2% reviewed/approved 3597 gpt-5.5
2468 Καπετώλιον 138 111 -14.552 17.2% 40.7% 43.9% 51.4% 87.7% 69.8% 50.5% reviewed/approved 3601 gpt-5.5
2328 Καλλίπολις 43 55 13.831 36.8% 63.4% 67.9% 63.3% 94.1% 75.2% 68.6% reviewed/approved 3841 gpt-5.5
2603 Κάρυστος 175 171 12.583 41.0% 65.0% 73.0% 68.2% 94.8% 78.8% 61.7% reviewed/approved 3595 gpt-5.5
2609 Κάσπειρος 135 134 11.113 44.0% 66.9% 55.2% 63.9% 89.9% 72.1% 62.9% reviewed/approved 3599 gpt-5.5
2346 Κάναι 77 82 10.631 62.5% 76.2% 89.4% 84.3% 95.6% 78.7% 71.7% reviewed/approved 3861 gpt-5.5
2342 Κάμιρος 54 60 9.061 42.6% 72.0% 69.8% 75.4% 96.0% 79.8% 68.3% reviewed/approved 3857 gpt-5.5
7259 Κύφος 54 60 9.061 49.9% 76.4% 84.2% 78.9% 96.3% 75.5% 69.9% reviewed/approved 4138 gpt-5.5
2116 Κάληρος 40 30 -8.504 17.3% 49.4% 53.6% 54.3% 92.9% 67.5% 62.4% reviewed/approved 3826 gpt-5.5
2455 Κάναστρον 60 48 -8.269 20.3% 51.5% 51.2% 63.0% 93.6% 68.1% 54.7% reviewed/approved 3863 gpt-5.5
2470 Καππαδοκία 68 71 7.625 23.1% 57.9% 82.9% 80.6% 95.2% 76.0% 71.5% reviewed/approved 4063 gpt-5.5
3496 Κοτιάειον 71 59 -7.039 30.3% 51.3% 57.3% 63.1% 92.7% 71.1% 67.0% reviewed/approved 4098 gpt-5.5
2605 Κάσιον 56 59 6.284 51.9% 75.2% 83.1% 78.3% 97.0% 79.1% 76.3% reviewed/approved 4071 gpt-5.5
7255 Κυτέριον 22 28 5.485 43.7% 69.3% 65.0% 68.0% 95.3% 77.1% 77.3% reviewed/approved 4130 gpt-5.5
2083 Καλαμένθη 24 19 -5.292 32.6% 53.7% 51.0% 60.5% 93.6% 77.2% 74.6% reviewed/approved 3810 gpt-5.5
2119 Κάλλατις 65 66 5.290 37.8% 63.9% 74.3% 68.7% 95.2% 72.1% 69.9% reviewed/approved 3833 gpt-5.5
7249 Κύρρος 45 38 -4.945 45.8% 60.1% 67.4% 69.9% 93.8% 72.6% 63.7% reviewed/approved 4119 gpt-5.5
7260 Κυχρεῖος πάγος 61 62 4.843 42.3% 65.4% 75.3% 61.8% 95.8% 75.7% 61.3% reviewed/approved 4140 gpt-5.5
2056 Καβειρία 112 98 -4.458 33.0% 70.8% 61.0% 67.6% 95.9% 78.0% 64.6% reviewed/approved 3605 gpt-5.5
7253 Κυρτώνιος 18 23 4.038 39.8% 69.1% 72.7% 63.4% 96.6% 77.5% 80.2% reviewed/approved 4126 gpt-5.5
7247 Κύρνος 50 44 -3.386 39.6% 65.8% 71.3% 72.3% 95.9% 78.5% 80.2% reviewed/approved 4115 gpt-5.5
2326 Καλλίαρος 47 48 3.278 60.1% 78.7% 88.9% 86.3% 97.9% 78.1% 77.9% reviewed/approved 3837 gpt-5.5
3530 Κριώα 26 23 -3.068 42.4% 70.3% 76.2% 77.6% 97.1% 78.0% 76.2% reviewed/approved 4104 gpt-5.5
2114 Κάλβιος 18 16 -2.962 19.5% 52.4% 64.5% 64.7% 94.5% 74.6% 67.0% reviewed/approved 3821 gpt-5.5
2327 Καλλιόπη 19 17 -2.851 61.3% 77.9% 75.7% 77.8% 95.3% 76.9% 77.1% reviewed/approved 3839 gpt-5.5
2624 Κατάβαθμος 20 18 -2.739 71.6% 82.5% 81.4% 84.2% 96.8% 82.9% 82.4% reviewed/approved 4084 gpt-5.5
7246 Κύρις 20 18 -2.739 77.3% 84.7% 90.4% 89.5% 97.9% 84.4% 81.4% reviewed/approved 4113 gpt-5.5
2079 Καισάρεια 39 35 -2.616 45.3% 67.7% 77.2% 81.1% 95.5% 78.9% 74.7% reviewed/approved 3802 gpt-5.5
7262 Κῶβρυς 22 20 -2.515 64.9% 76.7% 80.8% 81.0% 97.2% 82.2% 80.9% reviewed/approved 4144 gpt-5.5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.318.

Positive termCoefficientDocsMean presentMean absent
απο 1.7312 27 2.027 -0.750
τε 1.6029 5 4.651 -0.245
οι 1.5968 11 3.828 -0.473
δε και 1.4207 8 2.023 -0.176
τη 1.4189 15 2.687 -0.474
πολιχνιον 1.3621 3 9.918 -0.307
εν τη 1.3351 7 4.835 -0.364
πολις 1.3137 68 0.646 -1.372
παιδος 1.1931 4 5.583 -0.233
δυο 1.1668 4 6.071 -0.253
ου 1.1368 14 3.037 -0.494
εν 1.1358 37 1.165 -0.684
εστι δε και 1.0693 3 6.750 -0.209
εκαλειτο δε 0.9850 3 3.628 -0.112
εστι δε 0.9678 5 3.594 -0.189
εκ 0.9316 6 3.822 -0.244
αφ ου 0.9223 5 4.048 -0.213
αφ 0.9047 6 3.750 -0.239
και πολις 0.8892 5 3.326 -0.175
τον 0.8479 10 2.715 -0.302
Negative termCoefficientDocsMean presentMean absent
το -1.4934 70 -0.146 0.341
δια του -1.4091 10 -2.219 0.247
γαρ -1.3985 9 -0.462 0.046
δια -1.3617 16 0.185 -0.035
οτι -1.0938 5 -1.637 0.086
ηρωδιανος -1.0213 3 -12.252 0.379
θρακης -0.8996 5 -3.131 0.165
εθνος -0.8971 8 -3.848 0.335
τινες -0.8942 6 -3.169 0.202
περι -0.7661 11 -2.360 0.292
οικητωρ -0.7482 6 -4.076 0.260
το θηλυκον -0.6507 6 -3.299 0.211
το εθνικον -0.6318 56 -0.394 0.501
εθνικον -0.6318 56 -0.394 0.501
εκαταιος -0.6311 14 -0.881 0.143
του -0.6163 37 0.754 -0.443
ει -0.5782 4 -5.231 0.218
εστι και -0.5668 20 -0.191 0.048
την -0.5622 13 -0.425 0.064
τα -0.5511 12 -0.767 0.105

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.516.

Positive termCoefficientDocsMean presentMean absent
γαρ 1.5919 9 7.651 2.626
εκαλειτο 1.5349 7 9.060 2.628
δια 1.5184 16 6.560 2.416
δε 1.3547 46 4.121 2.191
του 1.3332 37 4.558 2.210
τα 1.2952 12 7.904 2.421
και 1.2781 66 3.667 1.936
εκαλειτο δε 1.2612 3 16.828 2.653
απο 1.2003 27 5.433 2.208
ος 1.1971 6 9.096 2.695
εν 1.1559 37 3.926 2.581
τη 1.1095 15 6.125 2.541
τω 1.0560 16 6.084 2.506
δια του 1.0410 10 7.291 2.611
περι 1.0331 11 6.717 2.629
πολιχνιον 0.9412 3 9.918 2.867
δε και 0.9039 8 7.627 2.683
εκαλειτο δε και 0.8626 3 16.828 2.653
απο της 0.8438 6 9.365 2.677
ζευς 0.8006 3 14.995 2.710
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.8320 56 3.118 3.029
εθνικον -0.8320 56 3.118 3.029
νησος -0.7560 9 1.389 3.246
προς -0.7353 15 2.516 3.178
και το -0.6999 9 3.009 3.086
ως -0.6486 52 3.250 2.893
μια -0.5674 3 0.499 3.158
προς τη -0.5615 4 1.727 3.135
δευτερω -0.5575 3 0.927 3.145
αυτην -0.4979 9 2.581 3.128
πλησιον -0.4862 8 2.359 3.141
τους -0.4819 4 1.577 3.141
καππαδοκιας -0.4795 3 1.844 3.117
του δε -0.4660 3 1.082 3.140
κωμη -0.4597 3 1.955 3.113
το -0.4166 70 2.902 3.491
εκαταιος -0.3953 14 2.067 3.243
πορρω -0.3866 3 0.472 3.159
ου πορρω -0.3866 3 0.472 3.159
βοιωτιας -0.3833 3 1.940 3.114

AI English terms predicting large absolute residuals

Features: 297; ridge alpha: 4.125; cross-validated R^2: -0.505.

Positive termCoefficientDocsMean presentMean absent
with 1.7064 16 6.440 2.438
and 1.6988 53 4.103 1.924
was 1.5320 16 6.703 2.388
by 1.5100 15 7.188 2.354
the 1.4066 91 3.163 2.222
on 1.3772 12 6.093 2.668
was called 1.2085 5 10.691 2.678
on the 1.1696 4 9.505 2.811
there 1.1279 36 4.633 2.204
for 1.0838 13 5.690 2.688
there is 1.0230 27 4.703 2.478
all 0.8162 5 8.071 2.816
son 0.8078 16 6.467 2.433
son of 0.8078 16 6.467 2.433
diphthong 0.8011 3 11.632 2.814
called 0.7905 28 4.957 2.348
zeus 0.7748 5 12.770 2.569
after 0.7634 12 7.280 2.506
it with 0.7490 4 10.652 2.763
among 0.7131 6 6.724 2.846
Negative termCoefficientDocsMean presentMean absent
as in -0.9408 32 2.354 3.420
the ethnonym is -0.7849 52 2.939 3.230
ethnonym is -0.7849 52 2.939 3.230
the ethnonym -0.7209 58 3.076 3.082
ethnonym -0.7209 58 3.076 3.082
island -0.6267 14 1.923 3.267
one -0.5964 10 1.889 3.211
name -0.5960 6 1.114 3.204
a city -0.5784 64 3.170 2.916
an island -0.5649 10 1.357 3.270
an -0.5393 24 2.665 3.209
near -0.5197 13 2.766 3.125
been -0.5158 6 1.708 3.166
is a -0.4757 27 2.971 3.119
one of the -0.4156 3 0.499 3.158
one of -0.4156 3 0.499 3.158
has been -0.3836 4 1.822 3.131
those -0.3758 6 1.884 3.155
people -0.3624 11 2.217 3.185
those who -0.3532 3 0.618 3.155

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 5.878; cross-validated R^2: -0.543.

Positive termCoefficientDocsMean presentMean absent
with 1.4928 16 6.289 2.467
the 1.3804 90 3.179 2.174
on 1.3244 16 6.714 2.386
and 1.1230 52 4.059 2.017
is with 0.8584 4 13.854 2.630
on the 0.7675 10 6.925 2.651
to 0.7031 31 4.393 2.488
used to be 0.6524 6 10.300 2.618
this 0.6066 15 6.777 2.426
be 0.5922 15 7.140 2.362
son 0.5821 17 6.349 2.409
son of 0.5821 17 6.349 2.409
to be 0.5654 9 7.913 2.601
used 0.5466 8 8.485 2.609
used to 0.5466 8 8.485 2.609
karia 0.5381 3 11.643 2.814
zeus 0.5364 5 12.770 2.569
his on 0.5322 3 11.744 2.811
who 0.5291 13 5.440 2.726
their 0.5094 3 16.828 2.653
Negative termCoefficientDocsMean presentMean absent
as in -0.9839 29 2.135 3.464
the ethnonym is -0.6767 52 2.896 3.277
ethnonym is -0.6767 52 2.896 3.277
an -0.6584 22 2.335 3.288
ethnonym -0.5917 57 3.090 3.063
the ethnonym -0.5853 56 3.106 3.044
a city -0.3973 64 3.170 2.916
island -0.3934 14 1.923 3.267
one of -0.3750 5 0.871 3.195
an island -0.3521 9 1.448 3.240
they -0.3379 12 3.004 3.089
one of the -0.3229 4 0.964 3.167
city -0.3215 70 3.246 2.687
is a -0.3186 31 3.478 2.899
was from -0.3139 4 1.048 3.163
of his -0.2762 18 2.998 3.096
and the -0.2754 15 3.542 2.997
near -0.2741 16 3.264 3.043
in book of -0.2684 14 2.133 3.233
in book -0.2530 16 3.150 3.065

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.