gpt-5.5 v2 Evaluation

Generated: 2026-07-25 12:53:49 UTC

Pairs100
Distinct lemmas100
Slope0.888
R^20.981
BLEU-451.9%
chrF++72.4%
METEOR77.3%
ROUGE-L78.0%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 274

First translation using this version: 2026-06-07

Total usable AI runs for this version: 102 runs across 102 lemmas

Created: 2026-03-06 21:55:13.641826+11:00

Active:

Notes: Reviewed house-style prompt based on 20 human-reviewed translations, 20 AI comparison translations, and reviewer guidance from Brady, Greta, and Gabriel. Activated 2026-03-06.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.888
Intercept2.957
R^20.981
P-value4.12e-86
Pearson r0.990
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.190
Mean absolute length residual3.084
Mean absolute percent error6.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.430, R^2 = 0.185, p = 7.97e-06.
chrF++ vs passage length
chrF++: significant negative; r = -0.450, R^2 = 0.202, p = 2.71e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.444, R^2 = 0.197, p = 3.78e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.443, R^2 = 0.196, p = 3.87e-06.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.290, R^2 = 0.084, p = 0.0035.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.317, R^2 = 0.101, p = 0.0013.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.305, R^2 = 0.093, p = 0.0021.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.314, R^2 = 0.098, p = 0.0015.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.430 0.185 7.97e-06 -0.00289 ok
chrF++ 100 significant negative -0.450 0.202 2.71e-06 -0.00191 ok
METEOR 100 significant negative -0.444 0.197 3.78e-06 -0.00193 ok
ROUGE-L 100 significant negative -0.443 0.196 3.87e-06 -0.00171 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.290 0.084 0.0035 -0.00211 ok
Trigram recall 100 significant negative -0.317 0.101 0.0013 -0.00230 ok
Trigram F1 100 significant negative -0.305 0.093 0.0021 -0.00221 ok
Trigram Jaccard 100 significant negative -0.314 0.098 0.0015 -0.00220 ok

Translation Similarity Metrics

Mean BLEU-451.9%
Mean chrF++72.4%
Mean METEOR77.3%
Mean ROUGE-L78.0%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU48.5%
Unigram F178.5%
Bigram F155.6%
Trigram precision43.3%
Trigram recall41.3%
Trigram F142.3%
Trigram Jaccard26.8%
4-gram F132.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 223 -19.832 33.1% 60.6% 56.7% 64.9% N/A N/A N/A reviewed/approved 3593 gpt-5.5
2604 Καρχηδών 117 125 18.097 35.8% 65.1% 78.3% 72.1% N/A N/A N/A reviewed/approved 3597 gpt-5.5
2468 Καπετώλιον 138 111 -14.560 17.2% 40.7% 43.9% 51.4% N/A N/A N/A reviewed/approved 3601 gpt-5.5
2328 Καλλίπολις 43 55 13.841 36.8% 63.4% 67.9% 63.3% N/A N/A N/A reviewed/approved 3841 gpt-5.5
2603 Κάρυστος 175 171 12.568 41.0% 65.0% 73.0% 68.2% N/A N/A N/A reviewed/approved 3595 gpt-5.5
2609 Κάσπειρος 135 134 11.105 44.0% 66.9% 55.2% 63.9% N/A N/A N/A reviewed/approved 3599 gpt-5.5
2346 Κάναι 77 82 10.634 62.5% 76.2% 89.4% 84.3% N/A N/A N/A reviewed/approved 3861 gpt-5.5
2342 Κάμιρος 54 60 9.068 42.6% 72.0% 69.8% 75.4% N/A N/A N/A reviewed/approved 3857 gpt-5.5
7259 Κύφος 54 60 9.068 49.9% 76.4% 84.2% 78.9% N/A N/A N/A reviewed/approved 4138 gpt-5.5
2116 Κάληρος 40 30 -8.494 17.3% 49.4% 53.6% 54.3% N/A N/A N/A reviewed/approved 3826 gpt-5.5
2455 Κάναστρον 60 48 -8.263 20.3% 51.5% 51.2% 63.0% N/A N/A N/A reviewed/approved 3863 gpt-5.5
2470 Καππαδοκία 68 71 7.630 23.1% 57.9% 82.9% 80.6% N/A N/A N/A reviewed/approved 4063 gpt-5.5
3496 Κοτιάειον 71 59 -7.035 30.3% 51.3% 57.3% 63.1% N/A N/A N/A reviewed/approved 4098 gpt-5.5
2605 Κάσιον 56 59 6.291 51.9% 75.2% 83.1% 78.3% N/A N/A N/A reviewed/approved 4071 gpt-5.5
7255 Κυτέριον 22 28 5.498 43.7% 69.3% 65.0% 68.0% N/A N/A N/A reviewed/approved 4130 gpt-5.5
2119 Κάλλατις 65 66 5.295 37.8% 63.9% 74.3% 68.7% N/A N/A N/A reviewed/approved 3833 gpt-5.5
2083 Καλαμένθη 24 19 -5.279 32.6% 53.7% 51.0% 60.5% N/A N/A N/A reviewed/approved 3810 gpt-5.5
7249 Κύρρος 45 38 -4.936 45.8% 60.1% 67.4% 69.9% N/A N/A N/A reviewed/approved 4119 gpt-5.5
7260 Κυχρεῖος πάγος 61 62 4.849 42.3% 65.4% 75.3% 61.8% N/A N/A N/A reviewed/approved 4140 gpt-5.5
2056 Καβειρία 112 98 -4.461 33.0% 70.8% 61.0% 67.6% N/A N/A N/A reviewed/approved 3605 gpt-5.5
7253 Κυρτώνιος 18 23 4.051 39.8% 69.1% 72.7% 63.4% N/A N/A N/A reviewed/approved 4126 gpt-5.5
7247 Κύρνος 50 44 -3.378 39.6% 65.8% 71.3% 72.3% N/A N/A N/A reviewed/approved 4115 gpt-5.5
2326 Καλλίαρος 47 48 3.287 60.1% 78.7% 88.9% 86.3% N/A N/A N/A reviewed/approved 3837 gpt-5.5
3530 Κριώα 26 23 -3.056 42.4% 70.3% 76.2% 77.6% N/A N/A N/A reviewed/approved 4104 gpt-5.5
2114 Κάλβιος 18 16 -2.949 19.5% 52.4% 64.5% 64.7% N/A N/A N/A reviewed/approved 3821 gpt-5.5
2327 Καλλιόπη 19 17 -2.837 61.3% 77.9% 75.7% 77.8% N/A N/A N/A reviewed/approved 3839 gpt-5.5
2599 Καρπήσιοι 11 10 -2.730 30.6% 67.0% 68.8% 66.7% N/A N/A N/A reviewed/approved 4067 gpt-5.5
2624 Κατάβαθμος 20 18 -2.726 71.6% 82.5% 81.4% 84.2% N/A N/A N/A reviewed/approved 4084 gpt-5.5
7246 Κύρις 20 18 -2.726 77.3% 84.7% 90.4% 89.5% N/A N/A N/A reviewed/approved 4113 gpt-5.5
2079 Καισάρεια 39 35 -2.606 45.3% 67.7% 77.2% 81.1% N/A N/A N/A reviewed/approved 3802 gpt-5.5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.313.

Positive termCoefficientDocsMean presentMean absent
απο 1.7324 27 2.030 -0.751
τε 1.6031 5 4.655 -0.245
οι 1.6004 11 3.833 -0.474
τη 1.4217 15 2.691 -0.475
δε και 1.4215 8 2.022 -0.176
πολιχνιον 1.3634 3 9.923 -0.307
εν τη 1.3366 7 4.843 -0.365
πολις 1.3279 68 0.654 -1.389
παιδος 1.1941 4 5.592 -0.233
δυο 1.1687 4 6.075 -0.253
εν 1.1476 37 1.171 -0.687
ου 1.1403 14 3.040 -0.495
εστι δε και 1.0699 3 6.751 -0.209
εκαλειτο δε 0.9838 3 3.611 -0.112
εστι δε 0.9689 5 3.595 -0.189
εκ 0.9335 6 3.829 -0.244
αφ ου 0.9215 5 4.047 -0.213
αφ 0.9049 6 3.751 -0.239
και πολις 0.8924 5 3.333 -0.175
και 0.8597 66 0.170 -0.330
Negative termCoefficientDocsMean presentMean absent
το -1.4816 70 -0.138 0.323
δια του -1.4086 10 -2.218 0.246
γαρ -1.4013 9 -0.468 0.046
δια -1.3610 16 0.186 -0.035
οτι -1.0916 5 -1.634 0.086
ηρωδιανος -1.0215 3 -12.261 0.379
εθνος -0.9765 8 -3.954 0.344
θρακης -0.8934 5 -3.120 0.164
τινες -0.8919 6 -3.159 0.202
περι -0.7641 11 -2.357 0.291
οικητωρ -0.7447 6 -4.072 0.260
το θηλυκον -0.6501 6 -3.300 0.211
εκαταιος -0.6221 14 -0.873 0.142
το εθνικον -0.6191 56 -0.386 0.491
εθνικον -0.6191 56 -0.386 0.491
του -0.6079 37 0.759 -0.446
ει -0.5789 4 -5.236 0.218
την -0.5639 13 -0.428 0.064
εστι και -0.5613 20 -0.188 0.047
τα -0.5509 12 -0.769 0.105

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.529.

Positive termCoefficientDocsMean presentMean absent
γαρ 1.5923 9 7.653 2.632
εκαλειτο 1.5296 7 9.058 2.634
δια 1.5164 16 6.560 2.422
δε 1.3509 46 4.122 2.200
του 1.3271 37 4.558 2.218
τα 1.2916 12 7.903 2.427
και 1.2628 66 3.666 1.955
εκαλειτο δε 1.2607 3 16.833 2.659
απο 1.2007 27 5.436 2.214
ος 1.1958 6 9.098 2.700
εν 1.1436 37 3.926 2.590
τη 1.1077 15 6.128 2.547
τω 1.0513 16 6.084 2.513
δια του 1.0403 10 7.294 2.616
περι 1.0301 11 6.717 2.635
πολιχνιον 0.9409 3 9.923 2.873
δε και 0.9034 8 7.631 2.689
εκαλειτο δε και 0.8624 3 16.833 2.659
απο της 0.8438 6 9.369 2.683
ζευς 0.8007 3 15.009 2.715
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.8435 56 3.116 3.044
εθνικον -0.8435 56 3.116 3.044
νησος -0.7605 9 1.385 3.252
προς -0.7429 15 2.512 3.185
και το -0.7017 9 3.006 3.092
ως -0.6596 52 3.248 2.906
μια -0.5715 3 0.497 3.164
δευτερω -0.5664 3 0.913 3.151
προς τη -0.5661 4 1.721 3.141
αυτην -0.5015 9 2.577 3.134
πλησιον -0.4942 8 2.353 3.148
τους -0.4841 4 1.575 3.147
καππαδοκιας -0.4801 3 1.844 3.122
του δε -0.4656 3 1.082 3.146
κωμη -0.4600 3 1.960 3.119
το -0.4292 70 2.900 3.513
εκαταιος -0.4063 14 2.060 3.251
πορρω -0.3931 3 0.461 3.165
ου πορρω -0.3931 3 0.461 3.165
βοιωτιας -0.3826 3 1.944 3.119

AI English terms predicting large absolute residuals

Features: 297; ridge alpha: 4.125; cross-validated R^2: -0.515.

Positive termCoefficientDocsMean presentMean absent
with 1.7063 16 6.442 2.445
and 1.6872 53 4.102 1.936
was 1.5289 16 6.703 2.395
by 1.5015 15 7.182 2.361
the 1.4080 91 3.171 2.208
on 1.3768 12 6.095 2.673
was called 1.2082 5 10.695 2.683
on the 1.1692 4 9.508 2.816
there 1.1227 36 4.632 2.213
for 1.0831 13 5.693 2.694
there is 1.0206 27 4.703 2.485
all 0.8156 5 8.072 2.822
son 0.8096 16 6.471 2.439
son of 0.8096 16 6.471 2.439
diphthong 0.8022 3 11.644 2.819
among 0.7917 6 6.874 2.842
called 0.7871 28 4.958 2.355
zeus 0.7752 5 12.777 2.574
after 0.7625 12 7.282 2.512
it with 0.7474 4 10.655 2.769
Negative termCoefficientDocsMean presentMean absent
as in -0.9437 32 2.354 3.428
the ethnonym is -0.7941 52 2.936 3.245
ethnonym is -0.7941 52 2.936 3.245
the ethnonym -0.7290 58 3.074 3.098
ethnonym -0.7290 58 3.074 3.098
island -0.6332 14 1.919 3.274
one -0.5984 10 1.888 3.217
name -0.5958 6 1.113 3.210
a city -0.5845 64 3.168 2.934
an island -0.5709 10 1.352 3.277
near -0.5239 13 2.764 3.132
been -0.5194 6 1.704 3.172
an -0.5006 24 2.695 3.207
is a -0.4818 27 2.968 3.127
one of the -0.4162 3 0.497 3.164
one of -0.4162 3 0.497 3.164
has been -0.3862 4 1.817 3.137
those who -0.3553 3 0.619 3.160
cappadocia -0.3536 3 1.844 3.122
if -0.3249 4 1.147 3.165

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 5.878; cross-validated R^2: -0.553.

Positive termCoefficientDocsMean presentMean absent
with 1.4900 16 6.289 2.474
the 1.3859 90 3.187 2.160
on 1.3221 16 6.715 2.392
and 1.1107 52 4.058 2.029
is with 0.8557 4 13.859 2.635
on the 0.7664 10 6.924 2.657
to 0.7005 31 4.394 2.496
used to be 0.6505 6 10.299 2.624
this 0.6030 15 6.775 2.433
be 0.5909 15 7.141 2.368
son 0.5800 17 6.352 2.415
son of 0.5800 17 6.352 2.415
to be 0.5640 9 7.912 2.607
used 0.5443 8 8.483 2.615
used to 0.5443 8 8.483 2.615
karia 0.5364 3 11.652 2.819
zeus 0.5358 5 12.777 2.574
his on 0.5313 3 11.755 2.816
who 0.5266 13 5.437 2.732
their 0.5088 3 16.833 2.659
Negative termCoefficientDocsMean presentMean absent
as in -0.9846 29 2.136 3.471
the ethnonym is -0.6853 52 2.893 3.291
ethnonym is -0.6853 52 2.893 3.291
an -0.6296 22 2.370 3.286
ethnonym -0.5995 57 3.088 3.079
the ethnonym -0.5932 56 3.104 3.059
a city -0.4017 64 3.168 2.934
island -0.3993 14 1.919 3.274
one of -0.3754 5 0.868 3.201
an island -0.3562 9 1.442 3.246
they -0.3417 12 3.001 3.095
city -0.3269 70 3.245 2.709
one of the -0.3233 4 0.962 3.172
is a -0.3232 31 3.476 2.908
was from -0.3143 4 1.046 3.169
of his -0.2819 18 2.995 3.104
near -0.2814 16 3.259 3.051
and the -0.2775 15 3.541 3.003
in book of -0.2735 14 2.128 3.240
in book -0.2582 16 3.147 3.072

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.