gpt-5.4 v2 Evaluation

Generated: 2026-09-20 11:21:55 UTC

Pairs100
Distinct lemmas100
Slope0.939
R^20.984
BLEU-446.8%
chrF++69.8%
METEOR76.7%
ROUGE-L77.4%
BERTScore96.1%
COMET79.1%
BLEURT73.7%

Profile version id: 2173

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.4 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.939
Intercept1.726
R^20.984
P-value9.63e-90
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words43.200
Mean absolute length residual3.016
Mean absolute percent error6.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.394, R^2 = 0.155, p = 5.08e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.421, R^2 = 0.177, p = 1.29e-05.
METEOR vs passage length
METEOR: significant negative; r = -0.475, R^2 = 0.226, p = 5.89e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.490, R^2 = 0.240, p = 2.29e-07.
BERTScore vs passage length
BERTScore: significant negative; r = -0.631, R^2 = 0.398, p = 1.93e-12.
COMET vs passage length
COMET: significant negative; r = -0.459, R^2 = 0.210, p = 1.60e-06.
BLEURT vs passage length
BLEURT: significant negative; r = -0.756, R^2 = 0.572, p = 9.18e-20.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.381, R^2 = 0.145, p = 9.19e-05.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.382, R^2 = 0.146, p = 8.98e-05.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.382, R^2 = 0.146, p = 8.61e-05.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.359, R^2 = 0.129, p = 0.0002.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.394 0.155 5.08e-05 -0.00227 ok
chrF++ 100 significant negative -0.421 0.177 1.29e-05 -0.00146 ok
METEOR 100 significant negative -0.475 0.226 5.89e-07 -0.00199 ok
ROUGE-L 100 significant negative -0.490 0.240 2.29e-07 -0.00186 ok
BERTScore 100 significant negative -0.631 0.398 1.93e-12 -0.00048 ok
COMET 100 significant negative -0.459 0.210 1.60e-06 -0.00088 ok
BLEURT 100 significant negative -0.756 0.572 9.18e-20 -0.00261 ok
Trigram precision 100 significant negative -0.381 0.145 9.19e-05 -0.00265 ok
Trigram recall 100 significant negative -0.382 0.146 8.98e-05 -0.00261 ok
Trigram F1 100 significant negative -0.382 0.146 8.61e-05 -0.00262 ok
Trigram Jaccard 100 significant negative -0.359 0.129 0.0002 -0.00242 ok

Translation Similarity Metrics

Mean BLEU-446.8%
Mean chrF++69.8%
Mean METEOR76.7%
Mean ROUGE-L77.4%
Mean BERTScore96.1%
Mean COMET79.1%
Mean BLEURT73.7%
Legacy smoothed corpus BLEU47.2%
Unigram F178.2%
Bigram F153.8%
Trigram precision40.6%
Trigram recall39.6%
Trigram F140.1%
Trigram Jaccard25.1%
4-gram F130.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 138 26.365 27.3% 62.1% 71.7% 65.4% 93.5% 78.4% 62.8% reviewed/approved 10894 gpt-5.4
2484 Καρία 270 238 -17.362 27.8% 56.2% 58.5% 61.4% 90.2% 78.3% 48.0% reviewed/approved 10879 gpt-5.4
2468 Καπετώλιον 138 121 -10.362 19.1% 49.7% 56.1% 59.5% 88.0% 67.3% 50.6% reviewed/approved 10870 gpt-5.4
2603 Κάρυστος 175 176 9.881 42.2% 64.9% 69.3% 68.4% 94.4% 77.3% 59.8% reviewed/approved 10891 gpt-5.4
2597 Καρπασία 112 98 -8.938 39.2% 66.7% 69.5% 71.4% 94.8% 74.9% 59.5% reviewed/approved 10882 gpt-5.4
2342 Κάμιρος 54 61 8.547 33.5% 65.2% 74.8% 69.6% 95.3% 78.1% 66.7% reviewed/approved 10855 gpt-5.4
2609 Κάσπειρος 135 137 8.456 40.2% 65.7% 58.4% 65.4% 89.6% 72.2% 59.9% reviewed/approved 10903 gpt-5.4
3288 Κορώνεια 110 113 7.941 33.8% 63.6% 64.2% 70.5% 95.2% 76.3% 65.0% reviewed/approved 10942 gpt-5.4
2056 Καβειρία 112 99 -7.938 26.4% 64.9% 56.3% 62.6% 95.0% 78.0% 59.9% reviewed/approved 10202 gpt-5.4
2328 Καλλίπολις 43 50 7.880 39.1% 64.3% 67.2% 64.5% 93.9% 73.7% 67.4% reviewed/approved 10831 gpt-5.4
7259 Κύφος 54 60 7.547 49.8% 76.3% 84.7% 82.5% 96.0% 75.1% 71.6% reviewed/approved 11005 gpt-5.4
7260 Κυχρεῖος πάγος 61 66 6.971 40.3% 65.4% 75.6% 66.1% 94.9% 77.3% 62.4% reviewed/approved 11008 gpt-5.4
2116 Κάληρος 40 33 -6.302 21.3% 54.0% 58.0% 57.5% 93.8% 74.8% 72.5% reviewed/approved 10810 gpt-5.4
2470 Καππαδοκία 68 71 5.396 19.2% 58.1% 83.5% 74.8% 95.4% 76.7% 69.3% reviewed/approved 10873 gpt-5.4
2083 Καλαμένθη 24 19 -5.271 35.2% 58.2% 57.0% 65.1% 94.6% 73.6% 71.0% reviewed/approved 10789 gpt-5.4
2115 Καλὴ ἀκτή 64 67 5.153 33.2% 58.1% 61.6% 63.1% 94.1% 72.0% 56.5% reviewed/approved 10807 gpt-5.4
2605 Κάσιον 56 59 4.668 46.8% 70.0% 72.2% 73.0% 95.8% 76.8% 68.5% reviewed/approved 10897 gpt-5.4
7255 Κυτέριον 22 27 4.608 32.2% 66.5% 65.3% 61.2% 95.5% 75.9% 75.3% reviewed/approved 10993 gpt-5.4
7246 Κύρις 20 16 -4.514 56.3% 75.6% 74.4% 83.3% 97.0% 82.5% 76.9% reviewed/approved 10966 gpt-5.4
2465 Κάνωπος 83 84 4.305 43.5% 66.5% 78.5% 79.0% 95.3% 75.6% 68.8% reviewed/approved 10867 gpt-5.4
2329 Κάλπη 57 51 -4.271 50.4% 70.7% 80.1% 82.6% 96.1% 74.9% 75.3% reviewed/approved 10834 gpt-5.4
7254 Κύτα 84 77 -3.635 41.0% 63.4% 55.4% 69.6% 93.9% 76.2% 62.5% reviewed/approved 10990 gpt-5.4
2078 Καινύς 22 19 -3.392 44.7% 62.1% 71.8% 68.3% 95.0% 80.7% 71.9% reviewed/approved 10771 gpt-5.4
7253 Κυρτώνιος 18 22 3.365 35.1% 65.1% 68.6% 65.0% 96.6% 76.8% 80.5% reviewed/approved 10987 gpt-5.4
2079 Καισάρεια 39 35 -3.362 43.6% 66.8% 79.6% 83.8% 95.2% 77.5% 73.2% reviewed/approved 10777 gpt-5.4
2602 Καρύανδα 34 37 3.335 48.8% 73.6% 87.3% 84.5% 97.0% 80.9% 73.2% reviewed/approved 10888 gpt-5.4
7250 Κυρταία 40 36 -3.302 61.8% 73.0% 74.7% 76.3% 96.8% 69.4% 71.3% reviewed/approved 10978 gpt-5.4
2330 Καλύβη 24 21 -3.271 44.8% 64.6% 64.1% 80.0% 96.4% 79.9% 76.4% reviewed/approved 10837 gpt-5.4
2119 Κάλλατις 65 66 3.214 35.8% 61.1% 68.2% 64.1% 94.7% 71.2% 66.2% reviewed/approved 10819 gpt-5.4
7257 Κυτώνιον 25 22 -3.211 66.4% 76.9% 75.7% 80.9% 97.9% 83.7% 79.8% reviewed/approved 10999 gpt-5.4

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.142.

Positive termCoefficientDocsMean presentMean absent
δε και 0.0011 8 4.327 -0.376
εστι δε και 0.0009 3 12.396 -0.383
εστι δε 0.0009 5 6.880 -0.362
και 0.0008 66 0.109 -0.212
δε 0.0008 46 0.687 -0.586
εκαλειτο δε 0.0007 3 6.295 -0.195
ος 0.0007 6 3.885 -0.248
απο 0.0007 27 1.352 -0.500
τε 0.0006 5 3.335 -0.176
οι 0.0006 11 3.051 -0.377
παιδος 0.0006 4 4.767 -0.199
τη 0.0005 15 1.513 -0.267
εν τη 0.0005 7 3.049 -0.230
πολιτης 0.0005 18 2.345 -0.515
εκληθη 0.0005 4 6.085 -0.254
πολις 0.0005 68 0.352 -0.748
πολις και 0.0004 6 4.991 -0.319
εκαλειτο 0.0004 7 0.898 -0.068
ου 0.0004 14 0.798 -0.130
της 0.0004 35 0.102 -0.055
Negative termCoefficientDocsMean presentMean absent
το -0.0009 70 -0.243 0.566
εστι και -0.0007 20 -1.344 0.336
γαρ -0.0006 9 -1.328 0.131
εθνος -0.0006 8 -4.408 0.383
θεοπομπος -0.0005 5 -3.653 0.192
το εθνικον -0.0005 56 -0.291 0.371
εθνικον -0.0005 56 -0.291 0.371
δια -0.0004 16 0.536 -0.102
οτι -0.0004 5 -2.604 0.137
το θηλυκον -0.0004 6 -4.665 0.298
πολις θρακης -0.0004 4 -1.915 0.080
οικητωρ -0.0004 6 -4.019 0.257
τινες -0.0004 6 -2.120 0.135
δεκατω -0.0004 3 -4.312 0.133
το κτητικον -0.0003 7 -3.851 0.290
κτητικον -0.0003 7 -3.851 0.290
δια του -0.0003 10 -0.044 0.005
θρακης -0.0003 5 -1.150 0.061
ποταμου -0.0003 3 -1.615 0.050
ζευς -0.0003 3 -8.928 0.276

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: -0.051.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 3.0446 7 9.458 2.531
την 2.9714 13 8.310 2.225
εκαλειτο δε 2.7786 3 17.869 2.557
δε και 2.6883 8 9.508 2.452
δε 2.6450 46 4.434 1.809
ος 2.0874 6 9.025 2.633
εστι δε 2.0856 5 10.055 2.646
εστι δε και 2.0568 3 12.396 2.726
δια 1.7633 16 6.367 2.378
γαρ 1.7180 9 6.548 2.667
εκαλειτο δε και 1.6913 3 17.869 2.557
και 1.5708 66 3.812 1.471
εν 1.5151 37 3.974 2.454
εστι 1.4607 24 5.314 2.291
μητροπολις 1.3748 3 11.414 2.757
δια του 1.3513 10 7.469 2.522
κατα 1.3281 10 7.403 2.529
απο 1.1397 27 5.539 2.083
τη 1.1292 15 5.112 2.647
φησιν 1.0528 10 7.736 2.492
Negative termCoefficientDocsMean presentMean absent
στραβων -1.3704 15 2.647 3.082
το εθνικον -1.1230 56 2.883 3.185
εθνικον -1.1230 56 2.883 3.185
προς τη -0.9188 4 1.908 3.063
καππαδοκιας -0.8937 3 1.918 3.050
δευτερω -0.8445 3 0.576 3.092
τους -0.8390 4 1.690 3.072
και πολις -0.8184 5 2.279 3.055
ειναι -0.7834 8 2.830 3.033
προς -0.7696 15 3.392 2.950
κωμη -0.7665 3 2.990 3.017
μια -0.7623 3 1.443 3.065
νησος -0.7002 9 2.505 3.067
ουδετερως -0.6750 3 1.567 3.061
και το -0.6191 9 3.950 2.924
το -0.6026 70 2.901 3.285
ειρηται -0.5784 3 2.343 3.037
εκαταιος -0.5425 14 2.398 3.117
αιγυπτου -0.5396 3 2.485 3.033
πολις προς -0.5288 3 4.466 2.972

AI English terms predicting large absolute residuals

Features: 326; ridge alpha: 2.031; cross-validated R^2: -0.034.

Positive termCoefficientDocsMean presentMean absent
was 2.8753 17 6.214 2.361
the 2.2444 91 3.132 1.847
and 2.1870 56 3.965 1.809
with 2.0720 15 6.056 2.480
was called 1.9499 4 10.439 2.707
diphthong 1.4262 3 12.220 2.732
there is 1.4175 25 5.384 2.227
he 1.3634 4 13.031 2.599
by 1.3556 13 6.494 2.497
there 1.3292 33 5.122 1.979
called 1.3055 26 5.367 2.190
and there 1.3041 3 14.283 2.668
whom 1.2861 6 10.167 2.560
all 1.2621 6 7.773 2.713
there is also 1.2490 25 5.384 2.227
on 1.2365 16 5.837 2.479
on the 1.2206 8 7.789 2.601
this 1.2157 10 6.964 2.578
when 1.2021 7 10.701 2.438
of 1.1873 85 3.184 2.064
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.2397 52 2.661 3.401
ethnonym is -1.2397 52 2.661 3.401
the ethnonym -1.0831 58 2.868 3.222
ethnonym -1.0831 58 2.868 3.222
strabo -0.8755 15 2.647 3.082
strabo book -0.7547 9 1.866 3.130
a city -0.6560 62 3.495 2.236
book -0.6498 39 2.885 3.100
one -0.6428 11 3.641 2.939
those -0.6422 4 2.515 3.037
is also called -0.6405 6 2.287 3.063
island -0.6295 14 2.822 3.048
near -0.6269 15 2.948 3.028
cappadocia -0.6107 3 1.918 3.050
from which -0.6050 7 3.204 3.002
is a -0.6009 31 3.563 2.771
syria -0.5897 3 1.140 3.074
in the -0.5624 30 3.550 2.788
they -0.5516 10 4.609 2.839
neuter -0.5443 4 1.433 3.082

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: -0.083.

Positive termCoefficientDocsMean presentMean absent
and 2.2998 52 4.230 1.702
with 2.0107 16 6.236 2.403
the 1.7561 90 3.161 1.717
is with 1.5560 4 14.181 2.551
was 1.3758 20 5.349 2.433
called 1.1644 21 5.523 2.350
who 1.1629 13 6.084 2.558
their 1.1021 3 17.869 2.557
also 1.0688 43 4.361 2.002
used to be 1.0684 6 10.717 2.525
there is 1.0586 24 5.347 2.280
to 0.9814 31 4.424 2.384
used 0.9733 8 9.268 2.473
used to 0.9733 8 9.268 2.473
there is also 0.9372 22 5.495 2.317
there 0.9249 34 4.833 2.081
on 0.9246 16 4.579 2.719
to be called 0.9207 5 10.884 2.602
them 0.9036 5 10.073 2.645
to be 0.9034 9 8.616 2.463
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9745 52 2.691 3.369
ethnonym is -0.9745 52 2.691 3.369
as in -0.9068 29 2.878 3.073
an -0.8496 22 2.235 3.237
ethnonym -0.8328 57 2.909 3.159
the ethnonym -0.8088 56 2.923 3.135
strabo -0.7334 15 2.647 3.082
they -0.5697 12 3.185 2.993
is a -0.5625 31 3.623 2.744
ethnonyms -0.5371 3 1.515 3.063
kappadokia -0.5350 4 2.787 3.026
in the -0.5311 25 3.288 2.926
book -0.5051 39 2.885 3.100
one of -0.4979 5 1.823 3.079
an island -0.4780 9 1.640 3.153
one of the -0.4461 4 1.840 3.065
in his asia -0.4352 5 0.920 3.127
his asia -0.4352 5 0.920 3.127
strabo book -0.4312 10 2.655 3.057
near -0.4157 16 2.903 3.038

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.