gpt-5.4 v2 Evaluation

Generated: 2026-07-25 12:53:41 UTC

Pairs100
Distinct lemmas100
Slope0.940
R^20.984
BLEU-446.8%
chrF++69.7%
METEOR76.5%
ROUGE-L77.2%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2173

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.4 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.940
Intercept1.708
R^20.984
P-value1.12e-89
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words43.200
Mean absolute length residual3.023
Mean absolute percent error6.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.393, R^2 = 0.154, p = 5.25e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.415, R^2 = 0.172, p = 1.76e-05.
METEOR vs passage length
METEOR: significant negative; r = -0.456, R^2 = 0.208, p = 1.85e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.474, R^2 = 0.225, p = 6.19e-07.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.373, R^2 = 0.139, p = 0.0001.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.372, R^2 = 0.138, p = 0.0001.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.373, R^2 = 0.139, p = 0.0001.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.352, R^2 = 0.124, p = 0.0003.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.393 0.154 5.25e-05 -0.00226 ok
chrF++ 100 significant negative -0.415 0.172 1.76e-05 -0.00145 ok
METEOR 100 significant negative -0.456 0.208 1.85e-06 -0.00194 ok
ROUGE-L 100 significant negative -0.474 0.225 6.19e-07 -0.00182 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.373 0.139 0.0001 -0.00260 ok
Trigram recall 100 significant negative -0.372 0.138 0.0001 -0.00256 ok
Trigram F1 100 significant negative -0.373 0.139 0.0001 -0.00257 ok
Trigram Jaccard 100 significant negative -0.352 0.124 0.0003 -0.00238 ok

Translation Similarity Metrics

Mean BLEU-446.8%
Mean chrF++69.7%
Mean METEOR76.5%
Mean ROUGE-L77.2%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU47.2%
Unigram F178.2%
Bigram F153.7%
Trigram precision40.5%
Trigram recall39.6%
Trigram F140.1%
Trigram Jaccard25.1%
4-gram F130.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 138 26.362 27.3% 62.1% 71.7% 65.4% N/A N/A N/A reviewed/approved 10894 gpt-5.4
2484 Καρία 270 238 -17.393 27.8% 56.2% 58.5% 61.4% N/A N/A N/A reviewed/approved 10879 gpt-5.4
2468 Καπετώλιον 138 121 -10.369 19.1% 49.7% 56.1% 59.5% N/A N/A N/A reviewed/approved 10870 gpt-5.4
2603 Κάρυστος 175 176 9.866 42.2% 64.9% 69.3% 68.4% N/A N/A N/A reviewed/approved 10891 gpt-5.4
2597 Καρπασία 112 98 -8.941 39.2% 66.7% 69.5% 71.4% N/A N/A N/A reviewed/approved 10882 gpt-5.4
2342 Κάμιρος 54 61 8.555 33.5% 65.2% 74.8% 69.6% N/A N/A N/A reviewed/approved 10855 gpt-5.4
2609 Κάσπειρος 135 137 8.449 40.2% 65.7% 58.4% 65.4% N/A N/A N/A reviewed/approved 10903 gpt-5.4
2056 Καβειρία 112 99 -7.941 26.4% 64.9% 56.3% 62.6% N/A N/A N/A reviewed/approved 10202 gpt-5.4
3288 Κορώνεια 110 113 7.939 33.8% 63.6% 64.2% 70.5% N/A N/A N/A reviewed/approved 10942 gpt-5.4
2328 Καλλίπολις 43 50 7.890 39.1% 64.3% 67.2% 64.5% N/A N/A N/A reviewed/approved 10831 gpt-5.4
7259 Κύφος 54 60 7.555 49.8% 76.3% 84.7% 82.5% N/A N/A N/A reviewed/approved 11005 gpt-5.4
7260 Κυχρεῖος πάγος 61 66 6.978 40.3% 65.4% 75.6% 66.1% N/A N/A N/A reviewed/approved 11008 gpt-5.4
2116 Κάληρος 40 33 -6.291 21.3% 54.0% 58.0% 57.5% N/A N/A N/A reviewed/approved 10810 gpt-5.4
2470 Καππαδοκία 68 71 5.401 19.2% 58.1% 83.5% 74.8% N/A N/A N/A reviewed/approved 10873 gpt-5.4
2083 Καλαμένθη 24 19 -5.258 35.2% 58.2% 57.0% 65.1% N/A N/A N/A reviewed/approved 10789 gpt-5.4
2115 Καλὴ ἀκτή 64 67 5.159 33.2% 58.1% 61.6% 63.1% N/A N/A N/A reviewed/approved 10807 gpt-5.4
2605 Κάσιον 56 59 4.675 46.8% 70.0% 72.2% 73.0% N/A N/A N/A reviewed/approved 10897 gpt-5.4
7255 Κυτέριον 22 27 4.621 32.2% 66.5% 65.3% 61.2% N/A N/A N/A reviewed/approved 10993 gpt-5.4
7246 Κύρις 20 16 -4.500 56.3% 75.6% 74.4% 83.3% N/A N/A N/A reviewed/approved 10966 gpt-5.4
2465 Κάνωπος 83 84 4.307 43.5% 66.5% 78.5% 79.0% N/A N/A N/A reviewed/approved 10867 gpt-5.4
2329 Κάλπη 57 51 -4.264 50.4% 70.7% 80.1% 82.6% N/A N/A N/A reviewed/approved 10834 gpt-5.4
2599 Καρπήσιοι 11 8 -4.044 22.4% 60.6% 55.0% 63.2% N/A N/A N/A reviewed/approved 10885 gpt-5.4
7254 Κύτα 84 77 -3.633 41.0% 63.4% 55.4% 68.3% N/A N/A N/A reviewed/approved 10990 gpt-5.4
7253 Κυρτώνιος 18 22 3.379 35.1% 65.1% 68.6% 65.0% N/A N/A N/A reviewed/approved 10987 gpt-5.4
2078 Καινύς 22 19 -3.379 44.7% 62.1% 71.8% 68.3% N/A N/A N/A reviewed/approved 10771 gpt-5.4
2079 Καισάρεια 39 35 -3.352 43.6% 66.8% 79.6% 83.8% N/A N/A N/A reviewed/approved 10777 gpt-5.4
2602 Καρύανδα 34 37 3.346 48.8% 73.6% 87.3% 84.5% N/A N/A N/A reviewed/approved 10888 gpt-5.4
7250 Κυρταία 40 36 -3.291 61.8% 73.0% 74.7% 76.3% N/A N/A N/A reviewed/approved 10978 gpt-5.4
2330 Καλύβη 24 21 -3.258 44.8% 64.6% 64.1% 80.0% N/A N/A N/A reviewed/approved 10837 gpt-5.4
2119 Κάλλατις 65 66 3.219 35.8% 61.1% 68.2% 64.1% N/A N/A N/A reviewed/approved 10819 gpt-5.4

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.138.

Positive termCoefficientDocsMean presentMean absent
δε και 0.0011 8 4.326 -0.376
εστι δε και 0.0009 3 12.397 -0.383
εστι δε 0.0009 5 6.882 -0.362
και 0.0008 66 0.116 -0.226
δε 0.0008 46 0.693 -0.590
εκαλειτο δε 0.0007 3 6.278 -0.194
ος 0.0007 6 3.887 -0.248
απο 0.0007 27 1.355 -0.501
τε 0.0006 5 3.339 -0.176
οι 0.0006 11 3.056 -0.378
παιδος 0.0006 4 4.776 -0.199
τη 0.0005 15 1.517 -0.268
εν τη 0.0005 7 3.058 -0.230
πολιτης 0.0005 18 2.352 -0.516
εκληθη 0.0005 4 6.087 -0.254
πολις 0.0005 68 0.360 -0.766
πολις και 0.0004 6 4.999 -0.319
εκαλειτο 0.0004 7 0.895 -0.067
της 0.0004 35 0.108 -0.058
ου 0.0004 14 0.801 -0.130
Negative termCoefficientDocsMean presentMean absent
το -0.0009 70 -0.234 0.547
εστι και -0.0007 20 -1.340 0.335
εθνος -0.0006 8 -4.520 0.393
γαρ -0.0006 9 -1.333 0.132
θεοπομπος -0.0005 5 -3.644 0.192
το εθνικον -0.0005 56 -0.282 0.359
εθνικον -0.0005 56 -0.282 0.359
δια -0.0004 16 0.538 -0.102
οτι -0.0004 5 -2.601 0.137
το θηλυκον -0.0004 6 -4.666 0.298
πολις θρακης -0.0004 4 -1.902 0.079
οικητωρ -0.0004 6 -4.014 0.256
ποταμου -0.0004 3 -1.918 0.059
τινες -0.0004 6 -2.110 0.135
δεκατω -0.0004 3 -4.304 0.133
το κτητικον -0.0003 7 -3.852 0.290
κτητικον -0.0003 7 -3.852 0.290
δια του -0.0003 10 -0.042 0.005
θρακης -0.0003 5 -1.139 0.060
ζευς -0.0003 3 -8.939 0.276

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: -0.058.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 3.0351 7 9.456 2.538
την 2.9716 13 8.312 2.232
εκαλειτο δε 2.7778 3 17.874 2.563
δε και 2.6883 8 9.512 2.458
δε 2.6393 46 4.435 1.820
ος 2.0847 6 9.027 2.639
εστι δε 2.0826 5 10.058 2.652
εστι δε και 2.0547 3 12.397 2.733
δια 1.7615 16 6.368 2.385
γαρ 1.7197 9 6.550 2.674
εκαλειτο δε και 1.6911 3 17.874 2.563
και 1.5490 66 3.811 1.493
εν 1.4909 37 3.974 2.464
εστι 1.4433 24 5.312 2.300
μητροπολις 1.3699 3 11.404 2.763
δια του 1.3517 10 7.471 2.528
κατα 1.3257 10 7.406 2.536
απο 1.1378 27 5.539 2.092
τη 1.1255 15 5.114 2.654
φησιν 1.0502 10 7.740 2.498
Negative termCoefficientDocsMean presentMean absent
στραβων -1.3825 15 2.643 3.090
το εθνικον -1.1425 56 2.881 3.202
εθνικον -1.1425 56 2.881 3.202
προς τη -0.9276 4 1.903 3.069
καππαδοκιας -0.8992 3 1.911 3.057
δευτερω -0.8547 3 0.581 3.098
τους -0.8410 4 1.696 3.078
και πολις -0.8260 5 2.275 3.062
ειναι -0.7840 8 2.835 3.039
προς -0.7831 15 3.388 2.958
μια -0.7759 3 1.432 3.072
κωμη -0.7682 3 2.994 3.023
νησος -0.7046 9 2.504 3.074
ουδετερως -0.6747 3 1.575 3.067
το -0.6230 70 2.899 3.311
και το -0.6196 9 3.949 2.931
ειρηται -0.5815 3 2.339 3.044
εκαταιος -0.5564 14 2.396 3.125
αιγυπτου -0.5381 3 2.491 3.039
εκαταιος ασια -0.5243 5 0.910 3.134

AI English terms predicting large absolute residuals

Features: 326; ridge alpha: 2.031; cross-validated R^2: -0.041.

Positive termCoefficientDocsMean presentMean absent
was 2.8717 17 6.216 2.369
the 2.2514 91 3.140 1.836
and 2.1715 56 3.964 1.824
with 2.0709 15 6.058 2.487
was called 1.9448 4 10.438 2.714
diphthong 1.4300 3 12.234 2.738
there is 1.4077 25 5.382 2.236
he 1.3670 4 13.043 2.605
by 1.3423 13 6.490 2.504
there 1.3188 33 5.120 1.989
and there 1.3045 3 14.286 2.674
called 1.2992 26 5.367 2.199
whom 1.2871 6 10.167 2.567
all 1.2627 6 7.776 2.719
there is also 1.2393 25 5.382 2.236
on 1.2369 16 5.837 2.487
on the 1.2196 8 7.789 2.608
this 1.2139 10 6.961 2.585
when 1.1999 7 10.705 2.444
city and 1.1779 5 6.861 2.821
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.2555 52 2.659 3.417
ethnonym is -1.2555 52 2.659 3.417
the ethnonym -1.0978 58 2.866 3.239
ethnonym -1.0978 58 2.866 3.239
strabo -0.8823 15 2.643 3.090
strabo book -0.7584 9 1.866 3.137
book -0.6709 39 2.883 3.112
a city -0.6634 62 3.493 2.255
one -0.6484 11 3.636 2.947
those -0.6461 4 2.512 3.044
is also called -0.6434 6 2.286 3.070
near -0.6407 15 2.945 3.036
island -0.6399 14 2.820 3.056
cappadocia -0.6148 3 1.911 3.057
is a -0.6099 31 3.561 2.781
from which -0.6044 7 3.205 3.009
syria -0.5925 3 1.130 3.081
in the -0.5705 30 3.550 2.797
they -0.5597 10 4.614 2.846
neuter -0.5435 4 1.437 3.089

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: -0.090.

Positive termCoefficientDocsMean presentMean absent
and 2.2771 52 4.229 1.715
with 2.0075 16 6.236 2.410
the 1.7668 90 3.169 1.709
is with 1.5512 4 14.186 2.557
was 1.3476 20 5.349 2.441
who 1.1555 13 6.084 2.565
called 1.1514 21 5.523 2.358
their 1.1019 3 17.874 2.563
used to be 1.0665 6 10.716 2.532
also 1.0479 43 4.359 2.014
there is 1.0466 24 5.344 2.290
to 0.9757 31 4.422 2.394
used 0.9708 8 9.267 2.480
used to 0.9708 8 9.267 2.480
there is also 0.9253 22 5.492 2.326
on 0.9201 16 4.578 2.726
to be called 0.9197 5 10.886 2.609
there 0.9103 34 4.830 2.091
says 0.9035 17 6.040 2.405
to be 0.9030 9 8.616 2.469
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9913 52 2.689 3.384
ethnonym is -0.9913 52 2.689 3.384
as in -0.9048 29 2.881 3.081
ethnonym -0.8475 57 2.907 3.176
the ethnonym -0.8243 56 2.921 3.152
an -0.7965 22 2.275 3.233
strabo -0.7488 15 2.643 3.090
they -0.5744 12 3.186 3.000
is a -0.5674 31 3.622 2.753
ethnonyms -0.5371 3 1.513 3.069
in the -0.5365 25 3.288 2.934
kappadokia -0.5303 3 2.594 3.036
book -0.5187 39 2.883 3.112
one of -0.4989 5 1.815 3.086
an island -0.4856 9 1.637 3.160
in his asia -0.4474 5 0.910 3.134
his asia -0.4474 5 0.910 3.134
one of the -0.4473 4 1.832 3.072
near -0.4349 16 2.900 3.046
strabo book -0.4343 10 2.652 3.064

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.