gpt-5.4 v1 Evaluation

Generated: 2026-07-25 12:53:38 UTC

Pairs100
Distinct lemmas100
Slope0.978
R^20.988
BLEU-423.2%
chrF++52.7%
METEOR57.5%
ROUGE-L60.1%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2172

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.4 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.978
Intercept2.550
R^20.988
P-value4.38e-95
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words45.720
Mean absolute length residual3.238
Mean absolute percent error8.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: positive; r = 0.089, R^2 = 0.008, p = 0.3795.
chrF++ vs passage length
chrF++: positive; r = 0.048, R^2 = 0.002, p = 0.6347.
METEOR vs passage length
METEOR: negative; r = -0.067, R^2 = 0.005, p = 0.5047.
ROUGE-L vs passage length
ROUGE-L: negative; r = -0.095, R^2 = 0.009, p = 0.3476.
Trigram precision vs passage length
Trigram precision: positive; r = 0.043, R^2 = 0.002, p = 0.6683.
Trigram recall vs passage length
Trigram recall: positive; r = 0.027, R^2 = 7.43e-04, p = 0.7878.
Trigram F1 vs passage length
Trigram F1: positive; r = 0.037, R^2 = 0.001, p = 0.7171.
Trigram Jaccard vs passage length
Trigram Jaccard: positive; r = 0.022, R^2 = 4.63e-04, p = 0.8317.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 positive 0.089 0.008 0.3795 0.00039 ok
chrF++ 100 positive 0.048 0.002 0.6347 0.00014 ok
METEOR 100 negative -0.067 0.005 0.5047 -0.00031 ok
ROUGE-L 100 negative -0.095 0.009 0.3476 -0.00037 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 positive 0.043 0.002 0.6683 0.00019 ok
Trigram recall 100 positive 0.027 7.43e-04 0.7878 0.00012 ok
Trigram F1 100 positive 0.037 0.001 0.7171 0.00016 ok
Trigram Jaccard 100 positive 0.022 4.63e-04 0.8317 0.00006 ok

Translation Similarity Metrics

Mean BLEU-423.2%
Mean chrF++52.7%
Mean METEOR57.5%
Mean ROUGE-L60.1%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU27.7%
Unigram F163.8%
Bigram F135.2%
Trigram precision21.0%
Trigram recall21.7%
Trigram F121.3%
Trigram Jaccard11.9%
4-gram F113.3%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 252 -14.496 21.2% 51.7% 54.3% 58.9% N/A N/A N/A reviewed/approved 10878 gpt-5.4
2603 Κάρυστος 175 187 13.374 33.1% 60.6% 66.7% 64.1% N/A N/A N/A reviewed/approved 10890 gpt-5.4
7266 Κώμη 79 91 11.221 10.0% 38.0% 34.5% 37.6% N/A N/A N/A reviewed/approved 11025 gpt-5.4
2328 Καλλίπολις 43 55 10.414 13.0% 50.7% 42.0% 46.9% N/A N/A N/A reviewed/approved 10830 gpt-5.4
2056 Καβειρία 112 102 -10.039 18.2% 53.4% 54.0% 57.9% N/A N/A N/A reviewed/approved 10201 gpt-5.4
2468 Καπετώλιον 138 128 -9.456 15.0% 41.8% 44.7% 45.1% N/A N/A N/A reviewed/approved 10869 gpt-5.4
2604 Καρχηδών 117 125 8.073 21.0% 54.6% 53.8% 53.9% N/A N/A N/A reviewed/approved 10893 gpt-5.4
2119 Κάλλατις 65 74 7.907 19.2% 49.1% 49.1% 48.9% N/A N/A N/A reviewed/approved 10818 gpt-5.4
2470 Καππαδοκία 68 76 6.975 14.1% 47.9% 60.2% 61.1% N/A N/A N/A reviewed/approved 10872 gpt-5.4
2477 Καρδαμύλη 43 51 6.414 25.3% 58.7% 62.3% 53.2% N/A N/A N/A reviewed/approved 10875 gpt-5.4
7254 Κύτα 84 91 6.333 26.8% 54.8% 47.7% 57.1% N/A N/A N/A reviewed/approved 10989 gpt-5.4
2597 Καρπασία 112 106 -6.039 16.6% 49.4% 44.8% 48.6% N/A N/A N/A reviewed/approved 10881 gpt-5.4
7255 Κυτέριον 22 30 5.943 17.6% 53.8% 67.3% 61.5% N/A N/A N/A reviewed/approved 10992 gpt-5.4
3288 Κορώνεια 110 116 5.916 15.1% 48.1% 41.4% 49.3% N/A N/A N/A reviewed/approved 10941 gpt-5.4
2079 Καισάρεια 39 35 -5.676 25.4% 54.6% 66.0% 67.6% N/A N/A N/A reviewed/approved 10776 gpt-5.4
2342 Κάμιρος 54 61 5.661 19.5% 52.2% 59.5% 59.1% N/A N/A N/A reviewed/approved 10854 gpt-5.4
7244 Κυρήνη 54 61 5.661 35.0% 60.5% 64.2% 60.9% N/A N/A N/A reviewed/approved 10959 gpt-5.4
7259 Κύφος 54 61 5.661 24.0% 52.2% 63.1% 59.1% N/A N/A N/A reviewed/approved 11004 gpt-5.4
2609 Κάσπειρος 135 129 -5.523 39.1% 62.7% 56.1% 67.4% N/A N/A N/A reviewed/approved 10902 gpt-5.4
2465 Κάνωπος 83 89 5.311 35.9% 60.0% 69.4% 68.6% N/A N/A N/A reviewed/approved 10866 gpt-5.4
7251 Κύρτος 66 62 -5.070 27.4% 58.7% 69.0% 68.2% N/A N/A N/A reviewed/approved 10980 gpt-5.4
2621 Κάστνιον 25 32 5.010 15.1% 46.9% 62.7% 56.1% N/A N/A N/A reviewed/approved 10908 gpt-5.4
7267 Κωνώπη 64 70 4.885 39.3% 59.2% 61.1% 65.7% N/A N/A N/A reviewed/approved 11028 gpt-5.4
7261 Κύψελα 34 31 -4.788 34.8% 61.5% 68.6% 70.8% N/A N/A N/A reviewed/approved 11010 gpt-5.4
2626 Κατάνη 95 100 4.580 17.8% 47.6% 45.6% 50.3% N/A N/A N/A reviewed/approved 10926 gpt-5.4
2081 Καλάθη 49 46 -4.451 30.9% 58.1% 67.0% 73.7% N/A N/A N/A reviewed/approved 10782 gpt-5.4
2622 Καστωλοῦ πεδίον 42 48 4.392 34.1% 55.0% 62.4% 62.2% N/A N/A N/A reviewed/approved 10911 gpt-5.4
2080 Καλαβρία 18 16 -4.147 21.9% 52.1% 39.5% 47.1% N/A N/A N/A reviewed/approved 10779 gpt-5.4
2327 Καλλιόπη 19 17 -4.124 17.3% 57.8% 59.8% 61.1% N/A N/A N/A reviewed/approved 10827 gpt-5.4
2624 Κατάβαθμος 20 18 -4.102 29.1% 54.8% 47.3% 52.6% N/A N/A N/A reviewed/approved 10917 gpt-5.4

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.321.

Positive termCoefficientDocsMean presentMean absent
και 0.7471 66 0.333 -0.646
απο 0.3658 27 2.108 -0.780
κωμη 0.2958 3 7.173 -0.222
λεγεται 0.2434 9 3.800 -0.376
οι 0.2395 11 3.470 -0.429
εν τη 0.2360 7 3.529 -0.266
πολιχνιον 0.2061 3 6.499 -0.201
της 0.1967 35 0.498 -0.268
αλλη 0.1961 4 6.162 -0.257
λεγεται και 0.1952 8 3.473 -0.302
δε και 0.1941 8 2.282 -0.198
εστι δε και 0.1899 3 6.801 -0.210
ου και 0.1839 4 6.356 -0.265
και το 0.1815 9 2.691 -0.266
εις 0.1773 7 1.428 -0.107
παιδος 0.1724 4 3.714 -0.155
απο του 0.1713 9 2.662 -0.263
δυο 0.1707 4 4.264 -0.178
θηλυκως 0.1696 5 4.253 -0.224
και θηλυκως 0.1696 5 4.253 -0.224
Negative termCoefficientDocsMean presentMean absent
το -0.2229 70 0.105 -0.244
θεοπομπος -0.1873 5 -2.887 0.152
εθνος -0.1733 8 -3.292 0.286
εκαταιος ασια -0.1667 5 -1.999 0.105
ασια -0.1667 5 -1.999 0.105
και του -0.1586 4 -3.224 0.134
πολις θρακης -0.1583 4 -2.473 0.103
δεκατω -0.1574 3 -3.954 0.122
θρακης -0.1565 5 -2.019 0.106
το εθνικον -0.1515 56 -0.163 0.207
εθνικον -0.1515 56 -0.163 0.207
ηρωδιανος -0.1446 3 -8.662 0.268
ως -0.1398 52 0.064 -0.070
μεταξυ -0.1357 5 -1.103 0.058
πλησιον -0.1349 8 -0.912 0.079
θηλυκον και -0.1310 3 -4.590 0.142
γαρ -0.1306 9 0.672 -0.066
ποταμου -0.1281 3 -1.972 0.061
ως το -0.1269 3 -6.072 0.188
προς τη -0.1224 4 -2.256 0.094

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.019.

Positive termCoefficientDocsMean presentMean absent
γαρ 1.8205 9 7.912 2.775
εν 1.6190 37 4.428 2.538
και 1.6031 66 3.753 2.238
τη 1.1785 15 5.399 2.856
απο 1.0968 27 5.313 2.470
την 1.0933 13 6.556 2.742
εστι 1.0914 24 5.181 2.624
εις 0.9880 7 8.271 2.859
κωμη 0.9780 3 7.173 3.116
τω 0.9764 16 4.972 2.907
της 0.8941 35 4.530 2.542
λεγεται 0.8743 9 7.398 2.826
αφ 0.8446 6 6.827 3.009
υπο 0.8356 4 6.838 3.088
παρα 0.8326 7 6.877 2.964
εκαλειτο δε 0.8035 3 11.981 2.967
τριτη 0.7832 3 7.335 3.111
δε και 0.7632 8 7.325 2.882
δια 0.7492 16 5.231 2.858
λεγεται και 0.7362 8 7.521 2.865
Negative termCoefficientDocsMean presentMean absent
και πολις -1.0152 5 1.216 3.344
στραβων -0.8532 15 3.198 3.245
δευτερω -0.8408 3 0.996 3.307
νησος -0.7577 9 2.373 3.323
το εθνικον -0.6887 56 3.332 3.118
εθνικον -0.6887 56 3.332 3.118
μια -0.6668 3 1.191 3.301
τεταρτω -0.5813 3 1.846 3.281
παυσανιας -0.5531 4 2.864 3.253
δε απο -0.5464 3 1.158 3.302
πολις και -0.5439 6 2.291 3.298
τοπικα -0.4983 3 2.319 3.266
τα τοπικα -0.4983 3 2.319 3.266
τινες -0.4843 6 2.100 3.310
εκ -0.4705 6 2.943 3.257
πολιτης -0.4590 18 3.527 3.174
αιγυπτου -0.4535 3 2.172 3.271
πολις -0.4498 68 3.384 2.927
θαλασση -0.4347 3 1.390 3.295
φησι το -0.4296 3 1.193 3.301

AI English terms predicting large absolute residuals

Features: 398; ridge alpha: 2.031; cross-validated R^2: 0.028.

Positive termCoefficientDocsMean presentMean absent
for 1.6470 11 7.076 2.763
there 1.5940 33 4.802 2.467
on 1.5921 18 5.735 2.690
from 1.5305 42 4.567 2.276
it 1.4839 36 4.689 2.422
when 1.3093 6 9.324 2.849
was 1.3071 19 5.214 2.774
to 1.2594 25 5.207 2.581
also 1.2563 47 4.274 2.319
is 1.1734 90 3.436 1.458
said 1.1049 4 9.756 2.966
there is 1.0597 24 5.255 2.601
in 1.0486 74 3.505 2.478
caria 1.0457 3 8.708 3.069
as 1.0257 49 4.277 2.239
with 0.9566 17 4.882 2.901
is also 0.9533 32 4.819 2.494
there is also 0.9451 24 5.255 2.601
the 0.9389 97 3.249 2.873
used 0.9072 5 7.733 3.001
Negative termCoefficientDocsMean presentMean absent
an -1.1233 22 2.262 3.513
island -0.8108 14 2.175 3.411
and a -0.8024 5 1.815 3.313
forms are -0.7643 3 1.352 3.296
the ethnic is -0.7094 62 3.155 3.372
the ethnic -0.6876 63 3.318 3.102
ethnic is -0.6601 64 3.242 3.230
ethnic -0.6542 66 3.394 2.935
strabo -0.6296 15 3.198 3.245
and a city -0.6272 3 0.870 3.311
book -0.6042 39 3.083 3.337
and also -0.5915 3 2.265 3.268
syria -0.5645 3 0.718 3.316
is also called -0.5430 3 0.982 3.308
inhabitants -0.5385 3 2.389 3.264
of his -0.5319 9 1.914 3.369
herodotus in the -0.5219 3 0.821 3.313
a city -0.4981 63 3.331 3.080
pausanias -0.4958 4 2.864 3.253
has been -0.4936 3 2.159 3.271

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 1.425; cross-validated R^2: 0.032.

Positive termCoefficientDocsMean presentMean absent
there 2.1220 34 5.080 2.289
with 1.9429 16 5.168 2.870
for 1.8498 11 6.278 2.862
on 1.8310 16 4.871 2.927
when 1.6649 7 8.578 2.836
from 1.5958 37 4.580 2.450
the 1.5508 90 3.340 2.317
also 1.4992 43 4.332 2.412
his on 1.3338 3 9.993 3.029
into 1.3198 3 8.755 3.067
also said 1.3176 6 9.677 2.827
said 1.3176 6 9.677 2.827
be 1.2502 15 6.136 2.726
situated 1.2460 4 8.837 3.004
is also said 1.2326 5 10.480 2.857
is also 1.1653 31 4.936 2.475
forms 1.1486 4 10.241 2.946
long 1.1423 3 11.724 2.975
and 1.1392 52 4.011 2.400
is with 1.1065 4 8.543 3.017
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.0547 52 3.065 3.425
ethnonym is -1.0547 52 3.065 3.425
but -1.0104 12 3.053 3.263
island -0.9815 14 2.175 3.411
strabo -0.9568 15 3.198 3.245
syria -0.9000 3 0.718 3.316
in syria -0.9000 3 0.718 3.316
herodotos book -0.8746 5 1.745 3.316
ethnonyms -0.8573 3 1.471 3.292
through -0.8197 3 0.844 3.312
territory -0.8029 5 1.984 3.304
one of the -0.7798 4 2.038 3.288
is a -0.7649 31 3.834 2.970
one of -0.7215 5 2.521 3.275
city and -0.7112 4 1.818 3.297
a city and -0.7112 4 1.818 3.297
an island -0.6736 9 1.920 3.368
it is also -0.6735 3 0.877 3.311
also a city -0.6719 4 4.786 3.173
also called -0.6689 6 2.826 3.264

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.