gpt-4o-2024-11-20 v1 Evaluation

Generated: 2026-07-25 12:52:51 UTC

Pairs100
Distinct lemmas100
Slope1.050
R^20.966
BLEU-416.5%
chrF++49.0%
METEOR54.7%
ROUGE-L54.6%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2413

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-11-20) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope1.050
Intercept4.802
R^20.966
P-value4.63e-74
Pearson r0.983
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words51.180
Mean absolute length residual5.150
Mean absolute percent error21.1%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: negative; r = -0.071, R^2 = 0.005, p = 0.4818.
chrF++ vs passage length
chrF++: negative; r = -0.009, R^2 = 7.71e-05, p = 0.9309.
METEOR vs passage length
METEOR: significant negative; r = -0.263, R^2 = 0.069, p = 0.0082.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.231, R^2 = 0.054, p = 0.0205.
Trigram precision vs passage length
Trigram precision: negative; r = -0.090, R^2 = 0.008, p = 0.3735.
Trigram recall vs passage length
Trigram recall: negative; r = -0.119, R^2 = 0.014, p = 0.2386.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.102, R^2 = 0.010, p = 0.3127.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.114, R^2 = 0.013, p = 0.2575.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 negative -0.071 0.005 0.4818 -0.00028 ok
chrF++ 100 negative -0.009 7.71e-05 0.9309 -0.00003 ok
METEOR 100 significant negative -0.263 0.069 0.0082 -0.00128 ok
ROUGE-L 100 significant negative -0.231 0.054 0.0205 -0.00095 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 negative -0.090 0.008 0.3735 -0.00038 ok
Trigram recall 100 negative -0.119 0.014 0.2386 -0.00054 ok
Trigram F1 100 negative -0.102 0.010 0.3127 -0.00044 ok
Trigram Jaccard 100 negative -0.114 0.013 0.2575 -0.00034 ok

Translation Similarity Metrics

Mean BLEU-416.5%
Mean chrF++49.0%
Mean METEOR54.7%
Mean ROUGE-L54.6%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU19.2%
Unigram F158.4%
Bigram F127.3%
Trigram precision13.2%
Trigram recall15.4%
Trigram F114.3%
Trigram Jaccard7.7%
4-gram F18.0%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2115 Καλὴ ἀκτή 64 106 33.983 8.4% 46.4% 40.5% 40.0% N/A N/A N/A reviewed/approved 12909 gpt-4o-2024-11-20
3288 Κορώνεια 110 149 28.673 9.0% 48.2% 47.7% 49.2% N/A N/A N/A reviewed/approved 13044 gpt-4o-2024-11-20
2484 Καρία 270 265 -23.364 9.3% 43.0% 36.0% 40.4% N/A N/A N/A reviewed/approved 12981 gpt-4o-2024-11-20
2465 Κάνωπος 83 112 20.029 18.9% 55.7% 64.5% 59.5% N/A N/A N/A reviewed/approved 12969 gpt-4o-2024-11-20
7249 Κύρρος 45 70 17.938 6.8% 47.5% 53.7% 43.5% N/A N/A N/A reviewed/approved 13077 gpt-4o-2024-11-20
2055 Καβασσός 109 103 -16.277 14.5% 46.3% 32.6% 43.2% N/A N/A N/A reviewed/approved 12837 gpt-4o-2024-11-20
2468 Καπετώλιον 138 137 -12.734 8.8% 36.3% 35.9% 40.0% N/A N/A N/A reviewed/approved 12972 gpt-4o-2024-11-20
3496 Κοτιάειον 71 68 -11.368 7.2% 43.7% 51.0% 48.9% N/A N/A N/A reviewed/approved 13047 gpt-4o-2024-11-20
7266 Κώμη 79 99 11.230 5.2% 38.0% 44.0% 34.8% N/A N/A N/A reviewed/approved 13128 gpt-4o-2024-11-20
7254 Κύτα 84 104 10.979 19.3% 52.3% 52.1% 53.2% N/A N/A N/A reviewed/approved 13092 gpt-4o-2024-11-20
2603 Κάρυστος 175 199 10.408 15.9% 54.7% 55.2% 52.4% N/A N/A N/A reviewed/approved 12993 gpt-4o-2024-11-20
2627 Κατάνειρα 21 37 10.143 6.4% 42.7% 47.1% 51.7% N/A N/A N/A reviewed/approved 13026 gpt-4o-2024-11-20
2625 Κατακεκαυμένη 46 63 9.888 11.0% 49.4% 57.5% 49.5% N/A N/A N/A reviewed/approved 13023 gpt-4o-2024-11-20
2455 Κάναστρον 60 58 -9.816 11.6% 49.5% 50.1% 45.8% N/A N/A N/A reviewed/approved 12966 gpt-4o-2024-11-20
7244 Κυρήνη 54 71 9.486 22.5% 55.8% 54.1% 56.0% N/A N/A N/A reviewed/approved 13062 gpt-4o-2024-11-20
2081 Καλάθη 49 47 -9.263 28.3% 54.4% 60.7% 66.7% N/A N/A N/A reviewed/approved 12885 gpt-4o-2024-11-20
2802 Κάδρεμα 24 21 -9.007 8.5% 43.6% 55.4% 62.2% N/A N/A N/A reviewed/approved 12858 gpt-4o-2024-11-20
2470 Καππαδοκία 68 85 8.783 7.6% 46.8% 65.1% 60.1% N/A N/A N/A reviewed/approved 12975 gpt-4o-2024-11-20
2114 Κάλβιος 18 16 -7.706 4.8% 33.4% 42.1% 41.2% N/A N/A N/A reviewed/approved 12906 gpt-4o-2024-11-20
2626 Κατάνη 95 97 -7.574 20.1% 51.4% 40.4% 50.0% N/A N/A N/A reviewed/approved 13029 gpt-4o-2024-11-20
2346 Κάναι 77 93 7.330 25.6% 53.2% 70.5% 70.6% N/A N/A N/A reviewed/approved 12963 gpt-4o-2024-11-20
2328 Καλλίπολις 43 57 7.038 7.0% 43.5% 41.9% 48.0% N/A N/A N/A reviewed/approved 12933 gpt-4o-2024-11-20
2078 Καινύς 22 21 -6.907 63.1% 70.7% 77.4% 79.1% N/A N/A N/A reviewed/approved 12873 gpt-4o-2024-11-20
2079 Καισάρεια 39 39 -6.761 26.0% 51.9% 59.6% 61.5% N/A N/A N/A reviewed/approved 12879 gpt-4o-2024-11-20
7256 Κύτινα 32 32 -6.409 47.3% 66.3% 69.7% 71.9% N/A N/A N/A reviewed/approved 13098 gpt-4o-2024-11-20
2599 Καρπήσιοι 11 10 -6.354 16.8% 45.8% 44.4% 47.6% N/A N/A N/A reviewed/approved 12987 gpt-4o-2024-11-20
7247 Κύρνος 50 51 -6.313 16.6% 46.9% 58.5% 51.5% N/A N/A N/A reviewed/approved 13071 gpt-4o-2024-11-20
2087 Καλαύρεια 30 30 -6.309 25.0% 56.6% 65.8% 73.3% N/A N/A N/A reviewed/approved 12903 gpt-4o-2024-11-20
3258 Κεκρυφάλεια 26 26 -6.108 52.0% 76.6% 79.8% 80.8% N/A N/A N/A reviewed/approved 13041 gpt-4o-2024-11-20
2330 Καλύβη 24 24 -6.007 34.8% 54.3% 80.7% 75.0% N/A N/A N/A reviewed/approved 12939 gpt-4o-2024-11-20

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.179.

Positive termCoefficientDocsMean presentMean absent
και 3.6480 66 0.569 -1.104
παρα 2.8346 7 6.389 -0.481
κωμη 2.7671 3 15.417 -0.477
εστι δε 2.4134 5 12.518 -0.659
δυο 2.3206 4 13.195 -0.550
ως 2.1924 52 0.678 -0.734
παρα το 2.1267 3 13.798 -0.427
δια του 2.0015 10 4.103 -0.456
βοιωτιας 1.9789 3 10.029 -0.310
θηλυκον 1.7891 12 5.237 -0.714
εκ 1.7686 6 7.460 -0.476
εστι 1.7165 24 3.011 -0.951
θεοπομπος 1.7103 5 4.217 -0.222
ου 1.6423 14 3.712 -0.604
πολυβιος 1.6032 5 3.900 -0.205
την 1.5866 13 1.115 -0.167
και νησος 1.4459 3 11.163 -0.345
οι 1.4431 11 4.418 -0.546
στραβων 1.4278 15 3.625 -0.640
δευτερα 1.4269 3 12.201 -0.377
Negative termCoefficientDocsMean presentMean absent
δε -2.4556 46 0.130 -0.110
εκαλειτο -2.1783 7 -7.100 0.534
λυκιας -2.0233 3 -10.997 0.340
περι -2.0088 11 -5.319 0.657
το εθνικον -1.9651 56 -1.100 1.400
εθνικον -1.9651 56 -1.100 1.400
δε της -1.6980 6 -8.123 0.519
τα -1.5944 12 -3.851 0.525
τινες δε -1.4084 3 -8.394 0.260
φησιν -1.3064 10 -5.875 0.653
καππαδοκιας -1.3019 3 -7.015 0.217
και του -1.2558 4 -7.213 0.301
εθνος -1.2370 8 -3.305 0.287
το -1.2063 70 0.486 -1.133
τους -1.1747 4 -4.325 0.180
ιταλιας -1.0896 3 -3.773 0.117
των -1.0499 17 -1.567 0.321
ην -1.0368 13 -2.227 0.333
μια -1.0136 3 -4.361 0.135
ηρωδιανος -1.0056 3 -13.018 0.403

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: 0.044.

Positive termCoefficientDocsMean presentMean absent
δια του 2.8509 10 13.400 4.233
δε 2.8406 46 7.399 3.234
και 2.7482 66 6.249 3.016
γαρ 2.7447 9 13.855 4.289
εστι 2.6908 24 9.191 3.874
δια 2.5832 16 10.353 4.159
παρα 2.0406 7 13.926 4.489
την 1.9063 13 8.527 4.645
κωμη 1.6625 3 15.417 4.832
εστι και 1.4891 20 9.066 4.171
εστι δε 1.3756 5 13.589 4.706
της 1.3693 35 7.121 4.089
εις 1.3680 7 14.107 4.476
παρα το 1.2395 3 13.798 4.882
των 1.2389 17 8.501 4.464
του 1.2274 37 8.082 3.428
δυο 1.2021 4 13.195 4.815
λυκιας 1.1476 3 10.997 4.969
θηλυκον 1.1460 12 9.760 4.521
δυναται 1.1460 3 16.348 4.804
Negative termCoefficientDocsMean presentMean absent
παυσανιας -1.2642 4 0.998 5.323
πλησιον -1.1624 8 2.759 5.358
ηροδοτος -1.0624 9 4.684 5.196
το εθνικον -1.0580 56 5.001 5.340
εθνικον -1.0580 56 5.001 5.340
δευτερω -1.0539 3 0.998 5.278
εκαταιος ασια -0.9987 5 0.835 5.377
ασια -0.9987 5 0.835 5.377
παιδος -0.8941 4 2.675 5.253
διονυσιος -0.8065 4 1.879 5.286
τε -0.7528 5 4.400 5.189
αφ -0.7187 6 4.113 5.216
πολις και -0.6987 6 3.550 5.252
του δε -0.6797 3 2.537 5.231
πολις προς -0.6497 3 4.387 5.173
οικητωρ -0.6434 6 6.930 5.036
τους -0.6243 4 4.325 5.184
φησι το -0.6058 3 0.955 5.280
τοις -0.6007 3 2.305 5.238
ενατω -0.5816 3 1.608 5.259

AI English terms predicting large absolute residuals

Features: 477; ridge alpha: 2.894; cross-validated R^2: 0.020.

Positive termCoefficientDocsMean presentMean absent
this 2.5842 16 10.460 4.138
inhabitant of 2.2360 3 13.951 4.878
a 2.1711 92 5.300 3.424
be 2.1696 7 17.964 4.185
or 2.0092 21 10.629 3.693
this name 1.9803 5 15.634 4.598
with 1.9717 27 9.521 3.533
is 1.8340 87 5.469 3.011
inhabitant 1.7939 11 8.663 4.716
from 1.6862 37 8.219 3.347
name 1.6848 22 9.060 4.047
is a 1.6552 13 11.575 4.190
of 1.6407 86 5.434 3.404
is in 1.6257 3 14.582 4.858
term 1.5707 7 11.131 4.700
the 1.5501 97 5.152 5.073
that 1.5432 19 9.305 4.175
i 1.5391 4 17.751 4.625
with the 1.5375 9 12.526 4.420
village 1.5156 4 13.076 4.820
Negative termCoefficientDocsMean presentMean absent
in the -1.4862 44 5.206 5.106
are -1.1896 24 4.582 5.329
ethnic designation is -1.0699 28 4.422 5.433
designation is -1.0699 28 4.422 5.433
book the -1.0621 9 1.670 5.494
pausanias -0.9966 4 0.998 5.323
the ethnic designation -0.9508 29 4.830 5.280
ethnic designation -0.9508 29 4.830 5.280
tribe -0.9328 3 1.555 5.261
asia -0.8789 6 0.958 5.417
mentioned by -0.8640 15 2.654 5.590
mentioned -0.8187 19 3.087 5.634
similar to -0.8024 8 2.807 5.354
similar -0.7275 9 3.651 5.298
are called -0.6909 6 2.978 5.288
analogous to -0.6724 12 4.799 5.198
analogous -0.6724 12 4.799 5.198
inhabitants -0.6669 11 3.068 5.407
genitive -0.6320 5 5.802 5.115
noted by -0.6249 5 1.885 5.322

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: 0.061.

Positive termCoefficientDocsMean presentMean absent
with 3.3417 16 10.699 4.093
is 2.7772 87 5.521 2.666
of 2.6465 66 6.151 3.206
there is 2.5817 24 9.514 3.772
there 2.4854 34 8.507 3.421
and 2.0851 52 6.507 3.680
from 1.7553 37 7.909 3.529
the 1.7186 90 5.352 3.330
this 1.6323 15 10.719 4.167
like 1.6255 5 14.172 4.675
is also 1.6245 31 7.969 3.883
there is also 1.5737 22 8.577 4.183
or 1.5665 18 10.476 3.981
has 1.5130 6 13.240 4.633
territory 1.4237 5 12.320 4.772
as well 1.3149 5 14.628 4.651
as well as 1.3149 5 14.628 4.651
well 1.3149 5 14.628 4.651
well as 1.3149 5 14.628 4.651
both 1.2670 5 11.425 4.820
Negative termCoefficientDocsMean presentMean absent
in book of -1.1776 14 2.788 5.534
near -1.1767 16 3.814 5.404
in book -1.1367 16 4.042 5.361
pausanias -1.0229 4 0.998 5.323
asia -1.0083 6 0.958 5.417
the ethnonym -0.9900 56 5.006 5.333
dionysios -0.9686 4 1.879 5.286
are -0.9563 25 6.591 4.669
herodotos -0.9351 9 4.684 5.196
also called -0.8670 6 2.540 5.316
in his asia -0.8597 5 0.835 5.377
his asia -0.8597 5 0.835 5.377
the ethnonym is -0.8572 52 5.053 5.255
ethnonym is -0.8572 52 5.053 5.255
ethnonym -0.8571 57 5.204 5.078
herodotos book -0.7896 5 1.396 5.347
pausanias book -0.7734 3 0.995 5.278
among -0.7673 4 6.349 5.100
that of the -0.7316 3 1.111 5.275
that of -0.7316 3 1.111 5.275

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.