gpt-4o-2024-08-06 v1 Evaluation

Generated: 2026-07-25 12:52:43 UTC

Pairs100
Distinct lemmas100
Slope0.965
R^20.977
BLEU-418.4%
chrF++49.7%
METEOR54.3%
ROUGE-L56.1%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2410

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-08-06) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.965
Intercept5.570
R^20.977
P-value4.46e-82
Pearson r0.988
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words48.170
Mean absolute length residual4.211
Mean absolute percent error15.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: negative; r = -0.029, R^2 = 8.35e-04, p = 0.7754.
chrF++ vs passage length
chrF++: positive; r = 2.25e-04, R^2 = 5.07e-08, p = 0.9982.
METEOR vs passage length
METEOR: negative; r = -0.161, R^2 = 0.026, p = 0.1099.
ROUGE-L vs passage length
ROUGE-L: negative; r = -0.190, R^2 = 0.036, p = 0.0579.
Trigram precision vs passage length
Trigram precision: negative; r = -0.026, R^2 = 6.90e-04, p = 0.7952.
Trigram recall vs passage length
Trigram recall: negative; r = -0.072, R^2 = 0.005, p = 0.4746.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.047, R^2 = 0.002, p = 0.6438.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.063, R^2 = 0.004, p = 0.5333.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 negative -0.029 8.35e-04 0.7754 -0.00012 ok
chrF++ 100 positive 2.25e-04 5.07e-08 0.9982 6.91e-07 ok
METEOR 100 negative -0.161 0.026 0.1099 -0.00085 ok
ROUGE-L 100 negative -0.190 0.036 0.0579 -0.00079 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 negative -0.026 6.90e-04 0.7952 -0.00012 ok
Trigram recall 100 negative -0.072 0.005 0.4746 -0.00036 ok
Trigram F1 100 negative -0.047 0.002 0.6438 -0.00022 ok
Trigram Jaccard 100 negative -0.063 0.004 0.5333 -0.00021 ok

Translation Similarity Metrics

Mean BLEU-418.4%
Mean chrF++49.7%
Mean METEOR54.3%
Mean ROUGE-L56.1%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU22.2%
Unigram F160.2%
Bigram F129.9%
Trigram precision15.6%
Trigram recall17.1%
Trigram F116.3%
Trigram Jaccard8.9%
4-gram F19.8%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 245 -21.033 11.0% 45.0% 39.0% 46.1% N/A N/A N/A reviewed/approved 12681 gpt-4o-2024-08-06
2346 Κάναι 77 99 19.150 19.9% 45.5% 55.5% 55.4% N/A N/A N/A reviewed/approved 12663 gpt-4o-2024-08-06
2054 Καβαλίς 65 83 14.726 19.4% 51.9% 49.3% 56.8% N/A N/A N/A reviewed/approved 12534 gpt-4o-2024-08-06
2115 Καλὴ ἀκτή 64 81 13.691 8.8% 40.4% 39.7% 44.1% N/A N/A N/A reviewed/approved 12609 gpt-4o-2024-08-06
2119 Κάλλατις 65 81 12.726 11.0% 45.6% 46.9% 43.8% N/A N/A N/A reviewed/approved 12621 gpt-4o-2024-08-06
2604 Καρχηδών 117 131 12.563 12.7% 48.5% 39.5% 45.0% N/A N/A N/A reviewed/approved 12696 gpt-4o-2024-08-06
2056 Καβειρία 112 126 12.386 13.0% 45.2% 38.5% 37.7% N/A N/A N/A reviewed/approved 12540 gpt-4o-2024-08-06
2468 Καπετώλιον 138 127 -11.695 8.9% 37.9% 38.9% 45.1% N/A N/A N/A reviewed/approved 12672 gpt-4o-2024-08-06
2470 Καππαδοκία 68 82 10.832 9.9% 45.8% 63.7% 56.0% N/A N/A N/A reviewed/approved 12675 gpt-4o-2024-08-06
7264 Κωλιάς 61 75 10.585 16.9% 43.2% 37.4% 45.6% N/A N/A N/A reviewed/approved 12822 gpt-4o-2024-08-06
2455 Κάναστρον 60 54 -9.451 13.9% 45.6% 46.6% 52.6% N/A N/A N/A reviewed/approved 12666 gpt-4o-2024-08-06
7254 Κύτα 84 96 9.397 18.2% 53.9% 52.8% 54.4% N/A N/A N/A reviewed/approved 12792 gpt-4o-2024-08-06
2116 Κάληρος 40 35 -9.157 10.4% 39.9% 41.9% 45.3% N/A N/A N/A reviewed/approved 12612 gpt-4o-2024-08-06
2609 Κάσπειρος 135 127 -8.801 27.1% 53.0% 49.4% 58.0% N/A N/A N/A reviewed/approved 12705 gpt-4o-2024-08-06
2055 Καβασσός 109 103 -7.720 11.6% 43.8% 39.3% 41.5% N/A N/A N/A reviewed/approved 12537 gpt-4o-2024-08-06
3288 Κορώνεια 110 119 7.316 19.4% 52.9% 51.4% 60.0% N/A N/A N/A reviewed/approved 12744 gpt-4o-2024-08-06
2599 Καρπήσιοι 11 9 -7.181 16.8% 42.3% 41.3% 50.0% N/A N/A N/A reviewed/approved 12687 gpt-4o-2024-08-06
2057 Καβελλιών 34 45 6.631 13.3% 43.7% 55.8% 50.6% N/A N/A N/A reviewed/approved 12543 gpt-4o-2024-08-06
2465 Κάνωπος 83 92 6.362 27.0% 58.8% 73.7% 67.4% N/A N/A N/A reviewed/approved 12669 gpt-4o-2024-08-06
7255 Κυτέριον 22 33 6.207 12.3% 44.0% 45.9% 47.3% N/A N/A N/A reviewed/approved 12795 gpt-4o-2024-08-06
7246 Κύρις 20 19 -5.863 26.9% 53.5% 50.8% 56.4% N/A N/A N/A reviewed/approved 12768 gpt-4o-2024-08-06
2081 Καλάθη 49 47 -5.839 22.0% 51.0% 56.2% 66.7% N/A N/A N/A reviewed/approved 12585 gpt-4o-2024-08-06
2630 Κάτρη 23 22 -5.757 9.4% 40.7% 38.8% 48.9% N/A N/A N/A reviewed/approved 12738 gpt-4o-2024-08-06
2326 Καλλίαρος 47 56 5.090 12.0% 49.6% 56.3% 60.2% N/A N/A N/A reviewed/approved 12627 gpt-4o-2024-08-06
7249 Κύρρος 45 54 5.020 5.2% 40.8% 28.2% 36.4% N/A N/A N/A reviewed/approved 12777 gpt-4o-2024-08-06
2328 Καλλίπολις 43 52 4.949 16.7% 50.8% 48.0% 54.2% N/A N/A N/A reviewed/approved 12633 gpt-4o-2024-08-06
2078 Καινύς 22 22 -4.793 52.1% 62.3% 67.9% 68.2% N/A N/A N/A reviewed/approved 12573 gpt-4o-2024-08-06
7263 Κώθων 22 22 -4.793 19.5% 55.0% 55.2% 63.6% N/A N/A N/A reviewed/approved 12819 gpt-4o-2024-08-06
7267 Κωνώπη 64 72 4.691 11.1% 45.8% 46.0% 49.6% N/A N/A N/A reviewed/approved 12831 gpt-4o-2024-08-06
3258 Κεκρυφάλεια 26 26 -4.651 45.9% 72.4% 79.8% 80.8% N/A N/A N/A reviewed/approved 12741 gpt-4o-2024-08-06

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.024.

Positive termCoefficientDocsMean presentMean absent
και 4.2755 66 0.630 -1.223
εστι δε 2.4399 5 9.781 -0.515
εν τη 2.4019 7 5.988 -0.451
δυο 2.1813 4 11.394 -0.475
πολιχνιον 2.1041 3 12.275 -0.380
παρα το 2.0997 3 10.389 -0.321
στραβων 2.0988 15 3.674 -0.648
παρα 2.0827 7 3.584 -0.270
απο 1.9803 27 2.809 -1.039
ως 1.7611 52 0.735 -0.796
εστι 1.4203 24 2.532 -0.800
απο του 1.3892 9 4.485 -0.444
τοπος 1.3888 4 5.424 -0.226
εκ 1.3746 6 6.306 -0.403
εστι δε και 1.3713 3 7.609 -0.235
και πολις 1.3711 5 3.632 -0.191
κωμη 1.3186 3 6.620 -0.205
θηλυκον 1.3032 12 3.700 -0.505
εισι 1.2859 5 8.416 -0.443
βοιωτιας 1.2836 3 7.362 -0.228
Negative termCoefficientDocsMean presentMean absent
περι -1.8878 11 -3.268 0.404
ηρωδιανος -1.7150 3 -12.829 0.397
γαρ -1.6375 9 -0.972 0.096
φησιν -1.5418 10 -3.273 0.364
το εθνικον -1.3658 56 -0.605 0.770
εθνικον -1.3658 56 -0.605 0.770
θρακης -1.3625 5 -3.818 0.201
ουτως -1.3146 5 -6.569 0.346
οτι -1.3033 5 -1.735 0.091
εν -1.3023 37 0.204 -0.120
εθνος -1.2342 8 -1.944 0.169
τα -1.1852 12 -1.427 0.195
εκαλειτο -1.1802 7 -3.686 0.277
οικητωρ -1.1625 6 -3.854 0.246
τε -1.1040 5 0.030 -0.002
τινες -1.1006 6 -2.718 0.173
ην -1.0883 13 -0.717 0.107
μετα -1.0640 3 -10.016 0.310
λυκιας -0.9724 3 -4.792 0.148
ιταλιας -0.9310 3 -2.220 0.069

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.164.

Positive termCoefficientDocsMean presentMean absent
γαρ 2.0520 9 9.973 3.641
δε 1.8891 46 5.663 2.973
και 1.7686 66 4.740 3.183
εστι 1.4902 24 6.430 3.510
την 1.1529 13 8.361 3.591
εκαλειτο 1.0426 7 8.307 3.902
απο της 1.0340 6 10.282 3.823
εις 1.0271 7 9.435 3.817
δια 0.9847 16 6.861 3.706
πολιχνιον 0.9533 3 12.275 3.961
παρα 0.9506 7 9.594 3.806
ζευς 0.9282 3 17.293 3.806
εστι δε 0.9224 5 9.781 3.918
δυο 0.9205 4 11.394 3.911
και πολις 0.9195 5 6.171 4.107
παρα το 0.9016 3 10.389 4.020
περι 0.8777 11 6.802 3.890
τα 0.8309 12 8.392 3.640
ακρα 0.8175 4 9.750 3.980
ει 0.8157 4 11.608 3.902
Negative termCoefficientDocsMean presentMean absent
δευτερω -0.9106 3 0.695 4.319
ηροδοτος -0.8532 9 5.139 4.119
το εθνικον -0.7747 56 4.315 4.078
εθνικον -0.7747 56 4.315 4.078
μεταξυ -0.6755 5 3.312 4.258
πολις -0.6583 68 4.340 3.935
προς τη -0.6576 4 1.979 4.304
μια -0.6394 3 1.906 4.282
ην -0.6227 13 3.640 4.296
και το -0.6133 9 3.856 4.246
πολις προς -0.5971 3 2.068 4.277
θεοπομπος -0.5948 5 2.467 4.302
δια το -0.5601 3 1.921 4.281
ιβηριας -0.5529 3 4.346 4.207
αιγυπτου -0.5494 3 1.760 4.286
νησος -0.5435 9 3.377 4.293
οικητωρ -0.4661 6 5.423 4.133
ουδετερως -0.4512 3 2.159 4.274
προς -0.4304 15 4.007 4.247
φασι -0.4256 8 3.939 4.234

AI English terms predicting large absolute residuals

Features: 441; ridge alpha: 2.894; cross-validated R^2: -0.138.

Positive termCoefficientDocsMean presentMean absent
the 2.4168 98 4.243 2.611
from 1.9505 41 6.037 2.941
on 1.4833 13 6.939 3.803
or 1.4171 16 7.886 3.511
with 1.3857 20 7.645 3.352
as 1.3672 49 5.662 2.817
there 1.3220 32 5.875 3.428
of the 1.2987 35 6.059 3.216
form 1.2007 18 7.320 3.528
is 1.0982 88 4.345 3.229
diphthong 1.0605 3 12.447 3.956
was 1.0450 20 6.140 3.728
derives 1.0022 6 8.966 3.907
there is 0.9985 25 5.977 3.622
but 0.9925 7 10.903 3.707
river 0.9876 7 8.259 3.906
on the 0.9560 6 8.923 3.910
strabo in 0.9383 7 8.765 3.868
and 0.9138 57 4.926 3.263
also 0.9056 45 5.156 3.437
Negative termCoefficientDocsMean presentMean absent
similar to -1.0089 18 2.385 4.612
similar -1.0089 18 2.385 4.612
mentioned -0.8812 23 2.765 4.643
are the -0.7302 3 1.140 4.306
the ethnonym -0.7133 6 1.828 4.363
ethnonym -0.7133 6 1.828 4.363
mentioned by -0.6956 19 2.951 4.506
sea -0.6460 7 3.092 4.295
island -0.6340 14 3.209 4.374
are called -0.6293 5 2.644 4.293
the ethnonym is -0.6251 5 1.762 4.340
ethnonym is -0.6251 5 1.762 4.340
is named -0.6022 6 2.440 4.324
a city -0.5883 64 4.142 4.333
which the -0.5819 4 2.085 4.299
mount -0.5775 4 3.206 4.253
second book -0.5588 3 0.547 4.324
inhabitants are -0.5412 3 1.064 4.308
is a city -0.5323 10 4.277 4.203
after -0.5099 18 4.514 4.144

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: -0.062.

Positive termCoefficientDocsMean presentMean absent
the 3.3651 90 4.502 1.587
form 2.5598 11 10.139 3.478
with 2.2007 16 7.649 3.556
like 2.1079 5 10.839 3.862
from the form 2.0313 5 11.669 3.818
is 1.9661 87 4.493 2.320
there is 1.8626 24 6.349 3.536
the form 1.8218 9 11.016 3.538
on 1.6509 16 7.515 3.581
from the 1.6382 13 8.639 3.549
there is also 1.6324 22 6.233 3.640
also 1.6318 43 5.272 3.410
from 1.5336 37 5.953 3.188
and 1.4751 52 5.127 3.218
genitive 1.4698 3 16.162 3.841
is with 1.4499 4 12.056 3.884
on the 1.4007 10 8.264 3.760
is also 1.3834 31 5.880 3.461
there 1.2772 34 6.324 3.122
in the 1.2102 25 5.991 3.617
Negative termCoefficientDocsMean presentMean absent
after -1.2497 22 4.821 4.039
as in -1.1136 29 4.362 4.149
was from -1.0212 4 1.171 4.337
herodotos book -0.9661 5 2.031 4.325
the ethnonym is -0.9246 52 4.123 4.305
ethnonym is -0.9246 52 4.123 4.305
one of the -0.9172 4 1.483 4.324
name -0.8694 7 3.525 4.262
one of -0.8168 5 2.354 4.308
island -0.7866 14 3.209 4.374
one -0.7846 12 4.445 4.179
in -0.7605 87 4.256 3.906
a city in -0.7561 40 4.412 4.077
city in -0.7496 41 4.387 4.088
a city -0.7312 64 4.312 4.030
herodotos -0.7167 9 5.139 4.119
another -0.6976 7 4.198 4.212
some -0.6707 10 3.923 4.243
or after -0.6689 3 1.294 4.301
named -0.6669 5 2.842 4.283

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.