gpt-5.6-sol v1 Evaluation

Generated: 2026-07-25 12:54:09 UTC

Pairs100
Distinct lemmas100
Slope0.998
R^20.974
BLEU-423.3%
chrF++51.8%
METEOR56.0%
ROUGE-L57.7%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2175

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.6 Sol using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.998
Intercept3.126
R^20.974
P-value1.50e-79
Pearson r0.987
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words47.190
Mean absolute length residual4.295
Mean absolute percent error13.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: negative; r = -0.085, R^2 = 0.007, p = 0.4001.
chrF++ vs passage length
chrF++: positive; r = 0.019, R^2 = 3.56e-04, p = 0.8523.
METEOR vs passage length
METEOR: negative; r = -0.107, R^2 = 0.011, p = 0.2914.
ROUGE-L vs passage length
ROUGE-L: negative; r = -0.079, R^2 = 0.006, p = 0.4340.
Trigram precision vs passage length
Trigram precision: negative; r = -0.065, R^2 = 0.004, p = 0.5213.
Trigram recall vs passage length
Trigram recall: negative; r = -0.074, R^2 = 0.005, p = 0.4662.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.068, R^2 = 0.005, p = 0.4992.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.095, R^2 = 0.009, p = 0.3448.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 negative -0.085 0.007 0.4001 -0.00039 ok
chrF++ 100 positive 0.019 3.56e-04 0.8523 0.00007 ok
METEOR 100 negative -0.107 0.011 0.2914 -0.00059 ok
ROUGE-L 100 negative -0.079 0.006 0.4340 -0.00039 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 negative -0.065 0.004 0.5213 -0.00031 ok
Trigram recall 100 negative -0.074 0.005 0.4662 -0.00037 ok
Trigram F1 100 negative -0.068 0.005 0.4992 -0.00034 ok
Trigram Jaccard 100 negative -0.095 0.009 0.3448 -0.00032 ok

Translation Similarity Metrics

Mean BLEU-423.3%
Mean chrF++51.8%
Mean METEOR56.0%
Mean ROUGE-L57.7%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU25.0%
Unigram F161.3%
Bigram F132.9%
Trigram precision18.7%
Trigram recall20.1%
Trigram F119.4%
Trigram Jaccard10.7%
4-gram F111.5%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 244 -28.539 19.1% 50.7% 43.8% 51.4% N/A N/A N/A reviewed/approved 10605 gpt-5.6-sol
3258 Κεκρυφάλεια 26 1 -28.069 0.0% 0.2% 0.0% 0.0% N/A N/A N/A reviewed/approved 10665 gpt-5.6-sol
2605 Κάσιον 56 74 14.996 14.8% 48.6% 46.0% 46.2% N/A N/A N/A reviewed/approved 10623 gpt-5.6-sol
2604 Καρχηδών 117 133 13.128 20.3% 53.8% 57.3% 55.6% N/A N/A N/A reviewed/approved 10620 gpt-5.6-sol
3288 Κορώνεια 110 126 13.113 15.1% 51.2% 43.0% 53.2% N/A N/A N/A reviewed/approved 10668 gpt-5.6-sol
2328 Καλλίπολις 43 58 11.967 14.9% 51.6% 56.0% 55.4% N/A N/A N/A reviewed/approved 10557 gpt-5.6-sol
2621 Κάστνιον 25 40 11.928 21.6% 52.5% 60.2% 49.2% N/A N/A N/A reviewed/approved 10635 gpt-5.6-sol
2116 Κάληρος 40 32 -11.039 16.4% 51.3% 56.4% 55.6% N/A N/A N/A reviewed/approved 10536 gpt-5.6-sol
2115 Καλὴ ἀκτή 64 78 11.013 17.1% 46.5% 59.5% 57.7% N/A N/A N/A reviewed/approved 10533 gpt-5.6-sol
2455 Κάναστρον 60 52 -10.996 13.7% 47.5% 56.1% 60.7% N/A N/A N/A reviewed/approved 10590 gpt-5.6-sol
2468 Καπετώλιον 138 150 9.174 16.5% 44.9% 45.5% 47.2% N/A N/A N/A reviewed/approved 10596 gpt-5.6-sol
7266 Κώμη 79 90 8.046 7.2% 38.7% 33.9% 37.9% N/A N/A N/A reviewed/approved 10752 gpt-5.6-sol
2119 Κάλλατις 65 76 8.015 16.1% 47.2% 47.2% 44.0% N/A N/A N/A reviewed/approved 10545 gpt-5.6-sol
2056 Καβειρία 112 107 -7.883 12.5% 46.0% 37.7% 42.0% N/A N/A N/A reviewed/approved 10466 gpt-5.6-sol
2603 Κάρυστος 175 185 7.254 31.6% 61.4% 67.5% 68.3% N/A N/A N/A reviewed/approved 10617 gpt-5.6-sol
2470 Καππαδοκία 68 78 7.022 12.8% 48.5% 54.1% 61.6% N/A N/A N/A reviewed/approved 10599 gpt-5.6-sol
7247 Κύρνος 50 46 -7.017 14.5% 42.7% 53.6% 52.1% N/A N/A N/A reviewed/approved 10695 gpt-5.6-sol
2342 Κάμιρος 54 64 6.991 27.4% 61.3% 70.9% 67.8% N/A N/A N/A reviewed/approved 10581 gpt-5.6-sol
2055 Καβασσός 109 118 6.111 16.7% 46.4% 39.2% 45.8% N/A N/A N/A reviewed/approved 10463 gpt-5.6-sol
7246 Κύρις 20 17 -6.083 25.9% 47.9% 43.7% 48.6% N/A N/A N/A reviewed/approved 10692 gpt-5.6-sol
7251 Κύρτος 66 75 6.017 36.9% 58.1% 68.4% 65.2% N/A N/A N/A reviewed/approved 10707 gpt-5.6-sol
7264 Κωλιάς 61 70 6.007 26.7% 54.5% 55.6% 59.5% N/A N/A N/A reviewed/approved 10746 gpt-5.6-sol
7259 Κύφος 54 63 5.991 20.9% 52.0% 61.2% 58.1% N/A N/A N/A reviewed/approved 10731 gpt-5.6-sol
2624 Κατάβαθμος 20 29 5.917 43.5% 73.2% 82.1% 69.4% N/A N/A N/A reviewed/approved 10644 gpt-5.6-sol
2078 Καινύς 22 20 -5.078 40.2% 55.6% 57.9% 57.1% N/A N/A N/A reviewed/approved 10497 gpt-5.6-sol
7262 Κῶβρυς 22 20 -5.078 8.2% 43.1% 29.2% 42.9% N/A N/A N/A reviewed/approved 10740 gpt-5.6-sol
7257 Κυτώνιον 25 23 -5.072 31.4% 55.6% 60.5% 66.7% N/A N/A N/A reviewed/approved 10725 gpt-5.6-sol
7261 Κύψελα 34 32 -5.052 31.4% 57.8% 59.7% 63.6% N/A N/A N/A reviewed/approved 10737 gpt-5.6-sol
2346 Κάναι 77 85 5.041 33.4% 60.0% 74.2% 74.1% N/A N/A N/A reviewed/approved 10587 gpt-5.6-sol
7249 Κύρρος 45 53 4.972 4.1% 42.0% 52.6% 49.0% N/A N/A N/A reviewed/approved 10701 gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.082.

Positive termCoefficientDocsMean presentMean absent
ου 2.0620 14 3.832 -0.624
της 1.7855 35 1.300 -0.700
του 1.6521 37 2.182 -1.281
ως 1.6428 52 1.253 -1.358
εν 1.6149 37 2.441 -1.433
εστι δε 1.5535 5 6.868 -0.361
εν τη 1.5454 7 5.979 -0.450
ου και 1.4914 4 8.809 -0.367
ορος 1.4696 6 6.716 -0.429
κωμη 1.4224 3 8.009 -0.248
και 1.4190 66 0.708 -1.375
εστι δε και 1.4063 3 10.403 -0.322
πολις 1.3862 68 0.289 -0.614
πολιτης 1.3671 18 3.885 -0.853
αφ ου 1.2795 5 4.900 -0.258
εξ 1.1974 3 7.386 -0.228
πολιχνιον 1.1654 3 8.341 -0.258
δευτερα 1.1594 3 8.337 -0.258
εκ 1.1575 6 5.333 -0.340
τον 1.0572 10 3.424 -0.380
Negative termCoefficientDocsMean presentMean absent
τινες -2.4277 6 -7.699 0.491
τινες δε -2.1747 3 -11.709 0.362
περι -2.0986 11 -3.326 0.411
το -2.0188 70 -0.201 0.469
φασι -1.8770 8 -3.721 0.324
πελοποννησου -1.7718 3 -6.021 0.186
τις -1.7297 3 -5.677 0.176
το εθνικον -1.6349 56 -1.523 1.938
εθνικον -1.6349 56 -1.523 1.938
εθνος -1.4829 8 -5.980 0.520
θρακης -1.4789 5 -5.455 0.287
εστι και -1.2961 20 -0.558 0.140
ακρα -1.2917 4 -3.726 0.155
το θηλυκον -1.1922 6 -5.742 0.367
ουτως -1.1824 5 -7.855 0.413
θεοπομπος -1.1457 5 -2.823 0.149
αυτην -1.0575 9 -1.655 0.164
πολις θρακης -0.8783 4 -4.070 0.170
εκαλειτο -0.8652 7 -2.735 0.206
οτι -0.7915 5 0.036 -0.002

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 8.377; cross-validated R^2: -0.299.

Positive termCoefficientDocsMean presentMean absent
δε 1.4418 46 5.920 2.910
περι 1.1524 11 9.721 3.624
πελοποννησου 1.0179 3 14.763 3.971
τις 0.9693 3 14.419 3.982
ακρα 0.8813 4 10.309 4.044
γαρ 0.8739 9 8.893 3.840
τινες 0.8404 6 9.024 3.993
εκαλειτο 0.8082 7 10.305 3.843
τινες δε 0.7971 3 11.709 4.066
και 0.7861 66 4.723 3.463
την 0.7712 13 7.560 3.807
απο της 0.6786 6 11.644 3.826
φασι 0.6780 8 8.813 3.902
εστι δε 0.5627 5 10.021 3.994
ου 0.5561 14 6.102 4.001
ει 0.5511 4 13.899 3.895
παρα 0.5403 7 10.068 3.860
δια 0.5379 16 6.461 3.882
εστι 0.5144 24 5.721 3.845
τα 0.5125 12 8.329 3.745
Negative termCoefficientDocsMean presentMean absent
δευτερω -0.7278 3 0.374 4.416
πολις -0.6639 68 4.063 4.788
εκαταιος -0.5619 14 3.122 4.486
τον -0.5133 10 3.677 4.364
στραβων -0.4597 15 3.490 4.437
εκαταιος ασια -0.3985 5 1.479 4.443
ασια -0.3985 5 1.479 4.443
νησος -0.3557 9 2.794 4.443
πολιτης -0.3529 18 4.474 4.256
οι -0.3521 11 4.286 4.296
ηροδοτος -0.3298 9 5.644 4.161
λυκιας -0.3255 3 3.091 4.332
ως στραβων -0.3068 4 1.773 4.400
και το -0.3043 9 4.364 4.288
ποσειδωνος -0.2912 3 1.667 4.376
φησι το -0.2870 3 1.037 4.396
προς τη -0.2788 4 2.529 4.368
πλησιον -0.2784 8 3.027 4.405
παυσανιας -0.2757 4 3.462 4.330
τριτω -0.2666 3 1.357 4.386

AI English terms predicting large absolute residuals

Features: 401; ridge alpha: 5.878; cross-validated R^2: -0.311.

Positive termCoefficientDocsMean presentMean absent
with 1.0693 19 6.731 3.724
and 1.0210 56 5.204 3.138
for 0.9282 12 7.414 3.870
usage 0.8987 5 10.063 3.991
caria 0.8742 3 14.533 3.978
on 0.8625 26 6.163 3.638
was called 0.7804 6 11.162 3.857
was 0.7681 24 6.675 3.543
diphthong 0.7251 3 13.277 4.017
he 0.6221 7 8.385 3.987
it 0.5883 40 5.637 3.400
is a 0.5815 6 11.114 3.860
by 0.5488 16 6.556 3.864
either 0.5479 3 17.570 3.884
was named 0.5289 5 10.897 3.947
it was named 0.5269 4 11.807 3.982
zeus 0.5227 5 13.001 3.837
it was 0.5202 13 7.357 3.837
from 0.4974 36 5.698 3.506
a mountain 0.4971 3 10.655 4.098
Negative termCoefficientDocsMean presentMean absent
a city -0.7200 66 3.792 5.272
book -0.6122 39 3.862 4.572
an -0.5648 24 3.350 4.593
of the -0.5421 40 4.468 4.179
city -0.5052 68 4.063 4.788
herodotus book -0.5012 5 1.639 4.435
is called -0.4932 24 3.885 4.424
hecataeus -0.4846 14 3.122 4.486
are called -0.4833 9 3.214 4.402
inhabitants -0.4138 6 2.186 4.430
of his -0.4126 10 3.022 4.436
are -0.3974 26 4.676 4.161
is called a -0.3903 21 3.825 4.420
called a -0.3903 21 3.825 4.420
settlement -0.3894 3 1.376 4.385
strabo -0.3891 15 3.490 4.437
book of the -0.3849 8 2.393 4.460
book of -0.3768 16 4.363 4.282
near -0.3746 14 3.452 4.432
in his asia -0.3699 5 1.479 4.443

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: -0.217.

Positive termCoefficientDocsMean presentMean absent
the 2.5393 90 4.542 2.073
however 2.5010 10 11.403 3.505
in the peloponnese 1.6846 3 14.763 3.971
peloponnese 1.6846 3 14.763 3.971
the peloponnese 1.6846 3 14.763 3.971
which the 1.5356 3 12.368 4.045
on 1.5037 16 6.552 3.865
this 1.4539 15 8.756 3.508
and 1.3585 52 5.295 3.211
promontory 1.3555 3 12.065 4.055
is with 1.3291 4 13.232 3.923
mountain 1.2473 3 11.393 4.075
his on 1.2275 3 15.508 3.948
used to be 1.2266 6 11.512 3.834
an island in 1.2192 3 10.988 4.088
island in 1.2192 3 10.988 4.088
island in the 1.2192 3 10.988 4.088
karia 1.2147 3 14.533 3.978
say that it 1.2140 3 11.709 4.066
with 1.1785 16 6.629 3.850
Negative termCoefficientDocsMean presentMean absent
a city -0.9858 64 3.848 5.090
in book -0.9468 16 3.865 4.377
in book of -0.9429 14 2.161 4.642
strabo -0.8524 15 3.490 4.437
book -0.8372 39 3.862 4.572
city -0.8023 70 4.105 4.738
but -0.7987 12 3.074 4.461
hekataios -0.7848 14 3.122 4.486
is a -0.7718 31 4.712 4.108
herodotos book -0.7409 5 1.639 4.435
are -0.7320 25 5.392 3.929
they -0.6923 12 4.801 4.226
near -0.6875 16 3.665 4.415
lykia -0.6470 3 3.091 4.332
in lykia -0.6470 3 3.091 4.332
written -0.6208 3 3.301 4.326
hekataios in -0.6184 12 3.128 4.454
they are -0.6179 7 3.469 4.357
one of -0.6022 5 2.217 4.404
was from -0.5914 4 2.540 4.368

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.