gpt-4o-2024-11-20 v1 Evaluation

Generated: 2026-09-20 11:20:35 UTC

Pairs100

Distinct lemmas100

Slope1.050

R^20.967

BLEU-416.5%

chrF++49.0%

METEOR54.7%

ROUGE-L54.6%

BERTScore91.9%

COMET72.7%

BLEURT64.7%

Profile version id: 2413

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-11-20) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.050
Intercept	4.821
R^2	0.967
P-value	4.18e-74
Pearson r	0.983
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	51.180
Mean absolute length residual	5.142
Mean absolute percent error	21.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.071, R^2 = 0.005, p = 0.4839.

chrF++ vs passage length — chrF++: negative; r = -0.009, R^2 = 8.57e-05, p = 0.9272.

METEOR vs passage length — METEOR: significant negative; r = -0.265, R^2 = 0.070, p = 0.0078.

ROUGE-L vs passage length — ROUGE-L: significant negative; r = -0.233, R^2 = 0.054, p = 0.0199.

BERTScore vs passage length — BERTScore: significant negative; r = -0.305, R^2 = 0.093, p = 0.0020.

COMET vs passage length — COMET: negative; r = -0.097, R^2 = 0.009, p = 0.3378.

BLEURT vs passage length — BLEURT: significant negative; r = -0.648, R^2 = 0.420, p = 3.05e-13.

Trigram precision vs passage length — Trigram precision: negative; r = -0.090, R^2 = 0.008, p = 0.3735.

Trigram recall vs passage length — Trigram recall: negative; r = -0.120, R^2 = 0.014, p = 0.2346.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.103, R^2 = 0.011, p = 0.3101.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.115, R^2 = 0.013, p = 0.2556.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.071	0.005	0.4839	-0.00028	ok
chrF++	100	negative	-0.009	8.57e-05	0.9272	-0.00003	ok
METEOR	100	significant negative	-0.265	0.070	0.0078	-0.00129	ok
ROUGE-L	100	significant negative	-0.233	0.054	0.0199	-0.00095	ok
BERTScore	100	significant negative	-0.305	0.093	0.0020	-0.00027	ok
COMET	100	negative	-0.097	0.009	0.3378	-0.00016	ok
BLEURT	100	significant negative	-0.648	0.420	3.05e-13	-0.00188	ok
Trigram precision	100	negative	-0.090	0.008	0.3735	-0.00038	ok
Trigram recall	100	negative	-0.120	0.014	0.2346	-0.00055	ok
Trigram F1	100	negative	-0.103	0.011	0.3101	-0.00045	ok
Trigram Jaccard	100	negative	-0.115	0.013	0.2556	-0.00034	ok

Translation Similarity Metrics

Mean BLEU-4	16.5%
Mean chrF++	49.0%
Mean METEOR	54.7%
Mean ROUGE-L	54.6%
Mean BERTScore	91.9%
Mean COMET	72.7%
Mean BLEURT	64.7%
Legacy smoothed corpus BLEU	19.2%
Unigram F1	58.4%
Bigram F1	27.3%
Trigram precision	13.2%
Trigram recall	15.4%
Trigram F1	14.3%
Trigram Jaccard	7.7%
4-gram F1	8.0%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2115	Καλὴ ἀκτή	64	106	33.977	8.4%	46.4%	40.5%	40.0%	88.4%	64.9%	48.5%	reviewed/approved	12909	gpt-4o-2024-11-20
3288	Κορώνεια	110	149	28.675	9.0%	48.2%	47.7%	49.2%	90.9%	70.0%	59.4%	reviewed/approved	13044	gpt-4o-2024-11-20
2484	Καρία	270	265	-23.332	9.3%	43.0%	36.0%	40.4%	86.3%	76.0%	44.6%	reviewed/approved	12981	gpt-4o-2024-11-20
2465	Κάνωπος	83	112	20.026	18.9%	55.7%	64.5%	59.5%	91.0%	71.2%	60.4%	reviewed/approved	12969	gpt-4o-2024-11-20
7249	Κύρρος	45	70	17.927	6.8%	47.5%	53.7%	43.5%	90.5%	68.9%	53.7%	reviewed/approved	13077	gpt-4o-2024-11-20
2055	Καβασσός	109	103	-16.275	14.5%	46.3%	32.6%	43.2%	90.7%	73.8%	47.5%	reviewed/approved	12837	gpt-4o-2024-11-20
2468	Καπετώλιον	138	137	-12.726	8.8%	36.3%	35.9%	40.0%	85.7%	67.2%	48.3%	reviewed/approved	12972	gpt-4o-2024-11-20
3496	Κοτιάειον	71	68	-11.374	7.2%	43.7%	51.0%	48.9%	91.0%	69.8%	59.3%	reviewed/approved	13047	gpt-4o-2024-11-20
7266	Κώμη	79	99	11.226	5.2%	38.0%	44.0%	34.8%	89.8%	67.9%	47.9%	reviewed/approved	13128	gpt-4o-2024-11-20
7254	Κύτα	84	104	10.976	19.6%	52.9%	53.7%	54.3%	90.3%	71.4%	54.7%	reviewed/approved	13092	gpt-4o-2024-11-20
2603	Κάρυστος	175	199	10.422	15.9%	54.7%	55.2%	52.4%	92.1%	76.4%	52.2%	reviewed/approved	12993	gpt-4o-2024-11-20
2627	Κατάνειρα	21	37	10.128	6.4%	42.7%	47.1%	51.7%	91.9%	68.1%	68.1%	reviewed/approved	13026	gpt-4o-2024-11-20
2625	Κατακεκαυμένη	46	63	9.877	11.0%	49.4%	57.5%	49.5%	90.6%	70.8%	66.1%	reviewed/approved	13023	gpt-4o-2024-11-20
2455	Κάναστρον	60	58	-9.823	11.6%	49.5%	50.1%	45.8%	92.1%	72.6%	56.0%	reviewed/approved	12966	gpt-4o-2024-11-20
7244	Κυρήνη	54	71	9.477	22.5%	55.8%	54.1%	56.0%	94.1%	73.0%	62.4%	reviewed/approved	13062	gpt-4o-2024-11-20
2081	Καλάθη	49	47	-9.273	28.3%	54.4%	60.7%	66.7%	93.4%	69.1%	66.1%	reviewed/approved	12885	gpt-4o-2024-11-20
2802	Κάδρεμα	24	21	-9.022	8.5%	43.6%	55.4%	62.2%	94.3%	72.5%	70.8%	reviewed/approved	12858	gpt-4o-2024-11-20
2470	Καππαδοκία	68	85	8.777	7.6%	46.8%	65.1%	60.1%	92.0%	71.1%	56.0%	reviewed/approved	12975	gpt-4o-2024-11-20
2114	Κάλβιος	18	16	-7.721	4.8%	33.4%	42.1%	41.2%	91.7%	68.9%	62.0%	reviewed/approved	12906	gpt-4o-2024-11-20
2626	Κατάνη	95	97	-7.575	20.1%	51.4%	40.4%	50.0%	91.2%	71.6%	58.2%	reviewed/approved	13029	gpt-4o-2024-11-20
2346	Κάναι	77	93	7.326	25.6%	53.2%	70.5%	70.6%	91.8%	74.1%	56.8%	reviewed/approved	12963	gpt-4o-2024-11-20
2328	Καλλίπολις	43	57	7.028	7.0%	43.5%	41.9%	48.0%	89.9%	68.5%	55.3%	reviewed/approved	12933	gpt-4o-2024-11-20
2078	Καινύς	22	21	-6.922	63.1%	70.7%	77.4%	79.1%	96.2%	82.9%	80.4%	reviewed/approved	12873	gpt-4o-2024-11-20
2079	Καισάρεια	39	39	-6.772	26.0%	51.9%	59.6%	61.5%	91.9%	74.2%	65.3%	reviewed/approved	12879	gpt-4o-2024-11-20
7256	Κύτινα	32	32	-6.422	47.3%	66.3%	69.7%	71.9%	94.7%	71.6%	75.4%	reviewed/approved	13098	gpt-4o-2024-11-20
7247	Κύρνος	50	51	-6.323	16.6%	46.9%	58.5%	51.5%	92.7%	75.0%	68.3%	reviewed/approved	13071	gpt-4o-2024-11-20
2087	Καλαύρεια	30	30	-6.322	25.0%	56.6%	65.8%	73.3%	93.2%	78.1%	71.9%	reviewed/approved	12903	gpt-4o-2024-11-20
3258	Κεκρυφάλεια	26	26	-6.122	52.0%	76.6%	79.8%	80.8%	97.2%	78.9%	80.1%	reviewed/approved	13041	gpt-4o-2024-11-20
2330	Καλύβη	24	24	-6.022	34.8%	54.3%	80.7%	75.0%	93.2%	73.4%	73.3%	reviewed/approved	12939	gpt-4o-2024-11-20
2326	Καλλίαρος	47	60	5.827	12.0%	49.5%	59.3%	55.6%	93.8%	68.8%	62.2%	reviewed/approved	12927	gpt-4o-2024-11-20

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.181.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	3.6294	66	0.560	-1.088
παρα	2.8337	7	6.389	-0.481
κωμη	2.7654	3	15.410	-0.477
εστι δε	2.4121	5	12.516	-0.659
δυο	2.3183	4	13.189	-0.550
ως	2.1789	52	0.669	-0.724
παρα το	2.1259	3	13.793	-0.427
δια του	2.0008	10	4.101	-0.456
βοιωτιας	1.9787	3	10.026	-0.310
θηλυκον	1.7865	12	5.234	-0.714
εκ	1.7663	6	7.452	-0.476
εστι	1.7099	24	3.007	-0.950
θεοπομπος	1.7075	5	4.207	-0.221
ου	1.6381	14	3.708	-0.604
πολυβιος	1.5985	5	3.888	-0.205
την	1.5881	13	1.117	-0.167
και νησος	1.4466	3	11.159	-0.345
οι	1.4387	11	4.413	-0.545
δευτερα	1.4255	3	12.194	-0.377
στραβων	1.4215	15	3.617	-0.638

Negative term	Coefficient	Docs	Mean present	Mean absent
δε	-2.4629	46	0.123	-0.105
εκαλειτο	-2.1808	7	-7.097	0.534
λυκιας	-2.0258	3	-11.006	0.340
περι	-2.0113	11	-5.324	0.658
το εθνικον	-1.9799	56	-1.110	1.413
εθνικον	-1.9799	56	-1.110	1.413
δε της	-1.6984	6	-8.124	0.519
τα	-1.5949	12	-3.850	0.525
τινες δε	-1.4108	3	-8.406	0.260
φησιν	-1.3075	10	-5.875	0.653
καππαδοκιας	-1.3038	3	-7.023	0.217
και του	-1.2620	4	-7.223	0.301
το	-1.2204	70	0.477	-1.112
τους	-1.1783	4	-4.335	0.181
εθνος	-1.1431	8	-3.180	0.276
ιταλιας	-1.0929	3	-3.788	0.117
ην	-1.0392	13	-2.235	0.334
μια	-1.0195	3	-4.372	0.135
ειναι	-1.0074	8	-2.917	0.254
ηρωδιανος	-1.0054	3	-13.010	0.402

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: 0.046.

Positive term	Coefficient	Docs	Mean present	Mean absent
δε	2.8527	46	7.401	3.218
δια του	2.8493	10	13.392	4.225
και	2.7669	66	6.250	2.991
γαρ	2.7436	9	13.849	4.281
εστι	2.6992	24	9.192	3.863
δια	2.5824	16	10.348	4.150
παρα	2.0414	7	13.919	4.481
την	1.9074	13	8.525	4.636
κωμη	1.6633	3	15.410	4.824
εστι και	1.4978	20	9.067	4.161
της	1.3813	35	7.123	4.075
εστι δε	1.3768	5	13.586	4.697
εις	1.3684	7	14.104	4.467
παρα το	1.2401	3	13.793	4.874
του	1.2367	37	8.081	3.416
δυο	1.2034	4	13.189	4.807
των	1.1635	17	8.440	4.466
λυκιας	1.1517	3	11.006	4.961
θηλυκον	1.1465	12	9.755	4.513
δυναται	1.1452	3	16.343	4.795

Negative term	Coefficient	Docs	Mean present	Mean absent
παυσανιας	-1.2589	4	1.010	5.314
πλησιον	-1.1518	8	2.767	5.348
ηροδοτος	-1.0588	9	4.677	5.188
δευτερω	-1.0474	3	0.993	5.270
το εθνικον	-1.0412	56	5.004	5.318
εθνικον	-1.0412	56	5.004	5.318
εκαταιος ασια	-0.9880	5	0.843	5.368
ασια	-0.9880	5	0.843	5.368
παιδος	-0.8960	4	2.665	5.245
διονυσιος	-0.8083	4	1.876	5.278
τε	-0.7517	5	4.396	5.181
αφ	-0.7162	6	4.116	5.207
πολις και	-0.6944	6	3.557	5.243
του δε	-0.6789	3	2.541	5.222
πολις προς	-0.6496	3	4.391	5.165
οικητωρ	-0.6434	6	6.919	5.028
τους	-0.6187	4	4.335	5.176
φησι το	-0.6045	3	0.958	5.271
τοις	-0.5993	3	2.308	5.230
ενατω	-0.5811	3	1.608	5.251

AI English terms predicting large absolute residuals

Features: 477; ridge alpha: 2.894; cross-validated R^2: 0.023.

Positive term	Coefficient	Docs	Mean present	Mean absent
this	2.5869	16	10.459	4.129
inhabitant of	2.2378	3	13.957	4.869
a	2.1934	92	5.303	3.293
be	2.1702	7	17.961	4.177
or	2.0145	21	10.628	3.684
this name	1.9789	5	15.627	4.590
with	1.9750	27	9.518	3.523
is	1.8524	87	5.471	2.942
inhabitant	1.7966	11	8.662	4.707
from	1.6944	37	8.217	3.336
name	1.6881	22	9.059	4.037
of	1.6615	86	5.436	3.335
is a	1.6551	13	11.569	4.182
is in	1.6240	3	14.576	4.850
term	1.5732	7	11.120	4.692
that	1.5466	19	9.304	4.166
i	1.5380	4	17.740	4.617
with the	1.5379	9	12.521	4.412
village	1.5151	4	13.075	4.811
the	1.5061	97	5.144	5.088

Negative term	Coefficient	Docs	Mean present	Mean absent
in the	-1.4775	44	5.204	5.093
are	-1.1784	24	4.583	5.318
book the	-1.0614	9	1.665	5.486
ethnic designation is	-1.0612	28	4.424	5.421
designation is	-1.0612	28	4.424	5.421
pausanias	-0.9920	4	1.010	5.314
the ethnic designation	-0.9422	29	4.833	5.268
ethnic designation	-0.9422	29	4.833	5.268
tribe	-0.9246	3	1.571	5.252
asia	-0.8700	6	0.965	5.409
mentioned by	-0.8557	15	2.655	5.581
mentioned	-0.8092	19	3.088	5.624
similar to	-0.7975	8	2.811	5.345
similar	-0.7225	9	3.657	5.289
are called	-0.6868	6	2.976	5.280
analogous to	-0.6700	12	4.796	5.189
analogous	-0.6700	12	4.796	5.189
inhabitants	-0.6672	11	3.059	5.399
genitive	-0.6314	5	5.796	5.107
noted by	-0.6237	5	1.885	5.313

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: 0.065.

Positive term	Coefficient	Docs	Mean present	Mean absent
with	3.3457	16	10.695	4.084
is	2.7953	87	5.523	2.593
of	2.6610	66	6.152	3.181
there is	2.5844	24	9.515	3.761
there	2.4855	34	8.507	3.408
and	2.1014	52	6.508	3.663
from	1.7591	37	7.907	3.518
the	1.7464	90	5.343	3.330
this	1.6295	15	10.721	4.157
like	1.6258	5	14.167	4.667
is also	1.6241	31	7.967	3.872
there is also	1.5756	22	8.577	4.173
or	1.5732	18	10.475	3.971
has	1.5137	6	13.241	4.625
territory	1.4246	5	12.316	4.764
as well	1.3176	5	14.625	4.643
as well as	1.3176	5	14.625	4.643
well	1.3176	5	14.625	4.643
well as	1.3176	5	14.625	4.643
however	1.2683	10	10.960	4.496

Negative term	Coefficient	Docs	Mean present	Mean absent
near	-1.1694	16	3.820	5.394
in book of	-1.1675	14	2.786	5.525
in book	-1.1270	16	4.038	5.352
pausanias	-1.0160	4	1.010	5.314
asia	-0.9966	6	0.965	5.409
the ethnonym	-0.9762	56	5.009	5.311
dionysios	-0.9688	4	1.876	5.278
are	-0.9575	25	6.591	4.659
herodotos	-0.9324	9	4.677	5.188
also called	-0.8656	6	2.535	5.308
in his asia	-0.8477	5	0.843	5.368
his asia	-0.8477	5	0.843	5.368
ethnonym	-0.8434	57	5.207	5.056
the ethnonym is	-0.8426	52	5.056	5.235
ethnonym is	-0.8426	52	5.056	5.235
herodotos book	-0.7868	5	1.395	5.339
pausanias book	-0.7683	3	1.006	5.270
among	-0.7660	4	6.339	5.092
that of the	-0.7294	3	1.122	5.266
that of	-0.7294	3	1.122	5.266

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.