gpt-5.1 v1 Evaluation

Generated: 2026-09-20 11:21:13 UTC

Pairs100

Distinct lemmas100

Slope1.130

R^20.987

BLEU-420.3%

chrF++51.9%

METEOR59.5%

ROUGE-L60.2%

BERTScore92.2%

COMET71.9%

BLEURT63.9%

Profile version id: 2346

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.1 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.130
Intercept	0.367
R^2	0.987
P-value	4.64e-94
Pearson r	0.993
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	50.270
Mean absolute length residual	3.793
Mean absolute percent error	15.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.128, R^2 = 0.016, p = 0.2055.

chrF++ vs passage length — chrF++: positive; r = 0.111, R^2 = 0.012, p = 0.2708.

METEOR vs passage length — METEOR: negative; r = -0.063, R^2 = 0.004, p = 0.5306.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.139, R^2 = 0.019, p = 0.1690.

BERTScore vs passage length — BERTScore: significant negative; r = -0.294, R^2 = 0.087, p = 0.0029.

COMET vs passage length — COMET: positive; r = 0.012, R^2 = 1.36e-04, p = 0.9084.

BLEURT vs passage length — BLEURT: significant negative; r = -0.631, R^2 = 0.399, p = 1.90e-12.

Trigram precision vs passage length — Trigram precision: positive; r = 0.048, R^2 = 0.002, p = 0.6385.

Trigram recall vs passage length — Trigram recall: positive; r = 0.061, R^2 = 0.004, p = 0.5442.

Trigram F1 vs passage length — Trigram F1: positive; r = 0.056, R^2 = 0.003, p = 0.5822.

Trigram Jaccard vs passage length — Trigram Jaccard: positive; r = 0.030, R^2 = 9.13e-04, p = 0.7654.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.128	0.016	0.2055	0.00052	ok
chrF++	100	positive	0.111	0.012	0.2708	0.00032	ok
METEOR	100	negative	-0.063	0.004	0.5306	-0.00028	ok
ROUGE-L	100	negative	-0.139	0.019	0.1690	-0.00054	ok
BERTScore	100	significant negative	-0.294	0.087	0.0029	-0.00023	ok
COMET	100	positive	0.012	1.36e-04	0.9084	0.00002	ok
BLEURT	100	significant negative	-0.631	0.399	1.90e-12	-0.00179	ok
Trigram precision	100	positive	0.048	0.002	0.6385	0.00021	ok
Trigram recall	100	positive	0.061	0.004	0.5442	0.00029	ok
Trigram F1	100	positive	0.056	0.003	0.5822	0.00026	ok
Trigram Jaccard	100	positive	0.030	9.13e-04	0.7654	0.00009	ok

Translation Similarity Metrics

Mean BLEU-4	20.3%
Mean chrF++	51.9%
Mean METEOR	59.5%
Mean ROUGE-L	60.2%
Mean BERTScore	92.2%
Mean COMET	71.9%
Mean BLEURT	63.9%
Legacy smoothed corpus BLEU	27.0%
Unigram F1	63.8%
Bigram F1	35.1%
Trigram precision	20.6%
Trigram recall	23.5%
Trigram F1	21.9%
Trigram Jaccard	12.3%
4-gram F1	14.0%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
3288	Κορώνεια	110	144	19.299	18.1%	50.8%	49.1%	48.6%	90.1%	72.4%	49.5%	reviewed/approved	12143	gpt-5.1-2025-11-13
2597	Καρπασία	112	111	-15.962	26.2%	57.6%	55.5%	66.4%	94.0%	73.7%	59.1%	reviewed/approved	12083	gpt-5.1-2025-11-13
2056	Καβειρία	112	115	-11.962	11.6%	53.6%	53.6%	54.6%	91.6%	73.4%	52.1%	reviewed/approved	11939	gpt-5.1-2025-11-13
7267	Κωνώπη	64	84	11.293	25.6%	49.9%	52.2%	52.7%	90.9%	64.9%	64.1%	reviewed/approved	12230	gpt-5.1-2025-11-13
2455	Κάναστρον	60	57	-11.185	18.8%	48.1%	58.9%	59.8%	92.0%	66.6%	58.5%	reviewed/approved	12065	gpt-5.1-2025-11-13
2470	Καππαδοκία	68	88	10.772	17.1%	51.2%	65.7%	61.5%	92.7%	71.8%	56.4%	reviewed/approved	12074	gpt-5.1-2025-11-13
7266	Κώμη	79	100	10.339	9.4%	39.9%	36.1%	35.8%	88.0%	63.2%	42.1%	reviewed/approved	12227	gpt-5.1-2025-11-13
2115	Καλὴ ἀκτή	64	83	10.293	12.9%	46.6%	69.9%	54.4%	89.7%	66.6%	52.3%	reviewed/approved	12008	gpt-5.1-2025-11-13
7264	Κωλιάς	61	78	8.684	17.1%	46.8%	47.6%	53.2%	90.5%	68.2%	50.8%	reviewed/approved	12221	gpt-5.1-2025-11-13
2609	Κάσπειρος	135	145	-7.959	38.9%	63.0%	67.2%	65.7%	89.4%	70.7%	57.1%	reviewed/approved	12104	gpt-5.1-2025-11-13
2622	Καστωλοῦ πεδίον	42	40	-7.840	23.3%	56.2%	76.3%	75.6%	94.0%	76.6%	64.2%	reviewed/approved	12113	gpt-5.1-2025-11-13
2055	Καβασσός	109	116	-7.571	16.5%	49.6%	47.1%	48.0%	91.3%	72.6%	49.0%	reviewed/approved	11936	gpt-5.1-2025-11-13
2621	Κάστνιον	25	36	7.375	14.5%	49.0%	65.9%	55.7%	92.7%	60.4%	64.4%	reviewed/approved	12110	gpt-5.1-2025-11-13
2328	Καλλίπολις	43	56	7.030	8.8%	46.8%	48.1%	52.5%	92.4%	72.4%	63.1%	reviewed/approved	12032	gpt-5.1-2025-11-13
2085	Καλάσιρις	13	22	6.939	5.3%	37.9%	25.2%	40.0%	89.3%	67.6%	66.0%	reviewed/approved	11996	gpt-5.1-2025-11-13
2602	Καρύανδα	34	32	-6.797	30.7%	55.4%	64.0%	63.6%	95.1%	73.5%	72.1%	reviewed/approved	12089	gpt-5.1-2025-11-13
2626	Κατάνη	95	101	-6.746	20.9%	50.2%	48.8%	52.0%	90.2%	70.9%	51.0%	reviewed/approved	12128	gpt-5.1-2025-11-13
7250	Κυρταία	40	39	-6.579	23.7%	46.6%	54.2%	60.8%	91.3%	62.0%	62.8%	reviewed/approved	12179	gpt-5.1-2025-11-13
7248	Κύρου πόλις	32	43	6.463	11.3%	44.1%	44.1%	45.3%	88.7%	65.6%	60.5%	reviewed/approved	12173	gpt-5.1-2025-11-13
2087	Καλαύρεια	30	28	-6.276	37.4%	60.8%	65.0%	69.0%	95.2%	81.3%	74.5%	reviewed/approved	12002	gpt-5.1-2025-11-13
7257	Κυτώνιον	25	23	-5.625	31.4%	52.5%	58.8%	58.3%	91.4%	68.8%	60.3%	reviewed/approved	12200	gpt-5.1-2025-11-13
7256	Κύτινα	32	31	-5.537	37.5%	64.5%	73.3%	76.2%	93.8%	72.4%	73.0%	reviewed/approved	12197	gpt-5.1-2025-11-13
2330	Καλύβη	24	22	-5.494	10.7%	42.5%	56.3%	65.2%	92.6%	66.8%	65.5%	reviewed/approved	12038	gpt-5.1-2025-11-13
7253	Κυρτώνιος	18	26	5.288	4.2%	41.1%	51.0%	45.5%	91.7%	71.2%	68.8%	reviewed/approved	12188	gpt-5.1-2025-11-13
7252	Κύρτωνες	27	36	5.115	4.0%	38.8%	41.9%	43.8%	89.6%	67.2%	57.3%	reviewed/approved	12185	gpt-5.1-2025-11-13
7245	Κύρης	13	20	4.939	6.6%	31.4%	46.1%	42.4%	85.2%	48.9%	51.8%	reviewed/approved	12164	gpt-5.1-2025-11-13
2465	Κάνωπος	83	99	4.817	28.6%	59.4%	63.0%	65.9%	92.4%	72.2%	58.4%	reviewed/approved	12068	gpt-5.1-2025-11-13
7254	Κύτα	84	100	4.687	18.1%	50.1%	51.6%	56.5%	89.7%	71.0%	52.4%	reviewed/approved	12191	gpt-5.1-2025-11-13
2625	Κατακεκαυμένη	46	57	4.639	26.2%	58.3%	70.8%	62.1%	92.8%	73.6%	67.3%	reviewed/approved	12122	gpt-5.1-2025-11-13
7259	Κύφος	54	66	4.596	17.3%	56.3%	72.7%	66.7%	91.8%	69.8%	54.6%	reviewed/approved	12206	gpt-5.1-2025-11-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.169.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.0012	66	-0.185	0.358
κωμη	0.0008	3	8.221	-0.254
παρα	0.0008	7	4.987	-0.375
εις	0.0007	7	4.587	-0.345
την	0.0007	13	0.834	-0.125
βοιωτιας	0.0007	3	4.151	-0.128
του	0.0006	37	0.582	-0.342
δευτερω	0.0006	3	3.302	-0.102
πολυβιος	0.0005	5	3.629	-0.191
λεγεται	0.0005	9	3.713	-0.367
δευτερα	0.0005	3	8.525	-0.264
δυο	0.0005	4	3.794	-0.158
δια του	0.0004	10	2.292	-0.255
τριτη	0.0004	3	7.960	-0.246
εστι δε	0.0004	5	3.809	-0.200
γαρ	0.0004	9	2.489	-0.246
ου και	0.0004	4	3.807	-0.159
οι	0.0004	11	2.353	-0.291
λεγεται και	0.0004	8	3.673	-0.319
ειρηται	0.0004	3	6.843	-0.212

Negative term	Coefficient	Docs	Mean present	Mean absent
ην	-0.0009	13	-3.901	0.583
φησιν	-0.0007	10	-2.268	0.252
αυτην	-0.0007	9	-2.383	0.236
οτι	-0.0006	5	-5.688	0.299
θρακης	-0.0006	5	-3.615	0.190
εκαλειτο	-0.0006	7	-2.365	0.178
λυκιας	-0.0006	3	-4.926	0.152
πλησιον	-0.0006	8	-2.030	0.177
μια	-0.0005	3	-4.275	0.132
θεοπομπος	-0.0005	5	-3.944	0.208
δε της	-0.0005	6	-4.609	0.294
αφ	-0.0005	6	-4.595	0.293
τα	-0.0005	12	-1.365	0.186
το κτητικον	-0.0005	7	-4.162	0.313
κτητικον	-0.0005	7	-4.162	0.313
αυτην φησι	-0.0005	3	-6.643	0.205
και πολις	-0.0005	5	-2.613	0.138
ελλανικος	-0.0004	4	-5.394	0.225
αφ ου	-0.0004	5	-5.519	0.290
και το κτητικον	-0.0004	4	-4.662	0.194

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.010.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	3.4782	66	4.657	2.114
την	1.9227	13	6.337	3.412
παρα	1.8858	7	8.545	3.435
βοιωτιας	1.7634	3	12.125	3.535
δε	1.4824	46	5.103	2.676
εις	1.2258	7	6.852	3.562
φησιν	1.1363	10	5.819	3.567
κωμη	1.0319	3	8.221	3.656
αλλ	1.0263	3	9.972	3.601
τριτω	0.9751	3	9.276	3.623
ου	0.9634	14	6.035	3.427
εστι δε	0.9631	5	9.639	3.485
τα	0.9262	12	5.630	3.542
εστι	0.8677	24	5.890	3.130
μεταξυ	0.8667	5	7.332	3.606
της	0.8528	35	4.603	3.356
τριτη	0.8266	3	9.593	3.613
τοις	0.8063	3	8.421	3.649
δυναται	0.7866	3	10.212	3.594
ακρα	0.7734	4	8.200	3.609

Negative term	Coefficient	Docs	Mean present	Mean absent
στραβων	-1.4530	15	2.613	4.001
εκαταιος ασια	-0.8884	5	0.988	3.940
ασια	-0.8884	5	0.988	3.940
προς τω	-0.8663	4	0.965	3.910
θηλυκον	-0.7315	12	3.891	3.779
ακροπολις	-0.7226	3	1.748	3.856
μητροπολις	-0.7069	3	2.012	3.848
ωστε	-0.6908	3	1.601	3.860
περι	-0.6679	11	2.662	3.932
του	-0.6500	37	4.091	3.617
και θηλυκον	-0.5440	3	1.629	3.859
απο της	-0.5386	6	2.303	3.888
και πολις	-0.5385	5	2.747	3.848
εθνος	-0.5325	8	3.167	3.847
λιβυης	-0.5223	4	2.832	3.833
δε και	-0.5207	8	4.471	3.734
τω	-0.5083	16	3.215	3.903
παιδος	-0.4997	4	2.972	3.827
εν τω	-0.4961	6	2.604	3.868
ως στραβων	-0.4845	4	2.404	3.850

AI English terms predicting large absolute residuals

Features: 470; ridge alpha: 2.894; cross-validated R^2: -0.034.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	2.5750	55	4.823	2.533
is a city	1.5211	5	7.886	3.577
one	1.4317	21	5.609	3.310
promontory	1.3796	4	12.550	3.428
should	1.3771	4	12.195	3.442
also	1.3118	50	5.251	2.334
is a	1.2644	12	6.278	3.454
to	1.2259	29	5.813	2.967
calls	1.1684	3	8.170	3.657
be	0.9839	10	8.238	3.299
says	0.9543	27	5.238	3.258
third	0.9216	7	9.224	3.384
as	0.9051	46	4.557	3.141
from	0.9039	48	4.942	2.732
if	0.8907	3	13.644	3.488
this	0.8673	16	5.606	3.447
into	0.8593	3	10.967	3.571
calls it	0.8156	3	8.170	3.657
is formed	0.7929	3	8.757	3.639
formed	0.7901	6	6.975	3.589

Negative term	Coefficient	Docs	Mean present	Mean absent
is called	-0.9632	13	2.046	4.054
strabo	-0.7751	15	2.613	4.001
book	-0.6319	41	3.617	3.915
he	-0.5933	6	1.703	3.926
comes	-0.5551	5	2.235	3.875
pontos	-0.5422	3	0.827	3.884
hecataeus in	-0.5255	9	2.270	3.943
herodotus in	-0.5186	4	1.149	3.903
the citizen	-0.5176	8	2.403	3.913
herodotus in the	-0.5100	3	0.715	3.888
citizen is called	-0.5031	3	0.384	3.898
are called	-0.4960	7	2.971	3.854
near the	-0.4880	3	0.991	3.879
book and	-0.4872	3	1.286	3.870
was	-0.4830	20	3.755	3.802
asia	-0.4620	6	2.817	3.855
acropolis	-0.4568	3	1.748	3.856
city the	-0.4350	3	2.514	3.832
is the	-0.4319	6	2.635	3.867
either from	-0.4250	3	1.523	3.863

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: -0.003.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	3.1224	52	5.027	2.455
from	1.8872	37	5.240	2.942
however	1.7791	10	7.995	3.326
boiotia	1.7436	3	12.125	3.535
as	1.6282	60	4.302	3.029
they	1.5972	12	5.987	3.493
but	1.4235	12	7.270	3.318
if	1.4208	3	13.644	3.488
also a	1.4050	15	5.973	3.408
for	1.3623	11	6.444	3.465
is from	1.3609	12	6.526	3.420
promontory	1.3234	3	9.467	3.617
is also a	1.3175	14	6.259	3.391
also	1.1944	43	4.947	2.922
into	1.1843	3	10.967	3.571
from the form	1.1227	5	7.367	3.604
all	1.1133	3	11.429	3.556
has	1.1035	6	7.789	3.537
both	1.0919	5	8.996	3.519
between	1.0716	6	7.905	3.530

Negative term	Coefficient	Docs	Mean present	Mean absent
strabo	-1.3149	15	2.613	4.001
in his asia	-0.8181	5	0.988	3.940
his asia	-0.8181	5	0.988	3.940
after	-0.8012	22	4.425	3.614
the city	-0.7271	6	2.298	3.888
most	-0.6583	4	2.517	3.846
near	-0.6446	16	2.832	3.976
metropolis	-0.6427	3	2.012	3.848
a metropolis	-0.6427	3	2.012	3.848
near the	-0.6304	4	1.331	3.895
just	-0.6134	6	2.572	3.870
the acropolis of	-0.6001	3	1.748	3.856
the acropolis	-0.6001	3	1.748	3.856
acropolis of	-0.6001	3	1.748	3.856
acropolis	-0.6001	3	1.748	3.856
the black sea	-0.5953	4	1.525	3.887
the black	-0.5953	4	1.525	3.887
black sea	-0.5953	4	1.525	3.887
black	-0.5953	4	1.525	3.887
sanctuary	-0.5835	3	2.168	3.843

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.