gpt-5.1 v1 Evaluation

Generated: 2026-07-25 12:53:15 UTC

Pairs100

Distinct lemmas100

Slope1.131

R^20.987

BLEU-420.3%

chrF++51.9%

METEOR59.5%

ROUGE-L60.2%

BERTScoreN/A

COMETN/A

BLEURTN/A

Profile version id: 2346

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.1 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.131
Intercept	0.345
R^2	0.987
P-value	5.31e-94
Pearson r	0.993
Mean source words	30.400
Length fallback rows	0
Mean human words	44.160
Mean AI words	50.270
Mean absolute length residual	3.802
Mean absolute percent error	15.9%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.127, R^2 = 0.016, p = 0.2067.

chrF++ vs passage length — chrF++: positive; r = 0.113, R^2 = 0.013, p = 0.2626.

METEOR vs passage length — METEOR: negative; r = -0.060, R^2 = 0.004, p = 0.5502.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.137, R^2 = 0.019, p = 0.1738.

Trigram precision vs passage length — Trigram precision: positive; r = 0.048, R^2 = 0.002, p = 0.6385.

Trigram recall vs passage length — Trigram recall: positive; r = 0.063, R^2 = 0.004, p = 0.5322.

Trigram F1 vs passage length — Trigram F1: positive; r = 0.057, R^2 = 0.003, p = 0.5744.

Trigram Jaccard vs passage length — Trigram Jaccard: positive; r = 0.031, R^2 = 9.81e-04, p = 0.7570.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.127	0.016	0.2067	0.00051	ok
chrF++	100	positive	0.113	0.013	0.2626	0.00033	ok
METEOR	100	negative	-0.060	0.004	0.5502	-0.00027	ok
ROUGE-L	100	negative	-0.137	0.019	0.1738	-0.00054	ok
BERTScore	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
COMET	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
BLEURT	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
Trigram precision	100	positive	0.048	0.002	0.6385	0.00021	ok
Trigram recall	100	positive	0.063	0.004	0.5322	0.00030	ok
Trigram F1	100	positive	0.057	0.003	0.5744	0.00026	ok
Trigram Jaccard	100	positive	0.031	9.81e-04	0.7570	0.00010	ok

Translation Similarity Metrics

Mean BLEU-4	20.3%
Mean chrF++	51.9%
Mean METEOR	59.5%
Mean ROUGE-L	60.2%
Mean BERTScore	N/A
Mean COMET	N/A
Mean BLEURT	N/A
Legacy smoothed corpus BLEU	27.0%
Unigram F1	63.8%
Bigram F1	35.1%
Trigram precision	20.6%
Trigram recall	23.5%
Trigram F1	21.9%
Trigram Jaccard	12.3%
4-gram F1	14.0%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
3288	Κορώνεια	110	144	19.295	18.1%	50.8%	49.1%	48.6%	N/A	N/A	N/A	reviewed/approved	12143	gpt-5.1-2025-11-13
2597	Καρπασία	112	111	-15.966	26.2%	57.6%	55.5%	66.4%	N/A	N/A	N/A	reviewed/approved	12083	gpt-5.1-2025-11-13
2056	Καβειρία	112	115	-11.966	11.6%	53.6%	53.6%	54.6%	N/A	N/A	N/A	reviewed/approved	11939	gpt-5.1-2025-11-13
7267	Κωνώπη	64	84	11.300	25.6%	49.9%	52.2%	52.7%	N/A	N/A	N/A	reviewed/approved	12230	gpt-5.1-2025-11-13
2455	Κάναστρον	60	57	-11.178	18.8%	48.1%	58.9%	59.8%	N/A	N/A	N/A	reviewed/approved	12065	gpt-5.1-2025-11-13
2470	Καππαδοκία	68	88	10.778	17.1%	51.2%	65.7%	61.5%	N/A	N/A	N/A	reviewed/approved	12074	gpt-5.1-2025-11-13
7266	Κώμη	79	100	10.342	9.4%	39.9%	36.1%	35.8%	N/A	N/A	N/A	reviewed/approved	12227	gpt-5.1-2025-11-13
2115	Καλὴ ἀκτή	64	83	10.300	12.9%	46.6%	69.9%	54.4%	N/A	N/A	N/A	reviewed/approved	12008	gpt-5.1-2025-11-13
7264	Κωλιάς	61	78	8.692	17.1%	46.8%	47.6%	53.2%	N/A	N/A	N/A	reviewed/approved	12221	gpt-5.1-2025-11-13
2609	Κάσπειρος	135	145	-7.968	38.9%	63.0%	67.2%	65.7%	N/A	N/A	N/A	reviewed/approved	12104	gpt-5.1-2025-11-13
2622	Καστωλοῦ πεδίον	42	40	-7.828	23.3%	56.2%	76.3%	75.6%	N/A	N/A	N/A	reviewed/approved	12113	gpt-5.1-2025-11-13
2055	Καβασσός	109	116	-7.574	16.5%	49.6%	47.1%	48.0%	N/A	N/A	N/A	reviewed/approved	11936	gpt-5.1-2025-11-13
2621	Κάστνιον	25	36	7.391	14.5%	49.0%	65.9%	55.7%	N/A	N/A	N/A	reviewed/approved	12110	gpt-5.1-2025-11-13
2328	Καλλίπολις	43	56	7.041	8.8%	46.8%	48.1%	52.5%	N/A	N/A	N/A	reviewed/approved	12032	gpt-5.1-2025-11-13
2085	Καλάσιρις	13	22	6.958	5.3%	37.9%	25.2%	40.0%	N/A	N/A	N/A	reviewed/approved	11996	gpt-5.1-2025-11-13
2602	Καρύανδα	34	32	-6.784	30.7%	55.4%	64.0%	63.6%	N/A	N/A	N/A	reviewed/approved	12089	gpt-5.1-2025-11-13
2626	Κατάνη	95	101	-6.747	20.9%	50.2%	48.8%	52.0%	N/A	N/A	N/A	reviewed/approved	12128	gpt-5.1-2025-11-13
7250	Κυρταία	40	39	-6.567	23.7%	46.6%	54.2%	60.8%	N/A	N/A	N/A	reviewed/approved	12179	gpt-5.1-2025-11-13
7248	Κύρου πόλις	32	43	6.477	11.3%	44.1%	44.1%	45.3%	N/A	N/A	N/A	reviewed/approved	12173	gpt-5.1-2025-11-13
2087	Καλαύρεια	30	28	-6.262	37.4%	60.8%	65.0%	69.0%	N/A	N/A	N/A	reviewed/approved	12002	gpt-5.1-2025-11-13
7257	Κυτώνιον	25	23	-5.609	31.4%	52.5%	58.8%	58.3%	N/A	N/A	N/A	reviewed/approved	12200	gpt-5.1-2025-11-13
7256	Κύτινα	32	31	-5.523	37.5%	64.5%	73.3%	76.2%	N/A	N/A	N/A	reviewed/approved	12197	gpt-5.1-2025-11-13
2330	Καλύβη	24	22	-5.478	10.7%	42.5%	56.3%	65.2%	N/A	N/A	N/A	reviewed/approved	12038	gpt-5.1-2025-11-13
7253	Κυρτώνιος	18	26	5.305	4.2%	41.1%	51.0%	45.5%	N/A	N/A	N/A	reviewed/approved	12188	gpt-5.1-2025-11-13
7252	Κύρτωνες	27	36	5.130	4.0%	38.8%	41.9%	43.8%	N/A	N/A	N/A	reviewed/approved	12185	gpt-5.1-2025-11-13
7245	Κύρης	13	20	4.958	6.6%	31.4%	46.1%	42.4%	N/A	N/A	N/A	reviewed/approved	12164	gpt-5.1-2025-11-13
2465	Κάνωπος	83	99	4.820	28.6%	59.4%	63.0%	65.9%	N/A	N/A	N/A	reviewed/approved	12068	gpt-5.1-2025-11-13
7254	Κύτα	84	100	4.689	17.7%	49.5%	50.9%	55.4%	N/A	N/A	N/A	reviewed/approved	12191	gpt-5.1-2025-11-13
2625	Κατακεκαυμένη	46	57	4.650	26.2%	58.3%	70.8%	62.1%	N/A	N/A	N/A	reviewed/approved	12122	gpt-5.1-2025-11-13
7259	Κύφος	54	66	4.606	17.3%	56.3%	72.7%	66.7%	N/A	N/A	N/A	reviewed/approved	12206	gpt-5.1-2025-11-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 10000.000; cross-validated R^2: -0.169.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.0012	66	-0.176	0.342
κωμη	0.0008	3	8.228	-0.254
παρα	0.0008	7	4.986	-0.375
εις	0.0007	7	4.579	-0.345
την	0.0007	13	0.831	-0.124
βοιωτιας	0.0007	3	4.153	-0.128
του	0.0006	37	0.588	-0.345
δευτερω	0.0006	3	3.320	-0.103
πολυβιος	0.0005	5	3.642	-0.192
λεγεται	0.0005	9	3.712	-0.367
δευτερα	0.0005	3	8.532	-0.264
δυο	0.0005	4	3.800	-0.158
δια του	0.0004	10	2.294	-0.255
τριτη	0.0004	3	7.967	-0.246
εστι δε	0.0004	5	3.811	-0.201
γαρ	0.0004	9	2.482	-0.246
οι	0.0004	11	2.359	-0.292
ου και	0.0004	4	3.809	-0.159
λεγεται και	0.0004	8	3.671	-0.319
ειρηται	0.0004	3	6.852	-0.212

Negative term	Coefficient	Docs	Mean present	Mean absent
ην	-0.0009	13	-3.893	0.582
φησιν	-0.0007	10	-2.269	0.252
αυτην	-0.0007	9	-2.376	0.235
οτι	-0.0006	5	-5.684	0.299
θρακης	-0.0006	5	-3.601	0.190
εκαλειτο	-0.0006	7	-2.370	0.178
λυκιας	-0.0006	3	-4.916	0.152
πλησιον	-0.0006	8	-2.018	0.175
μια	-0.0005	3	-4.262	0.132
θεοπομπος	-0.0005	5	-3.933	0.207
δε της	-0.0005	6	-4.609	0.294
αφ	-0.0005	6	-4.594	0.293
τα	-0.0005	12	-1.367	0.186
το κτητικον	-0.0005	7	-4.163	0.313
κτητικον	-0.0005	7	-4.163	0.313
αυτην φησι	-0.0005	3	-6.646	0.206
και πολις	-0.0005	5	-2.603	0.137
ποταμου	-0.0004	3	-3.321	0.103
ελλανικος	-0.0004	4	-5.403	0.225
αφ ου	-0.0004	5	-5.521	0.291

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.003.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	3.4504	66	4.655	2.147
την	1.9196	13	6.336	3.424
παρα	1.8829	7	8.542	3.445
βοιωτιας	1.7641	3	12.130	3.545
δε	1.4702	46	5.104	2.693
εις	1.2244	7	6.844	3.573
φησιν	1.1354	10	5.819	3.578
κωμη	1.0311	3	8.228	3.665
αλλ	1.0267	3	9.975	3.611
τριτω	0.9750	3	9.279	3.633
εστι δε	0.9614	5	9.644	3.495
ου	0.9561	14	6.035	3.439
τα	0.9216	12	5.627	3.553
μεταξυ	0.8535	5	7.322	3.617
εστι	0.8499	24	5.886	3.144
της	0.8354	35	4.599	3.373
τριτη	0.8270	3	9.591	3.623
τοις	0.8074	3	8.424	3.659
δυναται	0.7879	3	10.217	3.604
ακρα	0.7710	4	8.198	3.619

Negative term	Coefficient	Docs	Mean present	Mean absent
στραβων	-1.4621	15	2.610	4.012
εκαταιος ασια	-0.9065	5	0.981	3.951
ασια	-0.9065	5	0.981	3.951
προς τω	-0.8733	4	0.962	3.920
θηλυκον	-0.7364	12	3.887	3.791
ακροπολις	-0.7220	3	1.745	3.866
μητροπολις	-0.7115	3	2.004	3.858
ωστε	-0.6960	3	1.593	3.870
περι	-0.6777	11	2.655	3.944
του	-0.6621	37	4.091	3.632
και πολις	-0.5461	5	2.741	3.858
και θηλυκον	-0.5426	3	1.631	3.869
απο της	-0.5425	6	2.292	3.898
λιβυης	-0.5249	4	2.828	3.843
δε και	-0.5215	8	4.467	3.744
τω	-0.5171	16	3.213	3.914
εν τω	-0.4999	6	2.595	3.879
παιδος	-0.4981	4	2.977	3.836
ως στραβων	-0.4875	4	2.401	3.860
δια	-0.4624	16	4.357	3.696

AI English terms predicting large absolute residuals

Features: 470; ridge alpha: 2.894; cross-validated R^2: -0.041.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	2.5573	55	4.822	2.556
is a city	1.5222	5	7.885	3.587
one	1.4278	21	5.609	3.322
should	1.3774	4	12.196	3.452
promontory	1.3773	4	12.545	3.438
also	1.2957	50	5.250	2.354
is a	1.2604	12	6.275	3.465
to	1.2159	29	5.811	2.982
calls	1.1677	3	8.164	3.667
be	0.9845	10	8.240	3.309
says	0.9478	27	5.238	3.271
third	0.9213	7	9.226	3.394
from	0.8979	48	4.942	2.749
as	0.8971	46	4.556	3.160
if	0.8912	3	13.646	3.498
this	0.8638	16	5.601	3.459
into	0.8550	3	10.961	3.581
calls it	0.8147	3	8.164	3.667
is formed	0.7912	3	8.755	3.649
formed	0.7878	6	6.968	3.600

Negative term	Coefficient	Docs	Mean present	Mean absent
is called	-0.9702	13	2.043	4.065
strabo	-0.7816	15	2.610	4.012
book	-0.6461	41	3.618	3.930
he	-0.5926	6	1.702	3.936
comes	-0.5548	5	2.237	3.884
pontos	-0.5509	3	0.812	3.895
hecataeus in	-0.5401	9	2.266	3.954
the citizen	-0.5245	8	2.398	3.924
herodotus in	-0.5205	4	1.135	3.913
herodotus in the	-0.5117	3	0.710	3.898
citizen is called	-0.5074	3	0.375	3.908
near the	-0.4989	3	0.985	3.889
are called	-0.4948	7	2.980	3.864
was	-0.4894	20	3.750	3.815
book and	-0.4883	3	1.289	3.880
asia	-0.4759	6	2.812	3.865
acropolis	-0.4558	3	1.745	3.866
city the	-0.4344	3	2.515	3.842
is the	-0.4344	6	2.623	3.877
near	-0.4257	16	3.224	3.912

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: -0.009.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	3.1037	52	5.026	2.476
from	1.8729	37	5.240	2.958
however	1.7773	10	7.991	3.337
boiotia	1.7456	3	12.130	3.545
as	1.6152	60	4.301	3.054
they	1.5839	12	5.986	3.504
if	1.4219	3	13.646	3.498
but	1.4181	12	7.273	3.329
also a	1.4034	15	5.968	3.420
is from	1.3620	12	6.527	3.431
for	1.3579	11	6.445	3.475
promontory	1.3209	3	9.466	3.627
is also a	1.3173	14	6.255	3.403
also	1.1836	43	4.945	2.940
into	1.1718	3	10.961	3.581
from the form	1.1207	5	7.366	3.614
all	1.1150	3	11.435	3.566
has	1.1017	6	7.786	3.548
both	1.0925	5	9.000	3.528
in boiotia	1.0580	3	12.130	3.545

Negative term	Coefficient	Docs	Mean present	Mean absent
strabo	-1.3132	15	2.610	4.012
in his asia	-0.8370	5	0.981	3.951
his asia	-0.8370	5	0.981	3.951
after	-0.7896	22	4.424	3.627
the city	-0.7315	6	2.288	3.899
most	-0.6620	4	2.506	3.856
near	-0.6465	16	2.828	3.988
metropolis	-0.6462	3	2.004	3.858
a metropolis	-0.6462	3	2.004	3.858
near the	-0.6416	4	1.322	3.905
just	-0.6129	6	2.573	3.881
the black sea	-0.6038	4	1.512	3.897
the black	-0.6038	4	1.512	3.897
black sea	-0.6038	4	1.512	3.897
black	-0.6038	4	1.512	3.897
the acropolis of	-0.5985	3	1.745	3.866
the acropolis	-0.5985	3	1.745	3.866
acropolis of	-0.5985	3	1.745	3.866
acropolis	-0.5985	3	1.745	3.866
sanctuary	-0.5845	3	2.175	3.852

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.