gpt-5 v1 Evaluation

Generated: 2026-09-20 11:21:00 UTC

Pairs100

Distinct lemmas100

Slope0.981

R^20.980

BLEU-420.7%

chrF++51.9%

METEOR60.5%

ROUGE-L61.8%

BERTScore92.4%

COMET71.9%

BLEURT63.4%

Profile version id: 2343

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.981
Intercept	2.856
R^2	0.980
P-value	7.51e-85
Pearson r	0.990
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	46.170
Mean absolute length residual	4.077
Mean absolute percent error	10.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.056, R^2 = 0.003, p = 0.5810.

chrF++ vs passage length — chrF++: positive; r = 0.042, R^2 = 0.002, p = 0.6787.

METEOR vs passage length — METEOR: negative; r = -0.126, R^2 = 0.016, p = 0.2122.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.128, R^2 = 0.016, p = 0.2055.

BERTScore vs passage length — BERTScore: significant negative; r = -0.247, R^2 = 0.061, p = 0.0134.

COMET vs passage length — COMET: negative; r = -0.029, R^2 = 8.21e-04, p = 0.7771.

BLEURT vs passage length — BLEURT: significant negative; r = -0.628, R^2 = 0.394, p = 2.80e-12.

Trigram precision vs passage length — Trigram precision: positive; r = 0.024, R^2 = 5.71e-04, p = 0.8135.

Trigram recall vs passage length — Trigram recall: positive; r = 0.009, R^2 = 7.25e-05, p = 0.9330.

Trigram F1 vs passage length — Trigram F1: positive; r = 0.018, R^2 = 3.08e-04, p = 0.8625.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.011, R^2 = 1.19e-04, p = 0.9141.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.056	0.003	0.5810	0.00024	ok
chrF++	100	positive	0.042	0.002	0.6787	0.00013	ok
METEOR	100	negative	-0.126	0.016	0.2122	-0.00058	ok
ROUGE-L	100	negative	-0.128	0.016	0.2055	-0.00047	ok
BERTScore	100	significant negative	-0.247	0.061	0.0134	-0.00017	ok
COMET	100	negative	-0.029	8.21e-04	0.7771	-0.00005	ok
BLEURT	100	significant negative	-0.628	0.394	2.80e-12	-0.00181	ok
Trigram precision	100	positive	0.024	5.71e-04	0.8135	0.00011	ok
Trigram recall	100	positive	0.009	7.25e-05	0.9330	0.00004	ok
Trigram F1	100	positive	0.018	3.08e-04	0.8625	0.00008	ok
Trigram Jaccard	100	negative	-0.011	1.19e-04	0.9141	-0.00003	ok

Translation Similarity Metrics

Mean BLEU-4	20.7%
Mean chrF++	51.9%
Mean METEOR	60.5%
Mean ROUGE-L	61.8%
Mean BERTScore	92.4%
Mean COMET	71.9%
Mean BLEURT	63.4%
Legacy smoothed corpus BLEU	27.8%
Unigram F1	65.5%
Bigram F1	35.6%
Trigram precision	21.0%
Trigram recall	22.0%
Trigram F1	21.5%
Trigram Jaccard	12.0%
4-gram F1	13.0%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	248	-19.744	22.0%	51.2%	49.4%	57.5%	89.9%	75.8%	49.9%	reviewed/approved	11780	gpt-5-2025-08-07
2603	Κάρυστος	175	190	15.457	27.9%	59.0%	64.2%	62.5%	93.1%	77.3%	52.4%	reviewed/approved	11792	gpt-5-2025-08-07
2342	Κάμιρος	54	70	14.166	25.4%	63.6%	73.4%	72.6%	93.9%	73.8%	62.7%	reviewed/approved	11756	gpt-5-2025-08-07
7266	Κώμη	79	94	13.640	5.6%	40.0%	37.7%	37.0%	89.9%	65.9%	43.1%	reviewed/approved	11927	gpt-5-2025-08-07
2604	Καρχηδών	117	131	13.359	20.7%	54.8%	61.9%	60.0%	91.6%	76.4%	56.1%	reviewed/approved	11795	gpt-5-2025-08-07
2055	Καβασσός	109	97	-12.792	15.4%	44.0%	32.6%	45.6%	91.5%	71.9%	39.4%	reviewed/approved	11636	gpt-5-2025-08-07
3288	Κορώνεια	110	123	12.227	16.5%	52.1%	50.2%	54.7%	90.8%	71.1%	49.6%	reviewed/approved	11843	gpt-5-2025-08-07
2115	Καλὴ ἀκτή	64	77	11.356	13.8%	36.6%	46.1%	44.0%	88.0%	61.9%	48.7%	reviewed/approved	11708	gpt-5-2025-08-07
2056	Καβειρία	112	102	-10.735	21.2%	53.3%	56.1%	60.7%	92.8%	71.2%	58.2%	reviewed/approved	11639	gpt-5-2025-08-07
2468	Καπετώλιον	138	129	-9.243	14.1%	41.0%	40.0%	48.7%	86.2%	64.2%	44.9%	reviewed/approved	11771	gpt-5-2025-08-07
2116	Κάληρος	40	34	-8.099	12.1%	45.4%	51.9%	54.1%	92.5%	72.5%	67.6%	reviewed/approved	11711	gpt-5-2025-08-07
2625	Κατακεκαυμένη	46	56	8.015	15.8%	56.6%	70.0%	60.8%	92.3%	72.3%	67.6%	reviewed/approved	11822	gpt-5-2025-08-07
2328	Καλλίπολις	43	53	7.958	20.5%	53.5%	56.5%	58.3%	91.6%	72.4%	60.5%	reviewed/approved	11732	gpt-5-2025-08-07
2465	Κάνωπος	83	92	7.716	20.4%	54.9%	65.5%	64.0%	92.8%	75.2%	59.1%	reviewed/approved	11768	gpt-5-2025-08-07
2621	Κάστνιον	25	35	7.617	24.1%	48.0%	56.5%	56.7%	92.8%	60.0%	67.4%	reviewed/approved	11810	gpt-5-2025-08-07
7243	Κύρη	22	32	7.561	17.4%	46.1%	38.7%	55.6%	90.7%	71.3%	62.5%	reviewed/approved	11858	gpt-5-2025-08-07
2477	Καρδαμύλη	43	52	6.958	17.9%	54.6%	73.4%	63.2%	91.5%	71.5%	64.9%	reviewed/approved	11777	gpt-5-2025-08-07
7251	Κύρτος	66	61	-6.606	29.2%	56.4%	67.8%	67.2%	91.6%	68.8%	50.8%	reviewed/approved	11882	gpt-5-2025-08-07
2470	Καππαδοκία	68	76	6.432	15.4%	50.5%	72.3%	68.1%	92.8%	68.9%	54.3%	reviewed/approved	11774	gpt-5-2025-08-07
2330	Καλύβη	24	20	-6.401	8.4%	39.4%	56.8%	63.6%	91.1%	68.4%	60.0%	reviewed/approved	11738	gpt-5-2025-08-07
2079	Καισάρεια	39	35	-6.118	36.7%	59.5%	66.7%	70.3%	92.4%	73.5%	65.7%	reviewed/approved	11678	gpt-5-2025-08-07
7245	Κύρης	13	21	5.390	5.2%	28.9%	34.8%	41.2%	87.7%	56.8%	65.4%	reviewed/approved	11864	gpt-5-2025-08-07
2329	Κάλπη	57	64	5.223	16.4%	51.3%	65.2%	70.5%	92.2%	70.5%	67.6%	reviewed/approved	11735	gpt-5-2025-08-07
2057	Καβελλιών	34	31	-5.212	24.0%	58.8%	67.7%	76.9%	94.0%	69.3%	68.3%	reviewed/approved	11642	gpt-5-2025-08-07
7259	Κύφος	54	61	5.166	16.9%	51.5%	64.2%	60.9%	91.9%	67.9%	52.4%	reviewed/approved	11906	gpt-5-2025-08-07
7250	Κυρταία	40	37	-5.099	17.5%	45.2%	54.5%	59.7%	90.5%	66.3%	62.4%	reviewed/approved	11879	gpt-5-2025-08-07
2081	Καλάθη	49	46	-4.928	22.0%	48.3%	43.8%	54.7%	91.8%	64.6%	55.3%	reviewed/approved	11684	gpt-5-2025-08-07
7254	Κύτα	84	90	4.734	27.5%	55.7%	61.2%	62.1%	92.1%	72.4%	55.6%	reviewed/approved	11891	gpt-5-2025-08-07
2455	Κάναστρον	60	57	-4.720	20.0%	47.7%	52.9%	54.7%	92.2%	66.0%	59.8%	reviewed/approved	11765	gpt-5-2025-08-07
7252	Κύρτωνες	27	34	4.655	9.7%	44.7%	53.7%	49.2%	89.6%	69.5%	59.2%	reviewed/approved	11885	gpt-5-2025-08-07

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 49.239; cross-validated R^2: -0.347.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.2843	66	0.210	-0.407
κωμη	0.2142	3	10.651	-0.329
τε	0.1911	5	7.738	-0.407
λεγεται	0.1783	9	4.941	-0.489
εστι δε	0.1628	5	6.633	-0.349
δε και	0.1600	8	3.613	-0.314
λεγεται και	0.1541	8	4.688	-0.408
οι	0.1535	11	4.193	-0.518
απο	0.1498	27	1.860	-0.688
εστι δε και	0.1471	3	10.848	-0.336
δυο	0.1282	4	6.465	-0.269
ου και	0.1223	4	7.753	-0.323
ου	0.1204	14	1.861	-0.303
ορος	0.1198	6	5.675	-0.362
ως	0.1076	52	0.313	-0.339
εδει	0.1066	3	6.344	-0.196
νησος	0.0997	9	2.399	-0.237
παιδος	0.0986	4	4.751	-0.198
θηλυκως	0.0959	5	5.729	-0.302
και θηλυκως	0.0959	5	5.729	-0.302

Negative term	Coefficient	Docs	Mean present	Mean absent
καππαδοκιας	-0.1403	3	-6.437	0.199
το	-0.1333	70	0.007	-0.017
και του	-0.1304	4	-6.197	0.258
περι	-0.1220	11	-3.155	0.390
ως το	-0.1084	3	-10.045	0.311
λυκιας	-0.1065	3	-5.903	0.183
εθνος	-0.1025	8	-4.210	0.366
θρακης	-0.1012	5	-2.639	0.139
τους	-0.0981	4	-2.499	0.104
μεταξυ	-0.0909	5	-2.003	0.105
κατα	-0.0905	10	-0.866	0.096
το εθνικον	-0.0822	56	-0.402	0.512
εθνικον	-0.0822	56	-0.402	0.512
ελλανικος	-0.0802	4	-8.309	0.346
πολις θρακης	-0.0801	4	-2.118	0.088
οτι	-0.0767	5	-0.644	0.034
ηρωδιανος	-0.0744	3	-10.136	0.313
πλησιον	-0.0736	8	-0.606	0.053
ης	-0.0723	3	-4.072	0.126
πρωτη	-0.0682	3	-8.307	0.257

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: 0.053.

Positive term	Coefficient	Docs	Mean present	Mean absent
δε	3.5595	46	5.780	2.627
και	2.8004	66	4.798	2.679
γαρ	2.6996	9	10.330	3.459
εκαλειτο	2.3095	7	10.423	3.600
εστι	1.9710	24	6.590	3.284
κωμη	1.9180	3	10.651	3.874
εν	1.8881	37	5.502	3.241
της	1.8604	35	5.734	3.185
εν τω	1.8167	6	9.977	3.701
την	1.7628	13	7.836	3.516
εστι δε	1.7390	5	10.927	3.717
ομηρος	1.5530	3	11.306	3.854
εκαλειτο δε	1.5400	3	16.187	3.703
παρα	1.5160	7	9.385	3.678
δε και	1.5041	8	10.079	3.555
τω	1.4913	16	6.089	3.694
πολις εν	1.3752	5	8.367	3.852
λεγεται	1.3675	9	9.328	3.558
λεγεται και	1.3648	8	9.624	3.595
εις	1.2143	7	10.419	3.600

Negative term	Coefficient	Docs	Mean present	Mean absent
και το	-1.5110	9	3.651	4.119
τοις	-1.2269	3	0.453	4.189
στραβων	-1.2146	15	4.058	4.081
ουδετερως	-1.0915	3	1.145	4.168
οι	-1.0306	11	4.641	4.008
και πολις	-1.0206	5	2.485	4.161
μια	-0.9956	3	1.625	4.153
δεκατω	-0.7478	3	1.032	4.171
το εθνικον	-0.6927	56	4.103	4.045
εθνικον	-0.6927	56	4.103	4.045
πολυβιος	-0.6778	5	2.410	4.165
θεοπομπος	-0.6417	5	2.472	4.162
ηροδοτος	-0.6341	9	4.829	4.003
δε απο	-0.6268	3	2.313	4.132
ευρωπη	-0.5953	5	4.350	4.063
εκαταιος ευρωπη	-0.5953	5	4.350	4.063
ως στραβων	-0.5710	4	3.164	4.115
πολιτης	-0.5652	18	4.355	4.016
τους	-0.5236	4	3.045	4.120
αυτην	-0.5087	9	4.398	4.046

AI English terms predicting large absolute residuals

Features: 410; ridge alpha: 1.000; cross-validated R^2: 0.072.

Positive term	Coefficient	Docs	Mean present	Mean absent
for	5.2249	14	9.010	3.274
and	3.8182	56	5.233	2.607
is	2.7652	88	4.311	2.365
was called	2.4265	5	11.459	3.689
there	2.2618	30	6.179	3.177
this	2.1865	9	9.910	3.500
says it is	2.1407	5	12.839	3.616
that	2.1232	18	7.019	3.432
from	2.0895	43	5.709	2.846
kome	1.9958	3	10.651	3.874
was	1.9365	19	7.031	3.384
however	1.8497	7	7.454	3.823
is a city	1.8401	4	10.173	3.823
is a	1.8032	16	7.056	3.510
there is	1.7912	24	6.332	3.365
one has	1.7832	3	9.498	3.910
when	1.7671	7	10.038	3.629
and in	1.7324	3	12.284	3.823
says it	1.6646	6	11.043	3.633
such	1.6454	3	10.447	3.880

Negative term	Coefficient	Docs	Mean present	Mean absent
neuter	-1.6014	4	1.433	4.187
then	-1.5588	3	1.442	4.159
the ethnic is	-1.3165	29	3.453	4.332
are	-1.2919	22	4.260	4.026
syria	-1.2821	3	0.955	4.174
one of the	-1.2596	3	1.625	4.153
one of	-1.2596	3	1.625	4.153
ethnic is	-1.2173	31	3.741	4.229
promontory	-1.1989	3	2.585	4.123
city of the	-1.1191	9	3.984	4.087
is also called	-1.1093	3	1.024	4.172
europe	-1.0669	5	2.851	4.142
hecataeus in	-1.0614	7	1.634	4.261
inhabitant is a	-1.0446	4	2.022	4.163
demonym	-1.0360	7	2.533	4.193
one says	-1.0328	4	1.922	4.167
as the	-1.0043	3	2.240	4.134
the ethnicon is	-0.9946	16	3.903	4.110
and the	-0.9637	17	5.307	3.825
mountains	-0.9488	3	1.573	4.155

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 1.425; cross-validated R^2: 0.056.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	3.0534	52	5.430	2.612
for	2.9672	11	9.059	3.462
with	2.4870	16	7.135	3.495
the	2.4388	90	4.329	1.812
there	2.3032	34	6.362	2.901
also said	1.8563	6	12.362	3.548
said	1.8563	6	12.362	3.548
form	1.7912	11	7.898	3.605
on	1.7252	16	5.983	3.714
is	1.7017	87	4.412	1.839
used to be	1.6665	6	11.612	3.596
used	1.6538	8	10.462	3.522
used to	1.6538	8	10.462	3.522
homer	1.6054	3	11.306	3.854
for the	1.5856	7	10.741	3.576
is with	1.4893	4	10.965	3.790
after the	1.4241	6	9.733	3.716
this	1.4199	15	8.738	3.255
with the	1.4113	5	10.417	3.744
of	1.3802	66	4.520	3.217

Negative term	Coefficient	Docs	Mean present	Mean absent
also called	-1.2176	6	3.503	4.114
strabo	-1.1742	15	4.058	4.081
or after	-1.1356	3	2.505	4.126
those who	-1.0838	3	0.683	4.182
syria	-1.0711	3	0.955	4.174
in syria	-1.0711	3	0.955	4.174
one of the	-0.9768	4	1.954	4.166
those	-0.9571	5	2.286	4.172
hekataios in his	-0.9368	11	2.735	4.243
neuter	-0.9311	4	1.433	4.187
book	-0.9142	39	4.036	4.104
but	-0.8859	12	3.983	4.090
one of	-0.8818	5	2.549	4.158
the possessive is	-0.8549	4	2.119	4.159
possessive is	-0.8549	4	2.119	4.159
in book	-0.8344	16	3.771	4.136
strabo in	-0.7874	3	1.091	4.170
about	-0.7777	5	2.838	4.143
promontory	-0.7664	3	2.466	4.127
crete	-0.7629	3	3.148	4.106

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.