gpt-5 v1 Evaluation

Generated: 2026-07-25 12:53:06 UTC

Pairs100

Distinct lemmas100

Slope0.981

R^20.980

BLEU-420.7%

chrF++51.9%

METEOR60.4%

ROUGE-L61.8%

BERTScoreN/A

COMETN/A

BLEURTN/A

Profile version id: 2343

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.981
Intercept	2.837
R^2	0.980
P-value	8.03e-85
Pearson r	0.990
Mean source words	30.400
Length fallback rows	0
Mean human words	44.160
Mean AI words	46.170
Mean absolute length residual	4.084
Mean absolute percent error	10.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.055, R^2 = 0.003, p = 0.5837.

chrF++ vs passage length — chrF++: positive; r = 0.046, R^2 = 0.002, p = 0.6485.

METEOR vs passage length — METEOR: negative; r = -0.125, R^2 = 0.016, p = 0.2158.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.126, R^2 = 0.016, p = 0.2102.

Trigram precision vs passage length — Trigram precision: positive; r = 0.024, R^2 = 5.71e-04, p = 0.8135.

Trigram recall vs passage length — Trigram recall: positive; r = 0.011, R^2 = 1.29e-04, p = 0.9105.

Trigram F1 vs passage length — Trigram F1: positive; r = 0.019, R^2 = 3.58e-04, p = 0.8518.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.009, R^2 = 8.97e-05, p = 0.9255.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.055	0.003	0.5837	0.00024	ok
chrF++	100	positive	0.046	0.002	0.6485	0.00014	ok
METEOR	100	negative	-0.125	0.016	0.2158	-0.00057	ok
ROUGE-L	100	negative	-0.126	0.016	0.2102	-0.00046	ok
BERTScore	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
COMET	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
BLEURT	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
Trigram precision	100	positive	0.024	5.71e-04	0.8135	0.00011	ok
Trigram recall	100	positive	0.011	1.29e-04	0.9105	0.00005	ok
Trigram F1	100	positive	0.019	3.58e-04	0.8518	0.00009	ok
Trigram Jaccard	100	negative	-0.009	8.97e-05	0.9255	-0.00003	ok

Translation Similarity Metrics

Mean BLEU-4	20.7%
Mean chrF++	51.9%
Mean METEOR	60.4%
Mean ROUGE-L	61.8%
Mean BERTScore	N/A
Mean COMET	N/A
Mean BLEURT	N/A
Legacy smoothed corpus BLEU	27.8%
Unigram F1	65.5%
Bigram F1	35.6%
Trigram precision	21.0%
Trigram recall	22.0%
Trigram F1	21.5%
Trigram Jaccard	12.0%
4-gram F1	13.0%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	248	-19.779	22.0%	51.2%	49.4%	57.5%	N/A	N/A	N/A	reviewed/approved	11780	gpt-5-2025-08-07
2603	Κάρυστος	175	190	15.441	27.9%	59.0%	64.2%	62.5%	N/A	N/A	N/A	reviewed/approved	11792	gpt-5-2025-08-07
2342	Κάμιρος	54	70	14.174	25.4%	63.6%	73.4%	72.6%	N/A	N/A	N/A	reviewed/approved	11756	gpt-5-2025-08-07
7266	Κώμη	79	94	13.643	5.6%	40.0%	37.7%	37.0%	N/A	N/A	N/A	reviewed/approved	11927	gpt-5-2025-08-07
2604	Καρχηδών	117	131	13.355	20.7%	54.8%	61.9%	60.0%	N/A	N/A	N/A	reviewed/approved	11795	gpt-5-2025-08-07
2055	Καβασσός	109	97	-12.795	15.4%	44.0%	32.6%	45.6%	N/A	N/A	N/A	reviewed/approved	11636	gpt-5-2025-08-07
3288	Κορώνεια	110	123	12.223	16.5%	52.1%	50.2%	54.7%	N/A	N/A	N/A	reviewed/approved	11843	gpt-5-2025-08-07
2115	Καλὴ ἀκτή	64	77	11.362	13.8%	36.6%	46.1%	44.0%	N/A	N/A	N/A	reviewed/approved	11708	gpt-5-2025-08-07
2056	Καβειρία	112	102	-10.739	21.2%	53.3%	56.1%	60.7%	N/A	N/A	N/A	reviewed/approved	11639	gpt-5-2025-08-07
2468	Καπετώλιον	138	129	-9.252	14.1%	41.0%	40.0%	48.7%	N/A	N/A	N/A	reviewed/approved	11771	gpt-5-2025-08-07
2116	Κάληρος	40	34	-8.088	12.1%	45.4%	51.9%	54.1%	N/A	N/A	N/A	reviewed/approved	11711	gpt-5-2025-08-07
2625	Κατακεκαυμένη	46	56	8.024	15.8%	56.6%	70.0%	60.8%	N/A	N/A	N/A	reviewed/approved	11822	gpt-5-2025-08-07
2328	Καλλίπολις	43	53	7.968	20.5%	53.5%	56.5%	58.3%	N/A	N/A	N/A	reviewed/approved	11732	gpt-5-2025-08-07
2465	Κάνωπος	83	92	7.718	20.4%	54.9%	65.5%	64.0%	N/A	N/A	N/A	reviewed/approved	11768	gpt-5-2025-08-07
2621	Κάστνιον	25	35	7.631	24.1%	48.0%	56.5%	56.7%	N/A	N/A	N/A	reviewed/approved	11810	gpt-5-2025-08-07
7243	Κύρη	22	32	7.575	17.4%	46.1%	38.7%	55.6%	N/A	N/A	N/A	reviewed/approved	11858	gpt-5-2025-08-07
2477	Καρδαμύλη	43	52	6.968	17.9%	54.6%	73.4%	63.2%	N/A	N/A	N/A	reviewed/approved	11777	gpt-5-2025-08-07
7251	Κύρτος	66	61	-6.601	29.2%	56.4%	67.8%	67.2%	N/A	N/A	N/A	reviewed/approved	11882	gpt-5-2025-08-07
2470	Καππαδοκία	68	76	6.437	15.4%	50.5%	72.3%	68.1%	N/A	N/A	N/A	reviewed/approved	11774	gpt-5-2025-08-07
2330	Καλύβη	24	20	-6.388	8.4%	39.4%	56.8%	63.6%	N/A	N/A	N/A	reviewed/approved	11738	gpt-5-2025-08-07
2079	Καισάρεια	39	35	-6.107	36.7%	59.5%	66.7%	70.3%	N/A	N/A	N/A	reviewed/approved	11678	gpt-5-2025-08-07
7245	Κύρης	13	21	5.406	5.2%	28.9%	34.8%	41.2%	N/A	N/A	N/A	reviewed/approved	11864	gpt-5-2025-08-07
2329	Κάλπη	57	64	5.231	16.4%	51.3%	65.2%	70.5%	N/A	N/A	N/A	reviewed/approved	11735	gpt-5-2025-08-07
2057	Καβελλιών	34	31	-5.200	24.0%	58.8%	67.7%	76.9%	N/A	N/A	N/A	reviewed/approved	11642	gpt-5-2025-08-07
7259	Κύφος	54	61	5.174	16.9%	51.5%	64.2%	60.9%	N/A	N/A	N/A	reviewed/approved	11906	gpt-5-2025-08-07
7250	Κυρταία	40	37	-5.088	17.5%	45.2%	54.5%	59.7%	N/A	N/A	N/A	reviewed/approved	11879	gpt-5-2025-08-07
2081	Καλάθη	49	46	-4.919	22.0%	48.3%	43.8%	54.7%	N/A	N/A	N/A	reviewed/approved	11684	gpt-5-2025-08-07
7254	Κύτα	84	90	4.736	27.1%	55.0%	57.9%	60.9%	N/A	N/A	N/A	reviewed/approved	11891	gpt-5-2025-08-07
2455	Κάναστρον	60	57	-4.713	20.0%	47.7%	52.9%	54.7%	N/A	N/A	N/A	reviewed/approved	11765	gpt-5-2025-08-07
7252	Κύρτωνες	27	34	4.669	9.7%	44.7%	53.7%	49.2%	N/A	N/A	N/A	reviewed/approved	11885	gpt-5-2025-08-07

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 34.551; cross-validated R^2: -0.343.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.3912	66	0.217	-0.421
κωμη	0.3022	3	10.658	-0.330
τε	0.2680	5	7.742	-0.407
λεγεται	0.2505	9	4.940	-0.489
εστι δε	0.2296	5	6.634	-0.349
δε και	0.2253	8	3.611	-0.314
λεγεται και	0.2165	8	4.687	-0.408
οι	0.2164	11	4.198	-0.519
απο	0.2083	27	1.863	-0.689
εστι δε και	0.2077	3	10.849	-0.336
δυο	0.1807	4	6.469	-0.270
ου και	0.1724	4	7.755	-0.323
ου	0.1694	14	1.864	-0.303
ορος	0.1687	6	5.677	-0.362
ως	0.1570	52	0.321	-0.348
εδει	0.1502	3	6.347	-0.196
νησος	0.1419	9	2.408	-0.238
παιδος	0.1380	4	4.760	-0.198
θηλυκως	0.1344	5	5.735	-0.302
και θηλυκως	0.1344	5	5.735	-0.302

Negative term	Coefficient	Docs	Mean present	Mean absent
καππαδοκιας	-0.1984	3	-6.430	0.199
το	-0.1840	70	0.016	-0.037
και του	-0.1833	4	-6.188	0.258
περι	-0.1717	11	-3.151	0.389
εθνος	-0.1577	8	-4.327	0.376
ως το	-0.1540	3	-10.045	0.311
λυκιας	-0.1492	3	-5.894	0.182
θρακης	-0.1403	5	-2.627	0.138
τους	-0.1382	4	-2.490	0.104
κατα	-0.1278	10	-0.863	0.096
μεταξυ	-0.1269	5	-1.996	0.105
ελλανικος	-0.1137	4	-8.318	0.347
πολις θρακης	-0.1101	4	-2.105	0.088
το εθνικον	-0.1088	56	-0.393	0.500
εθνικον	-0.1088	56	-0.393	0.500
οτι	-0.1088	5	-0.640	0.034
ηρωδιανος	-0.1048	3	-10.146	0.314
πλησιον	-0.1036	8	-0.595	0.052
ης	-0.1034	3	-4.065	0.126
πρωτη	-0.0962	3	-8.312	0.257

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: 0.047.

Positive term	Coefficient	Docs	Mean present	Mean absent
δε	3.5516	46	5.780	2.639
και	2.7768	66	4.796	2.701
γαρ	2.7029	9	10.334	3.466
εκαλειτο	2.3013	7	10.423	3.607
εστι	1.9557	24	6.589	3.293
κωμη	1.9168	3	10.658	3.880
εν	1.8625	37	5.502	3.251
της	1.8454	35	5.732	3.196
εν τω	1.8166	6	9.981	3.707
την	1.7647	13	7.840	3.522
εστι δε	1.7359	5	10.929	3.723
ομηρος	1.5523	3	11.313	3.860
εκαλειτο δε	1.5391	3	16.192	3.709
παρα	1.5152	7	9.393	3.684
δε και	1.5041	8	10.082	3.562
τω	1.4879	16	6.092	3.701
πολις εν	1.3739	5	8.365	3.858
λεγεται	1.3689	9	9.336	3.564
λεγεται και	1.3673	8	9.632	3.601
εις	1.2145	7	10.422	3.607

Negative term	Coefficient	Docs	Mean present	Mean absent
και το	-1.5072	9	3.654	4.126
τοις	-1.2276	3	0.452	4.196
στραβων	-1.2258	15	4.056	4.089
ουδετερως	-1.0980	3	1.143	4.175
οι	-1.0354	11	4.641	4.015
και πολις	-1.0253	5	2.488	4.168
μια	-1.0084	3	1.619	4.160
δεκατω	-0.7510	3	1.032	4.178
το εθνικον	-0.7104	56	4.101	4.062
εθνικον	-0.7104	56	4.101	4.062
πολυβιος	-0.6869	5	2.404	4.172
θεοπομπος	-0.6444	5	2.471	4.169
ηροδοτος	-0.6420	9	4.832	4.010
δε απο	-0.6255	3	2.317	4.138
ευρωπη	-0.5939	5	4.352	4.070
εκαταιος ευρωπη	-0.5939	5	4.352	4.070
πολιτης	-0.5792	18	4.352	4.025
ως στραβων	-0.5724	4	3.167	4.122
τους	-0.5313	4	3.039	4.127
αυτην	-0.5148	9	4.394	4.053

AI English terms predicting large absolute residuals

Features: 410; ridge alpha: 1.000; cross-validated R^2: 0.064.

Positive term	Coefficient	Docs	Mean present	Mean absent
for	5.2350	14	9.016	3.281
and	3.8019	56	5.233	2.621
is	2.7480	88	4.309	2.434
was called	2.4251	5	11.464	3.695
there	2.2523	30	6.178	3.186
this	2.1880	9	9.910	3.508
says it is	2.1451	5	12.847	3.622
that	2.1089	18	7.019	3.439
from	2.0877	43	5.709	2.858
kome	1.9964	3	10.658	3.880
was	1.9313	19	7.031	3.392
is a city	1.8466	4	10.176	3.830
however	1.8447	7	7.453	3.830
is a	1.7969	16	7.054	3.518
there is	1.7846	24	6.331	3.374
one has	1.7841	3	9.496	3.916
when	1.7628	7	10.037	3.636
and in	1.7329	3	12.299	3.830
says it	1.6692	6	11.048	3.639
it is	1.6431	23	6.763	3.283

Negative term	Coefficient	Docs	Mean present	Mean absent
neuter	-1.6048	4	1.434	4.194
then	-1.5659	3	1.437	4.166
the ethnic is	-1.3283	29	3.448	4.343
are	-1.3208	22	4.260	4.034
syria	-1.2791	3	0.958	4.180
one of the	-1.2598	3	1.619	4.160
one of	-1.2598	3	1.619	4.160
ethnic is	-1.2294	31	3.736	4.240
promontory	-1.1954	3	2.586	4.130
city of the	-1.1146	9	3.981	4.094
is also called	-1.1126	3	1.027	4.178
hecataeus in	-1.0761	7	1.632	4.268
europe	-1.0709	5	2.854	4.148
inhabitant is a	-1.0516	4	2.014	4.170
one says	-1.0461	4	1.915	4.174
demonym	-1.0399	7	2.534	4.200
the ethnicon is	-1.0084	16	3.901	4.119
as the	-1.0036	3	2.237	4.141
and the	-0.9755	17	5.309	3.833
mountains	-0.9692	3	1.570	4.161

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 1.425; cross-validated R^2: 0.050.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	3.0300	52	5.429	2.626
for	2.9560	11	9.062	3.468
with	2.4813	16	7.136	3.502
the	2.4716	90	4.337	1.800
there	2.2958	34	6.361	2.911
also said	1.8561	6	12.368	3.555
said	1.8561	6	12.368	3.555
form	1.7872	11	7.901	3.612
on	1.7200	16	5.982	3.722
is	1.6704	87	4.409	1.907
used to be	1.6636	6	11.615	3.603
used	1.6498	8	10.463	3.529
used to	1.6498	8	10.463	3.529
homer	1.6105	3	11.313	3.860
for the	1.5800	7	10.748	3.582
is with	1.4834	4	10.971	3.797
after the	1.4214	6	9.731	3.723
this	1.4186	15	8.739	3.262
with the	1.4113	5	10.424	3.750
of	1.3606	66	4.519	3.240

Negative term	Coefficient	Docs	Mean present	Mean absent
also called	-1.2270	6	3.503	4.121
strabo	-1.1740	15	4.056	4.089
or after	-1.1371	3	2.500	4.133
those who	-1.0872	3	0.690	4.189
syria	-1.0737	3	0.958	4.180
in syria	-1.0737	3	0.958	4.180
one of the	-0.9803	4	1.950	4.173
hekataios in his	-0.9538	11	2.732	4.251
book	-0.9382	39	4.035	4.115
neuter	-0.9315	4	1.434	4.194
but	-0.8877	12	3.982	4.098
one of	-0.8839	5	2.544	4.165
the possessive is	-0.8594	4	2.116	4.166
possessive is	-0.8594	4	2.116	4.166
in book	-0.8460	16	3.772	4.143
strabo in	-0.7845	3	1.099	4.176
about	-0.7821	5	2.837	4.149
promontory	-0.7698	3	2.465	4.134
crete	-0.7647	3	3.139	4.113
those	-0.7609	5	2.484	4.168

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.