gpt-4o-2024-05-13 v1 Evaluation

Generated: 2026-07-25 12:52:35 UTC

Pairs100

Distinct lemmas100

Slope0.637

R^20.594

BLEU-419.8%

chrF++50.2%

METEOR55.3%

ROUGE-L57.2%

BERTScoreN/A

COMETN/A

BLEURTN/A

Profile version id: 2407

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-05-13) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.637
Intercept	16.785
R^2	0.594
P-value	6.89e-21
Pearson r	0.771
Mean source words	30.400
Length fallback rows	0
Mean human words	44.160
Mean AI words	44.910
Mean absolute length residual	11.343
Mean absolute percent error	15.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.069, R^2 = 0.005, p = 0.4933.

chrF++ vs passage length — chrF++: significant negative; r = -0.227, R^2 = 0.052, p = 0.0230.

METEOR vs passage length — METEOR: significant negative; r = -0.320, R^2 = 0.103, p = 0.0012.

ROUGE-L vs passage length — ROUGE-L: significant negative; r = -0.333, R^2 = 0.111, p = 0.0007.

Trigram precision vs passage length — Trigram precision: negative; r = -0.048, R^2 = 0.002, p = 0.6362.

Trigram recall vs passage length — Trigram recall: negative; r = -0.105, R^2 = 0.011, p = 0.2979.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.093, R^2 = 0.009, p = 0.3593.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.106, R^2 = 0.011, p = 0.2936.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.069	0.005	0.4933	-0.00029	ok
chrF++	100	significant negative	-0.227	0.052	0.0230	-0.00078	ok
METEOR	100	significant negative	-0.320	0.103	0.0012	-0.00160	ok
ROUGE-L	100	significant negative	-0.333	0.111	0.0007	-0.00147	ok
BERTScore	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
COMET	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
BLEURT	0	not enough data	N/A	N/A	N/A	N/A	needs at least two rows with metric scores
Trigram precision	100	negative	-0.048	0.002	0.6362	-0.00022	ok
Trigram recall	100	negative	-0.105	0.011	0.2979	-0.00051	ok
Trigram F1	100	negative	-0.093	0.009	0.3593	-0.00044	ok
Trigram Jaccard	100	negative	-0.106	0.011	0.2936	-0.00034	ok

Translation Similarity Metrics

Mean BLEU-4	19.8%
Mean chrF++	50.2%
Mean METEOR	55.3%
Mean ROUGE-L	57.2%
Mean BERTScore	N/A
Mean COMET	N/A
Mean BLEURT	N/A
Legacy smoothed corpus BLEU	24.0%
Unigram F1	59.9%
Bigram F1	30.7%
Trigram precision	17.5%
Trigram recall	17.8%
Trigram F1	17.7%
Trigram Jaccard	9.7%
4-gram F1	10.6%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	29	-159.743	0.0%	7.0%	5.9%	10.0%	N/A	N/A	N/A	reviewed/approved	13580	gpt-4o-2024-05-13
2603	Κάρυστος	175	176	47.761	19.9%	54.3%	53.3%	54.7%	N/A	N/A	N/A	reviewed/approved	13592	gpt-4o-2024-05-13
3288	Κορώνεια	110	130	43.158	22.0%	54.3%	52.8%	57.3%	N/A	N/A	N/A	reviewed/approved	13643	gpt-4o-2024-05-13
2346	Κάναι	77	99	33.175	17.7%	50.1%	57.6%	61.0%	N/A	N/A	N/A	reviewed/approved	13562	gpt-4o-2024-05-13
2597	Καρπασία	112	121	32.884	16.4%	51.6%	42.3%	49.8%	N/A	N/A	N/A	reviewed/approved	13583	gpt-4o-2024-05-13
2119	Κάλλατις	65	89	30.817	14.8%	48.7%	50.2%	46.8%	N/A	N/A	N/A	reviewed/approved	13520	gpt-4o-2024-05-13
2056	Καβειρία	112	118	29.884	12.8%	45.1%	41.6%	39.1%	N/A	N/A	N/A	reviewed/approved	13439	gpt-4o-2024-05-13
2465	Κάνωπος	83	99	29.354	19.9%	54.5%	61.4%	62.6%	N/A	N/A	N/A	reviewed/approved	13568	gpt-4o-2024-05-13
2604	Καρχηδών	117	119	27.700	26.5%	55.9%	60.0%	59.9%	N/A	N/A	N/A	reviewed/approved	13595	gpt-4o-2024-05-13
7266	Κώμη	79	91	23.901	12.5%	41.4%	36.4%	38.8%	N/A	N/A	N/A	reviewed/approved	13727	gpt-4o-2024-05-13
7264	Κωλιάς	61	77	21.365	18.5%	46.6%	42.6%	42.0%	N/A	N/A	N/A	reviewed/approved	13721	gpt-4o-2024-05-13
2609	Κάσπειρος	135	124	21.236	25.7%	55.3%	50.4%	62.5%	N/A	N/A	N/A	reviewed/approved	13604	gpt-4o-2024-05-13
2342	Κάμιρος	54	72	20.823	16.0%	53.4%	55.0%	57.1%	N/A	N/A	N/A	reviewed/approved	13556	gpt-4o-2024-05-13
7254	Κύτα	84	90	19.717	27.0%	53.1%	57.3%	56.3%	N/A	N/A	N/A	reviewed/approved	13691	gpt-4o-2024-05-13
2470	Καππαδοκία	68	77	16.907	8.5%	45.7%	64.5%	57.9%	N/A	N/A	N/A	reviewed/approved	13574	gpt-4o-2024-05-13
7251	Κύρτος	66	75	16.181	25.8%	51.8%	61.7%	59.6%	N/A	N/A	N/A	reviewed/approved	13682	gpt-4o-2024-05-13
2599	Καρπήσιοι	11	8	-15.791	37.0%	52.3%	55.0%	63.2%	N/A	N/A	N/A	reviewed/approved	13586	gpt-4o-2024-05-13
2115	Καλὴ ἀκτή	64	72	14.454	14.8%	44.9%	54.1%	55.9%	N/A	N/A	N/A	reviewed/approved	13508	gpt-4o-2024-05-13
7260	Κυχρεῖος πάγος	61	70	14.365	22.6%	56.2%	62.9%	53.4%	N/A	N/A	N/A	reviewed/approved	13709	gpt-4o-2024-05-13
7244	Κυρήνη	54	64	12.823	49.2%	70.1%	72.7%	74.6%	N/A	N/A	N/A	reviewed/approved	13661	gpt-4o-2024-05-13
2076	Καινοί	11	11	-12.791	13.3%	50.5%	57.7%	63.6%	N/A	N/A	N/A	reviewed/approved	13469	gpt-4o-2024-05-13
2075	Καινίνη	15	14	-12.339	46.8%	67.2%	79.9%	82.8%	N/A	N/A	N/A	reviewed/approved	13466	gpt-4o-2024-05-13
3513	Κρανίδες	15	14	-12.339	10.4%	43.1%	55.2%	62.1%	N/A	N/A	N/A	reviewed/approved	13649	gpt-4o-2024-05-13
2054	Καβαλίς	65	70	11.817	24.9%	54.3%	46.3%	53.3%	N/A	N/A	N/A	reviewed/approved	12233	gpt-4o-2024-05-13
2626	Κατάνη	95	89	11.711	20.8%	51.5%	44.0%	52.2%	N/A	N/A	N/A	reviewed/approved	13628	gpt-4o-2024-05-13
2084	Κάλαρνα	14	14	-11.702	19.7%	56.3%	70.5%	71.4%	N/A	N/A	N/A	reviewed/approved	13493	gpt-4o-2024-05-13
2082	Καλάμαι	9	11	-11.517	15.3%	57.2%	55.6%	50.0%	N/A	N/A	N/A	reviewed/approved	13487	gpt-4o-2024-05-13
2116	Κάληρος	40	31	-11.261	9.0%	41.4%	42.4%	50.7%	N/A	N/A	N/A	reviewed/approved	13511	gpt-4o-2024-05-13
2114	Κάλβιος	18	17	-11.249	9.8%	38.3%	53.5%	51.4%	N/A	N/A	N/A	reviewed/approved	13505	gpt-4o-2024-05-13
2629	Καταννοί	11	13	-10.791	19.2%	54.4%	78.9%	75.0%	N/A	N/A	N/A	reviewed/approved	13631	gpt-4o-2024-05-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: 0.046.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	10.2590	66	3.376	-6.553
ου	7.0631	14	19.531	-3.179
απο	5.3270	27	9.602	-3.551
απο του	5.1755	9	20.712	-2.048
αφ ου	5.0116	5	30.179	-1.588
αφ	4.5398	6	24.229	-1.547
της	4.3780	35	2.462	-1.326
εστι δε	4.1091	5	24.805	-1.306
τε	4.0291	5	15.340	-0.807
ος	3.5226	6	15.335	-0.979
βοιωτιας	3.4304	3	23.354	-0.722
και το	3.0789	9	14.936	-1.477
εστι δε και	3.0633	3	26.562	-0.822
πολιχνιον	2.9629	3	22.940	-0.709
οι	2.8581	11	12.294	-1.520
λιβυης	2.8176	4	18.451	-0.769
κωμη	2.8066	3	15.728	-0.486
ειναι	2.7624	8	8.031	-0.698
τοις	2.6528	3	22.845	-0.707
στραβων	2.6365	15	10.227	-1.805

Negative term	Coefficient	Docs	Mean present	Mean absent
εθνος	-4.7807	8	-20.116	1.749
περι	-4.4669	11	-10.386	1.284
τω περι	-4.4649	4	-37.336	1.556
εν τω περι	-4.4649	4	-37.336	1.556
το εθνικον	-4.2982	56	-1.524	1.940
εθνικον	-4.2982	56	-1.524	1.940
δια	-4.2306	16	2.927	-0.557
οικητωρ	-4.0690	6	-27.247	1.739
ακροπολις	-3.9063	3	-54.904	1.698
το	-3.8434	70	1.201	-2.803
το θηλυκον	-3.5437	6	-16.307	1.041
ηρωδιανος	-3.1976	3	-54.284	1.679
τω	-2.7410	16	-7.744	1.475
πρωτη	-2.6492	3	-53.634	1.659
εν πρωτη	-2.6492	3	-53.634	1.659
χωρα	-2.5619	6	-25.553	1.631
εν τω	-2.5389	6	-18.064	1.153
κατα	-2.4962	10	-5.908	0.656
δευτερω	-2.3207	3	-7.551	0.234
προς τη	-2.1801	4	-7.480	0.312

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.145.

Positive term	Coefficient	Docs	Mean present	Mean absent
την	7.3945	13	31.412	8.344
εστι	6.0360	24	22.411	7.848
δια	5.5708	16	26.374	8.480
απο	4.7403	27	23.114	6.989
απο της	4.5579	6	49.538	8.905
γαρ	4.5415	9	35.922	8.912
δε	4.4946	46	14.959	8.263
εστι και	4.3349	20	22.850	8.466
εν	4.0700	37	16.475	8.329
και	3.9190	66	12.731	8.650
περι	3.3726	11	24.843	9.674
εν τω περι	2.9122	4	47.841	9.822
τω περι	2.9122	4	47.841	9.822
εκαλειτο δε	2.9019	3	78.401	9.269
εν τω	2.8736	6	40.987	9.451
το θηλυκον	2.7919	6	38.389	9.617
εκαλειτο δε και	2.6553	3	78.401	9.269
τη	2.4363	15	22.464	9.380
παρα	2.4197	7	37.972	9.339
δε το	2.4021	3	73.107	9.433

Negative term	Coefficient	Docs	Mean present	Mean absent
το εθνικον	-1.7001	56	11.651	10.951
εθνικον	-1.7001	56	11.651	10.951
τους	-1.6298	4	4.314	11.636
πολυβιος	-1.4707	5	3.516	11.755
εκαταιος	-1.3086	14	9.192	11.693
παιδος	-1.3005	4	7.146	11.518
χωρας	-1.2697	4	1.787	11.741
μια	-1.2658	3	4.716	11.548
θρακης	-1.2206	5	3.748	11.743
εκ	-1.1822	6	11.635	11.324
της χωρας	-1.0837	3	1.571	11.645
νησος	-1.0657	9	10.999	11.377
τινες	-1.0247	6	7.447	11.592
πλησιον	-1.0199	8	10.492	11.417
του δε	-1.0079	3	7.691	11.456
τον	-0.9892	10	13.282	11.128
καππαδοκιας	-0.9595	3	6.496	11.493
προς τη	-0.9580	4	7.480	11.504
εν τη	-0.9536	7	8.771	11.537
πολιτης	-0.9288	18	13.503	10.869

AI English terms predicting large absolute residuals

Features: 420; ridge alpha: 17.013; cross-validated R^2: -0.266.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	2.3026	98	11.369	10.065
caria	2.0491	3	57.637	9.911
similarly to	2.0124	5	48.678	9.378
similarly	1.9140	6	42.031	9.384
ethnonym	1.7822	4	45.311	9.928
ethnonym is	1.7822	4	45.311	9.928
the ethnonym	1.7822	4	45.311	9.928
the ethnonym is	1.7822	4	45.311	9.928
feminine form	1.7600	8	35.365	9.254
the feminine form	1.7600	8	35.365	9.254
named	1.6800	24	20.617	8.415
the feminine	1.5739	12	28.036	9.067
form	1.5637	19	23.790	8.423
feminine	1.4302	15	26.348	8.695
the region	1.4261	7	28.842	10.026
to	1.1797	46	14.166	8.938
to the	1.1435	14	20.439	9.862
region	1.0428	12	20.846	10.047
according	0.8906	13	19.029	10.194
according to	0.8906	13	19.029	10.194

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnic	-0.7092	45	9.912	12.514
ethnic	-0.7092	45	9.912	12.514
a city	-0.6928	64	10.191	13.392
a city of	-0.6048	33	9.258	12.370
name	-0.5730	23	10.206	11.683
the ethnic name	-0.5178	12	7.043	11.929
ethnic name	-0.5178	12	7.043	11.929
the ethnic designation	-0.5063	32	10.363	11.804
ethnic designation	-0.5063	32	10.363	11.804
designation	-0.5063	32	10.363	11.804
they	-0.5023	6	7.688	11.576
name is	-0.4584	11	7.388	11.832
ethnic name is	-0.4584	11	7.388	11.832
book	-0.4541	38	10.752	11.705
the demonym	-0.4429	9	6.274	11.844
demonym	-0.4429	9	6.274	11.844
ethnic designation is	-0.4349	29	9.592	12.058
designation is	-0.4349	29	9.592	12.058
the demonym is	-0.4317	7	5.442	11.787
demonym is	-0.4317	7	5.442	11.787

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 8.377; cross-validated R^2: -0.198.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	3.6094	90	11.704	8.098
is	3.3985	87	11.837	8.037
is with	3.2501	4	51.254	9.680
for	3.2328	11	31.193	8.890
there is	3.2202	24	21.120	8.255
also	3.2054	43	15.827	7.961
with	3.1555	16	25.124	8.718
there	3.1349	34	19.114	7.340
is also	2.9855	31	19.117	7.850
there is also	2.7675	22	22.296	8.254
diphthong	2.6736	3	66.317	9.643
when	2.4783	7	37.761	9.355
with the	2.2924	5	42.456	9.705
according	2.2016	5	45.822	9.528
according to	2.2016	5	45.822	9.528
is also a	1.9358	14	31.393	8.079
he	1.8818	3	64.608	9.696
also a	1.8722	15	29.622	8.117
usage	1.7320	4	41.114	10.103
long	1.7251	3	63.323	9.735

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnonym is	-1.1742	52	11.131	11.573
ethnonym is	-1.1742	52	11.131	11.573
ethnonym	-1.1023	57	11.788	10.753
the ethnonym	-1.0868	56	11.841	10.709
as in	-0.9077	29	14.839	9.915
thrace	-0.7942	5	3.748	11.743
the same	-0.7795	4	2.898	11.695
same	-0.7795	4	2.898	11.695
polybios book	-0.7526	5	3.516	11.755
polybios	-0.7526	5	3.516	11.755
syria	-0.7256	3	2.271	11.624
in syria	-0.7256	3	2.271	11.624
local usage	-0.7190	3	1.571	11.645
in local usage	-0.7190	3	1.571	11.645
in local	-0.7190	3	1.571	11.645
also called	-0.7136	6	8.804	11.505
name	-0.7119	7	7.483	11.634
one of	-0.7111	5	5.573	11.647
hekataios	-0.7081	14	9.192	11.693
an island	-0.6899	9	8.272	11.647

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.