gpt-4o-2024-05-13 v1 Evaluation

Generated: 2026-09-20 11:20:09 UTC

Pairs100

Distinct lemmas100

Slope0.637

R^20.594

BLEU-419.8%

chrF++50.2%

METEOR55.3%

ROUGE-L57.2%

BERTScore92.3%

COMET72.4%

BLEURT65.4%

Profile version id: 2407

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-05-13) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.637
Intercept	16.793
R^2	0.594
P-value	6.74e-21
Pearson r	0.771
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	44.910
Mean absolute length residual	11.339
Mean absolute percent error	15.2%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.069, R^2 = 0.005, p = 0.4960.

chrF++ vs passage length — chrF++: significant negative; r = -0.228, R^2 = 0.052, p = 0.0223.

METEOR vs passage length — METEOR: significant negative; r = -0.321, R^2 = 0.103, p = 0.0011.

ROUGE-L vs passage length — ROUGE-L: significant negative; r = -0.334, R^2 = 0.111, p = 0.0007.

BERTScore vs passage length — BERTScore: significant negative; r = -0.439, R^2 = 0.193, p = 4.91e-06.

COMET vs passage length — COMET: significant negative; r = -0.408, R^2 = 0.166, p = 2.52e-05.

BLEURT vs passage length — BLEURT: significant negative; r = -0.644, R^2 = 0.414, p = 5.02e-13.

Trigram precision vs passage length — Trigram precision: negative; r = -0.048, R^2 = 0.002, p = 0.6362.

Trigram recall vs passage length — Trigram recall: negative; r = -0.107, R^2 = 0.011, p = 0.2901.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.094, R^2 = 0.009, p = 0.3533.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.107, R^2 = 0.012, p = 0.2882.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.069	0.005	0.4960	-0.00029	ok
chrF++	100	significant negative	-0.228	0.052	0.0223	-0.00079	ok
METEOR	100	significant negative	-0.321	0.103	0.0011	-0.00161	ok
ROUGE-L	100	significant negative	-0.334	0.111	0.0007	-0.00148	ok
BERTScore	100	significant negative	-0.439	0.193	4.91e-06	-0.00038	ok
COMET	100	significant negative	-0.408	0.166	2.52e-05	-0.00083	ok
BLEURT	100	significant negative	-0.644	0.414	5.02e-13	-0.00211	ok
Trigram precision	100	negative	-0.048	0.002	0.6362	-0.00022	ok
Trigram recall	100	negative	-0.107	0.011	0.2901	-0.00052	ok
Trigram F1	100	negative	-0.094	0.009	0.3533	-0.00045	ok
Trigram Jaccard	100	negative	-0.107	0.012	0.2882	-0.00034	ok

Translation Similarity Metrics

Mean BLEU-4	19.8%
Mean chrF++	50.2%
Mean METEOR	55.3%
Mean ROUGE-L	57.2%
Mean BERTScore	92.3%
Mean COMET	72.4%
Mean BLEURT	65.4%
Legacy smoothed corpus BLEU	24.0%
Unigram F1	59.9%
Bigram F1	30.7%
Trigram precision	17.5%
Trigram recall	17.8%
Trigram F1	17.7%
Trigram Jaccard	9.7%
4-gram F1	10.6%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	29	-159.740	0.0%	7.0%	5.9%	10.0%	82.2%	48.4%	35.2%	reviewed/approved	13580	gpt-4o-2024-05-13
2603	Κάρυστος	175	176	47.759	19.9%	54.3%	53.3%	54.7%	92.9%	76.8%	58.1%	reviewed/approved	13592	gpt-4o-2024-05-13
3288	Κορώνεια	110	130	43.154	22.0%	54.3%	52.8%	57.3%	91.4%	69.9%	62.3%	reviewed/approved	13643	gpt-4o-2024-05-13
2346	Κάναι	77	99	33.170	17.7%	50.1%	57.6%	61.0%	91.1%	73.9%	53.0%	reviewed/approved	13562	gpt-4o-2024-05-13
2597	Καρπασία	112	121	32.880	16.4%	51.6%	42.3%	49.8%	92.3%	71.3%	51.9%	reviewed/approved	13583	gpt-4o-2024-05-13
2119	Κάλλατις	65	89	30.812	14.8%	48.7%	50.2%	46.8%	90.6%	65.0%	59.7%	reviewed/approved	13520	gpt-4o-2024-05-13
2056	Καβειρία	112	118	29.880	12.8%	45.1%	41.6%	39.1%	90.8%	67.2%	48.4%	reviewed/approved	13439	gpt-4o-2024-05-13
2465	Κάνωπος	83	99	29.349	19.9%	54.5%	61.4%	62.6%	92.4%	72.2%	58.9%	reviewed/approved	13568	gpt-4o-2024-05-13
2604	Καρχηδών	117	119	27.696	26.5%	55.9%	60.0%	59.9%	91.6%	75.4%	57.4%	reviewed/approved	13595	gpt-4o-2024-05-13
7266	Κώμη	79	91	23.896	12.5%	41.4%	36.4%	38.8%	87.8%	59.3%	39.3%	reviewed/approved	13727	gpt-4o-2024-05-13
7264	Κωλιάς	61	77	21.359	18.5%	46.6%	42.6%	42.0%	90.7%	65.8%	53.6%	reviewed/approved	13721	gpt-4o-2024-05-13
2609	Κάσπειρος	135	124	21.233	25.7%	55.3%	50.4%	62.5%	88.6%	73.0%	60.0%	reviewed/approved	13604	gpt-4o-2024-05-13
2342	Κάμιρος	54	72	20.817	16.0%	53.4%	55.0%	57.1%	92.7%	73.1%	61.6%	reviewed/approved	13556	gpt-4o-2024-05-13
7254	Κύτα	84	90	19.712	27.5%	53.7%	59.4%	57.5%	91.3%	73.1%	59.4%	reviewed/approved	13691	gpt-4o-2024-05-13
2470	Καππαδοκία	68	77	16.901	8.5%	45.7%	64.5%	57.9%	93.2%	70.3%	59.7%	reviewed/approved	13574	gpt-4o-2024-05-13
7251	Κύρτος	66	75	16.175	25.8%	51.8%	61.7%	59.6%	90.8%	69.2%	54.0%	reviewed/approved	13682	gpt-4o-2024-05-13
2599	Καρπήσιοι	10	8	-15.162	37.0%	54.7%	60.1%	66.7%	94.7%	81.3%	82.1%	reviewed/approved	13586	gpt-4o-2024-05-13
2115	Καλὴ ἀκτή	64	72	14.449	14.8%	44.9%	54.1%	55.9%	90.1%	62.5%	51.7%	reviewed/approved	13508	gpt-4o-2024-05-13
7260	Κυχρεῖος πάγος	61	70	14.359	22.6%	56.2%	62.9%	53.4%	93.2%	74.3%	63.2%	reviewed/approved	13709	gpt-4o-2024-05-13
7244	Κυρήνη	54	64	12.817	49.2%	70.1%	72.7%	74.6%	94.8%	78.9%	73.1%	reviewed/approved	13661	gpt-4o-2024-05-13
2076	Καινοί	11	11	-12.799	13.3%	50.5%	57.7%	63.6%	94.6%	75.5%	74.5%	reviewed/approved	13469	gpt-4o-2024-05-13
2075	Καινίνη	15	14	-12.346	46.8%	67.2%	79.9%	82.8%	96.6%	80.8%	81.4%	reviewed/approved	13466	gpt-4o-2024-05-13
3513	Κρανίδες	15	14	-12.346	10.4%	43.1%	55.2%	62.1%	92.2%	68.2%	66.2%	reviewed/approved	13649	gpt-4o-2024-05-13
2054	Καβαλίς	65	70	11.812	24.9%	54.3%	46.3%	53.3%	91.9%	72.0%	65.3%	reviewed/approved	12233	gpt-4o-2024-05-13
2084	Κάλαρνα	14	14	-11.709	19.7%	56.3%	70.5%	71.4%	95.5%	84.8%	75.0%	reviewed/approved	13493	gpt-4o-2024-05-13
2626	Κατάνη	95	89	11.707	20.8%	51.5%	44.0%	52.2%	91.4%	72.3%	58.2%	reviewed/approved	13628	gpt-4o-2024-05-13
2082	Καλάμαι	9	11	-11.525	15.3%	57.2%	55.6%	50.0%	94.7%	76.1%	66.2%	reviewed/approved	13487	gpt-4o-2024-05-13
2116	Κάληρος	40	31	-11.267	9.0%	41.4%	42.4%	50.7%	91.0%	68.6%	63.2%	reviewed/approved	13511	gpt-4o-2024-05-13
2114	Κάλβιος	18	17	-11.257	9.8%	38.3%	53.5%	51.4%	92.0%	70.2%	63.9%	reviewed/approved	13505	gpt-4o-2024-05-13
2629	Καταννοί	11	13	-10.799	19.2%	54.4%	78.9%	75.0%	92.8%	72.0%	68.9%	reviewed/approved	13631	gpt-4o-2024-05-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: 0.045.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	10.2458	66	3.370	-6.541
ου	7.0598	14	19.526	-3.179
απο	5.3249	27	9.596	-3.549
απο του	5.1746	9	20.707	-2.048
αφ ου	5.0114	5	30.175	-1.588
αφ	4.5389	6	24.224	-1.546
της	4.3719	35	2.456	-1.323
εστι δε	4.1079	5	24.800	-1.305
τε	4.0281	5	15.335	-0.807
ος	3.5216	6	15.330	-0.979
βοιωτιας	3.4301	3	23.349	-0.722
και το	3.0777	9	14.931	-1.477
εστι δε και	3.0625	3	26.558	-0.821
πολιχνιον	2.9618	3	22.935	-0.709
οι	2.8551	11	12.289	-1.519
λιβυης	2.8172	4	18.446	-0.769
κωμη	2.8055	3	15.722	-0.486
ειναι	2.7607	8	8.026	-0.698
τοις	2.6524	3	22.840	-0.706
στραβων	2.6328	15	10.221	-1.804

Negative term	Coefficient	Docs	Mean present	Mean absent
εθνος	-4.7235	8	-20.041	1.743
περι	-4.4689	11	-10.391	1.284
τω περι	-4.4657	4	-37.341	1.556
εν τω περι	-4.4657	4	-37.341	1.556
το εθνικον	-4.3065	56	-1.530	1.948
εθνικον	-4.3065	56	-1.530	1.948
δια	-4.2325	16	2.922	-0.557
οικητωρ	-4.0712	6	-27.252	1.739
ακροπολις	-3.9069	3	-54.907	1.698
το	-3.8524	70	1.195	-2.789
το θηλυκον	-3.5447	6	-16.311	1.041
ηρωδιανος	-3.1978	3	-54.286	1.679
τω	-2.7445	16	-7.750	1.476
πρωτη	-2.6496	3	-53.637	1.659
εν πρωτη	-2.6496	3	-53.637	1.659
χωρα	-2.5642	6	-25.558	1.631
εν τω	-2.5401	6	-18.068	1.153
κατα	-2.4985	10	-5.913	0.657
δευτερω	-2.3253	3	-7.558	0.234
προς τη	-2.1832	4	-7.487	0.312

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.144.

Positive term	Coefficient	Docs	Mean present	Mean absent
την	7.3945	13	31.408	8.340
εστι	6.0388	24	22.408	7.843
δια	5.5720	16	26.372	8.475
απο	4.7404	27	23.110	6.985
απο της	4.5578	6	49.533	8.901
γαρ	4.5413	9	35.918	8.908
δε	4.4984	46	14.958	8.256
εστι και	4.3379	20	22.848	8.461
εν	4.0767	37	16.475	8.322
και	3.9256	66	12.730	8.637
περι	3.3742	11	24.843	9.670
εν τω περι	2.9127	4	47.839	9.818
τω περι	2.9127	4	47.839	9.818
εκαλειτο δε	2.9020	3	78.399	9.265
εν τω	2.8746	6	40.987	9.446
το θηλυκον	2.7925	6	38.386	9.612
εκαλειτο δε και	2.6554	3	78.399	9.265
τη	2.4381	15	22.463	9.375
παρα	2.4196	7	37.967	9.334
δε το	2.4021	3	73.104	9.428

Negative term	Coefficient	Docs	Mean present	Mean absent
το εθνικον	-1.6921	56	11.653	10.938
εθνικον	-1.6921	56	11.653	10.938
τους	-1.6286	4	4.314	11.631
πολυβιος	-1.4689	5	3.518	11.750
εκαταιος	-1.3047	14	9.192	11.688
παιδος	-1.3011	4	7.139	11.514
χωρας	-1.2682	4	1.790	11.736
μια	-1.2632	3	4.719	11.543
θρακης	-1.2168	5	3.754	11.738
εκ	-1.1817	6	11.632	11.320
της χωρας	-1.0830	3	1.573	11.641
νησος	-1.0634	9	11.000	11.372
τινες	-1.0237	6	7.447	11.587
πλησιον	-1.0166	8	10.493	11.412
του δε	-1.0079	3	7.690	11.451
τον	-0.9869	10	13.282	11.123
καππαδοκιας	-0.9581	3	6.500	11.488
προς τη	-0.9548	4	7.487	11.499
εν τη	-0.9541	7	8.766	11.532
πολιτης	-0.9260	18	13.501	10.864

AI English terms predicting large absolute residuals

Features: 420; ridge alpha: 17.013; cross-validated R^2: -0.265.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	2.3022	98	11.364	10.072
caria	2.0493	3	57.636	9.907
similarly to	2.0127	5	48.679	9.373
similarly	1.9143	6	42.031	9.380
ethnonym	1.7827	4	45.312	9.923
ethnonym is	1.7827	4	45.312	9.923
the ethnonym	1.7827	4	45.312	9.923
the ethnonym is	1.7827	4	45.312	9.923
feminine form	1.7601	8	35.363	9.250
the feminine form	1.7601	8	35.363	9.250
named	1.6803	24	20.613	8.410
the feminine	1.5741	12	28.034	9.062
form	1.5638	19	23.787	8.419
feminine	1.4305	15	26.345	8.690
the region	1.4265	7	28.842	10.021
to	1.1810	46	14.166	8.930
to the	1.1438	14	20.437	9.857
region	1.0431	12	20.845	10.042
according	0.8914	13	19.030	10.189
according to	0.8914	13	19.030	10.189

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnic	-0.7074	45	9.913	12.505
ethnic	-0.7074	45	9.913	12.505
a city	-0.6909	64	10.192	13.378
a city of	-0.6032	33	9.260	12.362
name	-0.5719	23	10.207	11.677
the ethnic name	-0.5169	12	7.045	11.924
ethnic name	-0.5169	12	7.045	11.924
the ethnic designation	-0.5048	32	10.364	11.797
ethnic designation	-0.5048	32	10.364	11.797
designation	-0.5048	32	10.364	11.797
they	-0.5016	6	7.686	11.572
name is	-0.4576	11	7.389	11.827
ethnic name is	-0.4576	11	7.389	11.827
book	-0.4523	38	10.754	11.697
the demonym	-0.4422	9	6.276	11.839
demonym	-0.4422	9	6.276	11.839
ethnic designation is	-0.4336	29	9.593	12.052
designation is	-0.4336	29	9.593	12.052
the demonym is	-0.4309	7	5.447	11.782
demonym is	-0.4309	7	5.447	11.782

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 8.377; cross-validated R^2: -0.198.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	3.6429	90	11.698	8.104
is	3.4230	87	11.838	7.993
is with	3.2691	4	51.254	9.675
for	3.2451	11	31.189	8.885
there is	3.2324	24	21.118	8.250
also	3.2262	43	15.824	7.954
with	3.1712	16	25.122	8.713
there	3.1454	34	19.112	7.334
is also	2.9946	31	19.115	7.845
there is also	2.7778	22	22.294	8.249
diphthong	2.6884	3	66.314	9.638
when	2.4897	7	37.759	9.350
with the	2.3044	5	42.452	9.701
according	2.2148	5	45.823	9.524
according to	2.2148	5	45.823	9.524
is also a	1.9364	14	31.389	8.075
also a	1.8723	15	29.618	8.113
usage	1.7468	4	41.115	10.098
long	1.7348	3	63.320	9.731
his on	1.6202	3	60.246	9.826

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnonym is	-1.1624	52	11.133	11.561
ethnonym is	-1.1624	52	11.133	11.561
ethnonym	-1.0916	57	11.789	10.741
the ethnonym	-1.0756	56	11.843	10.697
as in	-0.9073	29	14.839	9.909
thrace	-0.7929	5	3.754	11.738
the same	-0.7801	4	2.899	11.690
same	-0.7801	4	2.899	11.690
polybios book	-0.7516	5	3.518	11.750
polybios	-0.7516	5	3.518	11.750
syria	-0.7264	3	2.269	11.619
in syria	-0.7264	3	2.269	11.619
local usage	-0.7188	3	1.573	11.641
in local usage	-0.7188	3	1.573	11.641
in local	-0.7188	3	1.573	11.641
one of	-0.7117	5	5.573	11.642
name	-0.7075	7	7.481	11.629
hekataios	-0.7071	14	9.192	11.688
also called	-0.7043	6	8.800	11.501
an island	-0.6890	9	8.275	11.642

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.