gpt-4o-2024-08-06 v1 Evaluation

Generated: 2026-09-20 11:20:22 UTC

Pairs100

Distinct lemmas100

Slope0.965

R^20.977

BLEU-418.4%

chrF++49.8%

METEOR54.3%

ROUGE-L56.1%

BERTScore92.1%

COMET72.9%

BLEURT64.5%

Profile version id: 2410

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-08-06) using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.965
Intercept	5.587
R^2	0.977
P-value	3.70e-82
Pearson r	0.988
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	48.170
Mean absolute length residual	4.203
Mean absolute percent error	15.6%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.029, R^2 = 8.17e-04, p = 0.7778.

chrF++ vs passage length — chrF++: negative; r = -0.004, R^2 = 2.01e-05, p = 0.9647.

METEOR vs passage length — METEOR: negative; r = -0.161, R^2 = 0.026, p = 0.1089.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.192, R^2 = 0.037, p = 0.0563.

BERTScore vs passage length — BERTScore: significant negative; r = -0.316, R^2 = 0.100, p = 0.0013.

COMET vs passage length — COMET: negative; r = -0.154, R^2 = 0.024, p = 0.1257.

BLEURT vs passage length — BLEURT: significant negative; r = -0.670, R^2 = 0.449, p = 2.51e-14.

Trigram precision vs passage length — Trigram precision: negative; r = -0.026, R^2 = 6.90e-04, p = 0.7952.

Trigram recall vs passage length — Trigram recall: negative; r = -0.072, R^2 = 0.005, p = 0.4746.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.047, R^2 = 0.002, p = 0.6438.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.063, R^2 = 0.004, p = 0.5333.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.029	8.17e-04	0.7778	-0.00011	ok
chrF++	100	negative	-0.004	2.01e-05	0.9647	-0.00001	ok
METEOR	100	negative	-0.161	0.026	0.1089	-0.00085	ok
ROUGE-L	100	negative	-0.192	0.037	0.0563	-0.00080	ok
BERTScore	100	significant negative	-0.316	0.100	0.0013	-0.00026	ok
COMET	100	negative	-0.154	0.024	0.1257	-0.00024	ok
BLEURT	100	significant negative	-0.670	0.449	2.51e-14	-0.00214	ok
Trigram precision	100	negative	-0.026	6.90e-04	0.7952	-0.00012	ok
Trigram recall	100	negative	-0.072	0.005	0.4746	-0.00036	ok
Trigram F1	100	negative	-0.047	0.002	0.6438	-0.00022	ok
Trigram Jaccard	100	negative	-0.063	0.004	0.5333	-0.00021	ok

Translation Similarity Metrics

Mean BLEU-4	18.4%
Mean chrF++	49.8%
Mean METEOR	54.3%
Mean ROUGE-L	56.1%
Mean BERTScore	92.1%
Mean COMET	72.9%
Mean BLEURT	64.5%
Legacy smoothed corpus BLEU	22.2%
Unigram F1	60.2%
Bigram F1	29.9%
Trigram precision	15.6%
Trigram recall	17.1%
Trigram F1	16.3%
Trigram Jaccard	8.9%
4-gram F1	9.8%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	245	-21.005	11.0%	45.0%	39.0%	46.1%	87.8%	74.9%	40.9%	reviewed/approved	12681	gpt-4o-2024-08-06
2346	Κάναι	77	99	19.146	19.9%	45.5%	55.5%	55.4%	90.1%	72.3%	47.9%	reviewed/approved	12663	gpt-4o-2024-08-06
2054	Καβαλίς	65	83	14.720	19.4%	51.9%	49.3%	56.8%	90.5%	71.4%	59.1%	reviewed/approved	12534	gpt-4o-2024-08-06
2115	Καλὴ ἀκτή	64	81	13.684	8.8%	40.4%	39.7%	44.1%	89.1%	61.6%	48.0%	reviewed/approved	12609	gpt-4o-2024-08-06
2119	Κάλλατις	65	81	12.720	11.0%	45.6%	46.9%	43.8%	90.8%	65.3%	61.1%	reviewed/approved	12621	gpt-4o-2024-08-06
2604	Καρχηδών	117	131	12.565	12.7%	48.5%	39.5%	45.0%	90.4%	74.5%	51.1%	reviewed/approved	12696	gpt-4o-2024-08-06
2056	Καβειρία	112	126	12.388	13.0%	45.2%	38.5%	37.7%	90.5%	68.5%	51.5%	reviewed/approved	12540	gpt-4o-2024-08-06
2468	Καπετώλιον	138	127	-11.690	8.9%	37.9%	38.9%	45.1%	85.9%	66.7%	45.5%	reviewed/approved	12672	gpt-4o-2024-08-06
2470	Καππαδοκία	68	82	10.826	9.9%	45.8%	63.7%	56.0%	91.2%	69.0%	53.2%	reviewed/approved	12675	gpt-4o-2024-08-06
7264	Κωλιάς	61	75	10.578	16.9%	43.2%	37.4%	45.6%	90.6%	63.9%	51.4%	reviewed/approved	12822	gpt-4o-2024-08-06
2455	Κάναστρον	60	54	-9.458	13.9%	45.6%	46.6%	52.6%	92.5%	67.8%	60.0%	reviewed/approved	12666	gpt-4o-2024-08-06
7254	Κύτα	84	96	9.394	18.6%	54.4%	55.8%	55.6%	91.3%	71.4%	58.6%	reviewed/approved	12792	gpt-4o-2024-08-06
2116	Κάληρος	40	35	-9.167	10.4%	39.9%	41.9%	45.3%	90.6%	67.8%	62.9%	reviewed/approved	12612	gpt-4o-2024-08-06
2609	Κάσπειρος	135	127	-8.796	27.1%	53.0%	49.4%	58.0%	87.1%	69.7%	51.6%	reviewed/approved	12705	gpt-4o-2024-08-06
2055	Καβασσός	109	103	-7.719	11.6%	43.8%	39.3%	41.5%	90.9%	74.1%	42.5%	reviewed/approved	12537	gpt-4o-2024-08-06
3288	Κορώνεια	110	119	7.317	19.4%	52.9%	51.4%	60.0%	91.4%	73.8%	58.6%	reviewed/approved	12744	gpt-4o-2024-08-06
2057	Καβελλιών	34	45	6.620	13.3%	43.7%	55.8%	50.6%	90.3%	68.3%	56.1%	reviewed/approved	12543	gpt-4o-2024-08-06
2465	Κάνωπος	83	92	6.359	27.0%	58.8%	73.7%	67.4%	93.4%	73.8%	64.3%	reviewed/approved	12669	gpt-4o-2024-08-06
2599	Καρπήσιοι	10	9	-6.232	16.8%	47.5%	45.1%	52.6%	91.8%	70.7%	80.0%	reviewed/approved	12687	gpt-4o-2024-08-06
7255	Κυτέριον	22	33	6.194	12.3%	44.0%	45.9%	47.3%	90.9%	68.9%	65.4%	reviewed/approved	12795	gpt-4o-2024-08-06
7246	Κύρις	20	19	-5.877	26.9%	53.5%	50.8%	56.4%	93.5%	72.6%	61.0%	reviewed/approved	12768	gpt-4o-2024-08-06
2081	Καλάθη	49	47	-5.848	22.0%	51.0%	56.2%	66.7%	92.9%	70.2%	63.9%	reviewed/approved	12585	gpt-4o-2024-08-06
2630	Κάτρη	23	22	-5.771	9.4%	40.7%	38.8%	48.9%	92.1%	75.1%	68.7%	reviewed/approved	12738	gpt-4o-2024-08-06
2326	Καλλίαρος	47	56	5.081	12.0%	49.6%	56.3%	60.2%	93.7%	66.1%	63.2%	reviewed/approved	12627	gpt-4o-2024-08-06
7249	Κύρρος	45	54	5.010	5.2%	40.8%	28.2%	36.4%	90.3%	67.2%	48.5%	reviewed/approved	12777	gpt-4o-2024-08-06
2328	Καλλίπολις	43	52	4.939	16.7%	50.8%	48.0%	54.2%	91.4%	70.3%	64.1%	reviewed/approved	12633	gpt-4o-2024-08-06
2078	Καινύς	22	22	-4.806	52.1%	62.3%	67.9%	68.2%	92.5%	78.9%	65.7%	reviewed/approved	12573	gpt-4o-2024-08-06
7263	Κώθων	22	22	-4.806	19.5%	55.0%	55.2%	63.6%	93.5%	75.8%	71.8%	reviewed/approved	12819	gpt-4o-2024-08-06
7267	Κωνώπη	64	72	4.684	11.1%	45.8%	46.0%	49.6%	92.3%	71.8%	66.9%	reviewed/approved	12831	gpt-4o-2024-08-06
3258	Κεκρυφάλεια	26	26	-4.664	45.9%	72.4%	79.8%	80.8%	96.7%	77.6%	80.4%	reviewed/approved	12741	gpt-4o-2024-08-06

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.023.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	4.2548	66	0.622	-1.208
εστι δε	2.4379	5	9.779	-0.515
εν τη	2.4003	7	5.979	-0.450
δυο	2.1781	4	11.388	-0.475
πολιχνιον	2.1021	3	12.268	-0.379
παρα το	2.0986	3	10.384	-0.321
στραβων	2.0912	15	3.667	-0.647
παρα	2.0814	7	3.584	-0.270
απο	1.9790	27	2.805	-1.038
ως	1.7471	52	0.727	-0.787
εστι	1.4111	24	2.528	-0.798
τοπος	1.3894	4	5.421	-0.226
απο του	1.3881	9	4.480	-0.443
εκ	1.3720	6	6.299	-0.402
εστι δε και	1.3701	3	7.607	-0.235
και πολις	1.3661	5	3.623	-0.191
κωμη	1.3163	3	6.613	-0.205
θηλυκον	1.3002	12	3.697	-0.504
εισι	1.2841	5	8.411	-0.443
βοιωτιας	1.2837	3	7.359	-0.228

Negative term	Coefficient	Docs	Mean present	Mean absent
περι	-1.8908	11	-3.273	0.404
ηρωδιανος	-1.7146	3	-12.822	0.397
γαρ	-1.6338	9	-0.968	0.096
φησιν	-1.5430	10	-3.274	0.364
το εθνικον	-1.3820	56	-0.614	0.782
εθνικον	-1.3820	56	-0.614	0.782
θρακης	-1.3722	5	-3.830	0.202
εν	-1.3208	37	0.197	-0.116
ουτως	-1.3143	5	-6.565	0.346
οτι	-1.3068	5	-1.739	0.092
τα	-1.1858	12	-1.426	0.194
εκαλειτο	-1.1835	7	-3.684	0.277
οικητωρ	-1.1669	6	-3.859	0.246
εθνος	-1.1248	8	-1.830	0.159
τε	-1.1044	5	0.025	-0.001
τινες	-1.1035	6	-2.727	0.174
ην	-1.0906	13	-0.724	0.108
μετα	-1.0625	3	-10.006	0.309
λυκιας	-0.9754	3	-4.801	0.148
το	-0.9441	70	0.161	-0.377

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.159.

Positive term	Coefficient	Docs	Mean present	Mean absent
γαρ	2.0514	9	9.967	3.633
δε	1.8981	46	5.663	2.959
και	1.7853	66	4.740	3.160
εστι	1.4986	24	6.430	3.499
την	1.1545	13	8.359	3.582
εκαλειτο	1.0483	7	8.309	3.894
απο της	1.0338	6	10.275	3.815
εις	1.0276	7	9.432	3.809
δια	0.9851	16	6.857	3.697
πολιχνιον	0.9538	3	12.268	3.953
παρα	0.9511	7	9.586	3.797
ζευς	0.9284	3	17.280	3.798
εστι δε	0.9240	5	9.779	3.909
και πολις	0.9228	5	6.171	4.099
δυο	0.9219	4	11.388	3.903
παρα το	0.9023	3	10.384	4.011
περι	0.8823	11	6.803	3.881
τα	0.8321	12	8.388	3.632
ακρα	0.8190	4	9.750	3.972
ει	0.8155	4	11.600	3.894

Negative term	Coefficient	Docs	Mean present	Mean absent
δευτερω	-0.9002	3	0.709	4.311
ηροδοτος	-0.8515	9	5.132	4.111
το εθνικον	-0.7618	56	4.316	4.058
εθνικον	-0.7618	56	4.316	4.058
μεταξυ	-0.6724	5	3.310	4.250
προς τη	-0.6522	4	1.984	4.295
πολις	-0.6459	68	4.340	3.912
μια	-0.6325	3	1.917	4.273
ην	-0.6183	13	3.644	4.286
και το	-0.6128	9	3.855	4.237
πολις προς	-0.5988	3	2.063	4.269
θεοπομπος	-0.5951	5	2.463	4.294
δια το	-0.5591	3	1.924	4.273
ιβηριας	-0.5544	3	4.339	4.198
αιγυπτου	-0.5470	3	1.760	4.278
νησος	-0.5380	9	3.380	4.284
οικητωρ	-0.4632	6	5.418	4.125
ουδετερως	-0.4511	3	2.152	4.266
παιδος	-0.4258	4	3.269	4.242
προς	-0.4218	15	4.010	4.237

AI English terms predicting large absolute residuals

Features: 441; ridge alpha: 2.894; cross-validated R^2: -0.131.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	2.4258	98	4.235	2.625
from	1.9573	41	6.035	2.929
on	1.4896	13	6.939	3.794
or	1.4190	16	7.883	3.502
with	1.3879	20	7.641	3.343
as	1.3749	49	5.661	2.802
there	1.3280	32	5.875	3.416
of the	1.3046	35	6.059	3.203
form	1.2003	18	7.315	3.519
is	1.1145	88	4.345	3.157
diphthong	1.0596	3	12.436	3.948
was	1.0523	20	6.139	3.718
there is	1.0024	25	5.977	3.611
derives	1.0012	6	8.964	3.899
but	0.9920	7	10.894	3.699
on the	0.9582	6	8.923	3.901
strabo in	0.9442	7	8.765	3.859
and	0.9241	57	4.924	3.247
also	0.9107	45	5.155	3.424
be	0.8847	9	7.801	3.847

Negative term	Coefficient	Docs	Mean present	Mean absent
similar to	-1.0056	18	2.382	4.602
similar	-1.0056	18	2.382	4.602
mentioned	-0.8715	23	2.766	4.632
are the	-0.7296	3	1.136	4.298
the ethnonym	-0.7116	6	1.825	4.354
ethnonym	-0.7116	6	1.825	4.354
mentioned by	-0.6863	19	2.953	4.496
sea	-0.6355	7	3.096	4.286
are called	-0.6256	5	2.643	4.285
the ethnonym is	-0.6244	5	1.756	4.331
ethnonym is	-0.6244	5	1.756	4.331
island	-0.6242	14	3.214	4.364
is named	-0.5995	6	2.443	4.315
a city	-0.5798	64	4.141	4.312
which the	-0.5795	4	2.083	4.291
mount	-0.5743	4	3.206	4.244
second book	-0.5532	3	0.553	4.316
inhabitants are	-0.5380	3	1.068	4.300
is a city	-0.5341	10	4.273	4.195
after	-0.5091	18	4.510	4.135

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: -0.050.

Positive term	Coefficient	Docs	Mean present	Mean absent
the	3.3842	90	4.493	1.593
form	2.5598	11	10.135	3.469
with	2.2019	16	7.644	3.547
like	2.1122	5	10.833	3.854
from the form	2.0277	5	11.669	3.810
is	1.9816	87	4.494	2.250
there is	1.8637	24	6.348	3.525
the form	1.8198	9	11.011	3.529
on	1.6537	16	7.515	3.572
also	1.6427	43	5.271	3.397
there is also	1.6332	22	6.232	3.630
from the	1.6329	13	8.637	3.540
from	1.5397	37	5.952	3.175
and	1.5091	52	5.126	3.202
genitive	1.4675	3	16.148	3.833
is with	1.4551	4	12.052	3.876
on the	1.4017	10	8.264	3.751
is also	1.3830	31	5.878	3.450
there	1.2836	34	6.325	3.110
in the	1.2155	25	5.992	3.606

Negative term	Coefficient	Docs	Mean present	Mean absent
after	-1.2471	22	4.817	4.029
as in	-1.1156	29	4.359	4.139
was from	-1.0226	4	1.170	4.329
herodotos book	-0.9624	5	2.030	4.317
one of the	-0.9129	4	1.491	4.316
the ethnonym is	-0.9078	52	4.125	4.287
ethnonym is	-0.9078	52	4.125	4.287
name	-0.8635	7	3.526	4.254
one of	-0.8131	5	2.363	4.299
one	-0.7714	12	4.447	4.169
island	-0.7707	14	3.214	4.364
a city in	-0.7495	40	4.411	4.064
city in	-0.7436	41	4.387	4.075
in	-0.7394	87	4.257	3.839
a city	-0.7305	64	4.311	4.010
herodotos	-0.7102	9	5.132	4.111
another	-0.6990	7	4.198	4.203
some	-0.6752	10	3.922	4.234
or after	-0.6706	3	1.284	4.293
named	-0.6636	5	2.845	4.274

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.