gpt-5.4 v1 Evaluation

Generated: 2026-09-20 11:21:51 UTC

Pairs100

Distinct lemmas100

Slope0.977

R^20.988

BLEU-423.2%

chrF++52.7%

METEOR57.6%

ROUGE-L60.1%

BERTScore92.4%

COMET71.5%

BLEURT63.0%

Profile version id: 2172

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.4 using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.977
Intercept	2.569
R^2	0.988
P-value	3.99e-95
Pearson r	0.994
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	45.720
Mean absolute length residual	3.231
Mean absolute percent error	8.4%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.089, R^2 = 0.008, p = 0.3763.

chrF++ vs passage length — chrF++: positive; r = 0.046, R^2 = 0.002, p = 0.6490.

METEOR vs passage length — METEOR: negative; r = -0.069, R^2 = 0.005, p = 0.4971.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.096, R^2 = 0.009, p = 0.3413.

BERTScore vs passage length — BERTScore: significant negative; r = -0.308, R^2 = 0.095, p = 0.0018.

COMET vs passage length — COMET: negative; r = -0.044, R^2 = 0.002, p = 0.6661.

BLEURT vs passage length — BLEURT: significant negative; r = -0.599, R^2 = 0.359, p = 4.45e-11.

Trigram precision vs passage length — Trigram precision: positive; r = 0.044, R^2 = 0.002, p = 0.6614.

Trigram recall vs passage length — Trigram recall: positive; r = 0.025, R^2 = 6.33e-04, p = 0.8038.

Trigram F1 vs passage length — Trigram F1: positive; r = 0.036, R^2 = 0.001, p = 0.7207.

Trigram Jaccard vs passage length — Trigram Jaccard: positive; r = 0.021, R^2 = 4.34e-04, p = 0.8369.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.089	0.008	0.3763	0.00039	ok
chrF++	100	positive	0.046	0.002	0.6490	0.00013	ok
METEOR	100	negative	-0.069	0.005	0.4971	-0.00032	ok
ROUGE-L	100	negative	-0.096	0.009	0.3413	-0.00038	ok
BERTScore	100	significant negative	-0.308	0.095	0.0018	-0.00020	ok
COMET	100	negative	-0.044	0.002	0.6661	-0.00008	ok
BLEURT	100	significant negative	-0.599	0.359	4.45e-11	-0.00178	ok
Trigram precision	100	positive	0.044	0.002	0.6614	0.00019	ok
Trigram recall	100	positive	0.025	6.33e-04	0.8038	0.00011	ok
Trigram F1	100	positive	0.036	0.001	0.7207	0.00015	ok
Trigram Jaccard	100	positive	0.021	4.34e-04	0.8369	0.00006	ok

Translation Similarity Metrics

Mean BLEU-4	23.2%
Mean chrF++	52.7%
Mean METEOR	57.6%
Mean ROUGE-L	60.1%
Mean BERTScore	92.4%
Mean COMET	71.5%
Mean BLEURT	63.0%
Legacy smoothed corpus BLEU	27.8%
Unigram F1	63.8%
Bigram F1	35.2%
Trigram precision	21.0%
Trigram recall	21.8%
Trigram F1	21.4%
Trigram Jaccard	12.0%
4-gram F1	13.3%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	252	-14.460	21.2%	51.7%	54.3%	58.9%	89.1%	75.2%	47.2%	reviewed/approved	10878	gpt-5.4
2603	Κάρυστος	175	187	13.390	33.1%	60.6%	66.7%	64.1%	92.5%	76.1%	52.7%	reviewed/approved	10890	gpt-5.4
7266	Κώμη	79	91	11.218	10.0%	38.0%	34.5%	37.6%	89.6%	63.2%	43.5%	reviewed/approved	11025	gpt-5.4
2328	Καλλίπολις	43	55	10.404	13.0%	50.7%	42.0%	46.9%	91.1%	73.4%	62.4%	reviewed/approved	10830	gpt-5.4
2056	Καβειρία	112	102	-10.035	18.2%	53.4%	54.0%	57.9%	92.6%	72.4%	57.2%	reviewed/approved	10201	gpt-5.4
2468	Καπετώλιον	138	128	-9.447	15.0%	41.8%	44.7%	45.1%	89.3%	71.6%	43.3%	reviewed/approved	10869	gpt-5.4
2604	Καρχηδών	117	125	8.078	21.0%	54.6%	53.8%	53.9%	92.1%	77.0%	56.1%	reviewed/approved	10893	gpt-5.4
2119	Κάλλατις	65	74	7.902	19.2%	49.1%	49.1%	48.9%	90.5%	63.3%	61.6%	reviewed/approved	10818	gpt-5.4
2470	Καππαδοκία	68	76	6.970	14.1%	47.9%	60.2%	61.1%	92.2%	67.7%	53.3%	reviewed/approved	10872	gpt-5.4
2477	Καρδαμύλη	43	51	6.404	25.3%	58.7%	62.3%	53.2%	92.9%	76.0%	67.7%	reviewed/approved	10875	gpt-5.4
7254	Κύτα	84	91	6.332	27.5%	55.4%	50.8%	58.3%	91.5%	69.2%	54.2%	reviewed/approved	10989	gpt-5.4
2597	Καρπασία	112	106	-6.035	16.6%	49.4%	44.8%	48.6%	91.0%	69.2%	53.3%	reviewed/approved	10881	gpt-5.4
7255	Κυτέριον	22	30	5.929	17.6%	53.8%	67.3%	61.5%	92.7%	70.0%	67.4%	reviewed/approved	10992	gpt-5.4
3288	Κορώνεια	110	116	5.920	15.1%	48.1%	41.4%	49.3%	90.4%	72.0%	51.2%	reviewed/approved	10941	gpt-5.4
2079	Καισάρεια	39	35	-5.687	25.4%	54.6%	66.0%	67.6%	91.6%	71.8%	62.8%	reviewed/approved	10776	gpt-5.4
2342	Κάμιρος	54	61	5.653	19.5%	52.2%	59.5%	59.1%	92.6%	74.5%	56.2%	reviewed/approved	10854	gpt-5.4
7244	Κυρήνη	54	61	5.653	35.0%	60.5%	64.2%	60.9%	93.8%	76.1%	63.0%	reviewed/approved	10959	gpt-5.4
7259	Κύφος	54	61	5.653	24.0%	52.2%	63.1%	59.1%	92.4%	66.5%	50.0%	reviewed/approved	11004	gpt-5.4
2609	Κάσπειρος	135	129	-5.515	39.1%	62.7%	56.1%	67.4%	89.2%	72.3%	59.5%	reviewed/approved	10902	gpt-5.4
2465	Κάνωπος	83	89	5.309	35.9%	60.0%	69.4%	68.6%	93.2%	69.9%	58.1%	reviewed/approved	10866	gpt-5.4
7251	Κύρτος	66	62	-5.076	27.4%	58.7%	69.0%	68.2%	92.0%	68.7%	53.1%	reviewed/approved	10980	gpt-5.4
2621	Κάστνιον	25	32	4.997	15.1%	46.9%	62.7%	56.1%	92.9%	59.0%	67.2%	reviewed/approved	10908	gpt-5.4
7267	Κωνώπη	64	70	4.879	39.3%	59.2%	61.1%	65.7%	92.3%	69.6%	69.3%	reviewed/approved	11028	gpt-5.4
7261	Κύψελα	34	31	-4.800	34.8%	61.5%	68.6%	70.8%	95.5%	75.6%	71.8%	reviewed/approved	11010	gpt-5.4
2626	Κατάνη	95	100	4.580	17.8%	47.6%	45.6%	50.3%	90.4%	69.2%	50.5%	reviewed/approved	10926	gpt-5.4
2081	Καλάθη	49	46	-4.460	30.9%	58.1%	67.0%	73.7%	93.9%	65.8%	66.1%	reviewed/approved	10782	gpt-5.4
2622	Καστωλοῦ πεδίον	42	48	4.381	34.1%	55.0%	62.4%	62.2%	92.4%	71.1%	62.0%	reviewed/approved	10911	gpt-5.4
2080	Καλαβρία	18	16	-4.162	21.9%	52.1%	39.5%	47.1%	93.0%	72.5%	68.0%	reviewed/approved	10779	gpt-5.4
2327	Καλλιόπη	19	17	-4.139	17.3%	57.8%	59.8%	61.1%	91.5%	72.4%	62.9%	reviewed/approved	10827	gpt-5.4
2624	Κατάβαθμος	20	18	-4.116	29.1%	54.8%	47.3%	52.6%	91.9%	77.2%	66.2%	reviewed/approved	10917	gpt-5.4

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.324.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.7426	66	0.325	-0.632
απο	0.3651	27	2.105	-0.778
κωμη	0.2956	3	7.167	-0.222
λεγεται	0.2432	9	3.800	-0.376
οι	0.2387	11	3.465	-0.428
εν τη	0.2355	7	3.520	-0.265
πολιχνιον	0.2059	3	6.493	-0.201
αλλη	0.1960	4	6.161	-0.257
λεγεται και	0.1951	8	3.475	-0.302
της	0.1948	35	0.492	-0.265
δε και	0.1939	8	2.284	-0.199
εστι δε και	0.1899	3	6.801	-0.210
ου και	0.1837	4	6.354	-0.265
και το	0.1809	9	2.687	-0.266
εις	0.1774	7	1.434	-0.108
παιδος	0.1720	4	3.705	-0.154
απο του	0.1711	9	2.658	-0.263
δυο	0.1705	4	4.259	-0.177
θηλυκως	0.1692	5	4.247	-0.224
και θηλυκως	0.1692	5	4.247	-0.224

Negative term	Coefficient	Docs	Mean present	Mean absent
το	-0.2267	70	0.096	-0.225
θεοπομπος	-0.1881	5	-2.896	0.152
εκαταιος ασια	-0.1682	5	-2.012	0.106
ασια	-0.1682	5	-2.012	0.106
πολις θρακης	-0.1594	4	-2.486	0.104
και του	-0.1593	4	-3.233	0.135
δεκατω	-0.1578	3	-3.961	0.123
θρακης	-0.1576	5	-2.031	0.107
το εθνικον	-0.1555	56	-0.172	0.219
εθνικον	-0.1555	56	-0.172	0.219
εθνος	-0.1535	8	-3.175	0.276
ηρωδιανος	-0.1447	3	-8.652	0.268
ως	-0.1436	52	0.056	-0.060
μεταξυ	-0.1365	5	-1.111	0.058
πλησιον	-0.1361	8	-0.922	0.080
θηλυκον και	-0.1314	3	-4.598	0.142
γαρ	-0.1301	9	0.678	-0.067
ως το	-0.1269	3	-6.071	0.188
προς τη	-0.1234	4	-2.270	0.095
καππαδοκιας	-0.1179	3	-2.938	0.091

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.013.

Positive term	Coefficient	Docs	Mean present	Mean absent
γαρ	1.8186	9	7.906	2.769
εν	1.6324	37	4.426	2.530
και	1.6185	66	3.753	2.218
τη	1.1816	15	5.396	2.849
εστι	1.1036	24	5.181	2.616
απο	1.0980	27	5.310	2.462
την	1.0942	13	6.554	2.735
εις	0.9870	7	8.265	2.853
τω	0.9813	16	4.971	2.900
κωμη	0.9793	3	7.167	3.110
της	0.9060	35	4.530	2.532
λεγεται	0.8750	9	7.393	2.820
υπο	0.8470	4	6.847	3.081
αφ	0.8449	6	6.824	3.002
παρα	0.8325	7	6.867	2.958
εκαλειτο δε	0.8043	3	11.976	2.961
τριτη	0.7828	3	7.337	3.104
δε και	0.7634	8	7.321	2.876
δια	0.7507	16	5.230	2.851
δε	0.7386	46	3.957	2.613

Negative term	Coefficient	Docs	Mean present	Mean absent
και πολις	-1.0113	5	1.213	3.338
στραβων	-0.8437	15	3.202	3.237
δευτερω	-0.8329	3	0.991	3.301
νησος	-0.7556	9	2.372	3.316
το εθνικον	-0.6713	56	3.334	3.101
εθνικον	-0.6713	56	3.334	3.101
μια	-0.6576	3	1.197	3.294
τεταρτω	-0.5570	3	1.855	3.274
παυσανιας	-0.5505	4	2.867	3.247
δε απο	-0.5475	3	1.152	3.296
πολις και	-0.5398	6	2.296	3.291
τοπικα	-0.4987	3	2.311	3.260
τα τοπικα	-0.4987	3	2.311	3.260
τινες	-0.4796	6	2.108	3.303
εκ	-0.4695	6	2.941	3.250
αιγυπτου	-0.4542	3	2.166	3.264
πολιτης	-0.4481	18	3.530	3.166
θαλασση	-0.4277	3	1.395	3.288
πολις	-0.4273	68	3.387	2.901
φησι το	-0.4272	3	1.202	3.294

AI English terms predicting large absolute residuals

Features: 398; ridge alpha: 2.031; cross-validated R^2: 0.034.

Positive term	Coefficient	Docs	Mean present	Mean absent
for	1.6444	11	7.070	2.757
there	1.6017	33	4.803	2.458
on	1.5919	18	5.734	2.682
from	1.5303	42	4.564	2.266
it	1.4891	36	4.687	2.413
was	1.3095	19	5.211	2.767
when	1.3088	6	9.318	2.843
to	1.2716	25	5.206	2.573
also	1.2684	47	4.274	2.307
is	1.1956	90	3.438	1.372
said	1.1066	4	9.745	2.960
in	1.0670	74	3.507	2.448
there is	1.0665	24	5.255	2.592
caria	1.0436	3	8.698	3.062
as	1.0377	49	4.278	2.226
is also	0.9626	32	4.819	2.484
with	0.9585	17	4.880	2.894
there is also	0.9518	24	5.255	2.592
the	0.9496	97	3.242	2.889
used	0.9111	5	7.728	2.995

Negative term	Coefficient	Docs	Mean present	Mean absent
an	-1.1859	22	2.223	3.516
island	-0.8037	14	2.175	3.403
and a	-0.7988	5	1.816	3.306
forms are	-0.7647	3	1.347	3.290
the ethnic is	-0.6941	62	3.158	3.352
the ethnic	-0.6723	63	3.320	3.080
ethnic is	-0.6448	64	3.245	3.208
ethnic	-0.6391	66	3.396	2.911
and a city	-0.6256	3	0.864	3.305
strabo	-0.6211	15	3.202	3.237
and also	-0.5903	3	2.263	3.261
book	-0.5874	39	3.085	3.325
syria	-0.5635	3	0.717	3.309
is also called	-0.5386	3	0.986	3.301
inhabitants	-0.5338	3	2.385	3.258
of his	-0.5257	9	1.915	3.362
herodotus in the	-0.5173	3	0.836	3.306
pausanias	-0.4932	4	2.867	3.247
has been	-0.4889	3	2.162	3.264
a city	-0.4860	63	3.333	3.058

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 1.425; cross-validated R^2: 0.035.

Positive term	Coefficient	Docs	Mean present	Mean absent
there	2.1361	34	5.080	2.279
with	1.9438	16	5.165	2.863
for	1.8551	11	6.275	2.855
on	1.8322	16	4.869	2.920
when	1.6727	7	8.575	2.829
from	1.5926	37	4.579	2.440
the	1.5405	90	3.332	2.329
also	1.5104	43	4.333	2.401
his on	1.3338	3	9.980	3.023
also said	1.3150	6	9.671	2.820
said	1.3150	6	9.671	2.820
into	1.3145	3	8.760	3.060
be	1.2503	15	6.132	2.720
situated	1.2458	4	8.836	2.998
is also said	1.2299	5	10.474	2.850
is also	1.1729	31	4.936	2.465
and	1.1555	52	4.010	2.388
forms	1.1478	4	10.228	2.940
long	1.1445	3	11.709	2.969
is with	1.1118	4	8.537	3.010

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnonym is	-1.0338	52	3.068	3.409
ethnonym is	-1.0338	52	3.068	3.409
but	-1.0119	12	3.051	3.256
island	-0.9639	14	2.175	3.403
strabo	-0.9520	15	3.202	3.237
syria	-0.8975	3	0.717	3.309
in syria	-0.8975	3	0.717	3.309
herodotos book	-0.8698	5	1.749	3.309
ethnonyms	-0.8574	3	1.477	3.286
through	-0.8194	3	0.835	3.306
territory	-0.7979	5	1.990	3.297
one of the	-0.7768	4	2.043	3.281
is a	-0.7638	31	3.835	2.960
one of	-0.7190	5	2.526	3.269
city and	-0.7045	4	1.818	3.290
a city and	-0.7045	4	1.818	3.290
it is also	-0.6718	3	0.880	3.304
also a city	-0.6673	4	4.777	3.167
also called	-0.6673	6	2.826	3.257
an island	-0.6589	9	1.922	3.361

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.