gpt-5.6-sol v1 Evaluation

Generated: 2026-09-20 11:22:41 UTC

Pairs100

Distinct lemmas100

Slope0.998

R^20.974

BLEU-423.3%

chrF++51.8%

METEOR56.1%

ROUGE-L57.7%

BERTScore92.2%

COMET71.6%

BLEURT63.1%

Profile version id: 2175

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.6 Sol using prompt text from gpt-5.5 v1; approved-human-only for the 100-row Kappa corpus.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	0.998
Intercept	3.145
R^2	0.974
P-value	1.44e-79
Pearson r	0.987
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	47.190
Mean absolute length residual	4.288
Mean absolute percent error	13.1%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.084, R^2 = 0.007, p = 0.4037.

chrF++ vs passage length — chrF++: positive; r = 0.017, R^2 = 2.75e-04, p = 0.8700.

METEOR vs passage length — METEOR: negative; r = -0.107, R^2 = 0.012, p = 0.2878.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.080, R^2 = 0.006, p = 0.4285.

BERTScore vs passage length — BERTScore: significant negative; r = -0.212, R^2 = 0.045, p = 0.0339.

COMET vs passage length — COMET: negative; r = -0.053, R^2 = 0.003, p = 0.5980.

BLEURT vs passage length — BLEURT: significant negative; r = -0.422, R^2 = 0.178, p = 1.21e-05.

Trigram precision vs passage length — Trigram precision: negative; r = -0.064, R^2 = 0.004, p = 0.5271.

Trigram recall vs passage length — Trigram recall: negative; r = -0.076, R^2 = 0.006, p = 0.4544.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.069, R^2 = 0.005, p = 0.4968.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.096, R^2 = 0.009, p = 0.3418.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.084	0.007	0.4037	-0.00039	ok
chrF++	100	positive	0.017	2.75e-04	0.8700	0.00007	ok
METEOR	100	negative	-0.107	0.012	0.2878	-0.00060	ok
ROUGE-L	100	negative	-0.080	0.006	0.4285	-0.00039	ok
BERTScore	100	significant negative	-0.212	0.045	0.0339	-0.00020	ok
COMET	100	negative	-0.053	0.003	0.5980	-0.00014	ok
BLEURT	100	significant negative	-0.422	0.178	1.21e-05	-0.00158	ok
Trigram precision	100	negative	-0.064	0.004	0.5271	-0.00031	ok
Trigram recall	100	negative	-0.076	0.006	0.4544	-0.00039	ok
Trigram F1	100	negative	-0.069	0.005	0.4968	-0.00034	ok
Trigram Jaccard	100	negative	-0.096	0.009	0.3418	-0.00032	ok

Translation Similarity Metrics

Mean BLEU-4	23.3%
Mean chrF++	51.8%
Mean METEOR	56.1%
Mean ROUGE-L	57.7%
Mean BERTScore	92.2%
Mean COMET	71.6%
Mean BLEURT	63.1%
Legacy smoothed corpus BLEU	25.0%
Unigram F1	61.4%
Bigram F1	32.9%
Trigram precision	18.7%
Trigram recall	20.1%
Trigram F1	19.4%
Trigram Jaccard	10.7%
4-gram F1	11.5%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2484	Καρία	270	244	-28.503	19.1%	50.7%	43.8%	51.4%	89.6%	75.9%	48.8%	reviewed/approved	10605	gpt-5.6-sol
3258	Κεκρυφάλεια	26	1	-28.083	0.0%	0.2%	0.0%	0.0%	78.4%	28.0%	6.4%	reviewed/approved	10665	gpt-5.6-sol
2605	Κάσιον	56	74	14.988	14.8%	48.6%	46.0%	46.2%	90.9%	68.5%	58.3%	reviewed/approved	10623	gpt-5.6-sol
2604	Καρχηδών	117	133	13.133	20.3%	53.8%	57.3%	55.6%	91.4%	75.9%	56.0%	reviewed/approved	10620	gpt-5.6-sol
3288	Κορώνεια	110	126	13.117	15.1%	51.2%	43.0%	53.2%	90.9%	71.2%	56.6%	reviewed/approved	10668	gpt-5.6-sol
2328	Καλλίπολις	43	58	11.957	14.9%	51.6%	56.0%	55.4%	92.4%	73.2%	67.0%	reviewed/approved	10557	gpt-5.6-sol
2621	Κάστνιον	25	40	11.914	21.6%	52.5%	60.2%	49.2%	92.3%	65.9%	64.9%	reviewed/approved	10635	gpt-5.6-sol
2116	Κάληρος	40	32	-11.050	16.4%	51.3%	56.4%	55.6%	93.7%	76.3%	73.4%	reviewed/approved	10536	gpt-5.6-sol
2115	Καλὴ ἀκτή	64	78	11.007	17.1%	46.5%	59.5%	57.7%	89.5%	64.7%	48.6%	reviewed/approved	10533	gpt-5.6-sol
2455	Κάναστρον	60	52	-11.002	13.7%	47.5%	56.1%	60.7%	91.7%	65.2%	56.9%	reviewed/approved	10590	gpt-5.6-sol
2468	Καπετώλιον	138	150	9.183	16.5%	44.9%	45.5%	47.2%	86.6%	65.2%	45.4%	reviewed/approved	10596	gpt-5.6-sol
7266	Κώμη	79	90	8.043	7.2%	38.7%	33.9%	37.9%	89.5%	65.7%	42.7%	reviewed/approved	10752	gpt-5.6-sol
2119	Κάλλατις	65	76	8.010	16.1%	47.2%	47.2%	44.0%	90.4%	62.8%	56.7%	reviewed/approved	10545	gpt-5.6-sol
2056	Καβειρία	112	107	-7.879	12.5%	46.0%	37.7%	42.0%	90.8%	65.1%	44.7%	reviewed/approved	10466	gpt-5.6-sol
2603	Κάρυστος	175	185	7.271	31.6%	61.4%	67.5%	68.3%	93.2%	78.1%	57.8%	reviewed/approved	10617	gpt-5.6-sol
7247	Κύρνος	50	46	-7.026	14.5%	42.7%	53.6%	52.1%	92.2%	70.1%	62.3%	reviewed/approved	10695	gpt-5.6-sol
2470	Καππαδοκία	68	78	7.017	12.8%	48.5%	54.1%	61.6%	92.2%	66.8%	51.0%	reviewed/approved	10599	gpt-5.6-sol
2342	Κάμιρος	54	64	6.983	27.4%	61.3%	70.9%	67.8%	93.9%	74.4%	67.4%	reviewed/approved	10581	gpt-5.6-sol
2055	Καβασσός	109	118	6.114	16.7%	46.4%	39.2%	45.8%	88.2%	71.1%	43.7%	reviewed/approved	10463	gpt-5.6-sol
7246	Κύρις	20	17	-6.097	25.9%	47.9%	43.7%	48.6%	90.0%	59.9%	43.2%	reviewed/approved	10692	gpt-5.6-sol
7251	Κύρτος	66	75	6.012	36.9%	58.1%	68.4%	65.2%	91.3%	67.7%	53.6%	reviewed/approved	10707	gpt-5.6-sol
7264	Κωλιάς	61	70	6.000	26.7%	54.5%	55.6%	59.5%	91.7%	67.3%	56.9%	reviewed/approved	10746	gpt-5.6-sol
7259	Κύφος	54	63	5.983	20.9%	52.0%	61.2%	58.1%	91.8%	66.9%	51.5%	reviewed/approved	10731	gpt-5.6-sol
2624	Κατάβαθμος	20	29	5.903	43.5%	73.2%	82.1%	69.4%	94.9%	81.4%	79.8%	reviewed/approved	10644	gpt-5.6-sol
2078	Καινύς	22	20	-5.093	40.2%	55.6%	57.9%	57.1%	93.5%	79.3%	66.1%	reviewed/approved	10497	gpt-5.6-sol
7262	Κῶβρυς	22	20	-5.093	8.2%	43.1%	29.2%	42.9%	91.6%	67.4%	61.1%	reviewed/approved	10740	gpt-5.6-sol
7257	Κυτώνιον	25	23	-5.086	31.4%	55.6%	60.5%	66.7%	92.3%	72.0%	61.0%	reviewed/approved	10725	gpt-5.6-sol
7261	Κύψελα	34	32	-5.064	31.4%	57.8%	59.7%	63.6%	93.9%	73.6%	67.8%	reviewed/approved	10737	gpt-5.6-sol
2346	Κάναι	77	85	5.038	33.4%	60.0%	74.2%	74.1%	92.2%	75.2%	58.0%	reviewed/approved	10587	gpt-5.6-sol
3496	Κοτιάειον	71	69	-4.976	14.8%	45.3%	57.1%	55.7%	90.7%	59.9%	59.1%	reviewed/approved	10671	gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.085.

Positive term	Coefficient	Docs	Mean present	Mean absent
ου	2.0582	14	3.830	-0.623
της	1.7776	35	1.294	-0.697
του	1.6427	37	2.176	-1.278
ως	1.6300	52	1.244	-1.348
εν	1.6017	37	2.434	-1.430
εστι δε	1.5524	5	6.867	-0.361
εν τη	1.5437	7	5.970	-0.449
ου και	1.4907	4	8.807	-0.367
ορος	1.4683	6	6.714	-0.429
κωμη	1.4208	3	8.002	-0.247
εστι δε και	1.4056	3	10.402	-0.322
και	1.4020	66	0.701	-1.361
πολις	1.3702	68	0.280	-0.595
πολιτης	1.3594	18	3.877	-0.851
αφ ου	1.2805	5	4.902	-0.258
εξ	1.1966	3	7.384	-0.228
πολιχνιον	1.1640	3	8.335	-0.258
δευτερα	1.1580	3	8.330	-0.258
εκ	1.1553	6	5.326	-0.340
τον	1.0533	10	3.419	-0.380

Negative term	Coefficient	Docs	Mean present	Mean absent
τινες	-2.4303	6	-7.709	0.492
τινες δε	-2.1771	3	-11.721	0.362
περι	-2.1008	11	-3.329	0.411
το	-2.0320	70	-0.210	0.489
φασι	-1.8796	8	-3.725	0.324
πελοποννησου	-1.7748	3	-6.030	0.186
τις	-1.7317	3	-5.685	0.176
το εθνικον	-1.6493	56	-1.532	1.950
εθνικον	-1.6493	56	-1.532	1.950
θρακης	-1.4859	5	-5.467	0.288
εθνος	-1.3938	8	-5.860	0.510
εστι και	-1.3022	20	-0.561	0.140
ακρα	-1.2930	4	-3.731	0.155
το θηλυκον	-1.1928	6	-5.740	0.366
ουτως	-1.1819	5	-7.849	0.413
θεοπομπος	-1.1485	5	-2.832	0.149
αυτην	-1.0605	9	-1.661	0.164
πολις θρακης	-0.8857	4	-4.084	0.170
εκαλειτο	-0.8673	7	-2.731	0.206
οτι	-0.7939	5	0.032	-0.002

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.292.

Positive term	Coefficient	Docs	Mean present	Mean absent
δε	1.8766	46	5.921	2.898
περι	1.5366	11	9.722	3.617
πελοποννησου	1.3986	3	14.775	3.964
τις	1.3298	3	14.429	3.975
ακρα	1.2013	4	10.311	4.037
γαρ	1.1600	9	8.889	3.833
τινες	1.1163	6	9.031	3.986
εκαλειτο	1.0984	7	10.308	3.835
τινες δε	1.0646	3	11.721	4.059
την	0.9910	13	7.562	3.799
και	0.9258	66	4.724	3.442
απο της	0.9158	6	11.637	3.819
φασι	0.8622	8	8.822	3.894
εστι δε	0.7531	5	10.019	3.987
ει	0.7529	4	13.889	3.888
ου	0.7348	14	6.103	3.993
παρα	0.7140	7	10.060	3.854
δια	0.6785	16	6.463	3.874
αφ ου	0.6736	5	8.054	4.090
εστι	0.6690	24	5.722	3.836

Negative term	Coefficient	Docs	Mean present	Mean absent
δευτερω	-0.9355	3	0.380	4.409
πολις	-0.8264	68	4.066	4.762
εκαταιος	-0.7141	14	3.128	4.477
τον	-0.7020	10	3.683	4.356
στραβων	-0.6347	15	3.492	4.429
οι	-0.4964	11	4.284	4.289
νησος	-0.4734	9	2.796	4.436
εκαταιος ασια	-0.4665	5	1.490	4.436
ασια	-0.4665	5	1.490	4.436
λυκιας	-0.4610	3	3.102	4.325
πολιτης	-0.4536	18	4.474	4.248
και το	-0.4446	9	4.364	4.281
ως στραβων	-0.4295	4	1.779	4.393
ποσειδωνος	-0.4142	3	1.674	4.369
ηροδοτος	-0.3986	9	5.646	4.154
παυσανιας	-0.3824	4	3.458	4.323
φησι το	-0.3809	3	1.043	4.389
προς τη	-0.3710	4	2.535	4.361
τριτω	-0.3670	3	1.356	4.379
λιβυης	-0.3652	4	4.046	4.299

AI English terms predicting large absolute residuals

Features: 401; ridge alpha: 5.878; cross-validated R^2: -0.306.

Positive term	Coefficient	Docs	Mean present	Mean absent
with	1.0702	19	6.728	3.716
and	1.0270	56	5.203	3.124
for	0.9283	12	7.411	3.863
usage	0.8957	5	10.045	3.985
caria	0.8744	3	14.523	3.972
on	0.8659	26	6.164	3.629
was called	0.7816	6	11.162	3.850
was	0.7715	24	6.674	3.535
diphthong	0.7245	3	13.269	4.011
he	0.6211	7	8.378	3.981
it	0.5910	40	5.636	3.390
is a	0.5815	6	11.106	3.853
by	0.5504	16	6.555	3.857
either	0.5476	3	17.558	3.878
was named	0.5290	5	10.899	3.941
it was named	0.5268	4	11.805	3.975
zeus	0.5229	5	12.997	3.830
it was	0.5220	13	7.358	3.830
from	0.5010	36	5.698	3.496
a mountain	0.4971	3	10.647	4.092

Negative term	Coefficient	Docs	Mean present	Mean absent
a city	-0.7131	66	3.794	5.248
book	-0.6039	39	3.866	4.559
an	-0.5941	24	3.310	4.597
of the	-0.5376	40	4.470	4.168
herodotus book	-0.4974	5	1.649	4.427
city	-0.4973	68	4.066	4.762
is called	-0.4908	24	3.883	4.416
are called	-0.4814	9	3.210	4.395
hecataeus	-0.4797	14	3.128	4.477
inhabitants	-0.4120	6	2.185	4.423
of his	-0.4100	10	3.025	4.429
are	-0.3944	26	4.673	4.153
settlement	-0.3874	3	1.382	4.378
is called a	-0.3874	21	3.825	4.412
called a	-0.3874	21	3.825	4.412
strabo	-0.3858	15	3.492	4.429
book of the	-0.3816	8	2.396	4.453
book of	-0.3727	16	4.364	4.274
near	-0.3701	14	3.456	4.424
it is also	-0.3692	4	1.229	4.416

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: -0.210.

Positive term	Coefficient	Docs	Mean present	Mean absent
however	2.5036	10	11.400	3.498
the	2.5023	90	4.533	2.085
in the peloponnese	1.6885	3	14.775	3.964
peloponnese	1.6885	3	14.775	3.964
the peloponnese	1.6885	3	14.775	3.964
which the	1.5405	3	12.374	4.038
on	1.5088	16	6.553	3.857
this	1.4626	15	8.759	3.500
and	1.3728	52	5.295	3.198
promontory	1.3571	3	12.068	4.048
is with	1.3343	4	13.230	3.916
mountain	1.2479	3	11.391	4.069
used to be	1.2316	6	11.513	3.827
his on	1.2287	3	15.491	3.942
an island in	1.2201	3	10.985	4.081
island in	1.2201	3	10.985	4.081
island in the	1.2201	3	10.985	4.081
say that it	1.2172	3	11.721	4.059
karia	1.2171	3	14.523	3.972
with	1.1857	16	6.628	3.843

Negative term	Coefficient	Docs	Mean present	Mean absent
a city	-0.9785	64	3.851	5.067
in book	-0.9388	16	3.867	4.369
in book of	-0.9350	14	2.165	4.634
strabo	-0.8520	15	3.492	4.429
book	-0.8220	39	3.866	4.559
but	-0.7975	12	3.073	4.454
city	-0.7930	70	4.107	4.711
hekataios	-0.7739	14	3.128	4.477
is a	-0.7663	31	4.711	4.099
herodotos book	-0.7342	5	1.649	4.427
are	-0.7240	25	5.389	3.922
near	-0.6821	16	3.670	4.406
they	-0.6795	12	4.806	4.218
those	-0.6505	5	2.845	4.364
lykia	-0.6442	3	3.102	4.325
in lykia	-0.6442	3	3.102	4.325
written	-0.6192	3	3.295	4.319
they are	-0.6131	7	3.470	4.350
hekataios in	-0.6078	12	3.134	4.446
one of	-0.6020	5	2.220	4.397

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.