claude_sonnet_5 v1 Evaluation

Generated: 2026-09-20 11:19:44 UTC

Pairs100

Distinct lemmas100

Slope1.023

R^20.979

BLEU-421.0%

chrF++53.0%

METEOR62.3%

ROUGE-L62.2%

BERTScore92.7%

COMET73.2%

BLEURT66.0%

Profile version id: 2009

First translation using this version: 2026-07-03

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Sonnet 5; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.023
Intercept	2.794
R^2	0.979
P-value	2.53e-84
Pearson r	0.990
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	47.970
Mean absolute length residual	3.822
Mean absolute percent error	12.2%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: positive; r = 0.051, R^2 = 0.003, p = 0.6145.

chrF++ vs passage length — chrF++: positive; r = 0.022, R^2 = 4.92e-04, p = 0.8266.

METEOR vs passage length — METEOR: negative; r = -0.151, R^2 = 0.023, p = 0.1332.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.189, R^2 = 0.036, p = 0.0599.

BERTScore vs passage length — BERTScore: significant negative; r = -0.349, R^2 = 0.122, p = 0.0004.

COMET vs passage length — COMET: negative; r = -0.100, R^2 = 0.010, p = 0.3239.

BLEURT vs passage length — BLEURT: significant negative; r = -0.603, R^2 = 0.364, p = 3.16e-11.

Trigram precision vs passage length — Trigram precision: positive; r = 0.002, R^2 = 2.77e-06, p = 0.9869.

Trigram recall vs passage length — Trigram recall: negative; r = -0.023, R^2 = 5.40e-04, p = 0.8185.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.009, R^2 = 7.34e-05, p = 0.9326.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.024, R^2 = 5.66e-04, p = 0.8142.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	positive	0.051	0.003	0.6145	0.00021	ok
chrF++	100	positive	0.022	4.92e-04	0.8266	0.00007	ok
METEOR	100	negative	-0.151	0.023	0.1332	-0.00067	ok
ROUGE-L	100	negative	-0.189	0.036	0.0599	-0.00071	ok
BERTScore	100	significant negative	-0.349	0.122	0.0004	-0.00028	ok
COMET	100	negative	-0.100	0.010	0.3239	-0.00018	ok
BLEURT	100	significant negative	-0.603	0.364	3.16e-11	-0.00172	ok
Trigram precision	100	positive	0.002	2.77e-06	0.9869	7.62e-06	ok
Trigram recall	100	negative	-0.023	5.40e-04	0.8185	-0.00011	ok
Trigram F1	100	negative	-0.009	7.34e-05	0.9326	-0.00004	ok
Trigram Jaccard	100	negative	-0.024	5.66e-04	0.8142	-0.00007	ok

Translation Similarity Metrics

Mean BLEU-4	21.0%
Mean chrF++	53.0%
Mean METEOR	62.3%
Mean ROUGE-L	62.2%
Mean BERTScore	92.7%
Mean COMET	73.2%
Mean BLEURT	66.0%
Legacy smoothed corpus BLEU	27.7%
Unigram F1	65.8%
Bigram F1	36.4%
Trigram precision	20.6%
Trigram recall	22.5%
Trigram F1	21.5%
Trigram Jaccard	12.1%
4-gram F1	13.5%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2604	Καρχηδών	117	155	32.488	12.5%	51.0%	54.2%	48.2%	90.0%	74.2%	57.3%	reviewed/approved	6229	claude-sonnet-5
2484	Καρία	270	261	-18.066	16.5%	48.4%	47.9%	51.6%	86.5%	76.8%	49.3%	reviewed/approved	6224	claude-sonnet-5
2119	Κάλλατις	65	85	15.696	12.7%	48.3%	56.2%	48.0%	91.3%	63.3%	60.1%	reviewed/approved	6204	claude-sonnet-5
2056	Καβειρία	112	103	-14.396	14.1%	50.7%	56.3%	56.7%	92.3%	75.0%	59.4%	reviewed/approved	6177	claude-sonnet-5
2465	Κάνωπος	83	100	12.278	26.5%	60.9%	76.6%	68.9%	93.2%	74.1%	63.3%	reviewed/approved	6220	claude-sonnet-5
7266	Κώμη	79	95	11.370	3.9%	34.6%	37.0%	32.2%	89.9%	65.6%	43.5%	reviewed/approved	6273	claude-sonnet-5
7264	Κωλιάς	61	76	10.789	15.3%	50.3%	49.5%	51.1%	91.4%	69.3%	59.6%	reviewed/approved	6271	claude-sonnet-5
2115	Καλὴ ἀκτή	64	79	10.719	12.3%	47.2%	64.3%	58.7%	90.8%	64.8%	51.5%	reviewed/approved	6200	claude-sonnet-5
2328	Καλλίπολις	43	57	10.207	16.8%	52.4%	52.7%	56.0%	91.8%	74.7%	64.3%	reviewed/approved	6208	claude-sonnet-5
2116	Κάληρος	40	35	-8.724	16.2%	49.8%	50.0%	53.3%	93.9%	78.6%	75.9%	reviewed/approved	6201	claude-sonnet-5
2342	Κάμιρος	54	66	7.951	25.3%	62.9%	78.7%	68.3%	93.9%	74.8%	66.1%	reviewed/approved	6216	claude-sonnet-5
2455	Κάναστρον	60	57	-7.188	14.8%	48.1%	57.8%	59.8%	92.4%	66.7%	58.7%	reviewed/approved	6219	claude-sonnet-5
2625	Κατακεκαυμένη	46	57	7.137	18.7%	55.6%	65.9%	62.1%	93.8%	74.8%	69.9%	reviewed/approved	6238	claude-sonnet-5
2057	Καβελλιών	34	31	-6.584	23.8%	53.8%	52.3%	58.5%	92.6%	71.5%	59.6%	reviewed/approved	6178	claude-sonnet-5
3530	Κριώα	26	23	-6.398	10.9%	40.4%	43.4%	57.1%	91.6%	62.9%	61.6%	reviewed/approved	6248	claude-sonnet-5
7244	Κυρήνη	54	52	-6.049	27.9%	52.2%	49.9%	54.7%	94.2%	73.8%	63.2%	reviewed/approved	6251	claude-sonnet-5
2468	Καπετώλιον	138	150	6.000	10.0%	40.1%	32.5%	38.2%	86.0%	65.0%	44.7%	reviewed/approved	6221	claude-sonnet-5
2627	Κατάνειρα	21	30	5.718	6.3%	38.8%	43.2%	47.1%	89.6%	65.3%	61.7%	reviewed/approved	6239	claude-sonnet-5
3288	Κορώνεια	110	121	5.650	23.0%	56.1%	59.4%	62.1%	92.1%	71.1%	58.3%	reviewed/approved	6245	claude-sonnet-5
7248	Κύρου πόλις	32	41	5.462	11.4%	48.0%	41.8%	43.8%	88.2%	67.2%	61.7%	reviewed/approved	6255	claude-sonnet-5
2059	Καδμεία	20	28	4.741	25.6%	53.4%	41.0%	50.0%	93.6%	74.8%	66.8%	reviewed/approved	6180	claude-sonnet-5
7261	Κύψελα	34	33	-4.584	30.9%	57.6%	72.5%	68.7%	96.2%	74.7%	71.5%	reviewed/approved	6268	claude-sonnet-5
3496	Κοτιάειον	71	71	-4.444	16.5%	46.5%	56.7%	59.2%	91.1%	68.2%	63.8%	reviewed/approved	6246	claude-sonnet-5
7257	Κυτώνιον	25	24	-4.375	31.4%	57.6%	61.6%	65.3%	92.8%	71.4%	64.6%	reviewed/approved	6264	claude-sonnet-5
2078	Καινύς	22	21	-4.305	52.8%	67.8%	76.6%	74.4%	95.2%	80.5%	75.6%	reviewed/approved	6188	claude-sonnet-5
2075	Καινίνη	15	14	-4.143	39.8%	57.7%	73.1%	75.9%	95.2%	80.5%	75.9%	reviewed/approved	6186	claude-sonnet-5
2081	Καλάθη	49	49	-3.933	26.5%	53.7%	54.8%	63.3%	93.1%	67.5%	65.4%	reviewed/approved	6192	claude-sonnet-5
2603	Κάρυστος	175	178	-3.860	30.6%	57.3%	68.0%	61.2%	92.8%	75.8%	52.9%	reviewed/approved	6228	claude-sonnet-5
2117	Καλησία	19	26	3.764	11.8%	48.8%	70.2%	57.8%	92.6%	74.4%	67.8%	reviewed/approved	6202	claude-sonnet-5
2079	Καισάρεια	39	39	-3.700	25.0%	53.0%	60.6%	61.5%	91.8%	72.2%	63.8%	reviewed/approved	6190	claude-sonnet-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.184.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.4650	66	0.201	-0.391
δια του	0.3703	10	5.219	-0.580
εστι δε και	0.3620	3	12.115	-0.375
εστι δε	0.3590	5	6.533	-0.344
δε και	0.3551	8	1.665	-0.145
εν τη	0.3486	7	5.373	-0.404
ος	0.3269	6	6.619	-0.422
κωμη	0.2808	3	6.765	-0.209
πολιτης	0.2802	18	3.456	-0.759
εκαλειτο δε	0.2688	3	3.521	-0.109
πολιχνιον	0.2634	3	8.106	-0.251
του	0.2595	37	1.569	-0.921
και το	0.2420	9	2.241	-0.222
ως	0.2342	52	0.839	-0.909
εν	0.2241	37	1.502	-0.882
τοις	0.2231	3	5.921	-0.183
γαρ	0.2100	9	1.994	-0.197
απο	0.1974	27	1.151	-0.426
τοπος	0.1903	4	4.987	-0.208
δε	0.1887	46	0.351	-0.299

Negative term	Coefficient	Docs	Mean present	Mean absent
το	-0.4559	70	-0.519	1.211
το εθνικον	-0.4185	56	-1.309	1.666
εθνικον	-0.4185	56	-1.309	1.666
εθνος	-0.2606	8	-5.229	0.455
τω	-0.2545	16	-1.460	0.278
οτι	-0.2380	5	-2.727	0.144
θρακης	-0.2379	5	-3.752	0.197
τα	-0.2176	12	-1.995	0.272
εστι και	-0.2059	20	-0.889	0.222
υπο	-0.2027	4	-2.490	0.104
κατα	-0.1944	10	0.177	-0.020
πλησιον	-0.1844	8	-2.090	0.182
αυτην	-0.1829	9	-2.263	0.224
πολις θρακης	-0.1703	4	-2.893	0.121
ηροδοτος	-0.1650	9	-2.692	0.266
το θηλυκον	-0.1628	6	-5.117	0.327
εκαταιος ασια	-0.1509	5	-1.966	0.103
ασια	-0.1509	5	-1.966	0.103
πολις της	-0.1464	3	-6.808	0.211
εκει	-0.1454	3	-4.753	0.147

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.005.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	2.7450	66	4.301	2.892
εκαλειτο	2.2696	7	10.060	3.352
εστι δε	1.7890	5	13.009	3.338
εκαλειτο δε	1.7333	3	18.138	3.379
κατα	1.5709	10	9.454	3.196
εστι	1.5635	24	6.685	2.917
δε και	1.3823	8	8.520	3.413
εστι δε και	1.3815	3	13.310	3.528
δε	1.3502	46	4.741	3.039
δια του	1.2932	10	9.946	3.141
ει	1.1941	4	13.080	3.436
της	1.0463	35	5.505	2.915
γαρ	1.0298	9	7.296	3.478
δια	0.9582	16	6.921	3.232
εκαλειτο δε και	0.9572	3	18.138	3.379
ος	0.9483	6	8.100	3.549
πολιχνιον	0.8820	3	9.162	3.657
απο	0.8422	27	6.296	2.907
λιβυης	0.8319	4	13.291	3.427
ιβηριας	0.8084	3	13.932	3.509

Negative term	Coefficient	Docs	Mean present	Mean absent
το εθνικον	-1.0245	56	3.447	4.298
εθνικον	-1.0245	56	3.447	4.298
οι	-0.7665	11	3.053	3.917
και πολις	-0.7506	5	2.224	3.906
ηροδοτος	-0.7389	9	3.409	3.863
δευτερω	-0.6968	3	1.280	3.900
τους	-0.6815	4	1.368	3.924
εκαταιος	-0.6563	14	2.533	4.032
φασι	-0.6257	8	2.846	3.907
αιγυπτου	-0.5956	3	1.110	3.906
ορος	-0.5593	6	2.349	3.916
αυτην	-0.5319	9	2.733	3.929
λυκιας	-0.5115	3	1.746	3.886
δια το	-0.5111	3	1.502	3.894
καππαδοκιας	-0.5047	3	2.341	3.868
εκληθη	-0.5012	4	2.081	3.894
προς τη	-0.5002	4	2.513	3.876
του δε	-0.4839	3	1.544	3.892
το	-0.4504	70	3.459	4.669
και του	-0.4470	4	2.367	3.882

AI English terms predicting large absolute residuals

Features: 421; ridge alpha: 2.894; cross-validated R^2: -0.023.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	2.2055	58	4.644	2.687
was	1.9262	16	7.410	3.138
with	1.7840	19	6.894	3.101
on	1.7738	22	5.383	3.381
was also called	1.4806	3	13.280	3.529
is a	1.3254	7	11.441	3.248
written with	1.2969	4	15.911	3.318
name in	1.2945	3	15.403	3.464
by	1.2233	17	7.081	3.154
for	1.2202	11	6.874	3.445
there is	1.2055	25	6.267	3.007
written	1.1989	5	13.617	3.306
from	1.1562	42	5.499	2.607
this	1.1348	18	7.732	2.963
city and	1.1217	5	8.374	3.582
while	1.0807	3	16.435	3.432
states	1.0441	3	16.348	3.434
come	1.0385	5	12.288	3.376
it	1.0363	41	5.151	2.898
likewise	1.0250	3	18.851	3.357

Negative term	Coefficient	Docs	Mean present	Mean absent
one	-0.9282	11	2.881	3.938
a city	-0.8446	63	3.923	3.650
the ethnic	-0.8402	58	3.471	4.306
ethnic	-0.7798	66	3.659	4.137
the ethnic is	-0.7721	11	2.736	3.956
ethnic name	-0.7557	37	3.308	4.124
ethnic name is	-0.7269	34	3.345	4.067
river	-0.7102	6	1.962	3.940
near	-0.6853	19	2.905	4.037
cappadocia	-0.6758	4	2.412	3.881
the ethnic name	-0.6752	36	3.361	4.081
call	-0.6746	5	2.145	3.910
ethnic is	-0.6726	16	3.532	3.877
two	-0.6634	3	0.767	3.916
herodotus	-0.6321	9	3.409	3.863
name is	-0.6140	37	3.569	3.970
in the feminine	-0.6129	4	1.447	3.921
feminine form is	-0.6109	6	1.689	3.958
to the	-0.5853	7	1.944	3.963
inhabitants	-0.5349	3	1.179	3.903

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: -0.037.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	1.7943	52	4.737	2.830
like	1.3782	5	10.421	3.474
for	1.2578	11	7.327	3.389
is with	1.2296	4	14.942	3.358
also	1.1644	43	4.850	3.046
with	1.0665	16	7.054	3.206
it	1.0456	37	5.361	2.918
called	1.0169	21	5.499	3.376
there	1.0129	34	6.147	2.624
used	0.9813	8	10.602	3.232
used to	0.9813	8	10.602	3.232
on	0.9509	16	5.541	3.494
the	0.9463	90	4.004	2.181
used to be	0.9204	6	11.155	3.354
came	0.9180	4	13.819	3.405
to	0.9121	31	5.313	3.152
this	0.8617	15	7.968	3.090
to be called	0.8607	5	12.614	3.359
their	0.8473	3	18.138	3.379
there is	0.8456	24	5.777	3.204

Negative term	Coefficient	Docs	Mean present	Mean absent
the ethnonym is	-0.9796	52	3.175	4.522
ethnonym is	-0.9796	52	3.175	4.522
ethnonym	-0.8106	57	3.427	4.346
the ethnonym	-0.7850	56	3.458	4.285
herodotos	-0.6419	9	3.409	3.863
kappadokia	-0.6150	4	2.412	3.881
herodotos book	-0.5726	5	1.169	3.961
a city	-0.5611	64	3.918	3.650
city in	-0.5452	41	3.878	3.782
the river	-0.5413	6	1.962	3.940
river	-0.5413	6	1.962	3.940
a city in	-0.5185	40	3.928	3.751
egypt	-0.4957	3	1.110	3.906
the possessive is	-0.4932	4	1.582	3.915
possessive is	-0.4932	4	1.582	3.915
near	-0.4924	16	2.881	4.001
just as	-0.4770	5	1.503	3.944
hekataios	-0.4360	14	2.533	4.032
they	-0.4209	12	3.907	3.810
say that	-0.4206	9	2.668	3.936

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.