claude_opus_4_8 v1 Evaluation

Generated: 2026-09-20 11:19:31 UTC

Pairs100

Distinct lemmas100

Slope1.126

R^20.975

BLEU-420.5%

chrF++53.8%

METEOR61.6%

ROUGE-L61.4%

BERTScore92.1%

COMET71.9%

BLEURT64.6%

Profile version id: 2012

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Opus 4.8; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.126
Intercept	2.939
R^2	0.975
P-value	6.21e-80
Pearson r	0.987
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	52.660
Mean absolute length residual	5.166
Mean absolute percent error	21.9%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.008, R^2 = 6.92e-05, p = 0.9345.

chrF++ vs passage length — chrF++: negative; r = -7.12e-05, R^2 = 5.07e-09, p = 0.9994.

METEOR vs passage length — METEOR: negative; r = -0.172, R^2 = 0.029, p = 0.0876.

ROUGE-L vs passage length — ROUGE-L: negative; r = -0.190, R^2 = 0.036, p = 0.0580.

BERTScore vs passage length — BERTScore: significant negative; r = -0.283, R^2 = 0.080, p = 0.0043.

COMET vs passage length — COMET: positive; r = 0.027, R^2 = 7.11e-04, p = 0.7923.

BLEURT vs passage length — BLEURT: significant negative; r = -0.588, R^2 = 0.346, p = 1.28e-10.

Trigram precision vs passage length — Trigram precision: negative; r = -0.014, R^2 = 1.94e-04, p = 0.8907.

Trigram recall vs passage length — Trigram recall: negative; r = -0.043, R^2 = 0.002, p = 0.6741.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.024, R^2 = 5.63e-04, p = 0.8147.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.038, R^2 = 0.001, p = 0.7041.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.008	6.92e-05	0.9345	-0.00003	ok
chrF++	100	negative	-7.12e-05	5.07e-09	0.9994	-2.02e-07	ok
METEOR	100	negative	-0.172	0.029	0.0876	-0.00077	ok
ROUGE-L	100	negative	-0.190	0.036	0.0580	-0.00065	ok
BERTScore	100	significant negative	-0.283	0.080	0.0043	-0.00020	ok
COMET	100	positive	0.027	7.11e-04	0.7923	0.00004	ok
BLEURT	100	significant negative	-0.588	0.346	1.28e-10	-0.00154	ok
Trigram precision	100	negative	-0.014	1.94e-04	0.8907	-0.00005	ok
Trigram recall	100	negative	-0.043	0.002	0.6741	-0.00017	ok
Trigram F1	100	negative	-0.024	5.63e-04	0.8147	-0.00009	ok
Trigram Jaccard	100	negative	-0.038	0.001	0.7041	-0.00009	ok

Translation Similarity Metrics

Mean BLEU-4	20.5%
Mean chrF++	53.8%
Mean METEOR	61.6%
Mean ROUGE-L	61.4%
Mean BERTScore	92.1%
Mean COMET	71.9%
Mean BLEURT	64.6%
Legacy smoothed corpus BLEU	25.4%
Unigram F1	64.6%
Bigram F1	34.8%
Trigram precision	18.9%
Trigram recall	22.7%
Trigram F1	20.6%
Trigram Jaccard	11.5%
4-gram F1	12.6%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2115	Καλὴ ἀκτή	64	101	25.985	8.3%	40.9%	49.6%	46.1%	87.7%	63.8%	48.5%	reviewed/approved	6686	claude-opus-4.8
7266	Κώμη	79	114	22.093	6.6%	38.5%	30.7%	36.3%	88.2%	61.3%	42.5%	reviewed/approved	6759	claude-opus-4.8
2484	Καρία	270	287	-20.006	18.8%	51.2%	50.6%	55.6%	89.0%	77.4%	55.3%	reviewed/approved	6710	claude-opus-4.8
2056	Καβειρία	112	112	-17.071	15.2%	53.9%	57.1%	59.8%	92.0%	73.6%	60.1%	reviewed/approved	6663	claude-opus-4.8
2119	Κάλλατις	65	92	15.859	14.9%	51.9%	58.0%	48.4%	90.1%	64.9%	61.5%	reviewed/approved	6690	claude-opus-4.8
7264	Κωλιάς	61	87	15.364	14.8%	46.9%	43.6%	45.9%	89.7%	65.5%	53.3%	reviewed/approved	6757	claude-opus-4.8
7249	Κύρρος	45	68	14.383	3.4%	36.1%	26.3%	31.9%	87.2%	62.8%	49.8%	reviewed/approved	6742	claude-opus-4.8
2604	Καρχηδών	117	148	13.298	17.6%	55.3%	56.8%	58.4%	91.3%	74.9%	59.2%	reviewed/approved	6715	claude-opus-4.8
2597	Καρπασία	112	116	-13.071	15.5%	53.0%	47.4%	56.1%	93.1%	71.6%	55.7%	reviewed/approved	6711	claude-opus-4.8
2455	Κάναστρον	60	59	-11.510	15.6%	50.2%	61.8%	63.9%	92.3%	66.2%	54.1%	reviewed/approved	6705	claude-opus-4.8
7259	Κύφος	54	75	11.247	21.3%	56.1%	61.5%	57.4%	91.2%	66.3%	56.9%	reviewed/approved	6752	claude-opus-4.8
7253	Κυρτώνιος	18	34	10.789	3.7%	44.3%	64.8%	50.0%	90.9%	71.5%	67.2%	reviewed/approved	6746	claude-opus-4.8
2470	Καππαδοκία	68	90	10.481	9.8%	49.9%	64.6%	63.3%	92.7%	70.8%	57.0%	reviewed/approved	6708	claude-opus-4.8
2342	Κάμιρος	54	74	10.247	20.7%	60.1%	66.6%	65.6%	93.7%	74.7%	67.6%	reviewed/approved	6702	claude-opus-4.8
2116	Κάληρος	40	38	-9.986	13.2%	49.7%	52.9%	51.3%	92.4%	73.8%	69.2%	reviewed/approved	6687	claude-opus-4.8
2625	Κατακεκαυμένη	46	64	9.257	21.3%	61.9%	73.2%	63.6%	91.9%	72.5%	65.2%	reviewed/approved	6724	claude-opus-4.8
3530	Κριώα	26	23	-9.220	12.8%	42.5%	49.7%	61.2%	92.6%	66.2%	66.5%	reviewed/approved	6734	claude-opus-4.8
2626	Κατάνη	95	119	9.074	16.1%	50.2%	53.9%	51.4%	89.4%	67.7%	55.1%	reviewed/approved	6726	claude-opus-4.8
2328	Καλλίπολις	43	60	8.635	14.1%	52.7%	51.3%	56.3%	92.9%	72.4%	65.6%	reviewed/approved	6694	claude-opus-4.8
2465	Κάνωπος	83	105	8.588	27.2%	63.0%	68.6%	70.2%	92.7%	73.2%	61.5%	reviewed/approved	6706	claude-opus-4.8
2081	Καλάθη	49	50	-8.122	26.5%	56.7%	59.7%	66.7%	94.5%	72.7%	71.0%	reviewed/approved	6678	claude-opus-4.8
2609	Κάσπειρος	135	147	-7.973	23.3%	57.8%	64.3%	64.5%	88.8%	71.5%	57.0%	reviewed/approved	6718	claude-opus-4.8
2330	Καλύβη	24	22	-7.968	23.1%	51.2%	67.0%	73.9%	93.1%	72.7%	63.5%	reviewed/approved	6696	claude-opus-4.8
2055	Καβασσός	109	133	7.308	18.9%	51.3%	54.0%	50.4%	90.0%	71.5%	48.9%	reviewed/approved	6662	claude-opus-4.8
2602	Καρύανδα	34	34	-7.229	24.8%	56.8%	72.3%	64.7%	94.2%	73.1%	68.4%	reviewed/approved	6713	claude-opus-4.8
2079	Καισάρεια	39	40	-6.860	27.5%	55.9%	61.9%	68.4%	92.0%	75.2%	65.6%	reviewed/approved	6676	claude-opus-4.8
2630	Κάτρη	23	22	-6.841	25.3%	57.8%	63.7%	71.1%	92.5%	75.3%	66.2%	reviewed/approved	6729	claude-opus-4.8
2346	Κάναι	77	83	-6.655	40.8%	63.0%	72.4%	77.5%	93.3%	75.6%	58.5%	reviewed/approved	6704	claude-opus-4.8
7261	Κύψελα	34	35	-6.229	40.1%	62.5%	78.9%	69.6%	95.3%	73.7%	67.3%	reviewed/approved	6754	claude-opus-4.8
3288	Κορώνεια	110	133	6.181	21.2%	56.7%	56.5%	57.4%	91.2%	73.2%	57.3%	reviewed/approved	6731	claude-opus-4.8

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.154.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.7539	66	0.400	-0.776
κωμη	0.6023	3	14.571	-0.451
δια του	0.5571	10	6.473	-0.719
του	0.4879	37	2.286	-1.343
γαρ	0.4414	9	5.762	-0.570
εκ	0.4043	6	6.597	-0.421
δυο	0.4038	4	8.260	-0.344
δε	0.3645	46	0.990	-0.844
θηλυκως	0.3606	5	8.762	-0.461
και θηλυκως	0.3606	5	8.762	-0.461
εν τη	0.3451	7	5.272	-0.397
δια	0.3305	16	3.367	-0.641
εστι δε	0.2903	5	4.806	-0.253
ειρηται	0.2656	3	10.830	-0.335
πολυβιος	0.2630	5	3.996	-0.210
τεταρτω	0.2568	3	8.194	-0.253
απο	0.2462	27	1.218	-0.450
ος	0.2377	6	5.393	-0.344
χωρα	0.2239	6	1.635	-0.104
εν	0.2214	37	2.032	-1.193

Negative term	Coefficient	Docs	Mean present	Mean absent
το εθνικον	-0.5849	56	-1.473	1.875
εθνικον	-0.5849	56	-1.473	1.875
το	-0.4641	70	-0.101	0.235
θρακης	-0.4031	5	-5.853	0.308
αυτην	-0.3985	9	-4.250	0.420
πλησιον	-0.3598	8	-4.189	0.364
φησιν	-0.3118	10	-2.231	0.248
πολις θρακης	-0.2944	4	-4.438	0.185
κατα	-0.2776	10	-3.069	0.341
εστι και	-0.2708	20	-1.036	0.259
και πολις	-0.2402	5	-3.174	0.167
αυτην φησιν	-0.2244	3	-8.519	0.263
προς τη	-0.2101	4	-3.499	0.146
οτι	-0.2095	5	-2.152	0.113
τα	-0.2080	12	-1.251	0.171
πολις	-0.2062	68	-0.513	1.091
πολις της	-0.2012	3	-9.541	0.295
τινες δε	-0.1901	3	-4.776	0.148
αυτην φησι	-0.1882	3	-6.440	0.199
το κτητικον	-0.1839	7	-6.038	0.454

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 1.000; cross-validated R^2: 0.140.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	6.6968	66	6.183	3.193
κωμη	5.8485	3	17.481	4.785
ει	4.4675	4	17.367	4.658
γαρ	4.3623	9	11.980	4.492
παρα το	4.1386	3	15.396	4.850
ως	4.0889	52	6.462	3.762
εστι δε	3.6411	5	13.380	4.734
δε	3.1248	46	6.653	3.899
δια	3.0553	16	9.023	4.431
δυο	2.9065	4	11.587	4.898
δια του	2.8190	10	11.526	4.459
εκαλειτο	2.8017	7	8.776	4.894
εστι	2.6270	24	8.124	4.232
πολιχνιον	2.4617	3	10.383	5.005
ουτως	2.3116	5	11.126	4.852
ακρα	2.2782	4	10.077	4.961
παρα	2.2610	7	11.104	4.719
απο του	2.1269	9	9.128	4.774
ης	2.1170	3	11.330	4.975
μεν	2.0243	4	12.764	4.849

Negative term	Coefficient	Docs	Mean present	Mean absent
τους	-3.3637	4	1.022	5.339
οι	-3.2679	11	3.472	5.375
φασι	-2.7690	8	3.324	5.326
πολις προς	-2.6086	3	2.029	5.263
λιβυης	-2.3451	4	6.152	5.125
και πολις	-2.2553	5	3.572	5.250
ιβηριας	-1.9009	3	5.505	5.156
του δε	-1.8566	3	2.419	5.251
πλησιον	-1.8183	8	4.189	5.251
ου και	-1.7150	4	2.336	5.284
μια	-1.6703	3	3.359	5.222
εθνος	-1.6390	8	5.841	5.107
εκληθη	-1.6275	4	2.025	5.297
και θηλυκον	-1.6237	3	1.900	5.267
ουδετερως	-1.5939	3	4.265	5.194
ειναι	-1.5845	8	4.804	5.197
εκληθη δε	-1.5536	3	0.912	5.298
προς τω	-1.5518	4	0.975	5.341
το εθνικον	-1.5512	56	4.871	5.542
εθνικον	-1.5512	56	4.871	5.542

AI English terms predicting large absolute residuals

Features: 478; ridge alpha: 2.894; cross-validated R^2: -0.046.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	2.7499	57	6.701	3.131
from	2.6717	36	7.949	3.601
with	2.1825	18	8.994	4.326
the	2.1064	100	5.166	N/A
for	1.9588	13	10.118	4.426
there	1.6918	32	7.409	4.111
comes	1.6235	9	9.799	4.708
village	1.5669	4	14.938	4.759
on	1.5151	20	8.482	4.337
or	1.4738	14	8.386	4.642
was	1.3958	16	8.252	4.578
with the	1.3697	11	9.487	4.632
a	1.3474	88	5.592	2.042
at	1.2791	6	11.807	4.742
too	1.2566	7	10.927	4.732
from the	1.2469	10	10.085	4.619
there are	1.1881	8	11.713	4.597
this	1.1752	16	8.578	4.516
there is	1.1240	26	7.113	4.482
the form	1.1047	9	11.293	4.560

Negative term	Coefficient	Docs	Mean present	Mean absent
they	-0.9666	11	5.397	5.137
it is also	-0.9652	8	3.738	5.290
form is	-0.9124	11	3.852	5.328
near	-0.8412	14	3.821	5.385
mentions	-0.8177	32	3.905	5.759
ethnic is	-0.8159	51	5.193	5.138
that	-0.7912	26	5.036	5.212
city	-0.7635	68	5.377	4.717
ethnic	-0.7435	67	5.401	4.688
a city	-0.7424	63	5.339	4.871
the citizen ethnic	-0.6923	4	1.630	5.313
citizen ethnic is	-0.6923	4	1.630	5.313
citizen ethnic	-0.6923	4	1.630	5.313
and the	-0.6850	22	6.855	4.690
them in	-0.6672	4	1.011	5.339
mentions them in	-0.6672	4	1.011	5.339
mentions them	-0.6672	4	1.011	5.339
of his	-0.6626	7	2.581	5.361
mentions it in	-0.6553	26	3.992	5.578
mentions it	-0.6553	26	3.992	5.578

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 1.000; cross-validated R^2: 0.082.

Positive term	Coefficient	Docs	Mean present	Mean absent
and	6.0945	52	6.737	3.464
like	5.2625	5	15.256	4.635
for	4.9756	11	9.821	4.591
as	4.5881	60	6.438	3.258
because	3.2924	12	10.196	4.480
from the form	2.9391	5	12.028	4.805
were	2.7557	4	12.520	4.860
with	2.6494	16	9.921	4.260
for the	2.4689	7	11.305	4.704
usage	2.3909	4	9.767	4.974
from	2.3185	37	7.174	3.987
the form	2.3137	9	10.654	4.623
also	2.2449	43	6.731	3.986
from the	2.1793	13	8.967	4.598
long	2.1781	3	15.916	4.834
promontory	2.1473	3	11.218	4.979
karia	2.0333	3	11.158	4.981
there is	1.9334	24	8.157	4.221
form	1.9328	11	9.541	4.625
which is	1.8816	5	10.144	4.904

Negative term	Coefficient	Docs	Mean present	Mean absent
near	-2.7145	16	3.465	5.490
in book of	-2.4343	14	3.173	5.490
in book	-2.2602	16	4.135	5.362
they	-1.8550	12	4.841	5.210
or after	-1.7508	3	1.404	5.282
city	-1.7215	70	5.336	4.770
ethnonym	-1.7069	57	4.936	5.471
the ethnonym	-1.6935	56	4.893	5.513
the ethnonym is	-1.6071	52	4.778	5.586
ethnonym is	-1.6071	52	4.778	5.586
inhabitant is a	-1.5010	6	5.598	5.138
say that	-1.4049	9	3.627	5.318
say	-1.3995	12	4.984	5.191
that the	-1.3700	6	2.961	5.307
mount	-1.3695	4	5.210	5.164
that	-1.3605	29	5.739	4.932
a city	-1.3576	64	5.271	4.979
strabo in	-1.3077	3	1.668	5.274
hekataios in	-1.3033	12	3.151	5.441
city near	-1.2979	3	2.362	5.253

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.