claude_fable_5 v1 Evaluation

Generated: 2026-09-20 11:19:19 UTC

Pairs100

Distinct lemmas100

Slope1.040

R^20.982

BLEU-427.6%

chrF++58.4%

METEOR66.2%

ROUGE-L68.0%

BERTScore93.7%

COMET73.7%

BLEURT68.4%

Profile version id: 2015

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Fable 5; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Length Regression

Formula	AI words = intercept + slope * human words
Slope	1.040
Intercept	1.233
R^2	0.982
P-value	9.43e-87
Pearson r	0.991
Mean source words	30.400
Length fallback rows	0
Mean human words	44.150
Mean AI words	47.140
Mean absolute length residual	3.847
Mean absolute percent error	10.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length — BLEU-4: negative; r = -0.091, R^2 = 0.008, p = 0.3692.

chrF++ vs passage length — chrF++: negative; r = -0.101, R^2 = 0.010, p = 0.3197.

METEOR vs passage length — METEOR: significant negative; r = -0.210, R^2 = 0.044, p = 0.0358.

ROUGE-L vs passage length — ROUGE-L: significant negative; r = -0.258, R^2 = 0.067, p = 0.0095.

BERTScore vs passage length — BERTScore: significant negative; r = -0.481, R^2 = 0.231, p = 4.10e-07.

COMET vs passage length — COMET: negative; r = -0.168, R^2 = 0.028, p = 0.0938.

BLEURT vs passage length — BLEURT: significant negative; r = -0.678, R^2 = 0.459, p = 9.64e-15.

Trigram precision vs passage length — Trigram precision: negative; r = -0.013, R^2 = 1.62e-04, p = 0.9000.

Trigram recall vs passage length — Trigram recall: negative; r = -0.009, R^2 = 7.57e-05, p = 0.9315.

Trigram F1 vs passage length — Trigram F1: negative; r = -0.009, R^2 = 8.91e-05, p = 0.9257.

Trigram Jaccard vs passage length — Trigram Jaccard: negative; r = -0.033, R^2 = 0.001, p = 0.7421.

Metric	Rows	Pattern	Pearson r	R^2	P-value	Slope	Status
BLEU-4	100	negative	-0.091	0.008	0.3692	-0.00039	ok
chrF++	100	negative	-0.101	0.010	0.3197	-0.00030	ok
METEOR	100	significant negative	-0.210	0.044	0.0358	-0.00090	ok
ROUGE-L	100	significant negative	-0.258	0.067	0.0095	-0.00087	ok
BERTScore	100	significant negative	-0.481	0.231	4.10e-07	-0.00032	ok
COMET	100	negative	-0.168	0.028	0.0938	-0.00032	ok
BLEURT	100	significant negative	-0.678	0.459	9.64e-15	-0.00175	ok
Trigram precision	100	negative	-0.013	1.62e-04	0.9000	-0.00006	ok
Trigram recall	100	negative	-0.009	7.57e-05	0.9315	-0.00004	ok
Trigram F1	100	negative	-0.009	8.91e-05	0.9257	-0.00005	ok
Trigram Jaccard	100	negative	-0.033	0.001	0.7421	-0.00011	ok

Translation Similarity Metrics

Mean BLEU-4	27.6%
Mean chrF++	58.4%
Mean METEOR	66.2%
Mean ROUGE-L	68.0%
Mean BERTScore	93.7%
Mean COMET	73.7%
Mean BLEURT	68.4%
Legacy smoothed corpus BLEU	33.0%
Unigram F1	71.0%
Bigram F1	41.7%
Trigram precision	25.5%
Trigram recall	27.4%
Trigram F1	26.4%
Trigram Jaccard	15.2%
4-gram F1	17.4%

Largest Length Residuals

Lemma ID	Headword	Human words	AI words	Residual	BLEU-4	chrF++	METEOR	ROUGE-L	BERTScore	COMET	BLEURT	Human	Run	Model
2604	Καρχηδών	117	149	26.111	20.1%	55.4%	54.4%	53.7%	90.7%	75.2%	61.0%	reviewed/approved	6915	claude-fable-5
2484	Καρία	270	265	-16.979	23.3%	55.1%	62.3%	64.7%	89.8%	75.0%	50.5%	reviewed/approved	6910	claude-fable-5
7264	Κωλιάς	61	81	16.339	19.4%	48.6%	46.0%	46.5%	90.1%	62.4%	56.0%	reviewed/approved	6957	claude-fable-5
2468	Καπετώλιον	138	129	-15.725	11.5%	40.9%	44.3%	52.4%	85.7%	68.2%	48.5%	reviewed/approved	6907	claude-fable-5
7266	Κώμη	79	97	13.623	6.2%	42.2%	43.5%	47.7%	91.2%	66.8%	50.4%	reviewed/approved	6959	claude-fable-5
7251	Κύρτος	66	82	12.140	27.2%	56.3%	69.5%	62.2%	90.9%	65.0%	55.2%	reviewed/approved	6944	claude-fable-5
2081	Καλάθη	49	41	-11.183	39.8%	63.1%	61.0%	75.6%	95.1%	66.5%	70.0%	reviewed/approved	6878	claude-fable-5
2603	Κάρυστος	175	194	10.802	27.5%	60.1%	59.8%	61.2%	93.0%	75.4%	55.2%	reviewed/approved	6914	claude-fable-5
7267	Κωνώπη	64	78	10.220	33.6%	55.1%	58.6%	59.2%	91.1%	64.3%	66.1%	reviewed/approved	6960	claude-fable-5
2328	Καλλίπολις	43	54	8.056	17.0%	52.7%	45.6%	55.7%	93.4%	74.0%	66.1%	reviewed/approved	6894	claude-fable-5
2625	Κατακεκαυμένη	46	57	7.936	28.1%	66.7%	80.4%	71.8%	93.8%	75.4%	70.3%	reviewed/approved	6924	claude-fable-5
2116	Κάληρος	40	35	-7.825	21.6%	57.1%	62.6%	64.0%	94.0%	79.3%	73.6%	reviewed/approved	6887	claude-fable-5
2597	Καρπασία	112	111	-6.690	22.2%	60.0%	65.5%	64.6%	94.2%	72.4%	60.6%	reviewed/approved	6911	claude-fable-5
7255	Κυτέριον	22	30	5.892	43.4%	68.2%	77.3%	65.4%	95.1%	75.3%	73.1%	reviewed/approved	6948	claude-fable-5
2055	Καβασσός	109	109	-5.571	20.7%	52.9%	48.3%	55.0%	92.8%	75.1%	53.7%	reviewed/approved	6862	claude-fable-5
2329	Κάλπη	57	55	-5.501	32.4%	55.2%	64.1%	70.8%	93.1%	65.6%	65.9%	reviewed/approved	6895	claude-fable-5
3288	Κορώνεια	110	121	5.389	26.0%	59.6%	54.7%	62.1%	93.0%	73.7%	60.2%	reviewed/approved	6931	claude-fable-5
3530	Κριώα	26	23	-5.268	22.2%	45.9%	54.7%	61.2%	93.8%	66.7%	69.6%	reviewed/approved	6934	claude-fable-5
2330	Καλύβη	24	21	-5.188	26.8%	51.7%	59.1%	71.1%	93.9%	73.3%	72.3%	reviewed/approved	6896	claude-fable-5
2119	Κάλλατις	65	74	5.180	19.6%	53.6%	61.1%	57.6%	92.4%	65.5%	61.6%	reviewed/approved	6890	claude-fable-5
2624	Κατάβαθμος	20	17	-5.029	39.3%	66.1%	67.9%	75.7%	94.9%	79.6%	72.0%	reviewed/approved	6923	claude-fable-5
2059	Καδμεία	20	27	4.971	31.4%	58.4%	71.0%	72.3%	93.9%	78.7%	74.9%	reviewed/approved	6866	claude-fable-5
2079	Καισάρεια	39	37	-4.785	41.6%	63.9%	78.9%	81.6%	93.6%	77.4%	71.1%	reviewed/approved	6876	claude-fable-5
2056	Καβειρία	112	113	-4.690	20.0%	56.4%	49.8%	60.4%	93.3%	73.7%	62.5%	reviewed/approved	6863	claude-fable-5
2602	Καρύανδα	34	32	-4.586	44.9%	66.7%	76.2%	72.7%	95.8%	74.6%	73.8%	reviewed/approved	6913	claude-fable-5
7261	Κύψελα	34	32	-4.586	35.8%	59.8%	76.4%	72.7%	95.7%	71.6%	68.0%	reviewed/approved	6954	claude-fable-5
7260	Κυχρεῖος πάγος	61	69	4.339	33.6%	60.8%	67.7%	58.5%	94.1%	76.0%	59.3%	reviewed/approved	6953	claude-fable-5
7262	Κῶβρυς	22	20	-4.108	12.9%	47.1%	58.6%	66.7%	92.6%	67.9%	66.0%	reviewed/approved	6955	claude-fable-5
7253	Κυρτώνιος	18	24	4.051	4.6%	39.5%	44.5%	42.9%	91.0%	66.2%	68.0%	reviewed/approved	6946	claude-fable-5
2626	Κατάνη	95	104	3.986	20.3%	55.0%	62.6%	61.3%	91.5%	71.6%	58.3%	reviewed/approved	6926	claude-fable-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 49.239; cross-validated R^2: -0.223.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	0.4721	66	0.421	-0.818
απο	0.1963	27	2.294	-0.848
δε και	0.1951	8	3.378	-0.294
εστι δε και	0.1519	3	10.185	-0.315
εν τη	0.1426	7	4.137	-0.311
της	0.1390	35	0.483	-0.260
εκαλειτο δε	0.1369	3	6.645	-0.206
οι	0.1185	11	2.625	-0.324
εκ	0.1168	6	3.965	-0.253
εστι δε	0.1151	5	4.817	-0.254
και το	0.1140	9	3.195	-0.316
τη	0.1109	15	1.430	-0.252
ος	0.1020	6	2.692	-0.172
κωμη	0.0981	3	3.633	-0.112
της χωρας	0.0928	3	4.373	-0.135
αιγυπτου	0.0923	3	5.976	-0.185
εν τω	0.0921	6	2.167	-0.138
και αλλη	0.0920	3	10.507	-0.325
λιβυης	0.0918	4	7.751	-0.323
λεγεται και	0.0916	8	2.482	-0.216

Negative term	Coefficient	Docs	Mean present	Mean absent
ως	-0.1755	52	0.120	-0.130
θρακης	-0.1488	5	-3.730	0.196
θεοπομπος	-0.1434	5	-4.319	0.227
πολις θρακης	-0.1351	4	-3.758	0.157
φησιν	-0.1209	10	-3.073	0.341
πλησιον	-0.1194	8	-2.435	0.212
και του	-0.1186	4	-5.393	0.225
το	-0.1143	70	-0.058	0.134
αυτην	-0.1105	9	-1.860	0.184
ηρωδιανος	-0.1078	3	-11.951	0.370
το εθνικον	-0.1027	56	-0.535	0.682
εθνικον	-0.1027	56	-0.535	0.682
γαρ	-0.0967	9	0.392	-0.039
πορρω	-0.0952	3	-5.147	0.159
ου πορρω	-0.0952	3	-5.147	0.159
οτι	-0.0932	5	-2.002	0.105
αυτην φησιν	-0.0869	3	-6.273	0.194
μεταξυ	-0.0787	5	-1.646	0.087
τινες	-0.0742	6	-2.111	0.135
ζευς	-0.0742	3	-10.667	0.330

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.024.

Positive term	Coefficient	Docs	Mean present	Mean absent
και	2.3270	66	4.732	2.129
γαρ	2.0297	9	9.293	3.308
εκαλειτο	1.9357	7	10.317	3.360
δε	1.9018	46	5.263	2.641
εκαλειτο δε	1.5158	3	17.964	3.410
την	1.3696	13	8.840	3.101
δε και	1.3469	8	9.055	3.394
της	1.3273	35	5.724	2.836
εν	1.3111	37	5.288	3.001
εστι	1.1592	24	6.467	3.020
τω	1.0945	16	6.048	3.428
δια	1.0422	16	7.125	3.223
δια του	1.0297	10	8.751	3.302
ος	0.9848	6	8.953	3.521
εστι δε και	0.9737	3	10.815	3.631
εκαλειτο δε και	0.9154	3	17.964	3.410
του	0.8805	37	5.329	2.977
τα	0.8407	12	7.762	3.313
εν τω	0.7915	6	8.529	3.548
τοπος	0.7860	4	8.909	3.636

Negative term	Coefficient	Docs	Mean present	Mean absent
εθνος	-0.8524	8	3.438	3.883
ηροδοτος	-0.7534	9	3.366	3.895
το εθνικον	-0.7469	56	3.803	3.903
εθνικον	-0.7469	56	3.803	3.903
τεταρτω	-0.7209	3	0.953	3.936
και πολις	-0.6927	5	2.181	3.935
δυο	-0.6877	4	1.381	3.950
στραβων	-0.6875	15	3.679	3.877
θηλυκον	-0.6367	12	4.234	3.794
πολις	-0.5953	68	3.865	3.809
δευτερω	-0.5581	3	2.010	3.904
παυσανιας	-0.5119	4	2.245	3.914
παιδος	-0.4983	4	2.216	3.915
οι	-0.4793	11	3.939	3.836
πλησιον	-0.4739	8	2.611	3.955
εκαταιος ασια	-0.4668	5	1.791	3.955
ασια	-0.4668	5	1.791	3.955
εισι	-0.4467	5	2.644	3.910
ορος	-0.4434	6	3.626	3.861
εδει	-0.4216	3	2.029	3.903

AI English terms predicting large absolute residuals

Features: 402; ridge alpha: 2.031; cross-validated R^2: 0.023.

Positive term	Coefficient	Docs	Mean present	Mean absent
for	2.3643	12	7.838	3.303
it	2.1926	40	5.716	2.601
at	2.1466	10	9.983	3.165
on	1.8697	18	6.490	3.267
called	1.8009	44	5.257	2.739
is	1.7654	88	4.158	1.565
was	1.5691	18	6.453	3.275
he	1.5643	4	11.860	3.513
in	1.5506	73	4.131	3.079
from	1.4887	37	5.679	2.771
there is	1.4739	25	6.264	3.041
there is also	1.4739	25	6.264	3.041
name	1.4735	13	8.580	3.140
was called	1.4733	6	9.628	3.478
diphthong	1.4680	3	13.132	3.560
i	1.4299	8	8.651	3.429
their	1.4208	4	12.132	3.502
by	1.3753	13	7.778	3.260
the	1.3261	99	3.881	0.449
is also	1.2936	34	5.781	2.851

Negative term	Coefficient	Docs	Mean present	Mean absent
one	-1.2271	12	2.900	3.976
and the	-1.1437	18	2.520	4.138
book	-1.1378	39	3.440	4.107
book of	-0.8558	16	3.620	3.890
been	-0.8278	4	1.388	3.949
a city	-0.8121	63	3.953	3.667
the ethnic is	-0.7983	52	3.823	3.873
herodotus	-0.7927	9	3.366	3.895
near	-0.7811	17	2.970	4.027
of his	-0.7765	6	2.193	3.953
city of the	-0.7677	11	2.816	3.974
the ethnic	-0.7656	57	3.958	3.700
and the feminine	-0.7494	8	1.902	4.016
ethnic is	-0.7010	53	3.839	3.856
book of his	-0.7000	5	1.763	3.957
strabo	-0.6885	15	3.679	3.877
there are	-0.6671	7	2.884	3.920
feminine form is	-0.6629	5	1.540	3.968
fourth book	-0.6589	3	0.953	3.936
ethnic	-0.6582	59	3.997	3.631

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: -0.032.

Positive term	Coefficient	Docs	Mean present	Mean absent
with	2.2444	16	7.277	3.194
and	1.6089	52	5.316	2.256
the	1.5452	90	4.051	2.012
is also	1.5452	31	6.006	2.877
on	1.5301	16	6.270	3.385
called	1.5032	21	6.910	3.033
place	1.4265	6	8.861	3.527
is with	1.3069	4	14.941	3.385
when	1.2629	7	10.641	3.336
was called	1.2350	3	15.359	3.491
for	1.2093	11	6.603	3.506
is	1.1938	87	4.081	2.282
also	1.1232	43	5.117	2.889
of	1.1196	66	4.447	2.682
used to be	1.1078	6	11.631	3.350
there is also	1.0499	22	6.533	3.089
their	1.0368	3	17.964	3.410
was	1.0263	20	6.032	3.301
there is	0.9991	24	6.167	3.114
at	0.9768	6	11.039	3.388

Negative term	Coefficient	Docs	Mean present	Mean absent
people	-0.9257	9	3.106	3.920
the ethnonym is	-0.8225	52	3.811	3.886
ethnonym is	-0.8225	52	3.811	3.886
a city	-0.8204	64	3.925	3.708
book	-0.7503	39	3.440	4.107
herodotos	-0.7188	9	3.366	3.895
citizen is a	-0.6591	15	3.030	3.991
citizen is	-0.6591	15	3.030	3.991
a citizen is	-0.6591	15	3.030	3.991
the ethnonym	-0.6577	56	3.984	3.672
ethnonym	-0.6510	57	4.012	3.628
and the	-0.6487	15	3.833	3.849
a people	-0.6385	5	1.845	3.952
strabo	-0.6250	15	3.679	3.877
near	-0.5991	16	3.308	3.950
city in	-0.5704	41	4.009	3.734
herodotos book	-0.5600	5	1.299	3.981
kappadokia	-0.5500	4	3.557	3.859
city	-0.5468	70	3.897	3.731
a city in	-0.5442	40	4.046	3.714

Prompt Text

Show prompt text

You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.