claude_fable_5 v1 Evaluation

Generated: 2026-07-25 12:52:06 UTC

Pairs100
Distinct lemmas100
Slope1.040
R^20.982
BLEU-427.5%
chrF++58.3%
METEOR66.0%
ROUGE-L67.9%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2015

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Fable 5; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope1.040
Intercept1.213
R^20.982
P-value9.78e-87
Pearson r0.991
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words47.140
Mean absolute length residual3.855
Mean absolute percent error10.2%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: negative; r = -0.086, R^2 = 0.007, p = 0.3924.
chrF++ vs passage length
chrF++: negative; r = -0.095, R^2 = 0.009, p = 0.3471.
METEOR vs passage length
METEOR: significant negative; r = -0.202, R^2 = 0.041, p = 0.0442.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.252, R^2 = 0.063, p = 0.0115.
Trigram precision vs passage length
Trigram precision: negative; r = -0.007, R^2 = 4.49e-05, p = 0.9473.
Trigram recall vs passage length
Trigram recall: positive; r = 0.002, R^2 = 2.63e-06, p = 0.9872.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.001, R^2 = 1.71e-06, p = 0.9897.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.023, R^2 = 5.20e-04, p = 0.8218.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 negative -0.086 0.007 0.3924 -0.00037 ok
chrF++ 100 negative -0.095 0.009 0.3471 -0.00028 ok
METEOR 100 significant negative -0.202 0.041 0.0442 -0.00085 ok
ROUGE-L 100 significant negative -0.252 0.063 0.0115 -0.00083 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 negative -0.007 4.49e-05 0.9473 -0.00003 ok
Trigram recall 100 positive 0.002 2.63e-06 0.9872 8.00e-06 ok
Trigram F1 100 negative -0.001 1.71e-06 0.9897 -6.38e-06 ok
Trigram Jaccard 100 negative -0.023 5.20e-04 0.8218 -0.00008 ok

Translation Similarity Metrics

Mean BLEU-427.5%
Mean chrF++58.3%
Mean METEOR66.0%
Mean ROUGE-L67.9%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU33.0%
Unigram F170.9%
Bigram F141.6%
Trigram precision25.5%
Trigram recall27.3%
Trigram F126.4%
Trigram Jaccard15.2%
4-gram F117.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 149 26.105 20.1% 55.4% 54.4% 53.7% N/A N/A N/A reviewed/approved 6915 claude-fable-5
2484 Καρία 270 265 -17.018 23.3% 55.1% 62.3% 64.7% N/A N/A N/A reviewed/approved 6910 claude-fable-5
7264 Κωλιάς 61 81 16.346 19.4% 48.6% 46.0% 46.5% N/A N/A N/A reviewed/approved 6957 claude-fable-5
2468 Καπετώλιον 138 129 -15.735 11.5% 40.9% 44.3% 52.4% N/A N/A N/A reviewed/approved 6907 claude-fable-5
7266 Κώμη 79 97 13.626 6.2% 42.2% 43.5% 47.7% N/A N/A N/A reviewed/approved 6959 claude-fable-5
7251 Κύρτος 66 82 12.146 27.2% 56.3% 69.5% 62.2% N/A N/A N/A reviewed/approved 6944 claude-fable-5
2081 Καλάθη 49 41 -11.174 39.8% 63.1% 61.0% 75.6% N/A N/A N/A reviewed/approved 6878 claude-fable-5
2603 Κάρυστος 175 194 10.784 27.5% 60.1% 59.8% 61.2% N/A N/A N/A reviewed/approved 6914 claude-fable-5
7267 Κωνώπη 64 78 10.226 33.6% 55.1% 58.6% 59.2% N/A N/A N/A reviewed/approved 6960 claude-fable-5
2328 Καλλίπολις 43 54 8.066 17.0% 52.7% 45.6% 55.7% N/A N/A N/A reviewed/approved 6894 claude-fable-5
2625 Κατακεκαυμένη 46 57 7.946 28.1% 66.7% 80.4% 71.8% N/A N/A N/A reviewed/approved 6924 claude-fable-5
2116 Κάληρος 40 35 -7.814 21.6% 57.1% 62.6% 64.0% N/A N/A N/A reviewed/approved 6887 claude-fable-5
2597 Καρπασία 112 111 -6.695 22.2% 60.0% 65.5% 64.6% N/A N/A N/A reviewed/approved 6911 claude-fable-5
7255 Κυτέριον 22 30 5.907 43.4% 68.2% 77.3% 65.4% N/A N/A N/A reviewed/approved 6948 claude-fable-5
2055 Καβασσός 109 109 -5.575 20.7% 52.9% 48.3% 55.0% N/A N/A N/A reviewed/approved 6862 claude-fable-5
2329 Κάλπη 57 55 -5.494 32.4% 55.2% 64.1% 70.8% N/A N/A N/A reviewed/approved 6895 claude-fable-5
3288 Κορώνεια 110 121 5.385 26.0% 59.6% 54.7% 62.1% N/A N/A N/A reviewed/approved 6931 claude-fable-5
3530 Κριώα 26 23 -5.253 22.2% 45.9% 54.7% 61.2% N/A N/A N/A reviewed/approved 6934 claude-fable-5
2119 Κάλλατις 65 74 5.186 19.6% 53.6% 61.1% 57.6% N/A N/A N/A reviewed/approved 6890 claude-fable-5
2330 Καλύβη 24 21 -5.173 26.8% 51.7% 59.1% 71.1% N/A N/A N/A reviewed/approved 6896 claude-fable-5
2624 Κατάβαθμος 20 17 -5.013 39.3% 66.1% 67.9% 75.7% N/A N/A N/A reviewed/approved 6923 claude-fable-5
2059 Καδμεία 20 27 4.987 31.4% 58.4% 71.0% 72.3% N/A N/A N/A reviewed/approved 6866 claude-fable-5
2079 Καισάρεια 39 37 -4.774 41.6% 63.9% 78.9% 81.6% N/A N/A N/A reviewed/approved 6876 claude-fable-5
2056 Καβειρία 112 113 -4.695 20.0% 56.4% 49.8% 60.4% N/A N/A N/A reviewed/approved 6863 claude-fable-5
2602 Καρύανδα 34 32 -4.573 44.9% 66.7% 76.2% 72.7% N/A N/A N/A reviewed/approved 6913 claude-fable-5
7261 Κύψελα 34 32 -4.573 35.8% 59.8% 76.4% 72.7% N/A N/A N/A reviewed/approved 6954 claude-fable-5
7260 Κυχρεῖος πάγος 61 69 4.346 33.6% 60.8% 67.7% 58.5% N/A N/A N/A reviewed/approved 6953 claude-fable-5
7262 Κῶβρυς 22 20 -4.093 12.9% 47.1% 58.6% 66.7% N/A N/A N/A reviewed/approved 6955 claude-fable-5
7253 Κυρτώνιος 18 24 4.067 4.6% 39.5% 44.5% 42.9% N/A N/A N/A reviewed/approved 6946 claude-fable-5
2626 Κατάνη 95 104 3.985 20.3% 55.0% 62.6% 61.3% N/A N/A N/A reviewed/approved 6926 claude-fable-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 49.239; cross-validated R^2: -0.222.

Positive termCoefficientDocsMean presentMean absent
και 0.4746 66 0.429 -0.833
απο 0.1967 27 2.297 -0.850
δε και 0.1952 8 3.375 -0.294
εστι δε και 0.1519 3 10.186 -0.315
εν τη 0.1429 7 4.146 -0.312
της 0.1400 35 0.489 -0.263
εκαλειτο δε 0.1368 3 6.624 -0.205
οι 0.1189 11 2.630 -0.325
εκ 0.1171 6 3.973 -0.254
εστι δε 0.1151 5 4.818 -0.254
και το 0.1143 9 3.199 -0.316
τη 0.1113 15 1.434 -0.253
ος 0.1021 6 2.692 -0.172
κωμη 0.0982 3 3.639 -0.113
της χωρας 0.0931 3 4.387 -0.136
αιγυπτου 0.0926 3 5.986 -0.185
εν τω 0.0923 6 2.165 -0.138
λιβυης 0.0920 4 7.756 -0.323
και αλλη 0.0920 3 10.505 -0.325
λεγεται και 0.0917 8 2.480 -0.216
Negative termCoefficientDocsMean presentMean absent
ως -0.1733 52 0.129 -0.140
θρακης -0.1483 5 -3.717 0.196
θεοπομπος -0.1430 5 -4.310 0.227
πολις θρακης -0.1345 4 -3.743 0.156
φησιν -0.1207 10 -3.075 0.342
πλησιον -0.1188 8 -2.424 0.211
και του -0.1182 4 -5.384 0.224
το -0.1122 70 -0.049 0.113
αυτην -0.1102 9 -1.854 0.183
ηρωδιανος -0.1077 3 -11.962 0.370
το εθνικον -0.1005 56 -0.526 0.669
εθνικον -0.1005 56 -0.526 0.669
γαρ -0.0970 9 0.385 -0.038
πορρω -0.0948 3 -5.133 0.159
ου πορρω -0.0948 3 -5.133 0.159
οτι -0.0930 5 -1.998 0.105
αυτην φησιν -0.0868 3 -6.267 0.194
μεταξυ -0.0783 5 -1.638 0.086
ζευς -0.0743 3 -10.682 0.330
τινες -0.0739 6 -2.100 0.134

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.030.

Positive termCoefficientDocsMean presentMean absent
και 2.3087 66 4.731 2.154
γαρ 2.0301 9 9.296 3.317
εκαλειτο 1.9297 7 10.315 3.369
δε 1.8904 46 5.262 2.656
εκαλειτο δε 1.5153 3 17.969 3.419
την 1.3687 13 8.841 3.110
δε και 1.3453 8 9.057 3.403
της 1.3173 35 5.724 2.849
εν 1.2935 37 5.287 3.014
εστι 1.1490 24 6.464 3.031
τω 1.0895 16 6.050 3.437
δια 1.0428 16 7.129 3.232
δια του 1.0307 10 8.758 3.310
ος 0.9832 6 8.955 3.530
εστι δε και 0.9714 3 10.808 3.640
εκαλειτο δε και 0.9152 3 17.969 3.419
του 0.8746 37 5.332 2.988
τα 0.8374 12 7.764 3.322
εν τω 0.7911 6 8.536 3.556
τοπος 0.7843 4 8.902 3.645
Negative termCoefficientDocsMean presentMean absent
εθνος -0.7613 8 3.567 3.880
το εθνικον -0.7603 56 3.800 3.925
εθνικον -0.7603 56 3.800 3.925
ηροδοτος -0.7539 9 3.374 3.903
τεταρτω -0.7351 3 0.951 3.945
και πολις -0.6961 5 2.181 3.943
στραβων -0.6912 15 3.681 3.886
δυο -0.6900 4 1.383 3.958
θηλυκον -0.6379 12 4.238 3.803
πολις -0.6098 68 3.864 3.837
δευτερω -0.5670 3 2.004 3.912
παυσανιας -0.5114 4 2.250 3.922
παιδος -0.4966 4 2.226 3.923
πλησιον -0.4852 8 2.600 3.964
οι -0.4816 11 3.939 3.845
εκαταιος ασια -0.4767 5 1.784 3.964
ασια -0.4767 5 1.784 3.964
εισι -0.4484 5 2.645 3.919
ορος -0.4461 6 3.621 3.870
εδει -0.4232 3 2.021 3.912

AI English terms predicting large absolute residuals

Features: 402; ridge alpha: 2.031; cross-validated R^2: 0.016.

Positive termCoefficientDocsMean presentMean absent
for 2.3600 12 7.841 3.312
it 2.1843 40 5.717 2.614
at 2.1447 10 9.983 3.174
on 1.8744 18 6.494 3.276
called 1.7908 44 5.258 2.753
is 1.7435 88 4.156 1.649
he 1.5669 4 11.874 3.521
was 1.5634 18 6.454 3.285
in 1.5298 73 4.131 3.111
from 1.4848 37 5.680 2.783
name 1.4733 13 8.582 3.149
diphthong 1.4717 3 13.149 3.568
was called 1.4704 6 9.627 3.487
there is 1.4657 25 6.263 3.053
there is also 1.4657 25 6.263 3.053
i 1.4322 8 8.661 3.437
their 1.4183 4 12.125 3.511
by 1.3656 13 7.775 3.269
the 1.3072 99 3.889 0.467
is also 1.2873 34 5.782 2.862
Negative termCoefficientDocsMean presentMean absent
one -1.2328 12 2.894 3.986
book -1.1536 39 3.441 4.120
and the -1.1467 18 2.524 4.147
book of -0.8700 16 3.618 3.900
been -0.8271 4 1.390 3.958
a city -0.8191 63 3.952 3.690
the ethnic is -0.8145 52 3.820 3.893
near -0.8019 17 2.962 4.038
herodotus -0.7969 9 3.374 3.903
of his -0.7926 6 2.182 3.962
the ethnic -0.7807 57 3.955 3.722
city of the -0.7704 11 2.813 3.984
and the feminine -0.7489 8 1.908 4.024
book of his -0.7173 5 1.749 3.966
ethnic is -0.7166 53 3.837 3.876
strabo -0.6939 15 3.681 3.886
ethnic -0.6723 59 3.995 3.654
there are -0.6708 7 2.884 3.928
fourth book -0.6668 3 0.951 3.945
feminine form is -0.6636 5 1.544 3.977

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: -0.038.

Positive termCoefficientDocsMean presentMean absent
with 2.2436 16 7.281 3.203
and 1.5928 52 5.315 2.273
the 1.5820 90 4.061 2.005
is also 1.5335 31 6.007 2.888
on 1.5299 16 6.273 3.395
called 1.4956 21 6.911 3.043
place 1.4252 6 8.857 3.536
is with 1.3047 4 14.948 3.393
when 1.2571 7 10.644 3.344
was called 1.2343 3 15.355 3.499
for 1.2055 11 6.608 3.515
is 1.1669 87 4.077 2.367
also 1.1075 43 5.117 2.903
used to be 1.1059 6 11.631 3.359
of 1.1022 66 4.445 2.709
there is also 1.0382 22 6.531 3.101
their 1.0362 3 17.969 3.419
was 1.0255 20 6.030 3.311
there is 0.9867 24 6.164 3.126
at 0.9809 6 11.042 3.396
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.8348 52 3.808 3.906
ethnonym is -0.8348 52 3.808 3.906
people -0.8325 9 3.223 3.918
a city -0.8243 64 3.924 3.733
book -0.7622 39 3.441 4.120
herodotos -0.7181 9 3.374 3.903
the ethnonym -0.6698 56 3.982 3.694
citizen is a -0.6630 15 3.028 4.001
citizen is -0.6630 15 3.028 4.001
a citizen is -0.6630 15 3.028 4.001
ethnonym -0.6621 57 4.010 3.650
and the -0.6561 15 3.834 3.859
a people -0.6492 5 1.843 3.961
strabo -0.6263 15 3.681 3.886
near -0.6150 16 3.300 3.961
kappadokia -0.6083 3 3.156 3.877
city in -0.5762 41 4.009 3.748
herodotos book -0.5613 5 1.304 3.989
city -0.5565 70 3.896 3.760
a city in -0.5498 40 4.046 3.728

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.