claude_sonnet_5 v1 Evaluation

Generated: 2026-07-25 12:52:19 UTC

Pairs100
Distinct lemmas100
Slope1.023
R^20.979
BLEU-421.0%
chrF++53.0%
METEOR62.3%
ROUGE-L62.2%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2009

First translation using this version: 2026-07-03

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Sonnet 5; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope1.023
Intercept2.775
R^20.979
P-value2.57e-84
Pearson r0.990
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words47.970
Mean absolute length residual3.829
Mean absolute percent error12.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: positive; r = 0.051, R^2 = 0.003, p = 0.6171.
chrF++ vs passage length
chrF++: positive; r = 0.022, R^2 = 4.69e-04, p = 0.8307.
METEOR vs passage length
METEOR: negative; r = -0.153, R^2 = 0.023, p = 0.1289.
ROUGE-L vs passage length
ROUGE-L: negative; r = -0.188, R^2 = 0.035, p = 0.0614.
Trigram precision vs passage length
Trigram precision: positive; r = 0.002, R^2 = 2.77e-06, p = 0.9869.
Trigram recall vs passage length
Trigram recall: negative; r = -0.021, R^2 = 4.58e-04, p = 0.8326.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.008, R^2 = 6.18e-05, p = 0.9381.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.023, R^2 = 5.34e-04, p = 0.8195.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 positive 0.051 0.003 0.6171 0.00021 ok
chrF++ 100 positive 0.022 4.69e-04 0.8307 0.00006 ok
METEOR 100 negative -0.153 0.023 0.1289 -0.00068 ok
ROUGE-L 100 negative -0.188 0.035 0.0614 -0.00070 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 positive 0.002 2.77e-06 0.9869 7.62e-06 ok
Trigram recall 100 negative -0.021 4.58e-04 0.8326 -0.00010 ok
Trigram F1 100 negative -0.008 6.18e-05 0.9381 -0.00004 ok
Trigram Jaccard 100 negative -0.023 5.34e-04 0.8195 -0.00007 ok

Translation Similarity Metrics

Mean BLEU-421.0%
Mean chrF++53.0%
Mean METEOR62.3%
Mean ROUGE-L62.2%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU27.7%
Unigram F165.8%
Bigram F136.4%
Trigram precision20.6%
Trigram recall22.5%
Trigram F121.5%
Trigram Jaccard12.1%
4-gram F113.5%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 155 32.482 12.5% 51.0% 54.2% 48.2% N/A N/A N/A reviewed/approved 6229 claude-sonnet-5
2484 Καρία 270 261 -18.106 16.5% 48.4% 47.9% 51.6% N/A N/A N/A reviewed/approved 6224 claude-sonnet-5
2119 Κάλλατις 65 85 15.701 12.7% 48.3% 56.2% 48.0% N/A N/A N/A reviewed/approved 6204 claude-sonnet-5
2056 Καβειρία 112 103 -14.401 14.1% 50.7% 56.3% 56.7% N/A N/A N/A reviewed/approved 6177 claude-sonnet-5
2465 Κάνωπος 83 100 12.279 26.5% 60.9% 76.6% 68.9% N/A N/A N/A reviewed/approved 6220 claude-sonnet-5
7266 Κώμη 79 95 11.373 3.9% 34.6% 37.0% 32.2% N/A N/A N/A reviewed/approved 6273 claude-sonnet-5
7264 Κωλιάς 61 76 10.795 15.3% 50.3% 49.5% 51.1% N/A N/A N/A reviewed/approved 6271 claude-sonnet-5
2115 Καλὴ ἀκτή 64 79 10.725 12.3% 47.2% 64.3% 58.7% N/A N/A N/A reviewed/approved 6200 claude-sonnet-5
2328 Καλλίπολις 43 57 10.217 16.8% 52.4% 52.7% 56.0% N/A N/A N/A reviewed/approved 6208 claude-sonnet-5
2116 Κάληρος 40 35 -8.712 16.2% 49.8% 50.0% 53.3% N/A N/A N/A reviewed/approved 6201 claude-sonnet-5
2342 Κάμιρος 54 66 7.959 25.3% 62.9% 78.7% 68.3% N/A N/A N/A reviewed/approved 6216 claude-sonnet-5
2455 Κάναστρον 60 57 -7.181 14.8% 48.1% 57.8% 59.8% N/A N/A N/A reviewed/approved 6219 claude-sonnet-5
2625 Κατακεκαυμένη 46 57 7.147 18.7% 55.6% 65.9% 62.1% N/A N/A N/A reviewed/approved 6238 claude-sonnet-5
2057 Καβελλιών 34 31 -6.572 23.8% 53.8% 52.3% 58.5% N/A N/A N/A reviewed/approved 6178 claude-sonnet-5
3530 Κριώα 26 23 -6.384 10.9% 40.4% 43.4% 57.1% N/A N/A N/A reviewed/approved 6248 claude-sonnet-5
7244 Κυρήνη 54 52 -6.041 27.9% 52.2% 49.9% 54.7% N/A N/A N/A reviewed/approved 6251 claude-sonnet-5
2468 Καπετώλιον 138 150 5.990 10.0% 40.1% 32.5% 38.2% N/A N/A N/A reviewed/approved 6221 claude-sonnet-5
2627 Κατάνειρα 21 30 5.733 6.3% 38.8% 43.2% 47.1% N/A N/A N/A reviewed/approved 6239 claude-sonnet-5
3288 Κορώνεια 110 121 5.646 23.0% 56.1% 59.4% 62.1% N/A N/A N/A reviewed/approved 6245 claude-sonnet-5
7248 Κύρου πόλις 32 41 5.475 11.4% 48.0% 41.8% 43.8% N/A N/A N/A reviewed/approved 6255 claude-sonnet-5
2059 Καδμεία 20 28 4.757 25.6% 53.4% 41.0% 50.0% N/A N/A N/A reviewed/approved 6180 claude-sonnet-5
7261 Κύψελα 34 33 -4.572 30.9% 57.6% 72.5% 68.7% N/A N/A N/A reviewed/approved 6268 claude-sonnet-5
3496 Κοτιάειον 71 71 -4.439 16.5% 46.5% 56.7% 59.2% N/A N/A N/A reviewed/approved 6246 claude-sonnet-5
7257 Κυτώνιον 25 24 -4.361 31.4% 57.6% 61.6% 65.3% N/A N/A N/A reviewed/approved 6264 claude-sonnet-5
2078 Καινύς 22 21 -4.290 52.8% 67.8% 76.6% 74.4% N/A N/A N/A reviewed/approved 6188 claude-sonnet-5
2075 Καινίνη 15 14 -4.126 39.8% 57.7% 73.1% 75.9% N/A N/A N/A reviewed/approved 6186 claude-sonnet-5
2081 Καλάθη 49 49 -3.923 26.5% 53.7% 54.8% 63.3% N/A N/A N/A reviewed/approved 6192 claude-sonnet-5
2603 Κάρυστος 175 178 -3.878 30.6% 57.3% 68.0% 61.2% N/A N/A N/A reviewed/approved 6228 claude-sonnet-5
2117 Καλησία 19 26 3.780 11.8% 48.8% 70.2% 57.8% N/A N/A N/A reviewed/approved 6202 claude-sonnet-5
7255 Κυτέριον 22 29 3.710 17.6% 50.3% 55.6% 58.8% N/A N/A N/A reviewed/approved 6262 claude-sonnet-5

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 17.013; cross-validated R^2: -0.182.

Positive termCoefficientDocsMean presentMean absent
και 0.6148 66 0.209 -0.406
δια του 0.5097 10 5.221 -0.580
εστι δε και 0.5084 3 12.115 -0.375
εστι δε 0.5019 5 6.534 -0.344
δε και 0.4947 8 1.662 -0.145
εν τη 0.4904 7 5.382 -0.405
ος 0.4564 6 6.619 -0.422
κωμη 0.3894 3 6.772 -0.209
πολιτης 0.3871 18 3.464 -0.760
εκαλειτο δε 0.3772 3 3.500 -0.108
πολιχνιον 0.3709 3 8.113 -0.251
του 0.3441 37 1.574 -0.924
ως 0.3423 52 0.848 -0.918
και το 0.3376 9 2.246 -0.222
τοις 0.3116 3 5.925 -0.183
εν 0.3104 37 1.508 -0.886
γαρ 0.2864 9 1.987 -0.197
τοπος 0.2662 4 4.988 -0.208
απο 0.2621 27 1.153 -0.427
δε 0.2496 46 0.357 -0.304
Negative termCoefficientDocsMean presentMean absent
το -0.6172 70 -0.510 1.191
το εθνικον -0.5529 56 -1.300 1.654
εθνικον -0.5529 56 -1.300 1.654
εθνος -0.3915 8 -5.351 0.465
τω -0.3498 16 -1.454 0.277
οτι -0.3375 5 -2.724 0.143
θρακης -0.3216 5 -3.739 0.197
τα -0.3105 12 -1.998 0.272
εστι και -0.2942 20 -0.886 0.221
υπο -0.2763 4 -2.486 0.104
κατα -0.2763 10 0.180 -0.020
αυτην -0.2575 9 -2.257 0.223
πλησιον -0.2558 8 -2.079 0.181
το θηλυκον -0.2339 6 -5.119 0.327
πολις θρακης -0.2252 4 -2.878 0.120
ηροδοτος -0.2236 9 -2.687 0.266
πολις της -0.2090 3 -6.807 0.211
εκει -0.2038 3 -4.752 0.147
εκαταιος ασια -0.2023 5 -1.951 0.103
ασια -0.2023 5 -1.951 0.103

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.007.

Positive termCoefficientDocsMean presentMean absent
και 2.7294 66 4.301 2.913
εκαλειτο 2.2641 7 10.062 3.360
εστι δε 1.7865 5 13.007 3.346
εκαλειτο δε 1.7336 3 18.155 3.386
κατα 1.5664 10 9.455 3.204
εστι 1.5550 24 6.686 2.927
δε και 1.3815 8 8.526 3.421
εστι δε και 1.3792 3 13.304 3.536
δε 1.3405 46 4.741 3.052
δια του 1.2914 10 9.949 3.149
ει 1.1944 4 13.090 3.443
της 1.0377 35 5.505 2.927
γαρ 1.0296 9 7.304 3.486
εκαλειτο δε και 0.9579 3 18.155 3.386
δια 0.9577 16 6.926 3.239
ος 0.9462 6 8.099 3.557
πολιχνιον 0.8811 3 9.166 3.664
απο 0.8421 27 6.299 2.916
λιβυης 0.8295 4 13.285 3.435
ιβηριας 0.8055 3 13.922 3.517
Negative termCoefficientDocsMean presentMean absent
το εθνικον -1.0416 56 3.444 4.320
εθνικον -1.0416 56 3.444 4.320
οι -0.7695 11 3.055 3.925
και πολις -0.7549 5 2.222 3.914
ηροδοτος -0.7463 9 3.408 3.871
δευτερω -0.7057 3 1.274 3.908
τους -0.6858 4 1.362 3.932
εκαταιος -0.6675 14 2.527 4.041
φασι -0.6293 8 2.845 3.915
αιγυπτου -0.5988 3 1.105 3.913
ορος -0.5596 6 2.354 3.923
αυτην -0.5368 9 2.729 3.938
λυκιας -0.5157 3 1.734 3.894
δια το -0.5099 3 1.516 3.901
προς τη -0.5060 4 2.507 3.884
καππαδοκιας -0.5056 3 2.340 3.875
εκληθη -0.5021 4 2.086 3.902
του δε -0.4850 3 1.540 3.900
το -0.4658 70 3.457 4.698
και του -0.4528 4 2.363 3.890

AI English terms predicting large absolute residuals

Features: 421; ridge alpha: 2.894; cross-validated R^2: -0.026.

Positive termCoefficientDocsMean presentMean absent
and 2.1915 58 4.645 2.703
was 1.9195 16 7.412 3.147
with 1.7809 19 6.897 3.110
on 1.7725 22 5.388 3.390
was also called 1.4797 3 13.279 3.537
is a 1.3247 7 11.442 3.256
written with 1.2980 4 15.923 3.325
name in 1.2943 3 15.400 3.471
for 1.2190 11 6.880 3.452
by 1.2180 17 7.083 3.163
written 1.1997 5 13.627 3.314
there is 1.1990 25 6.267 3.016
from 1.1522 42 5.502 2.618
this 1.1308 18 7.735 2.972
city and 1.1198 5 8.367 3.590
while 1.0792 3 16.430 3.439
states 1.0441 3 16.351 3.442
come 1.0381 5 12.290 3.384
it 1.0278 41 5.152 2.910
likewise 1.0250 3 18.859 3.364
Negative termCoefficientDocsMean presentMean absent
a city -0.8549 63 3.921 3.673
the ethnic -0.8526 58 3.468 4.329
one -0.8493 11 2.974 3.935
ethnic -0.7925 66 3.656 4.165
the ethnic is -0.7763 11 2.734 3.965
ethnic name -0.7651 37 3.304 4.137
ethnic name is -0.7368 34 3.341 4.081
near -0.6964 19 2.899 4.047
the ethnic name -0.6831 36 3.358 4.094
cappadocia -0.6806 4 2.413 3.888
call -0.6781 5 2.145 3.918
ethnic is -0.6766 16 3.532 3.886
two -0.6672 3 0.765 3.924
herodotus -0.6369 9 3.408 3.871
name is -0.6236 37 3.565 3.984
river -0.6163 6 2.129 3.938
in the feminine -0.6120 4 1.453 3.928
feminine form is -0.6093 6 1.696 3.965
to the -0.5884 7 1.943 3.971
inhabitants -0.5398 3 1.175 3.911

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 4.125; cross-validated R^2: -0.040.

Positive termCoefficientDocsMean presentMean absent
and 1.7866 52 4.738 2.845
like 1.3780 5 10.426 3.482
for 1.2543 11 7.334 3.396
is with 1.2276 4 14.943 3.366
also 1.1549 43 4.851 3.058
with 1.0623 16 7.055 3.215
it 1.0421 37 5.361 2.930
called 1.0130 21 5.498 3.385
there 1.0005 34 6.146 2.636
used 0.9788 8 10.605 3.240
used to 0.9788 8 10.605 3.240
the 0.9785 90 4.013 2.171
on 0.9476 16 5.541 3.503
used to be 0.9189 6 11.160 3.361
came 0.9186 4 13.821 3.413
to 0.9067 31 5.312 3.163
to be called 0.8597 5 12.616 3.367
this 0.8522 15 7.968 3.099
their 0.8471 3 18.155 3.386
there is 0.8377 24 5.776 3.214
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9908 52 3.170 4.543
ethnonym is -0.9908 52 3.170 4.543
ethnonym -0.8207 57 3.423 4.368
the ethnonym -0.7957 56 3.454 4.307
herodotos -0.6461 9 3.408 3.871
kappadokia -0.6114 3 1.988 3.886
herodotos book -0.5770 5 1.163 3.969
a city -0.5671 64 3.917 3.673
city in -0.5525 41 3.875 3.798
a city in -0.5254 40 3.924 3.766
near -0.4988 16 2.876 4.011
egypt -0.4985 3 1.105 3.913
the possessive is -0.4954 4 1.579 3.923
possessive is -0.4954 4 1.579 3.923
just as -0.4781 5 1.499 3.952
the river -0.4521 6 2.129 3.938
river -0.4521 6 2.129 3.938
hekataios -0.4436 14 2.527 4.041
they -0.4294 12 3.906 3.819
say that -0.4234 9 2.663 3.945

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.