claude_opus_4_8 v1 Evaluation

Generated: 2026-07-25 12:52:14 UTC

Pairs100
Distinct lemmas100
Slope1.126
R^20.975
BLEU-420.5%
chrF++53.7%
METEOR61.5%
ROUGE-L61.3%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2012

First translation using this version: 2026-07-05

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-02 10:20:24.063701+10:00

Active: True

Notes: External-only Claude workspace variant for Claude Opus 4.8; copied from legacy_scholarly v1 (source profile_version_id=1); source_active=False; uses_guidance_context=False.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope1.126
Intercept2.917
R^20.975
P-value5.91e-80
Pearson r0.987
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words52.660
Mean absolute length residual5.150
Mean absolute percent error21.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: negative; r = -0.008, R^2 = 7.03e-05, p = 0.9340.
chrF++ vs passage length
chrF++: positive; r = 0.004, R^2 = 1.60e-05, p = 0.9685.
METEOR vs passage length
METEOR: negative; r = -0.167, R^2 = 0.028, p = 0.0971.
ROUGE-L vs passage length
ROUGE-L: negative; r = -0.182, R^2 = 0.033, p = 0.0704.
Trigram precision vs passage length
Trigram precision: negative; r = -0.014, R^2 = 1.94e-04, p = 0.8907.
Trigram recall vs passage length
Trigram recall: negative; r = -0.040, R^2 = 0.002, p = 0.6895.
Trigram F1 vs passage length
Trigram F1: negative; r = -0.023, R^2 = 5.35e-04, p = 0.8193.
Trigram Jaccard vs passage length
Trigram Jaccard: negative; r = -0.038, R^2 = 0.001, p = 0.7082.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 negative -0.008 7.03e-05 0.9340 -0.00003 ok
chrF++ 100 positive 0.004 1.60e-05 0.9685 0.00001 ok
METEOR 100 negative -0.167 0.028 0.0971 -0.00075 ok
ROUGE-L 100 negative -0.182 0.033 0.0704 -0.00063 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 negative -0.014 1.94e-04 0.8907 -0.00005 ok
Trigram recall 100 negative -0.040 0.002 0.6895 -0.00017 ok
Trigram F1 100 negative -0.023 5.35e-04 0.8193 -0.00009 ok
Trigram Jaccard 100 negative -0.038 0.001 0.7082 -0.00009 ok

Translation Similarity Metrics

Mean BLEU-420.5%
Mean chrF++53.7%
Mean METEOR61.5%
Mean ROUGE-L61.3%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU25.4%
Unigram F164.5%
Bigram F134.8%
Trigram precision18.9%
Trigram recall22.7%
Trigram F120.6%
Trigram Jaccard11.5%
4-gram F112.6%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2115 Καλὴ ἀκτή 64 101 25.992 8.3% 40.9% 49.6% 46.1% N/A N/A N/A reviewed/approved 6686 claude-opus-4.8
7266 Κώμη 79 114 22.095 6.6% 38.5% 30.7% 36.3% N/A N/A N/A reviewed/approved 6759 claude-opus-4.8
2484 Καρία 270 287 -20.053 18.8% 51.2% 50.6% 55.6% N/A N/A N/A reviewed/approved 6710 claude-opus-4.8
2056 Καβειρία 112 112 -17.077 15.2% 53.9% 57.1% 59.8% N/A N/A N/A reviewed/approved 6663 claude-opus-4.8
2119 Κάλλατις 65 92 15.865 14.9% 51.9% 58.0% 48.4% N/A N/A N/A reviewed/approved 6690 claude-opus-4.8
7264 Κωλιάς 61 87 15.371 14.8% 46.9% 43.6% 45.9% N/A N/A N/A reviewed/approved 6757 claude-opus-4.8
7249 Κύρρος 45 68 14.394 3.4% 36.1% 26.3% 31.9% N/A N/A N/A reviewed/approved 6742 claude-opus-4.8
2604 Καρχηδών 117 148 13.291 17.6% 55.3% 56.8% 58.4% N/A N/A N/A reviewed/approved 6715 claude-opus-4.8
2597 Καρπασία 112 116 -13.077 15.5% 53.0% 47.4% 56.1% N/A N/A N/A reviewed/approved 6711 claude-opus-4.8
2455 Κάναστρον 60 59 -11.503 15.6% 50.2% 61.8% 63.9% N/A N/A N/A reviewed/approved 6705 claude-opus-4.8
7259 Κύφος 54 75 11.256 21.3% 56.1% 61.5% 57.4% N/A N/A N/A reviewed/approved 6752 claude-opus-4.8
7253 Κυρτώνιος 18 34 10.807 3.7% 44.3% 64.8% 50.0% N/A N/A N/A reviewed/approved 6746 claude-opus-4.8
2470 Καππαδοκία 68 90 10.486 9.8% 49.9% 64.6% 63.3% N/A N/A N/A reviewed/approved 6708 claude-opus-4.8
2342 Κάμιρος 54 74 10.256 20.7% 60.1% 66.6% 65.6% N/A N/A N/A reviewed/approved 6702 claude-opus-4.8
2116 Κάληρος 40 38 -9.974 13.2% 49.7% 52.9% 51.3% N/A N/A N/A reviewed/approved 6687 claude-opus-4.8
2625 Κατακεκαυμένη 46 64 9.267 21.3% 61.9% 73.2% 63.6% N/A N/A N/A reviewed/approved 6724 claude-opus-4.8
3530 Κριώα 26 23 -9.204 12.8% 42.5% 49.7% 61.2% N/A N/A N/A reviewed/approved 6734 claude-opus-4.8
2626 Κατάνη 95 119 9.072 16.1% 50.2% 53.9% 51.4% N/A N/A N/A reviewed/approved 6726 claude-opus-4.8
2328 Καλλίπολις 43 60 8.647 14.1% 52.7% 51.3% 56.3% N/A N/A N/A reviewed/approved 6694 claude-opus-4.8
2465 Κάνωπος 83 105 8.589 27.2% 63.0% 68.6% 70.2% N/A N/A N/A reviewed/approved 6706 claude-opus-4.8
2081 Καλάθη 49 50 -8.112 26.5% 56.7% 59.7% 66.7% N/A N/A N/A reviewed/approved 6678 claude-opus-4.8
2609 Κάσπειρος 135 147 -7.985 23.3% 57.8% 64.3% 64.5% N/A N/A N/A reviewed/approved 6718 claude-opus-4.8
2330 Καλύβη 24 22 -7.951 23.1% 51.2% 67.0% 73.9% N/A N/A N/A reviewed/approved 6696 claude-opus-4.8
2055 Καβασσός 109 133 7.302 18.9% 51.3% 54.0% 50.4% N/A N/A N/A reviewed/approved 6662 claude-opus-4.8
2602 Καρύανδα 34 34 -7.215 24.8% 56.8% 72.3% 64.7% N/A N/A N/A reviewed/approved 6713 claude-opus-4.8
2079 Καισάρεια 39 40 -6.848 27.5% 55.9% 61.9% 68.4% N/A N/A N/A reviewed/approved 6676 claude-opus-4.8
2630 Κάτρη 23 22 -6.825 25.3% 57.8% 63.7% 71.1% N/A N/A N/A reviewed/approved 6729 claude-opus-4.8
2346 Κάναι 77 83 -6.652 40.8% 63.0% 72.4% 77.5% N/A N/A N/A reviewed/approved 6704 claude-opus-4.8
7261 Κύψελα 34 35 -6.215 40.1% 62.5% 78.9% 69.6% N/A N/A N/A reviewed/approved 6754 claude-opus-4.8
3288 Κορώνεια 110 133 6.176 21.2% 56.7% 56.5% 57.4% N/A N/A N/A reviewed/approved 6731 claude-opus-4.8

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.154.

Positive termCoefficientDocsMean presentMean absent
και 0.7589 66 0.408 -0.792
κωμη 0.6025 3 14.578 -0.451
δια του 0.5574 10 6.473 -0.719
του 0.4900 37 2.291 -1.346
γαρ 0.4407 9 5.753 -0.569
εκ 0.4049 6 6.605 -0.422
δυο 0.4041 4 8.265 -0.344
δε 0.3660 46 0.996 -0.848
θηλυκως 0.3611 5 8.769 -0.462
και θηλυκως 0.3611 5 8.769 -0.462
εν τη 0.3457 7 5.282 -0.398
δια 0.3307 16 3.368 -0.642
εστι δε 0.2903 5 4.805 -0.253
ειρηται 0.2660 3 10.838 -0.335
πολυβιος 0.2641 5 4.010 -0.211
τεταρτω 0.2584 3 8.210 -0.254
απο 0.2470 27 1.220 -0.451
ος 0.2377 6 5.392 -0.344
χωρα 0.2246 6 1.637 -0.105
εν 0.2237 37 2.039 -1.197
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.5802 56 -1.463 1.861
εθνικον -0.5802 56 -1.463 1.861
το -0.4597 70 -0.091 0.213
θρακης -0.4018 5 -5.838 0.307
αυτην -0.3978 9 -4.243 0.420
πλησιον -0.3584 8 -4.177 0.363
φησιν -0.3116 10 -2.234 0.248
πολις θρακης -0.2931 4 -4.422 0.184
κατα -0.2767 10 -3.067 0.341
εστι και -0.2699 20 -1.034 0.258
και πολις -0.2396 5 -3.165 0.167
αυτην φησιν -0.2241 3 -8.514 0.263
οτι -0.2091 5 -2.149 0.113
προς τη -0.2090 4 -3.483 0.145
τα -0.2080 12 -1.255 0.171
πολις -0.2012 68 -0.503 1.069
πολις της -0.2012 3 -9.541 0.295
τινες δε -0.1896 3 -4.763 0.147
αυτην φησι -0.1882 3 -6.445 0.199
το κτητικον -0.1837 7 -6.041 0.455

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 1.000; cross-validated R^2: 0.146.

Positive termCoefficientDocsMean presentMean absent
και 6.7494 66 6.180 3.149
κωμη 5.8543 3 17.480 4.768
ει 4.4643 4 17.379 4.640
γαρ 4.3596 9 11.984 4.474
παρα το 4.1455 3 15.403 4.833
ως 4.1070 52 6.458 3.733
εστι δε 3.6486 5 13.378 4.717
δε 3.1597 46 6.652 3.870
δια 3.0669 16 9.022 4.412
δυο 2.9199 4 11.591 4.881
δια του 2.8118 10 11.526 4.441
εκαλειτο 2.8108 7 8.773 4.877
εστι 2.6666 24 8.121 4.211
πολιχνιον 2.4726 3 10.388 4.988
ουτως 2.3172 5 11.135 4.835
ακρα 2.2854 4 10.076 4.944
παρα 2.2710 7 11.112 4.701
απο του 2.1300 9 9.129 4.756
ης 2.1290 3 11.342 4.958
μεν 2.0299 4 12.781 4.832
Negative termCoefficientDocsMean presentMean absent
τους -3.3504 4 1.020 5.322
οι -3.2500 11 3.473 5.357
φασι -2.7676 8 3.316 5.309
πολις προς -2.6092 3 2.020 5.246
λιβυης -2.3491 4 6.144 5.108
και πολις -2.2438 5 3.566 5.233
ιβηριας -1.9009 3 5.493 5.139
του δε -1.8474 3 2.428 5.234
εθνος -1.8423 8 5.710 5.101
πλησιον -1.7966 8 4.177 5.234
ου και -1.7108 4 2.337 5.267
μια -1.6516 3 3.346 5.205
και θηλυκον -1.6276 3 1.890 5.250
εκληθη -1.6175 4 2.026 5.280
ουδετερως -1.5812 3 4.262 5.177
ειναι -1.5777 8 4.799 5.180
εκληθη δε -1.5529 3 0.911 5.281
προς τω -1.5444 4 0.963 5.324
το εθνικον -1.5346 56 4.864 5.514
εθνικον -1.5346 56 4.864 5.514

AI English terms predicting large absolute residuals

Features: 478; ridge alpha: 2.894; cross-validated R^2: -0.041.

Positive termCoefficientDocsMean presentMean absent
and 2.7783 57 6.701 3.093
from 2.6856 36 7.950 3.574
with 2.1911 18 8.994 4.306
the 2.0873 100 5.150 N/A
for 1.9649 13 10.122 4.407
there 1.7053 32 7.407 4.088
comes 1.6279 9 9.799 4.690
village 1.5670 4 14.936 4.742
on 1.5259 20 8.486 4.315
or 1.4784 14 8.384 4.623
was 1.4066 16 8.255 4.558
with the 1.3737 11 9.490 4.613
a 1.3674 88 5.587 1.943
at 1.2822 6 11.806 4.725
too 1.2611 7 10.930 4.715
from the 1.2491 10 10.087 4.601
there are 1.1927 8 11.718 4.578
this 1.1827 16 8.577 4.497
there is 1.1343 26 7.111 4.460
the form 1.1086 9 11.302 4.541
Negative termCoefficientDocsMean presentMean absent
it is also -0.9652 8 3.736 5.273
they -0.9573 11 5.395 5.119
form is -0.9068 11 3.848 5.311
near -0.8365 14 3.809 5.368
ethnic is -0.8094 51 5.187 5.111
mentions -0.8063 32 3.901 5.737
that -0.7812 26 5.032 5.191
city -0.7560 68 5.372 4.677
a city -0.7367 63 5.334 4.835
ethnic -0.7355 67 5.395 4.651
the citizen ethnic -0.6899 4 1.624 5.297
citizen ethnic is -0.6899 4 1.624 5.297
citizen ethnic -0.6899 4 1.624 5.297
people -0.6830 12 4.600 5.225
and the -0.6742 22 6.858 4.668
of his -0.6595 7 2.578 5.343
them in -0.6592 4 1.001 5.323
mentions them in -0.6592 4 1.001 5.323
mentions them -0.6592 4 1.001 5.323
says that -0.6498 7 3.463 5.277

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 1.425; cross-validated R^2: 0.084.

Positive termCoefficientDocsMean presentMean absent
and 5.2511 52 6.736 3.431
like 4.1764 5 15.261 4.618
for 3.9398 11 9.828 4.571
as 3.7995 60 6.433 3.225
because 2.5901 12 10.198 4.461
from the form 2.4732 5 12.027 4.788
with 2.3283 16 9.920 4.241
were 2.1767 4 12.517 4.843
from 2.1747 37 7.171 3.962
also 2.0705 43 6.729 3.958
the form 1.9940 9 10.658 4.605
for the 1.9493 7 11.313 4.686
from the 1.8963 13 8.967 4.579
usage 1.8879 4 9.784 4.957
form 1.7442 11 9.542 4.607
there is 1.7335 24 8.155 4.201
long 1.7095 3 15.928 4.816
there 1.6899 34 8.430 3.460
promontory 1.6623 3 11.217 4.962
is 1.6216 87 5.551 2.463
Negative termCoefficientDocsMean presentMean absent
near -2.2077 16 3.454 5.473
in book of -1.9766 14 3.169 5.472
in book -1.8366 16 4.134 5.343
they -1.5868 12 4.835 5.193
the ethnonym -1.4902 56 4.887 5.484
ethnonym -1.4890 57 4.929 5.442
the ethnonym is -1.4633 52 4.771 5.560
ethnonym is -1.4633 52 4.771 5.560
city -1.4305 70 5.330 4.729
or after -1.4236 3 1.398 5.266
people -1.2342 9 5.178 5.147
inhabitant is a -1.1572 6 5.602 5.121
a city -1.1253 64 5.266 4.943
hekataios in -1.1240 12 3.140 5.424
say that -1.1041 9 3.623 5.301
say -1.0890 12 4.983 5.172
that the -1.0729 6 2.952 5.290
city near -1.0650 3 2.350 5.236
book the -1.0363 10 4.664 5.204
strabo in -1.0307 3 1.666 5.257

Prompt Text

Show prompt text 
You are an expert classical philologist and translator specializing in Byzantine Greek geographical texts.
You will receive Greek text from a lemma entry in Stephanos of Byzantium's Ethnika.
Translate the Greek text into clear, scholarly English.
Preserve technical terminology and place names appropriately.
Generated by generate_translation_prompt_evaluation.py.