gpt-4.1-2025-04-14 v2 Evaluation

Generated: 2026-09-20 11:20:01 UTC

Pairs100
Distinct lemmas100
Slope0.948
R^20.988
BLEU-441.3%
chrF++66.1%
METEOR72.4%
ROUGE-L73.1%
BERTScore95.4%
COMET78.2%
BLEURT72.2%

Profile version id: 2417

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4.1 (2025-04-14) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.948
Intercept1.775
R^20.988
P-value2.18e-95
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words43.620
Mean absolute length residual2.911
Mean absolute percent error6.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.288, R^2 = 0.083, p = 0.0036.
chrF++ vs passage length
chrF++: significant negative; r = -0.289, R^2 = 0.084, p = 0.0035.
METEOR vs passage length
METEOR: significant negative; r = -0.370, R^2 = 0.137, p = 0.0002.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.386, R^2 = 0.149, p = 7.36e-05.
BERTScore vs passage length
BERTScore: significant negative; r = -0.562, R^2 = 0.316, p = 1.15e-09.
COMET vs passage length
COMET: significant negative; r = -0.391, R^2 = 0.153, p = 5.68e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.701, R^2 = 0.492, p = 4.59e-16.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.272, R^2 = 0.074, p = 0.0061.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.287, R^2 = 0.082, p = 0.0038.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.279, R^2 = 0.078, p = 0.0049.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.285, R^2 = 0.081, p = 0.0040.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.288 0.083 0.0036 -0.00178 ok
chrF++ 100 significant negative -0.289 0.084 0.0035 -0.00116 ok
METEOR 100 significant negative -0.370 0.137 0.0002 -0.00190 ok
ROUGE-L 100 significant negative -0.386 0.149 7.36e-05 -0.00172 ok
BERTScore 100 significant negative -0.562 0.316 1.15e-09 -0.00044 ok
COMET 100 significant negative -0.391 0.153 5.68e-05 -0.00072 ok
BLEURT 100 significant negative -0.701 0.492 4.59e-16 -0.00231 ok
Trigram precision 100 significant negative -0.272 0.074 0.0061 -0.00197 ok
Trigram recall 100 significant negative -0.287 0.082 0.0038 -0.00205 ok
Trigram F1 100 significant negative -0.279 0.078 0.0049 -0.00200 ok
Trigram Jaccard 100 significant negative -0.285 0.081 0.0040 -0.00181 ok

Translation Similarity Metrics

Mean BLEU-441.3%
Mean chrF++66.1%
Mean METEOR72.4%
Mean ROUGE-L73.1%
Mean BERTScore95.4%
Mean COMET78.2%
Mean BLEURT72.2%
Legacy smoothed corpus BLEU42.4%
Unigram F175.2%
Bigram F148.5%
Trigram precision35.0%
Trigram recall34.6%
Trigram F134.8%
Trigram Jaccard21.1%
4-gram F126.1%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2603 Κάρυστος 175 184 16.362 26.6% 57.6% 53.9% 56.8% 93.0% 79.2% 59.3% reviewed/approved 13294 gpt-4.1-2025-04-14
2055 Καβασσός 109 92 -13.084 21.8% 56.4% 60.6% 59.7% 93.8% 73.6% 48.5% reviewed/approved 13138 gpt-4.1-2025-04-14
2604 Καρχηδών 117 125 12.334 35.5% 64.7% 69.8% 70.0% 94.0% 78.1% 62.9% reviewed/approved 13297 gpt-4.1-2025-04-14
2342 Κάμιρος 54 65 12.044 45.1% 72.4% 67.7% 70.6% 95.4% 77.9% 68.3% reviewed/approved 13258 gpt-4.1-2025-04-14
2484 Καρία 270 246 -11.678 27.0% 56.2% 58.1% 59.3% 90.1% 77.9% 49.2% reviewed/approved 13282 gpt-4.1-2025-04-14
3288 Κορώνεια 110 116 9.968 35.6% 64.5% 71.1% 70.5% 94.8% 75.8% 63.1% reviewed/approved 13345 gpt-4.1-2025-04-14
3496 Κοτιάειον 71 61 -8.068 30.4% 50.2% 58.1% 56.1% 93.1% 71.6% 63.2% reviewed/approved 13348 gpt-4.1-2025-04-14
2346 Κάναι 77 82 7.245 46.6% 65.8% 79.3% 79.2% 95.0% 76.7% 63.5% reviewed/approved 13264 gpt-4.1-2025-04-14
2115 Καλὴ ἀκτή 64 69 6.566 31.9% 55.9% 75.0% 67.7% 93.3% 68.8% 56.0% reviewed/approved 13210 gpt-4.1-2025-04-14
2119 Κάλλατις 65 69 5.619 33.1% 60.2% 60.5% 59.7% 95.0% 71.3% 68.9% reviewed/approved 13222 gpt-4.1-2025-04-14
2468 Καπετώλιον 138 127 -5.570 14.4% 43.1% 43.5% 47.5% 86.2% 64.7% 47.8% reviewed/approved 13273 gpt-4.1-2025-04-14
2328 Καλλίπολις 43 48 5.470 18.6% 54.8% 53.4% 56.5% 93.0% 72.1% 65.7% reviewed/approved 13234 gpt-4.1-2025-04-14
7255 Κυτέριον 22 28 5.373 37.2% 60.6% 51.8% 56.0% 94.4% 77.6% 75.0% reviewed/approved 13396 gpt-4.1-2025-04-14
7247 Κύρνος 50 44 -5.165 36.5% 59.5% 67.0% 61.7% 95.9% 75.9% 76.5% reviewed/approved 13372 gpt-4.1-2025-04-14
7259 Κύφος 54 58 5.044 45.3% 71.3% 78.5% 76.8% 95.8% 74.5% 69.7% reviewed/approved 13408 gpt-4.1-2025-04-14
2626 Κατάνη 95 87 -4.815 28.9% 54.2% 48.0% 52.7% 92.5% 73.5% 56.6% reviewed/approved 13330 gpt-4.1-2025-04-14
7266 Κώμη 79 72 -4.650 18.3% 45.3% 39.3% 46.4% 92.9% 73.6% 54.2% reviewed/approved 13429 gpt-4.1-2025-04-14
2625 Κατακεκαυμένη 46 50 4.627 36.5% 67.4% 76.5% 72.9% 94.6% 74.3% 72.6% reviewed/approved 13324 gpt-4.1-2025-04-14
2465 Κάνωπος 83 85 4.559 44.6% 70.1% 81.9% 82.1% 94.8% 76.0% 67.0% reviewed/approved 13270 gpt-4.1-2025-04-14
2059 Καδμεία 20 25 4.269 25.0% 55.3% 71.7% 75.6% 93.9% 78.7% 72.9% reviewed/approved 13150 gpt-4.1-2025-04-14
7253 Κυρτώνιος 18 23 4.165 9.1% 52.9% 64.8% 58.5% 94.3% 73.9% 77.5% reviewed/approved 13390 gpt-4.1-2025-04-14
2597 Καρπασία 112 104 -3.927 39.0% 65.2% 61.4% 70.4% 95.3% 74.5% 63.7% reviewed/approved 13285 gpt-4.1-2025-04-14
2326 Καλλίαρος 47 50 3.679 46.0% 68.6% 83.4% 78.4% 96.7% 74.8% 71.1% reviewed/approved 13228 gpt-4.1-2025-04-14
2062 Κάθαια 25 29 3.530 30.4% 59.8% 73.8% 59.3% 97.1% 80.5% 80.9% reviewed/approved 13162 gpt-4.1-2025-04-14
2802 Κάδρεμα 24 21 -3.522 41.2% 61.4% 68.0% 71.1% 96.5% 79.9% 76.5% reviewed/approved 13159 gpt-4.1-2025-04-14
2330 Καλύβη 24 21 -3.522 41.4% 62.9% 64.1% 80.0% 96.4% 79.8% 77.2% reviewed/approved 13240 gpt-4.1-2025-04-14
7260 Κυχρεῖος πάγος 61 63 3.410 41.0% 65.6% 70.3% 64.5% 95.7% 77.0% 62.0% reviewed/approved 13411 gpt-4.1-2025-04-14
7243 Κύρη 22 26 3.373 46.0% 67.8% 88.1% 83.3% 95.6% 77.3% 75.6% reviewed/approved 13360 gpt-4.1-2025-04-14
7251 Κύρτος 66 61 -3.329 15.1% 47.8% 58.7% 59.8% 90.8% 65.2% 50.8% reviewed/approved 13384 gpt-4.1-2025-04-14
2345 Κάναθα 17 21 3.112 50.3% 76.0% 89.5% 78.9% 96.4% 85.8% 64.9% reviewed/approved 13261 gpt-4.1-2025-04-14

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.063.

Positive termCoefficientDocsMean presentMean absent
δε και 1.8001 8 3.655 -0.318
απο 1.7075 27 1.603 -0.593
παιδος 1.5019 4 5.587 -0.233
και 1.4852 66 0.118 -0.228
τη 1.4585 15 1.858 -0.328
εστι δε 1.4080 5 4.882 -0.257
οι 1.3460 11 3.325 -0.411
εν τη 1.3380 7 3.596 -0.271
εκ 1.2759 6 3.744 -0.239
πολιτης 1.2350 18 2.602 -0.571
πολις 1.2085 68 0.378 -0.803
εστι δε και 1.2022 3 6.924 -0.214
εκαλειτο δε 1.1766 3 5.673 -0.175
δυο 1.1647 4 4.867 -0.203
ου 1.1222 14 2.154 -0.351
πολιχνιον 1.0959 3 6.111 -0.189
αφ 1.0573 6 3.591 -0.229
της 1.0424 35 0.253 -0.136
της χωρας 0.9794 3 3.950 -0.122
χωρας 0.9621 4 3.069 -0.128
Negative termCoefficientDocsMean presentMean absent
καππαδοκιας -1.6276 3 -5.448 0.169
γαρ -1.5563 9 -1.751 0.173
οικητωρ -1.4121 6 -4.340 0.277
περι -1.3586 11 -2.381 0.294
το -1.3246 70 -0.305 0.711
ειναι -1.2568 8 -2.278 0.198
λυκιας -1.2483 3 -6.147 0.190
τοπικα -1.1297 3 -6.031 0.187
τα τοπικα -1.1297 3 -6.031 0.187
ως το -1.1057 3 -6.447 0.199
εθνος -1.0110 8 -2.645 0.230
φησιν -0.9126 10 -2.006 0.223
πλησιον -0.8685 8 -0.581 0.051
το θηλυκον -0.8637 6 -2.387 0.152
οτι -0.8625 5 -2.080 0.109
και του -0.8566 4 -4.019 0.167
δε -0.8484 46 0.252 -0.215
μεταξυ -0.8135 5 -1.872 0.099
ελλανικος -0.7837 4 -6.252 0.261
δε απο -0.7800 3 -2.239 0.069

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.044.

Positive termCoefficientDocsMean presentMean absent
δε 2.0307 46 3.863 2.100
της 1.5424 35 4.369 2.126
απο 1.4719 27 5.068 2.113
την 1.2643 13 5.823 2.476
εκαλειτο δε 1.2055 3 13.458 2.585
εκαλειτο 1.1951 7 8.307 2.505
και 1.1682 66 3.514 1.741
γαρ 1.1329 9 6.982 2.508
δε και 1.1310 8 7.642 2.500
εν 1.0623 37 4.151 2.183
δια 1.0459 16 5.745 2.371
δυναται 0.9996 3 9.475 2.708
του 0.9726 37 4.595 1.922
ομηρος 0.9444 3 8.886 2.726
εκαλειτο δε και 0.8961 3 13.458 2.585
τω 0.8154 16 4.359 2.635
πολις εν 0.8071 5 6.484 2.723
αφ 0.8001 6 5.876 2.722
εστι δε 0.7988 5 6.665 2.713
περι 0.7857 11 5.017 2.651
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.9509 56 2.706 3.173
εθνικον -0.9509 56 2.706 3.173
ως -0.7006 52 2.966 2.852
παυσανιας -0.5083 4 1.530 2.969
μια -0.5007 3 1.142 2.966
εθνος -0.4853 8 2.798 2.921
φησι το -0.4792 3 0.724 2.979
εξ -0.4705 3 2.237 2.932
αυτην -0.4687 9 3.324 2.870
του δε -0.4627 3 1.831 2.944
εκαταιος ασια -0.4473 5 1.091 3.007
ασια -0.4473 5 1.091 3.007
πολις και -0.4470 6 3.553 2.870
δε οτι -0.4388 3 1.238 2.963
προς τη -0.4255 4 2.065 2.946
ενατω -0.4215 3 1.306 2.961
πλησιον -0.3927 8 2.393 2.956
οτι -0.3895 5 2.491 2.933
πορρω -0.3681 3 0.945 2.972
ου πορρω -0.3681 3 0.945 2.972

AI English terms predicting large absolute residuals

Features: 325; ridge alpha: 4.125; cross-validated R^2: -0.025.

Positive termCoefficientDocsMean presentMean absent
the 1.3799 91 3.040 1.609
and 1.0452 51 4.030 1.747
from 0.8245 37 4.616 1.910
most 0.8053 5 9.531 2.563
that 0.8030 12 6.327 2.445
on 0.7983 11 5.313 2.614
by 0.7426 20 5.158 2.349
for 0.6984 16 5.154 2.484
of 0.6797 83 3.039 2.285
with 0.6428 19 4.664 2.500
was 0.6367 18 4.989 2.455
used 0.5889 3 11.688 2.640
at the 0.5691 4 9.011 2.657
at 0.5596 7 7.096 2.596
called 0.5582 32 4.561 2.134
renowned 0.5421 3 9.731 2.700
there 0.5349 33 4.266 2.244
son 0.5318 18 5.511 2.340
son of 0.5318 18 5.511 2.340
was called 0.5254 4 7.879 2.704
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.7335 51 2.481 3.359
ethnonym is -0.7335 51 2.481 3.359
as in -0.6348 40 2.561 3.145
the ethnonym -0.5100 57 2.919 2.901
ethnonym -0.5100 57 2.919 2.901
a city of -0.4691 43 2.701 3.069
city of -0.4426 46 2.666 3.120
but -0.4082 17 3.058 2.881
pausanias -0.3848 4 1.530 2.969
in book of -0.3466 8 1.608 3.024
book of -0.3316 15 2.190 3.038
calls -0.3115 5 1.572 2.982
book -0.3102 39 3.009 2.849
of his -0.3095 15 2.252 3.027
book of his -0.3069 14 2.170 3.032
calls it -0.3044 4 1.517 2.969
asia -0.3043 6 1.397 3.008
in book -0.2946 20 3.338 2.804
in his asia -0.2917 5 1.091 3.007
his asia -0.2917 5 1.091 3.007

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 5.878; cross-validated R^2: -0.051.

Positive termCoefficientDocsMean presentMean absent
there 0.9134 34 4.668 2.006
the 0.8976 90 3.065 1.528
and 0.7736 52 3.892 1.849
of 0.7577 66 3.479 1.809
after 0.6670 22 5.597 2.154
with 0.6062 16 5.245 2.467
that 0.5346 29 3.876 2.517
when 0.4863 7 6.745 2.623
says 0.4805 17 5.271 2.428
for 0.4690 11 5.900 2.542
on 0.4670 16 4.435 2.621
who 0.4543 13 5.850 2.472
also 0.4524 43 3.859 2.196
to be 0.4474 9 7.830 2.425
used to be 0.4359 6 9.491 2.491
is also 0.4316 31 4.304 2.285
too 0.4308 3 8.949 2.724
this 0.4145 15 6.491 2.279
son 0.4022 17 5.788 2.322
son of 0.4022 17 5.788 2.322
Negative termCoefficientDocsMean presentMean absent
as in -0.5898 29 2.457 3.097
the ethnonym is -0.5590 52 2.470 3.389
ethnonym is -0.5590 52 2.470 3.389
the ethnonym -0.4636 56 2.752 3.114
ethnonym -0.3925 57 2.933 2.882
as -0.3028 60 2.947 2.858
in book of -0.2949 14 1.877 3.079
but -0.2726 12 2.468 2.971
asia -0.2590 6 1.397 3.008
and the -0.2560 15 2.669 2.954
in his asia -0.2559 5 1.091 3.007
his asia -0.2559 5 1.091 3.007
or after -0.2532 3 1.612 2.951
people -0.2392 9 2.638 2.938
in book -0.2350 16 2.484 2.992
a people -0.2338 5 1.451 2.988
one of -0.2330 5 1.805 2.969
pausanias book -0.2265 3 1.199 2.964
pausanias -0.2208 4 1.530 2.969
per -0.2204 20 2.838 2.929

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.