gpt-4.1-2025-04-14 v2 Evaluation

Generated: 2026-07-25 12:52:30 UTC

Pairs100
Distinct lemmas100
Slope0.948
R^20.988
BLEU-441.2%
chrF++66.0%
METEOR72.4%
ROUGE-L73.1%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2417

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4.1 (2025-04-14) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.948
Intercept1.757
R^20.988
P-value2.53e-95
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words43.620
Mean absolute length residual2.918
Mean absolute percent error6.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.289, R^2 = 0.083, p = 0.0036.
chrF++ vs passage length
chrF++: significant negative; r = -0.285, R^2 = 0.081, p = 0.0041.
METEOR vs passage length
METEOR: significant negative; r = -0.367, R^2 = 0.135, p = 0.0002.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.384, R^2 = 0.148, p = 7.91e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.272, R^2 = 0.074, p = 0.0061.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.287, R^2 = 0.082, p = 0.0038.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.279, R^2 = 0.078, p = 0.0049.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.285, R^2 = 0.081, p = 0.0040.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.289 0.083 0.0036 -0.00178 ok
chrF++ 100 significant negative -0.285 0.081 0.0041 -0.00115 ok
METEOR 100 significant negative -0.367 0.135 0.0002 -0.00188 ok
ROUGE-L 100 significant negative -0.384 0.148 7.91e-05 -0.00172 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.272 0.074 0.0061 -0.00197 ok
Trigram recall 100 significant negative -0.287 0.082 0.0038 -0.00205 ok
Trigram F1 100 significant negative -0.279 0.078 0.0049 -0.00200 ok
Trigram Jaccard 100 significant negative -0.285 0.081 0.0040 -0.00181 ok

Translation Similarity Metrics

Mean BLEU-441.2%
Mean chrF++66.0%
Mean METEOR72.4%
Mean ROUGE-L73.1%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU42.4%
Unigram F175.1%
Bigram F148.4%
Trigram precision35.0%
Trigram recall34.6%
Trigram F134.8%
Trigram Jaccard21.1%
4-gram F126.1%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2603 Κάρυστος 175 184 16.347 26.6% 57.6% 53.9% 56.8% N/A N/A N/A reviewed/approved 13294 gpt-4.1-2025-04-14
2055 Καβασσός 109 92 -13.087 21.8% 56.4% 60.6% 59.7% N/A N/A N/A reviewed/approved 13138 gpt-4.1-2025-04-14
2604 Καρχηδών 117 125 12.330 35.5% 64.7% 69.8% 70.0% N/A N/A N/A reviewed/approved 13297 gpt-4.1-2025-04-14
2342 Κάμιρος 54 65 12.052 45.1% 72.4% 67.7% 70.6% N/A N/A N/A reviewed/approved 13258 gpt-4.1-2025-04-14
2484 Καρία 270 246 -11.710 27.0% 56.2% 58.1% 59.3% N/A N/A N/A reviewed/approved 13282 gpt-4.1-2025-04-14
3288 Κορώνεια 110 116 9.965 35.6% 64.5% 71.1% 70.5% N/A N/A N/A reviewed/approved 13345 gpt-4.1-2025-04-14
3496 Κοτιάειον 71 61 -8.064 30.4% 50.2% 58.1% 56.1% N/A N/A N/A reviewed/approved 13348 gpt-4.1-2025-04-14
2346 Κάναι 77 82 7.249 46.6% 65.8% 79.3% 79.2% N/A N/A N/A reviewed/approved 13264 gpt-4.1-2025-04-14
2115 Καλὴ ἀκτή 64 69 6.572 31.9% 55.9% 75.0% 67.7% N/A N/A N/A reviewed/approved 13210 gpt-4.1-2025-04-14
2119 Κάλλατις 65 69 5.624 33.1% 60.2% 60.5% 59.7% N/A N/A N/A reviewed/approved 13222 gpt-4.1-2025-04-14
2468 Καπετώλιον 138 127 -5.578 14.4% 43.1% 43.5% 47.5% N/A N/A N/A reviewed/approved 13273 gpt-4.1-2025-04-14
2328 Καλλίπολις 43 48 5.480 18.6% 54.8% 53.4% 56.5% N/A N/A N/A reviewed/approved 13234 gpt-4.1-2025-04-14
7255 Κυτέριον 22 28 5.387 37.2% 60.6% 51.8% 56.0% N/A N/A N/A reviewed/approved 13396 gpt-4.1-2025-04-14
7247 Κύρνος 50 44 -5.156 36.5% 59.5% 67.0% 61.7% N/A N/A N/A reviewed/approved 13372 gpt-4.1-2025-04-14
7259 Κύφος 54 58 5.052 45.3% 71.3% 78.5% 76.8% N/A N/A N/A reviewed/approved 13408 gpt-4.1-2025-04-14
2626 Κατάνη 95 87 -4.815 28.9% 54.2% 48.0% 52.7% N/A N/A N/A reviewed/approved 13330 gpt-4.1-2025-04-14
7266 Κώμη 79 72 -4.647 18.3% 45.3% 39.3% 46.4% N/A N/A N/A reviewed/approved 13429 gpt-4.1-2025-04-14
2625 Κατακεκαυμένη 46 50 4.636 36.5% 67.4% 76.5% 72.9% N/A N/A N/A reviewed/approved 13324 gpt-4.1-2025-04-14
2465 Κάνωπος 83 85 4.561 44.6% 70.1% 81.9% 82.1% N/A N/A N/A reviewed/approved 13270 gpt-4.1-2025-04-14
2059 Καδμεία 20 25 4.283 25.0% 55.3% 71.7% 75.6% N/A N/A N/A reviewed/approved 13150 gpt-4.1-2025-04-14
7253 Κυρτώνιος 18 23 4.179 9.1% 52.9% 64.8% 58.5% N/A N/A N/A reviewed/approved 13390 gpt-4.1-2025-04-14
2597 Καρπασία 112 104 -3.931 39.0% 65.2% 61.4% 70.4% N/A N/A N/A reviewed/approved 13285 gpt-4.1-2025-04-14
2326 Καλλίαρος 47 50 3.688 46.0% 68.6% 83.4% 78.4% N/A N/A N/A reviewed/approved 13228 gpt-4.1-2025-04-14
2062 Κάθαια 25 29 3.543 30.4% 59.8% 73.8% 59.3% N/A N/A N/A reviewed/approved 13162 gpt-4.1-2025-04-14
2802 Κάδρεμα 24 21 -3.509 41.2% 61.4% 68.0% 71.1% N/A N/A N/A reviewed/approved 13159 gpt-4.1-2025-04-14
2330 Καλύβη 24 21 -3.509 41.4% 62.9% 64.1% 80.0% N/A N/A N/A reviewed/approved 13240 gpt-4.1-2025-04-14
7260 Κυχρεῖος πάγος 61 63 3.416 41.0% 65.6% 70.3% 64.5% N/A N/A N/A reviewed/approved 13411 gpt-4.1-2025-04-14
7243 Κύρη 22 26 3.387 46.0% 67.8% 88.1% 83.3% N/A N/A N/A reviewed/approved 13360 gpt-4.1-2025-04-14
7251 Κύρτος 66 61 -3.324 15.1% 47.8% 58.7% 59.8% N/A N/A N/A reviewed/approved 13384 gpt-4.1-2025-04-14
2599 Καρπήσιοι 11 9 -3.185 4.8% 27.7% 13.9% 30.0% N/A N/A N/A reviewed/approved 13288 gpt-4.1-2025-04-14

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.060.

Positive termCoefficientDocsMean presentMean absent
δε και 1.8012 8 3.653 -0.318
απο 1.7083 27 1.606 -0.594
και 1.5049 66 0.125 -0.242
παιδος 1.5029 4 5.597 -0.233
τη 1.4620 15 1.862 -0.329
εστι δε 1.4098 5 4.883 -0.257
οι 1.3512 11 3.329 -0.412
εν τη 1.3396 7 3.604 -0.271
εκ 1.2785 6 3.751 -0.239
πολιτης 1.2442 18 2.609 -0.573
πολις 1.2264 68 0.386 -0.821
εστι δε και 1.2033 3 6.925 -0.214
εκαλειτο δε 1.1749 3 5.655 -0.175
δυο 1.1678 4 4.871 -0.203
ου 1.1274 14 2.157 -0.351
πολιχνιον 1.0979 3 6.117 -0.189
αφ 1.0575 6 3.592 -0.229
της 1.0513 35 0.259 -0.139
της χωρας 0.9816 3 3.963 -0.123
χωρας 0.9667 4 3.082 -0.128
Negative termCoefficientDocsMean presentMean absent
καππαδοκιας -1.6254 3 -5.441 0.168
γαρ -1.5607 9 -1.756 0.174
οικητωρ -1.4076 6 -4.336 0.277
περι -1.3558 11 -2.377 0.294
το -1.3098 70 -0.297 0.692
ειναι -1.2544 8 -2.273 0.198
λυκιας -1.2453 3 -6.139 0.190
τοπικα -1.1274 3 -6.025 0.186
τα τοπικα -1.1274 3 -6.025 0.186
εθνος -1.1184 8 -2.759 0.240
ως το -1.1056 3 -6.447 0.199
φησιν -0.9116 10 -2.007 0.223
το θηλυκον -0.8628 6 -2.388 0.152
οτι -0.8591 5 -2.076 0.109
πλησιον -0.8577 8 -0.571 0.050
και του -0.8483 4 -4.010 0.167
δε -0.8410 46 0.257 -0.219
μεταξυ -0.8063 5 -1.865 0.098
ελλανικος -0.7850 4 -6.260 0.261
δε απο -0.7798 3 -2.233 0.069

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: 0.038.

Positive termCoefficientDocsMean presentMean absent
δε 1.6803 46 3.863 2.112
της 1.2248 35 4.369 2.136
απο 1.1929 27 5.069 2.122
και 1.0580 66 3.513 1.763
την 1.0231 13 5.824 2.483
εκαλειτο 0.9170 7 8.306 2.512
εκαλειτο δε 0.9058 3 13.462 2.591
γαρ 0.9048 9 6.986 2.515
δε και 0.8921 8 7.643 2.507
του 0.8723 37 4.594 1.933
δια 0.8538 16 5.746 2.379
εν 0.8440 37 4.151 2.193
δυναται 0.7583 3 9.480 2.715
ομηρος 0.7162 3 8.886 2.733
εκαλειτο δε και 0.6723 3 13.462 2.591
τω 0.6535 16 4.360 2.643
εστι 0.6356 24 4.441 2.436
περι 0.6353 11 5.017 2.658
εστι δε 0.6254 5 6.664 2.720
τη 0.6212 15 4.386 2.658
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.8422 56 2.703 3.190
εθνικον -0.8422 56 2.703 3.190
ως -0.6402 52 2.965 2.867
εκαταιος ασια -0.4144 5 1.078 3.014
ασια -0.4144 5 1.078 3.014
παυσανιας -0.4123 4 1.520 2.976
μια -0.3881 3 1.131 2.973
φησι το -0.3631 3 0.723 2.985
πολις και -0.3519 6 3.549 2.877
ενατω -0.3320 3 1.299 2.968
αυτην -0.3313 9 3.321 2.878
του δε -0.3304 3 1.823 2.951
εξ -0.3294 3 2.245 2.938
εθνος -0.3239 8 2.912 2.918
προς τη -0.3228 4 2.059 2.953
δε οτι -0.3176 3 1.239 2.970
πλησιον -0.3165 8 2.386 2.964
πορρω -0.3030 3 0.948 2.979
ου πορρω -0.3030 3 0.948 2.979
ηροδοτος -0.2965 9 2.825 2.927

AI English terms predicting large absolute residuals

Features: 325; ridge alpha: 4.125; cross-validated R^2: -0.030.

Positive termCoefficientDocsMean presentMean absent
the 1.3721 91 3.048 1.598
and 1.0342 51 4.030 1.760
from 0.8210 37 4.618 1.919
most 0.8053 5 9.540 2.569
that 0.8022 12 6.330 2.452
on 0.7979 11 5.317 2.621
by 0.7404 20 5.160 2.357
for 0.6980 16 5.156 2.491
of 0.6865 83 3.049 2.277
with 0.6412 19 4.665 2.508
was 0.6318 18 4.990 2.463
used 0.5888 3 11.686 2.646
at the 0.5671 4 9.004 2.664
at 0.5575 7 7.094 2.603
called 0.5536 32 4.562 2.144
renowned 0.5416 3 9.732 2.707
there 0.5291 33 4.266 2.254
son 0.5290 18 5.513 2.348
son of 0.5290 18 5.513 2.348
was called 0.5229 4 7.877 2.711
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.7437 51 2.478 3.375
ethnonym is -0.7437 51 2.478 3.375
as in -0.6418 40 2.561 3.156
the ethnonym -0.5198 57 2.916 2.919
ethnonym -0.5198 57 2.916 2.919
a city of -0.4796 43 2.699 3.083
city of -0.4531 46 2.663 3.134
but -0.4109 17 3.060 2.888
pausanias -0.3898 4 1.520 2.976
in book of -0.3493 8 1.606 3.032
book of -0.3338 15 2.191 3.046
book -0.3169 39 3.008 2.860
asia -0.3132 6 1.386 3.015
of his -0.3119 15 2.253 3.035
calls -0.3106 5 1.576 2.988
book of his -0.3095 14 2.170 3.039
calls it -0.3032 4 1.524 2.976
in his asia -0.3005 5 1.078 3.014
his asia -0.3005 5 1.078 3.014
in book -0.2984 20 3.339 2.812

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 5.878; cross-validated R^2: -0.055.

Positive termCoefficientDocsMean presentMean absent
the 0.9093 90 3.073 1.520
there 0.9078 34 4.667 2.017
and 0.7646 52 3.892 1.862
of 0.7469 66 3.478 1.830
after 0.6621 22 5.598 2.162
with 0.6038 16 5.247 2.474
that 0.5298 29 3.875 2.526
when 0.4840 7 6.747 2.629
says 0.4777 17 5.273 2.435
on 0.4656 16 4.439 2.628
for 0.4648 11 5.901 2.549
who 0.4526 13 5.848 2.480
to be 0.4450 9 7.829 2.432
also 0.4447 43 3.858 2.208
used to be 0.4334 6 9.490 2.498
too 0.4305 3 8.954 2.731
is also 0.4267 31 4.303 2.295
this 0.4123 15 6.491 2.287
son 0.4000 17 5.791 2.329
son of 0.4000 17 5.791 2.329
Negative termCoefficientDocsMean presentMean absent
as in -0.5919 29 2.458 3.105
the ethnonym is -0.5663 52 2.468 3.404
ethnonym is -0.5663 52 2.468 3.404
the ethnonym -0.4707 56 2.750 3.131
ethnonym -0.3999 57 2.931 2.899
as -0.3107 60 2.946 2.875
in book of -0.2983 14 1.878 3.087
but -0.2739 12 2.470 2.979
asia -0.2654 6 1.386 3.015
in his asia -0.2625 5 1.078 3.014
his asia -0.2625 5 1.078 3.014
and the -0.2593 15 2.671 2.961
or after -0.2546 3 1.607 2.958
in book -0.2383 16 2.486 3.000
a people -0.2382 5 1.443 2.995
one of -0.2341 5 1.800 2.976
pausanias book -0.2289 3 1.191 2.971
pausanias -0.2236 4 1.520 2.976
per -0.2218 20 2.840 2.937
as per -0.2218 20 2.840 2.937

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.