gpt-5.1 v2 Evaluation

Generated: 2026-07-25 12:53:17 UTC

Pairs100
Distinct lemmas100
Slope0.966
R^20.990
BLEU-445.3%
chrF++68.7%
METEOR73.5%
ROUGE-L74.6%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2347

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.1 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.966
Intercept1.024
R^20.990
P-value1.67e-99
Pearson r0.995
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words43.670
Mean absolute length residual2.670
Mean absolute percent error6.1%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.309, R^2 = 0.095, p = 0.0018.
chrF++ vs passage length
chrF++: significant negative; r = -0.327, R^2 = 0.107, p = 0.0009.
METEOR vs passage length
METEOR: significant negative; r = -0.401, R^2 = 0.161, p = 3.56e-05.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.361, R^2 = 0.130, p = 0.0002.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.285, R^2 = 0.081, p = 0.0041.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.281, R^2 = 0.079, p = 0.0046.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.283, R^2 = 0.080, p = 0.0043.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.288, R^2 = 0.083, p = 0.0037.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.309 0.095 0.0018 -0.00200 ok
chrF++ 100 significant negative -0.327 0.107 0.0009 -0.00129 ok
METEOR 100 significant negative -0.401 0.161 3.56e-05 -0.00189 ok
ROUGE-L 100 significant negative -0.361 0.130 0.0002 -0.00151 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.285 0.081 0.0041 -0.00211 ok
Trigram recall 100 significant negative -0.281 0.079 0.0046 -0.00207 ok
Trigram F1 100 significant negative -0.283 0.080 0.0043 -0.00208 ok
Trigram Jaccard 100 significant negative -0.288 0.083 0.0037 -0.00203 ok

Translation Similarity Metrics

Mean BLEU-445.3%
Mean chrF++68.7%
Mean METEOR73.5%
Mean ROUGE-L74.6%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU45.0%
Unigram F175.9%
Bigram F151.4%
Trigram precision37.9%
Trigram recall37.5%
Trigram F137.7%
Trigram Jaccard23.2%
4-gram F128.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 246 -15.766 34.6% 60.6% 56.5% 65.5% N/A N/A N/A reviewed/approved 12081 gpt-5.1-2025-11-13
2056 Καβειρία 112 95 -14.184 23.3% 64.9% 51.2% 59.9% N/A N/A N/A reviewed/approved 11940 gpt-5.1-2025-11-13
2604 Καρχηδών 117 124 9.988 37.1% 65.5% 70.0% 71.6% N/A N/A N/A reviewed/approved 12096 gpt-5.1-2025-11-13
2603 Κάρυστος 175 179 8.977 34.1% 61.8% 61.7% 65.0% N/A N/A N/A reviewed/approved 12093 gpt-5.1-2025-11-13
2342 Κάμιρος 54 62 8.827 50.1% 74.7% 72.2% 72.4% N/A N/A N/A reviewed/approved 12057 gpt-5.1-2025-11-13
2328 Καλλίπολις 43 51 8.450 23.0% 52.6% 60.9% 59.6% N/A N/A N/A reviewed/approved 12033 gpt-5.1-2025-11-13
2609 Κάσπειρος 135 139 7.605 45.9% 66.2% 58.0% 65.7% N/A N/A N/A reviewed/approved 12105 gpt-5.1-2025-11-13
2468 Καπετώλιον 138 141 6.708 19.9% 48.1% 44.9% 50.2% N/A N/A N/A reviewed/approved 12072 gpt-5.1-2025-11-13
2597 Καρπασία 112 103 -6.184 47.6% 74.3% 78.6% 79.1% N/A N/A N/A reviewed/approved 12084 gpt-5.1-2025-11-13
7255 Κυτέριον 22 28 5.730 40.2% 67.2% 55.9% 64.0% N/A N/A N/A reviewed/approved 12195 gpt-5.1-2025-11-13
7266 Κώμη 79 83 5.685 25.0% 52.2% 50.1% 53.1% N/A N/A N/A reviewed/approved 12228 gpt-5.1-2025-11-13
2620 Κάσταξ 20 15 -5.338 27.1% 51.3% 60.4% 68.6% N/A N/A N/A reviewed/approved 12108 gpt-5.1-2025-11-13
2054 Καβαλίς 65 69 5.205 48.7% 70.3% 75.5% 79.1% N/A N/A N/A reviewed/approved 11934 gpt-5.1-2025-11-13
2330 Καλύβη 24 19 -5.201 27.3% 55.3% 56.2% 69.8% N/A N/A N/A reviewed/approved 12039 gpt-5.1-2025-11-13
7260 Κυχρεῖος πάγος 61 65 5.067 38.7% 64.0% 70.8% 60.3% N/A N/A N/A reviewed/approved 12210 gpt-5.1-2025-11-13
7259 Κύφος 54 58 4.827 34.0% 62.2% 64.8% 64.3% N/A N/A N/A reviewed/approved 12207 gpt-5.1-2025-11-13
2116 Κάληρος 40 35 -4.653 15.6% 49.4% 58.8% 56.0% N/A N/A N/A reviewed/approved 12012 gpt-5.1-2025-11-13
3496 Κοτιάειον 71 65 -4.590 28.7% 48.6% 46.0% 57.4% N/A N/A N/A reviewed/approved 12147 gpt-5.1-2025-11-13
2081 Καλάθη 49 44 -4.344 68.9% 79.3% 83.9% 88.2% N/A N/A N/A reviewed/approved 11985 gpt-5.1-2025-11-13
7254 Κύτα 84 78 -4.144 36.2% 58.4% 56.3% 65.4% N/A N/A N/A reviewed/approved 12192 gpt-5.1-2025-11-13
2605 Κάσιον 56 59 3.896 50.9% 73.9% 75.6% 74.8% N/A N/A N/A reviewed/approved 12099 gpt-5.1-2025-11-13
3288 Κορώνεια 110 111 3.748 33.5% 63.1% 65.5% 69.4% N/A N/A N/A reviewed/approved 12144 gpt-5.1-2025-11-13
7250 Κυρταία 40 36 -3.653 45.0% 65.1% 62.3% 68.4% N/A N/A N/A reviewed/approved 12180 gpt-5.1-2025-11-13
2346 Κάναι 77 79 3.616 60.5% 72.1% 83.5% 85.9% N/A N/A N/A reviewed/approved 12063 gpt-5.1-2025-11-13
2470 Καππαδοκία 68 70 3.307 30.8% 63.0% 73.7% 73.9% N/A N/A N/A reviewed/approved 12075 gpt-5.1-2025-11-13
2626 Κατάνη 95 96 3.233 37.8% 63.3% 63.7% 67.0% N/A N/A N/A reviewed/approved 12129 gpt-5.1-2025-11-13
2119 Κάλλατις 65 67 3.205 44.6% 69.1% 75.5% 72.7% N/A N/A N/A reviewed/approved 12021 gpt-5.1-2025-11-13
2802 Κάδρεμα 24 21 -3.201 47.0% 65.0% 68.0% 71.1% N/A N/A N/A reviewed/approved 11958 gpt-5.1-2025-11-13
7244 Κυρήνη 54 50 -3.173 40.0% 65.8% 64.3% 67.3% N/A N/A N/A reviewed/approved 12162 gpt-5.1-2025-11-13
7257 Κυτώνιον 25 22 -3.167 75.1% 80.0% 84.7% 89.4% N/A N/A N/A reviewed/approved 12201 gpt-5.1-2025-11-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.087.

Positive termCoefficientDocsMean presentMean absent
εν 1.0645 37 1.335 -0.784
τε 1.0105 5 4.700 -0.247
εν τη 0.7744 7 3.484 -0.262
δε και 0.6829 8 1.542 -0.134
οι 0.6714 11 2.733 -0.338
τη 0.6479 15 1.649 -0.291
παιδος 0.6129 4 3.777 -0.157
θηλυκως 0.5825 5 4.522 -0.238
και θηλυκως 0.5825 5 4.522 -0.238
πολιχνιον 0.5661 3 5.090 -0.157
εστι δε και 0.5488 3 5.062 -0.157
στραβων 0.4978 15 1.627 -0.287
πολιτης 0.4896 18 1.513 -0.332
ος 0.4685 6 2.723 -0.174
δευτερα 0.4529 3 4.022 -0.124
αι 0.4399 3 5.123 -0.158
ειναι 0.4358 8 2.230 -0.194
ωστε 0.4321 3 5.839 -0.181
δια του 0.4146 10 1.108 -0.123
δυναται 0.4107 3 3.397 -0.105
Negative termCoefficientDocsMean presentMean absent
το -0.9709 70 -0.223 0.520
εθνος -0.7537 8 -4.718 0.410
εστι και -0.7151 20 -0.875 0.219
το θηλυκον -0.6615 6 -4.929 0.315
οτι -0.6300 5 -2.666 0.140
εστι -0.5939 24 -0.795 0.251
το κτητικον -0.5864 7 -4.905 0.369
κτητικον -0.5864 7 -4.905 0.369
ως -0.5631 52 -0.156 0.169
το εθνικον -0.5363 56 -0.448 0.570
εθνικον -0.5363 56 -0.448 0.570
παρα το -0.5174 3 -4.142 0.128
θρακης -0.4991 5 -1.906 0.100
πολις θρακης -0.4805 4 -2.141 0.089
παρα -0.4798 7 -3.546 0.267
πολις της -0.4254 3 -5.052 0.156
θηλυκον και -0.4249 3 -5.380 0.166
ης -0.4203 3 -3.668 0.113
δε οτι -0.4192 3 -3.460 0.107
ιβηριας -0.4135 3 0.492 -0.015

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.036.

Positive termCoefficientDocsMean presentMean absent
δε 1.7654 46 3.736 1.761
την 1.5869 13 6.541 2.091
εν 1.1895 37 3.523 2.168
γαρ 1.1588 9 5.871 2.353
τη 1.1316 15 4.949 2.267
και 1.1052 66 3.162 1.713
απο 1.0889 27 4.748 1.901
κατα 1.0502 10 5.751 2.327
αφ 1.0267 6 6.351 2.435
εκαλειτο δε 0.9214 3 11.577 2.394
εκαλειτο 0.8287 7 6.023 2.417
της 0.8280 35 3.849 2.035
ιβηριας 0.8129 3 6.166 2.561
αφ ου 0.8099 5 7.289 2.426
περι 0.7881 11 4.889 2.395
μεν 0.7845 4 9.385 2.390
τε 0.7683 5 4.727 2.561
φησι 0.7572 12 4.963 2.357
δια 0.7529 16 4.507 2.320
ος 0.7137 6 5.634 2.480
Negative termCoefficientDocsMean presentMean absent
και το -0.7381 9 2.933 2.644
μια -0.6990 3 0.399 2.740
τεταρτω -0.6912 3 0.746 2.729
το εθνικον -0.6691 56 2.763 2.551
εθνικον -0.6691 56 2.763 2.551
καππαδοκιας -0.5662 3 1.534 2.705
νησος -0.5191 9 2.107 2.725
πολυβιος -0.4952 5 1.216 2.746
πλησιον -0.4599 8 2.108 2.718
και πολις -0.4548 5 1.880 2.711
προς τη -0.4504 4 1.722 2.709
στραβων -0.4341 15 2.884 2.632
φασι -0.4128 8 3.115 2.631
ηροδοτος -0.4085 9 3.529 2.585
ειναι -0.3987 8 2.807 2.658
εκαταιος ασια -0.3921 5 1.463 2.733
ασια -0.3921 5 1.463 2.733
δευτερω -0.3848 3 1.502 2.706
τους -0.3672 4 1.889 2.702
ιταλιας -0.3415 3 1.090 2.718

AI English terms predicting large absolute residuals

Features: 327; ridge alpha: 2.894; cross-validated R^2: -0.035.

Positive termCoefficientDocsMean presentMean absent
the 1.9519 92 2.805 1.112
on 1.1675 16 4.818 2.260
and 1.1672 59 3.284 1.785
there 1.0303 31 4.068 2.041
after 1.0231 14 5.506 2.208
whom 0.9407 7 6.687 2.367
after whom 0.9265 5 7.289 2.426
says 0.8905 18 5.141 2.127
diphthong 0.8447 3 9.552 2.457
of 0.8422 86 2.879 1.385
from 0.7942 41 3.845 1.853
to 0.7435 18 4.344 2.302
and there is 0.7185 3 10.118 2.439
also 0.7152 41 3.543 2.063
as 0.7127 58 3.330 1.758
was called 0.7062 4 5.105 2.568
by 0.6974 13 4.550 2.389
on the 0.6895 9 5.067 2.432
and there 0.6893 6 7.395 2.368
form 0.6737 7 6.901 2.351
Negative termCoefficientDocsMean presentMean absent
one -0.6671 9 1.720 2.763
the ethnonym is -0.6409 55 2.457 2.929
the ethnonym -0.6263 57 2.539 2.843
ethnonym -0.4976 59 2.705 2.618
ethnonym is -0.4813 56 2.667 2.673
near -0.4705 16 2.531 2.696
one of the -0.4610 4 1.108 2.735
one of -0.4610 4 1.108 2.735
a city and -0.4526 5 1.637 2.724
italy -0.4209 4 0.906 2.743
kappadokia -0.4076 4 1.977 2.698
headland -0.3888 3 1.722 2.699
syria -0.3741 3 0.574 2.734
a city -0.3678 67 2.994 2.010
in his asia -0.3677 5 1.463 2.733
his asia -0.3677 5 1.463 2.733
and in the -0.3618 10 3.212 2.609
herodotus -0.3579 8 1.999 2.728
city of the -0.3558 9 2.485 2.688
village -0.3557 3 0.788 2.728

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: 0.029.

Positive termCoefficientDocsMean presentMean absent
the 2.3659 90 2.848 1.067
for 1.5095 11 5.956 2.263
on 1.4033 16 4.653 2.292
with 1.1475 16 4.294 2.360
from the 1.1329 13 5.427 2.257
according 1.1253 5 7.381 2.422
according to 1.1253 5 7.381 2.422
from the form 1.0958 5 5.663 2.512
also 1.0747 43 3.580 1.983
are also 1.0093 4 6.926 2.492
is with 1.0003 4 8.217 2.438
is from 0.9939 12 4.409 2.432
for the 0.9693 7 7.471 2.308
form 0.9562 11 5.157 2.362
and 0.9558 52 3.521 1.747
to 0.9410 31 4.050 2.049
diphthong 0.9216 3 9.552 2.457
usage 0.8892 4 6.435 2.513
the form 0.8613 9 5.737 2.366
situated 0.8526 4 7.804 2.456
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.0332 52 2.392 2.970
ethnonym is -1.0332 52 2.392 2.970
kappadokia -0.8559 3 1.741 2.698
and the -0.7384 15 3.102 2.593
ethnonym -0.6664 57 2.752 2.561
one of the -0.6333 4 1.108 2.735
is a city -0.6200 3 0.868 2.725
ethnonyms -0.6156 3 1.006 2.721
the ethnonym -0.6123 56 2.788 2.519
an -0.5590 22 2.219 2.797
near -0.5555 16 2.608 2.681
syria -0.5445 3 0.574 2.734
in syria -0.5445 3 0.574 2.734
as in -0.5199 29 2.588 2.703
territory -0.5043 5 1.550 2.728
in his asia -0.4989 5 1.463 2.733
his asia -0.4989 5 1.463 2.733
one of -0.4929 5 1.755 2.718
it is -0.4768 24 3.068 2.544
one -0.4640 12 2.561 2.684

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.