gpt-5.1 v2 Evaluation

Generated: 2026-09-20 11:21:17 UTC

Pairs100
Distinct lemmas100
Slope0.966
R^20.990
BLEU-445.4%
chrF++68.7%
METEOR73.6%
ROUGE-L74.7%
BERTScore95.7%
COMET78.8%
BLEURT73.3%

Profile version id: 2347

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.1 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.966
Intercept1.043
R^20.990
P-value1.46e-99
Pearson r0.995
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words43.670
Mean absolute length residual2.663
Mean absolute percent error6.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.308, R^2 = 0.095, p = 0.0018.
chrF++ vs passage length
chrF++: significant negative; r = -0.331, R^2 = 0.109, p = 0.0008.
METEOR vs passage length
METEOR: significant negative; r = -0.406, R^2 = 0.165, p = 2.78e-05.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.363, R^2 = 0.132, p = 0.0002.
BERTScore vs passage length
BERTScore: significant negative; r = -0.575, R^2 = 0.331, p = 3.90e-10.
COMET vs passage length
COMET: significant negative; r = -0.414, R^2 = 0.171, p = 1.88e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.732, R^2 = 0.536, p = 4.85e-18.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.284, R^2 = 0.081, p = 0.0042.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.283, R^2 = 0.080, p = 0.0044.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.284, R^2 = 0.080, p = 0.0042.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.288, R^2 = 0.083, p = 0.0036.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.308 0.095 0.0018 -0.00200 ok
chrF++ 100 significant negative -0.331 0.109 0.0008 -0.00131 ok
METEOR 100 significant negative -0.406 0.165 2.78e-05 -0.00190 ok
ROUGE-L 100 significant negative -0.363 0.132 0.0002 -0.00152 ok
BERTScore 100 significant negative -0.575 0.331 3.90e-10 -0.00045 ok
COMET 100 significant negative -0.414 0.171 1.88e-05 -0.00080 ok
BLEURT 100 significant negative -0.732 0.536 4.85e-18 -0.00258 ok
Trigram precision 100 significant negative -0.284 0.081 0.0042 -0.00211 ok
Trigram recall 100 significant negative -0.283 0.080 0.0044 -0.00208 ok
Trigram F1 100 significant negative -0.284 0.080 0.0042 -0.00209 ok
Trigram Jaccard 100 significant negative -0.288 0.083 0.0036 -0.00203 ok

Translation Similarity Metrics

Mean BLEU-445.4%
Mean chrF++68.7%
Mean METEOR73.6%
Mean ROUGE-L74.7%
Mean BERTScore95.7%
Mean COMET78.8%
Mean BLEURT73.3%
Legacy smoothed corpus BLEU45.0%
Unigram F175.9%
Bigram F151.5%
Trigram precision37.9%
Trigram recall37.5%
Trigram F137.7%
Trigram Jaccard23.2%
4-gram F128.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 246 -15.731 34.6% 60.6% 56.5% 65.5% 91.1% 78.7% 48.9% reviewed/approved 12081 gpt-5.1-2025-11-13
2056 Καβειρία 112 95 -14.180 23.3% 64.9% 51.2% 59.9% 95.2% 77.2% 59.7% reviewed/approved 11940 gpt-5.1-2025-11-13
2604 Καρχηδών 117 124 9.992 37.1% 65.5% 70.0% 71.6% 94.2% 78.2% 64.2% reviewed/approved 12096 gpt-5.1-2025-11-13
2603 Κάρυστος 175 179 8.992 34.1% 61.8% 61.7% 65.0% 93.6% 75.8% 56.4% reviewed/approved 12093 gpt-5.1-2025-11-13
2342 Κάμιρος 54 62 8.820 50.1% 74.7% 72.2% 72.4% 96.1% 79.5% 67.9% reviewed/approved 12057 gpt-5.1-2025-11-13
2328 Καλλίπολις 43 51 8.440 23.0% 52.6% 60.9% 59.6% 93.3% 74.9% 67.8% reviewed/approved 12033 gpt-5.1-2025-11-13
2609 Κάσπειρος 135 139 7.613 45.9% 66.2% 58.0% 65.7% 90.0% 72.1% 61.0% reviewed/approved 12105 gpt-5.1-2025-11-13
2468 Καπετώλιον 138 141 6.716 19.9% 48.1% 44.9% 50.2% 88.2% 72.3% 53.6% reviewed/approved 12072 gpt-5.1-2025-11-13
2597 Καρπασία 112 103 -6.180 47.6% 74.3% 78.6% 79.1% 95.5% 78.0% 67.0% reviewed/approved 12084 gpt-5.1-2025-11-13
7255 Κυτέριον 22 28 5.716 40.2% 67.2% 55.9% 64.0% 95.0% 77.6% 75.1% reviewed/approved 12195 gpt-5.1-2025-11-13
7266 Κώμη 79 83 5.682 25.0% 52.2% 50.1% 53.1% 92.6% 70.9% 55.9% reviewed/approved 12228 gpt-5.1-2025-11-13
2620 Κάσταξ 20 15 -5.353 27.1% 51.3% 60.4% 68.6% 95.3% 76.9% 73.6% reviewed/approved 12108 gpt-5.1-2025-11-13
2330 Καλύβη 24 19 -5.215 27.3% 55.3% 56.2% 69.8% 94.8% 76.6% 74.7% reviewed/approved 12039 gpt-5.1-2025-11-13
2054 Καβαλίς 65 69 5.199 48.7% 70.3% 75.5% 79.1% 95.7% 75.4% 73.7% reviewed/approved 11934 gpt-5.1-2025-11-13
7260 Κυχρεῖος πάγος 61 65 5.061 38.7% 64.0% 70.8% 60.3% 95.6% 76.5% 59.2% reviewed/approved 12210 gpt-5.1-2025-11-13
7259 Κύφος 54 58 4.820 34.0% 62.2% 64.8% 64.3% 94.0% 73.2% 64.8% reviewed/approved 12207 gpt-5.1-2025-11-13
2116 Κάληρος 40 35 -4.663 15.6% 49.4% 58.8% 56.0% 94.0% 78.1% 71.7% reviewed/approved 12012 gpt-5.1-2025-11-13
3496 Κοτιάειον 71 65 -4.594 28.7% 48.6% 46.0% 57.4% 92.4% 71.6% 59.7% reviewed/approved 12147 gpt-5.1-2025-11-13
2081 Καλάθη 49 44 -4.353 68.9% 79.3% 83.9% 88.2% 97.5% 78.1% 78.3% reviewed/approved 11985 gpt-5.1-2025-11-13
7254 Κύτα 84 78 -4.146 36.2% 58.4% 56.3% 66.7% 91.1% 74.6% 63.9% reviewed/approved 12192 gpt-5.1-2025-11-13
2605 Κάσιον 56 59 3.889 50.9% 73.9% 75.6% 74.8% 96.3% 77.5% 71.0% reviewed/approved 12099 gpt-5.1-2025-11-13
3288 Κορώνεια 110 111 3.751 33.5% 63.1% 65.5% 69.4% 94.8% 75.0% 60.4% reviewed/approved 12144 gpt-5.1-2025-11-13
7250 Κυρταία 40 36 -3.663 45.0% 65.1% 62.3% 68.4% 95.5% 69.6% 67.4% reviewed/approved 12180 gpt-5.1-2025-11-13
2346 Κάναι 77 79 3.613 60.5% 72.1% 83.5% 85.9% 95.9% 77.7% 65.4% reviewed/approved 12063 gpt-5.1-2025-11-13
2470 Καππαδοκία 68 70 3.302 30.8% 63.0% 73.7% 73.9% 95.9% 76.3% 67.2% reviewed/approved 12075 gpt-5.1-2025-11-13
2626 Κατάνη 95 96 3.234 37.8% 63.3% 63.7% 67.0% 93.2% 71.2% 57.0% reviewed/approved 12129 gpt-5.1-2025-11-13
2802 Κάδρεμα 24 21 -3.215 47.0% 65.0% 68.0% 71.1% 96.9% 80.6% 77.9% reviewed/approved 11958 gpt-5.1-2025-11-13
2119 Κάλλατις 65 67 3.199 44.6% 69.1% 75.5% 72.7% 95.2% 71.2% 70.1% reviewed/approved 12021 gpt-5.1-2025-11-13
7257 Κυτώνιον 25 22 -3.180 75.1% 80.0% 84.7% 89.4% 97.6% 81.1% 77.6% reviewed/approved 12201 gpt-5.1-2025-11-13
7244 Κυρήνη 54 50 -3.180 40.0% 65.8% 64.3% 67.3% 96.2% 78.1% 74.3% reviewed/approved 12162 gpt-5.1-2025-11-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.090.

Positive termCoefficientDocsMean presentMean absent
εν 1.0554 37 1.329 -0.780
τε 1.0103 5 4.696 -0.247
εν τη 0.7729 7 3.475 -0.262
δε και 0.6823 8 1.544 -0.134
οι 0.6685 11 2.729 -0.337
τη 0.6455 15 1.645 -0.290
παιδος 0.6118 4 3.768 -0.157
θηλυκως 0.5812 5 4.516 -0.238
και θηλυκως 0.5812 5 4.516 -0.238
πολιχνιον 0.5652 3 5.084 -0.157
εστι δε και 0.5484 3 5.061 -0.157
στραβων 0.4934 15 1.620 -0.286
πολιτης 0.4840 18 1.506 -0.330
ος 0.4682 6 2.722 -0.174
δευτερα 0.4519 3 4.015 -0.124
αι 0.4381 3 5.119 -0.158
ειναι 0.4344 8 2.225 -0.193
ωστε 0.4313 3 5.843 -0.181
δια του 0.4139 10 1.106 -0.123
δυναται 0.4104 3 3.394 -0.105
Negative termCoefficientDocsMean presentMean absent
το -0.9814 70 -0.231 0.539
εστι και -0.7191 20 -0.879 0.220
εθνος -0.6873 8 -4.603 0.400
το θηλυκον -0.6619 6 -4.927 0.315
οτι -0.6316 5 -2.670 0.141
εστι -0.5977 24 -0.798 0.252
το κτητικον -0.5870 7 -4.904 0.369
κτητικον -0.5870 7 -4.904 0.369
ως -0.5735 52 -0.164 0.178
το εθνικον -0.5476 56 -0.457 0.581
εθνικον -0.5476 56 -0.457 0.581
παρα το -0.5179 3 -4.146 0.128
θρακης -0.5038 5 -1.918 0.101
πολις θρακης -0.4854 4 -2.155 0.090
παρα -0.4803 7 -3.545 0.267
θηλυκον και -0.4261 3 -5.387 0.167
πολις της -0.4256 3 -5.054 0.156
ης -0.4214 3 -3.675 0.114
δε οτι -0.4205 3 -3.468 0.107
ιβηριας -0.4143 3 0.486 -0.015

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.028.

Positive termCoefficientDocsMean presentMean absent
δε 1.7747 46 3.737 1.748
την 1.5868 13 6.541 2.083
εν 1.2071 37 3.523 2.158
γαρ 1.1591 9 5.869 2.346
τη 1.1352 15 4.947 2.260
και 1.1271 66 3.165 1.689
απο 1.0911 27 4.748 1.892
κατα 1.0548 10 5.750 2.320
αφ 1.0261 6 6.348 2.428
εκαλειτο δε 0.9219 3 11.572 2.387
της 0.8421 35 3.852 2.023
εκαλειτο 0.8361 7 6.026 2.410
ιβηριας 0.8163 3 6.175 2.554
αφ ου 0.8098 5 7.288 2.419
περι 0.7908 11 4.888 2.388
μεν 0.7839 4 9.374 2.383
τε 0.7712 5 4.726 2.554
φησι 0.7569 12 4.960 2.350
δια 0.7555 16 4.507 2.312
ος 0.7160 6 5.635 2.473
Negative termCoefficientDocsMean presentMean absent
και το -0.7369 9 2.934 2.636
μια -0.6891 3 0.410 2.733
τεταρτω -0.6728 3 0.745 2.722
το εθνικον -0.6534 56 2.765 2.532
εθνικον -0.6534 56 2.765 2.532
καππαδοκιας -0.5617 3 1.543 2.697
νησος -0.5147 9 2.108 2.718
πολυβιος -0.4923 5 1.212 2.739
και πολις -0.4493 5 1.884 2.704
πλησιον -0.4473 8 2.114 2.711
προς τη -0.4432 4 1.728 2.702
στραβων -0.4271 15 2.885 2.624
ηροδοτος -0.4109 9 3.522 2.578
φασι -0.4069 8 3.123 2.623
ειναι -0.3975 8 2.804 2.651
δευτερω -0.3839 3 1.486 2.699
εκαταιος ασια -0.3757 5 1.474 2.725
ασια -0.3757 5 1.474 2.725
τους -0.3637 4 1.892 2.695
ιταλιας -0.3385 3 1.095 2.711

AI English terms predicting large absolute residuals

Features: 327; ridge alpha: 2.894; cross-validated R^2: -0.028.

Positive termCoefficientDocsMean presentMean absent
the 1.9508 92 2.797 1.122
and 1.1883 59 3.286 1.766
on 1.1671 16 4.818 2.252
there 1.0378 31 4.069 2.031
after 1.0224 14 5.504 2.200
whom 0.9406 7 6.689 2.360
after whom 0.9261 5 7.288 2.419
says 0.8938 18 5.138 2.119
of 0.8576 86 2.881 1.324
diphthong 0.8429 3 9.543 2.450
from 0.8019 41 3.846 1.841
to 0.7522 18 4.346 2.293
as 0.7240 58 3.332 1.739
also 0.7225 41 3.544 2.051
and there is 0.7184 3 10.118 2.432
was called 0.7085 4 5.108 2.561
by 0.7037 13 4.554 2.380
on the 0.6919 9 5.070 2.425
and there 0.6908 6 7.397 2.361
was 0.6784 20 4.212 2.276
Negative termCoefficientDocsMean presentMean absent
one -0.6638 9 1.725 2.756
the ethnonym is -0.6288 55 2.460 2.911
the ethnonym -0.6147 57 2.541 2.824
ethnonym -0.4861 59 2.708 2.599
ethnonym is -0.4693 56 2.669 2.655
one of the -0.4587 4 1.116 2.727
one of -0.4587 4 1.116 2.727
near -0.4567 16 2.536 2.687
a city and -0.4493 5 1.643 2.717
an -0.4208 21 2.321 2.754
italy -0.4167 4 0.905 2.736
kappadokia -0.4013 4 1.983 2.691
headland -0.3856 3 1.727 2.692
syria -0.3722 3 0.577 2.727
a city -0.3577 67 2.996 1.987
and in the -0.3567 10 3.214 2.602
herodotus -0.3565 8 1.996 2.721
village -0.3550 3 0.791 2.721
a village -0.3550 3 0.791 2.721
is also called -0.3545 5 1.595 2.719

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: 0.032.

Positive termCoefficientDocsMean presentMean absent
the 2.3532 90 2.840 1.072
for 1.5113 11 5.951 2.256
on 1.4058 16 4.654 2.284
with 1.1517 16 4.293 2.352
from the 1.1317 13 5.426 2.250
according 1.1308 5 7.384 2.414
according to 1.1308 5 7.384 2.414
also 1.0970 43 3.581 1.970
from the form 1.0959 5 5.664 2.505
are also 1.0232 4 6.922 2.485
is with 1.0067 4 8.215 2.431
is from 0.9957 12 4.412 2.424
and 0.9893 52 3.523 1.732
for the 0.9703 7 7.465 2.301
form 0.9609 11 5.159 2.354
to 0.9469 31 4.053 2.038
diphthong 0.9257 3 9.543 2.450
usage 0.8915 4 6.422 2.506
the form 0.8632 9 5.736 2.359
as 0.8580 60 3.140 1.947
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.0140 52 2.394 2.954
ethnonym is -1.0140 52 2.394 2.954
kappadokia -0.8495 4 1.983 2.691
and the -0.7361 15 3.105 2.585
ethnonym -0.6492 57 2.754 2.542
one of the -0.6319 4 1.116 2.727
an -0.6295 22 2.177 2.800
is a city -0.6239 3 0.871 2.718
ethnonyms -0.6156 3 1.014 2.714
the ethnonym -0.5939 56 2.791 2.500
syria -0.5414 3 0.577 2.727
in syria -0.5414 3 0.577 2.727
near -0.5265 16 2.613 2.672
as in -0.5190 29 2.588 2.693
territory -0.5041 5 1.547 2.722
one of -0.4920 5 1.763 2.710
in his asia -0.4820 5 1.474 2.725
his asia -0.4820 5 1.474 2.725
it is -0.4793 24 3.070 2.534
one -0.4508 12 2.565 2.676

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.