gpt-5.6-sol v2 Evaluation

Generated: 2026-09-20 11:22:45 UTC

Pairs100
Distinct lemmas100
Slope0.925
R^20.988
BLEU-452.4%
chrF++72.9%
METEOR77.5%
ROUGE-L77.6%
BERTScore96.5%
COMET80.6%
BLEURT77.1%

Profile version id: 2176

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.6 Sol using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.925
Intercept2.093
R^20.988
P-value1.07e-96
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.930
Mean absolute length residual2.704
Mean absolute percent error6.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.407, R^2 = 0.165, p = 2.70e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.440, R^2 = 0.194, p = 4.59e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.461, R^2 = 0.213, p = 1.35e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.409, R^2 = 0.167, p = 2.42e-05.
BERTScore vs passage length
BERTScore: significant negative; r = -0.630, R^2 = 0.397, p = 2.25e-12.
COMET vs passage length
COMET: significant negative; r = -0.514, R^2 = 0.264, p = 4.63e-08.
BLEURT vs passage length
BLEURT: significant negative; r = -0.734, R^2 = 0.538, p = 3.94e-18.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.312, R^2 = 0.097, p = 0.0016.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.331, R^2 = 0.109, p = 0.0008.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.322, R^2 = 0.104, p = 0.0011.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.325, R^2 = 0.106, p = 0.0010.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.407 0.165 2.70e-05 -0.00271 ok
chrF++ 100 significant negative -0.440 0.194 4.59e-06 -0.00184 ok
METEOR 100 significant negative -0.461 0.213 1.35e-06 -0.00199 ok
ROUGE-L 100 significant negative -0.409 0.167 2.42e-05 -0.00167 ok
BERTScore 100 significant negative -0.630 0.397 2.25e-12 -0.00048 ok
COMET 100 significant negative -0.514 0.264 4.63e-08 -0.00098 ok
BLEURT 100 significant negative -0.734 0.538 3.94e-18 -0.00268 ok
Trigram precision 100 significant negative -0.312 0.097 0.0016 -0.00230 ok
Trigram recall 100 significant negative -0.331 0.109 0.0008 -0.00239 ok
Trigram F1 100 significant negative -0.322 0.104 0.0011 -0.00234 ok
Trigram Jaccard 100 significant negative -0.325 0.106 0.0010 -0.00237 ok

Translation Similarity Metrics

Mean BLEU-452.4%
Mean chrF++72.9%
Mean METEOR77.5%
Mean ROUGE-L77.6%
Mean BERTScore96.5%
Mean COMET80.6%
Mean BLEURT77.1%
Legacy smoothed corpus BLEU48.6%
Unigram F178.3%
Bigram F155.3%
Trigram precision42.7%
Trigram recall41.4%
Trigram F142.1%
Trigram Jaccard26.6%
4-gram F132.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 130 19.686 28.3% 60.0% 62.1% 63.5% 92.9% 77.9% 63.4% reviewed/approved 10621 gpt-5.6-sol
2484 Καρία 270 240 -11.834 37.2% 62.4% 56.5% 64.7% 91.3% 79.7% 51.6% reviewed/approved 10606 gpt-5.6-sol
2342 Κάμιρος 54 63 10.959 49.1% 74.8% 78.4% 73.5% 95.4% 78.7% 67.7% reviewed/approved 10582 gpt-5.6-sol
3288 Κορώνεια 110 112 8.161 38.5% 64.5% 71.6% 70.9% 95.1% 76.4% 65.6% reviewed/approved 10669 gpt-5.6-sol
2328 Καλλίπολις 43 50 8.134 46.5% 65.9% 73.7% 64.5% 94.4% 76.6% 66.1% reviewed/approved 10558 gpt-5.6-sol
7259 Κύφος 54 60 7.959 51.5% 76.0% 90.1% 84.2% 96.4% 76.8% 78.2% reviewed/approved 10732 gpt-5.6-sol
2116 Κάληρος 40 32 -7.091 22.3% 55.0% 59.3% 58.3% 93.8% 75.3% 72.7% reviewed/approved 10537 gpt-5.6-sol
2346 Κάναι 77 80 6.685 58.5% 74.0% 82.9% 82.8% 94.8% 76.2% 67.1% reviewed/approved 10588 gpt-5.6-sol
7260 Κυχρεῖος πάγος 61 65 6.484 44.6% 68.9% 78.7% 65.1% 95.3% 76.9% 63.4% reviewed/approved 10735 gpt-5.6-sol
7251 Κύρτος 66 69 5.859 45.9% 62.7% 75.5% 72.6% 92.8% 70.9% 57.5% reviewed/approved 10708 gpt-5.6-sol
2056 Καβειρία 112 100 -5.689 34.7% 69.2% 62.5% 67.9% 95.9% 79.8% 68.2% reviewed/approved 10467 gpt-5.6-sol
2603 Κάρυστος 175 169 5.038 46.4% 67.3% 67.8% 73.3% 95.3% 78.6% 61.9% reviewed/approved 10618 gpt-5.6-sol
2470 Καππαδοκία 68 70 5.009 18.8% 53.2% 78.6% 75.4% 94.1% 72.8% 63.7% reviewed/approved 10600 gpt-5.6-sol
2626 Κατάνη 95 85 -4.965 36.5% 60.9% 68.8% 70.0% 94.1% 74.2% 64.3% reviewed/approved 10654 gpt-5.6-sol
3496 Κοτιάειον 71 63 -4.765 32.9% 51.7% 56.2% 56.7% 92.5% 70.9% 66.4% reviewed/approved 10672 gpt-5.6-sol
2597 Καρπασία 112 101 -4.689 33.3% 61.1% 59.9% 59.2% 94.7% 74.9% 59.9% reviewed/approved 10609 gpt-5.6-sol
2455 Κάναστρον 60 53 -4.591 18.5% 51.4% 61.3% 65.5% 93.2% 67.9% 61.8% reviewed/approved 10591 gpt-5.6-sol
7247 Κύρνος 50 44 -4.341 34.5% 58.4% 69.7% 66.0% 96.8% 78.8% 79.1% reviewed/approved 10696 gpt-5.6-sol
7253 Κυρτώνιος 18 23 4.258 39.8% 69.1% 72.7% 63.4% 96.6% 77.5% 80.2% reviewed/approved 10714 gpt-5.6-sol
7257 Κυτώνιον 25 21 -4.217 70.8% 75.3% 71.8% 78.3% 97.6% 81.0% 78.8% reviewed/approved 10726 gpt-5.6-sol
7254 Κύτα 84 84 4.210 48.1% 71.5% 69.2% 75.0% 95.2% 78.0% 68.9% reviewed/approved 10717 gpt-5.6-sol
7258 Κύτωρος 30 34 4.158 49.3% 80.1% 86.3% 81.2% 97.4% 80.0% 82.3% reviewed/approved 10729 gpt-5.6-sol
3530 Κριώα 26 22 -4.142 41.7% 65.7% 76.5% 79.2% 96.7% 77.6% 74.7% reviewed/approved 10678 gpt-5.6-sol
2609 Κάσπειρος 135 123 -3.963 43.7% 65.5% 60.6% 66.7% 89.9% 72.2% 64.8% reviewed/approved 10630 gpt-5.6-sol
2055 Καβασσός 109 99 -3.914 34.0% 59.1% 64.5% 66.3% 94.8% 76.4% 52.2% reviewed/approved 10464 gpt-5.6-sol
7246 Κύρις 20 17 -3.592 67.0% 80.8% 85.2% 86.5% 97.4% 84.2% 77.9% reviewed/approved 10693 gpt-5.6-sol
7255 Κυτέριον 22 26 3.558 55.7% 74.9% 73.5% 75.0% 96.5% 78.7% 79.3% reviewed/approved 10720 gpt-5.6-sol
2078 Καινύς 22 19 -3.442 55.5% 67.9% 67.8% 73.2% 95.6% 82.1% 77.8% reviewed/approved 10498 gpt-5.6-sol
2326 Καλλίαρος 47 49 3.434 57.9% 80.2% 92.8% 89.6% 97.8% 77.8% 80.8% reviewed/approved 10552 gpt-5.6-sol
7248 Κύρου πόλις 32 35 3.308 34.4% 69.6% 74.2% 68.7% 96.7% 81.4% 81.8% reviewed/approved 10699 gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.243.

Positive termCoefficientDocsMean presentMean absent
δε και 2.2147 8 3.336 -0.290
απο 1.8991 27 2.086 -0.771
εστι δε και 1.7738 3 9.327 -0.288
παιδος 1.7228 4 6.303 -0.263
ος 1.6874 6 4.981 -0.318
πολις 1.6490 68 0.395 -0.840
πολιτης 1.4303 18 3.011 -0.661
εκαλειτο δε 1.3314 3 4.297 -0.133
εστι δε 1.2291 5 4.200 -0.221
πολιχνιον 1.0372 3 5.868 -0.181
δευτερα 1.0320 3 5.384 -0.167
οι 1.0279 11 2.419 -0.299
και 1.0246 66 0.114 -0.222
λεγεται και 0.9576 8 2.139 -0.186
ου 0.9466 14 1.576 -0.257
δυο 0.9425 4 4.391 -0.183
πολις και 0.9080 6 5.097 -0.325
λεγεται 0.8821 9 1.916 -0.189
εις 0.8102 7 1.393 -0.105
της χωρας 0.7994 3 2.575 -0.080
Negative termCoefficientDocsMean presentMean absent
το -1.6992 70 -0.261 0.610
οτι -1.4533 5 -2.495 0.131
ουτως -1.4150 5 -5.744 0.302
τα -1.3070 12 -1.927 0.263
τινες -1.0406 6 -2.700 0.172
οικητωρ -1.0174 6 -3.178 0.203
γαρ -0.9478 9 -1.353 0.134
μετα -0.9138 3 -6.921 0.214
το κτητικον -0.8984 7 -3.500 0.263
κτητικον -0.8984 7 -3.500 0.263
εστι και -0.8881 20 -0.496 0.124
καππαδοκιας -0.8448 3 -2.458 0.076
δια -0.8407 16 0.516 -0.098
μεταξυ -0.8302 5 -0.721 0.038
το εθνικον -0.7961 56 -0.526 0.669
εθνικον -0.7961 56 -0.526 0.669
νησος -0.7538 9 -1.441 0.143
εκαταιος -0.7445 14 -0.745 0.121
πλησιον -0.7427 8 -0.495 0.043
τοπικα -0.6980 3 -2.857 0.088

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: 0.052.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 2.3294 7 7.227 2.364
την 2.0463 13 5.735 2.252
εκαλειτο δε 1.8741 3 12.186 2.411
δε και 1.8709 8 6.836 2.345
απο 1.8213 27 5.255 1.761
και 1.8113 66 3.324 1.502
δε 1.7029 46 3.546 1.987
ος 1.6302 6 6.816 2.442
εστι δε και 1.3906 3 9.327 2.500
παιδος 1.1883 4 6.303 2.555
εστι δε 1.1515 5 6.992 2.479
κατα 1.0993 10 6.442 2.289
εκαλειτο δε και 1.0804 3 12.186 2.411
της 1.0620 35 3.935 2.042
μεν 1.0540 4 7.835 2.491
μητροπολις 0.9543 3 8.481 2.526
περι 0.9216 11 4.421 2.492
τω 0.8441 16 3.578 2.538
και αλλη 0.8198 3 9.620 2.491
πολιχνιον 0.7383 3 5.868 2.607
Negative termCoefficientDocsMean presentMean absent
και το -1.1927 9 2.001 2.774
κωμη -1.0839 3 0.753 2.765
το εθνικον -1.0783 56 2.345 3.161
εθνικον -1.0783 56 2.345 3.161
το -1.0164 70 2.452 3.294
στραβων -0.8985 15 2.642 2.715
δευτερω -0.8983 3 0.864 2.761
προς τη -0.8565 4 1.365 2.760
ως -0.8453 52 2.818 2.581
αυτην -0.6651 9 2.058 2.768
ειρηται -0.6532 3 2.009 2.726
μια -0.6210 3 1.261 2.749
προς -0.6136 15 2.760 2.695
φησι το -0.5502 3 0.505 2.772
πλησιον -0.5440 8 2.199 2.748
αυτην φησιν -0.5259 3 1.802 2.732
νησος -0.4996 9 2.084 2.766
τεταρτω -0.4938 3 1.325 2.747
πορρω -0.4904 3 0.464 2.774
ου πορρω -0.4904 3 0.464 2.774

AI English terms predicting large absolute residuals

Features: 307; ridge alpha: 2.031; cross-validated R^2: 0.067.

Positive termCoefficientDocsMean presentMean absent
was 2.2016 20 5.141 2.095
and 1.9344 54 3.659 1.584
was called 1.5546 6 9.008 2.302
called 1.4486 31 4.222 2.023
by 1.4455 16 4.619 2.340
while 1.3798 4 10.767 2.368
with 1.3589 18 4.960 2.209
of 1.3257 82 2.844 2.069
most 1.2674 4 11.066 2.356
the 1.1511 93 2.798 1.464
he 1.1015 3 12.668 2.396
says 1.0801 18 5.098 2.179
after 1.0247 18 5.680 2.051
son 0.9728 17 5.789 2.073
son of 0.9728 17 5.789 2.073
also called 0.9194 13 5.326 2.313
at 0.8847 7 4.907 2.539
there 0.8719 35 4.098 1.954
metropolis 0.8318 3 8.481 2.526
metropolis of 0.8318 3 8.481 2.526
Negative termCoefficientDocsMean presentMean absent
the ethnonym -0.9472 54 2.313 3.164
the ethnonym is -0.9444 50 2.301 3.108
ethnonym is -0.8980 51 2.367 3.056
ethnonym -0.8856 56 2.401 3.090
name -0.8458 9 1.660 2.808
as in -0.6870 35 2.718 2.697
near -0.6758 14 2.278 2.774
according to -0.6671 12 2.766 2.696
according -0.6671 12 2.766 2.696
book -0.6593 39 2.285 2.972
island -0.6444 14 2.251 2.778
syria -0.5027 3 1.347 2.746
strabo -0.5021 15 2.642 2.715
book of his -0.4954 15 1.780 2.868
in his asia -0.4720 5 1.047 2.792
his asia -0.4720 5 1.047 2.792
one -0.4587 7 3.445 2.649
those -0.4515 4 1.794 2.742
place -0.4507 7 3.201 2.667
sea -0.4354 9 2.042 2.770

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: 0.072.

Positive termCoefficientDocsMean presentMean absent
and 2.2090 52 3.665 1.663
was 1.5953 20 4.515 2.252
is with 1.2633 4 8.750 2.453
too 1.2167 3 10.926 2.450
son 1.1760 17 5.789 2.073
son of 1.1760 17 5.789 2.073
used to be 1.1605 6 8.291 2.348
also 1.1501 43 3.666 1.979
who 1.1499 13 4.857 2.383
of 1.1131 66 3.149 1.842
this 1.0626 15 5.808 2.157
was called 1.0445 3 10.879 2.452
them 1.0427 5 7.954 2.428
to be called 1.0124 5 8.942 2.376
the 0.9666 90 2.817 1.690
to be 0.9476 9 6.593 2.320
called 0.9414 21 4.144 2.322
their 0.9368 3 12.186 2.411
on 0.9253 16 4.065 2.445
who was 0.9024 3 10.028 2.478
Negative termCoefficientDocsMean presentMean absent
as in -1.3567 29 1.955 3.010
and the -0.9984 15 2.228 2.789
the ethnonym is -0.9207 52 2.328 3.112
ethnonym is -0.9207 52 2.328 3.112
an -0.9102 22 2.116 2.870
the ethnonym -0.8636 56 2.425 3.060
ethnonym -0.8556 57 2.451 3.040
it is -0.7803 24 2.496 2.770
is -0.6442 87 2.801 2.060
or after -0.6338 3 1.853 2.731
one of -0.6046 5 1.867 2.749
near -0.6028 16 2.412 2.760
book -0.6013 39 2.285 2.972
strabo -0.5103 15 2.642 2.715
in his asia -0.5005 5 1.047 2.792
his asia -0.5005 5 1.047 2.792
with a -0.4802 4 1.776 2.743
in book of -0.4783 14 1.815 2.849
says that -0.4778 12 2.621 2.716
one of the -0.4761 4 2.187 2.726

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.