gpt-5.6-sol v2 Evaluation

Generated: 2026-07-25 12:54:12 UTC

Pairs100
Distinct lemmas100
Slope0.925
R^20.988
BLEU-452.3%
chrF++72.9%
METEOR77.4%
ROUGE-L77.4%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2176

First translation using this version: 2026-07-14

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-10 11:03:49.528703+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.6 Sol using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.925
Intercept2.075
R^20.988
P-value1.19e-96
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.930
Mean absolute length residual2.711
Mean absolute percent error6.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.403, R^2 = 0.163, p = 3.16e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.436, R^2 = 0.191, p = 5.64e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.450, R^2 = 0.203, p = 2.62e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.402, R^2 = 0.161, p = 3.44e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.307, R^2 = 0.094, p = 0.0019.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.324, R^2 = 0.105, p = 0.0010.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.316, R^2 = 0.100, p = 0.0013.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.319, R^2 = 0.102, p = 0.0012.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.403 0.163 3.16e-05 -0.00268 ok
chrF++ 100 significant negative -0.436 0.191 5.64e-06 -0.00182 ok
METEOR 100 significant negative -0.450 0.203 2.62e-06 -0.00193 ok
ROUGE-L 100 significant negative -0.402 0.161 3.44e-05 -0.00164 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.307 0.094 0.0019 -0.00226 ok
Trigram recall 100 significant negative -0.324 0.105 0.0010 -0.00233 ok
Trigram F1 100 significant negative -0.316 0.100 0.0013 -0.00229 ok
Trigram Jaccard 100 significant negative -0.319 0.102 0.0012 -0.00232 ok

Translation Similarity Metrics

Mean BLEU-452.3%
Mean chrF++72.9%
Mean METEOR77.4%
Mean ROUGE-L77.4%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU48.6%
Unigram F178.2%
Bigram F155.2%
Trigram precision42.7%
Trigram recall41.4%
Trigram F142.0%
Trigram Jaccard26.6%
4-gram F132.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2604 Καρχηδών 117 130 19.682 28.3% 60.0% 62.1% 63.5% N/A N/A N/A reviewed/approved 10621 gpt-5.6-sol
2484 Καρία 270 240 -11.868 37.2% 62.4% 56.5% 64.7% N/A N/A N/A reviewed/approved 10606 gpt-5.6-sol
2342 Κάμιρος 54 63 10.966 49.1% 74.8% 78.4% 73.5% N/A N/A N/A reviewed/approved 10582 gpt-5.6-sol
3288 Κορώνεια 110 112 8.158 38.5% 64.5% 71.6% 70.9% N/A N/A N/A reviewed/approved 10669 gpt-5.6-sol
2328 Καλλίπολις 43 50 8.143 46.5% 65.9% 73.7% 64.5% N/A N/A N/A reviewed/approved 10558 gpt-5.6-sol
7259 Κύφος 54 60 7.966 51.5% 76.0% 90.1% 84.2% N/A N/A N/A reviewed/approved 10732 gpt-5.6-sol
2116 Κάληρος 40 32 -7.081 22.3% 55.0% 59.3% 58.3% N/A N/A N/A reviewed/approved 10537 gpt-5.6-sol
2346 Κάναι 77 80 6.688 58.5% 74.0% 82.9% 82.8% N/A N/A N/A reviewed/approved 10588 gpt-5.6-sol
7260 Κυχρεῖος πάγος 61 65 6.490 44.6% 68.9% 78.7% 65.1% N/A N/A N/A reviewed/approved 10735 gpt-5.6-sol
7251 Κύρτος 66 69 5.865 45.9% 62.7% 75.5% 72.6% N/A N/A N/A reviewed/approved 10708 gpt-5.6-sol
2056 Καβειρία 112 100 -5.693 34.7% 69.2% 62.5% 67.9% N/A N/A N/A reviewed/approved 10467 gpt-5.6-sol
2603 Κάρυστος 175 169 5.022 46.4% 67.3% 67.8% 73.3% N/A N/A N/A reviewed/approved 10618 gpt-5.6-sol
2470 Καππαδοκία 68 70 5.014 18.8% 53.2% 78.6% 75.4% N/A N/A N/A reviewed/approved 10600 gpt-5.6-sol
2626 Κατάνη 95 85 -4.965 36.5% 60.9% 68.8% 70.0% N/A N/A N/A reviewed/approved 10654 gpt-5.6-sol
3496 Κοτιάειον 71 63 -4.761 32.9% 51.7% 56.2% 56.7% N/A N/A N/A reviewed/approved 10672 gpt-5.6-sol
2597 Καρπασία 112 101 -4.693 33.3% 61.1% 59.9% 59.2% N/A N/A N/A reviewed/approved 10609 gpt-5.6-sol
2455 Κάναστρον 60 53 -4.584 18.5% 51.4% 61.3% 65.5% N/A N/A N/A reviewed/approved 10591 gpt-5.6-sol
7247 Κύρνος 50 44 -4.333 34.5% 58.4% 69.7% 66.0% N/A N/A N/A reviewed/approved 10696 gpt-5.6-sol
7253 Κυρτώνιος 18 23 4.272 39.8% 69.1% 72.7% 63.4% N/A N/A N/A reviewed/approved 10714 gpt-5.6-sol
7254 Κύτα 84 84 4.212 48.1% 71.5% 71.1% 73.8% N/A N/A N/A reviewed/approved 10717 gpt-5.6-sol
7257 Κυτώνιον 25 21 -4.204 70.8% 75.3% 71.8% 78.3% N/A N/A N/A reviewed/approved 10726 gpt-5.6-sol
7258 Κύτωρος 30 34 4.170 49.3% 80.1% 86.3% 81.2% N/A N/A N/A reviewed/approved 10729 gpt-5.6-sol
3530 Κριώα 26 22 -4.129 41.7% 65.7% 76.5% 79.2% N/A N/A N/A reviewed/approved 10678 gpt-5.6-sol
2609 Κάσπειρος 135 123 -3.971 43.7% 65.5% 60.6% 66.7% N/A N/A N/A reviewed/approved 10630 gpt-5.6-sol
2055 Καβασσός 109 99 -3.917 34.0% 59.1% 64.5% 66.3% N/A N/A N/A reviewed/approved 10464 gpt-5.6-sol
7246 Κύρις 20 17 -3.578 67.0% 80.8% 85.2% 86.5% N/A N/A N/A reviewed/approved 10693 gpt-5.6-sol
7255 Κυτέριον 22 26 3.571 55.7% 74.9% 73.5% 75.0% N/A N/A N/A reviewed/approved 10720 gpt-5.6-sol
2326 Καλλίαρος 47 49 3.443 57.9% 80.2% 92.8% 89.6% N/A N/A N/A reviewed/approved 10552 gpt-5.6-sol
2078 Καινύς 22 19 -3.429 55.5% 67.9% 67.8% 73.2% N/A N/A N/A reviewed/approved 10498 gpt-5.6-sol
7248 Κύρου πόλις 32 35 3.320 34.4% 69.6% 74.2% 68.7% N/A N/A N/A reviewed/approved 10699 gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.239.

Positive termCoefficientDocsMean presentMean absent
δε και 2.2157 8 3.334 -0.290
απο 1.8998 27 2.088 -0.772
εστι δε και 1.7748 3 9.327 -0.288
παιδος 1.7238 4 6.312 -0.263
ος 1.6885 6 4.982 -0.318
πολις 1.6666 68 0.404 -0.858
πολιτης 1.4394 18 3.017 -0.662
εκαλειτο δε 1.3296 3 4.279 -0.132
εστι δε 1.2308 5 4.201 -0.221
και 1.0436 66 0.121 -0.236
πολιχνιον 1.0391 3 5.874 -0.182
δευτερα 1.0336 3 5.391 -0.167
οι 1.0329 11 2.423 -0.299
λεγεται και 0.9572 8 2.137 -0.186
ου 0.9516 14 1.579 -0.257
δυο 0.9455 4 4.395 -0.183
πολις και 0.9100 6 5.104 -0.326
λεγεται 0.8831 9 1.916 -0.189
εις 0.8084 7 1.387 -0.104
της χωρας 0.8015 3 2.587 -0.080
Negative termCoefficientDocsMean presentMean absent
το -1.6847 70 -0.253 0.591
οτι -1.4500 5 -2.492 0.131
ουτως -1.4156 5 -5.750 0.303
τα -1.3069 12 -1.929 0.263
τινες -1.0377 6 -2.690 0.172
οικητωρ -1.0130 6 -3.174 0.203
γαρ -0.9522 9 -1.358 0.134
μετα -0.9156 3 -6.935 0.214
το κτητικον -0.8979 7 -3.501 0.264
κτητικον -0.8979 7 -3.501 0.264
εστι και -0.8796 20 -0.493 0.123
καππαδοκιας -0.8427 3 -2.451 0.076
δια -0.8401 16 0.518 -0.099
μεταξυ -0.8232 5 -0.714 0.038
το εθνικον -0.7803 56 -0.517 0.658
εθνικον -0.7803 56 -0.517 0.658
εθνος -0.7733 8 -2.542 0.221
νησος -0.7489 9 -1.433 0.142
εκαταιος -0.7322 14 -0.736 0.120
πλησιον -0.7321 8 -0.484 0.042

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.031; cross-validated R^2: 0.047.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 2.3200 7 7.227 2.371
την 2.0458 13 5.737 2.259
εκαλειτο δε 1.8731 3 12.190 2.418
δε και 1.8707 8 6.840 2.352
απο 1.8229 27 5.258 1.769
και 1.7888 66 3.323 1.523
δε 1.6945 46 3.547 1.999
ος 1.6272 6 6.818 2.449
εστι δε και 1.3884 3 9.327 2.506
παιδος 1.1920 4 6.312 2.561
εστι δε 1.1474 5 6.992 2.486
κατα 1.0972 10 6.445 2.296
εκαλειτο δε και 1.0801 3 12.190 2.418
μεν 1.0553 4 7.848 2.497
της 1.0458 35 3.934 2.053
μητροπολις 0.9493 3 8.472 2.533
περι 0.9187 11 4.425 2.499
τω 0.8396 16 3.580 2.546
και αλλη 0.8173 3 9.619 2.497
πολιχνιον 0.7369 3 5.874 2.613
Negative termCoefficientDocsMean presentMean absent
και το -1.1925 9 2.002 2.781
το εθνικον -1.0960 56 2.344 3.178
εθνικον -1.0960 56 2.344 3.178
κωμη -1.0864 3 0.755 2.772
το -1.0353 70 2.451 3.319
στραβων -0.9123 15 2.638 2.724
δευτερω -0.9118 3 0.859 2.768
προς τη -0.8653 4 1.359 2.767
ως -0.8555 52 2.818 2.595
αυτην -0.6708 9 2.055 2.776
ειρηται -0.6514 3 2.016 2.733
μια -0.6303 3 1.263 2.756
προς -0.6275 15 2.756 2.703
πλησιον -0.5616 8 2.192 2.756
φησι το -0.5507 3 0.503 2.779
τεταρτω -0.5288 3 1.313 2.754
αυτην φησιν -0.5262 3 1.809 2.739
νησος -0.5074 9 2.080 2.773
πορρω -0.5033 3 0.458 2.781
ου πορρω -0.5033 3 0.458 2.781

AI English terms predicting large absolute residuals

Features: 307; ridge alpha: 1.425; cross-validated R^2: 0.069.

Positive termCoefficientDocsMean presentMean absent
was 2.6176 20 5.142 2.103
and 2.2147 54 3.658 1.599
was called 1.9054 6 9.010 2.309
by 1.8162 16 4.615 2.348
while 1.8126 4 10.777 2.375
with 1.7338 18 4.963 2.217
called 1.6814 31 4.225 2.031
most 1.6680 4 11.078 2.362
of 1.6145 82 2.842 2.113
he 1.4194 3 12.680 2.403
the 1.3577 93 2.806 1.449
says 1.3469 18 5.101 2.186
son 1.1530 17 5.792 2.080
son of 1.1530 17 5.792 2.080
after 1.1185 18 5.684 2.058
metropolis 1.0962 3 8.472 2.533
metropolis of 1.0962 3 8.472 2.533
also called 1.0940 13 5.330 2.320
at 1.0904 7 4.911 2.545
is named 1.0312 5 7.699 2.449
Negative termCoefficientDocsMean presentMean absent
name -1.1094 9 1.658 2.815
the ethnonym -1.1028 54 2.312 3.180
the ethnonym is -1.0869 50 2.300 3.122
ethnonym is -1.0312 51 2.366 3.070
ethnonym -1.0311 56 2.401 3.106
near -0.8973 14 2.274 2.782
island -0.8647 14 2.248 2.786
according to -0.8264 12 2.770 2.703
according -0.8264 12 2.770 2.703
as in -0.8137 35 2.717 2.708
book -0.7746 39 2.283 2.985
syria -0.6968 3 1.342 2.753
one -0.6822 7 3.443 2.656
for -0.6475 13 3.225 2.634
strabo -0.6378 15 2.638 2.724
from the -0.6230 10 3.233 2.653
book of his -0.5865 15 1.779 2.876
and in the -0.5749 8 2.956 2.690
in his asia -0.5748 5 1.034 2.799
his asia -0.5748 5 1.034 2.799

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: 0.069.

Positive termCoefficientDocsMean presentMean absent
and 2.1807 52 3.666 1.677
was 1.5608 20 4.515 2.260
is with 1.2589 4 8.756 2.459
too 1.2207 3 10.930 2.457
son 1.1703 17 5.792 2.080
son of 1.1703 17 5.792 2.080
used to be 1.1573 6 8.294 2.355
who 1.1380 13 4.857 2.390
also 1.1268 43 3.667 1.990
of 1.0948 66 3.148 1.864
this 1.0549 15 5.808 2.164
was called 1.0462 3 10.876 2.459
them 1.0413 5 7.952 2.435
to be called 1.0099 5 8.948 2.383
the 0.9673 90 2.825 1.682
to be 0.9468 9 6.595 2.327
their 0.9377 3 12.190 2.418
called 0.9305 21 4.147 2.329
on 0.9281 16 4.069 2.452
used 0.8985 8 7.035 2.335
Negative termCoefficientDocsMean presentMean absent
as in -1.3569 29 1.957 3.019
and the -0.9967 15 2.230 2.796
the ethnonym is -0.9379 52 2.326 3.128
ethnonym is -0.9379 52 2.326 3.128
the ethnonym -0.8787 56 2.423 3.077
ethnonym -0.8709 57 2.450 3.058
an -0.8476 22 2.155 2.868
it is -0.7751 24 2.497 2.779
is -0.6666 87 2.798 2.127
or after -0.6316 3 1.857 2.737
near -0.6253 16 2.407 2.769
book -0.6186 39 2.283 2.985
one of -0.6016 5 1.869 2.755
strabo -0.5332 15 2.638 2.724
in his asia -0.5167 5 1.034 2.799
his asia -0.5167 5 1.034 2.799
in book of -0.4901 14 1.814 2.857
says that -0.4825 12 2.623 2.723
with a -0.4810 4 1.777 2.750
one of the -0.4750 4 2.189 2.733

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.