paper_guidance_ablation_gpt56 v1 Evaluation

Generated: 2026-07-25 12:54:20 UTC

Pairs100
Distinct lemmas100
Slope0.919
R^20.987
BLEU-451.9%
chrF++72.7%
METEOR77.0%
ROUGE-L77.6%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2586

First translation using this version: 2026-07-18

Total usable AI runs for this version: 300 runs across 100 lemmas

Created: 2026-07-18 21:14:55.545407+10:00

Active: True

Notes: Arm A: Stephanos v2 static prompt with no matched entry-specific guidance. Model gpt-5.6-sol; Responses API; medium reasoning; top_p 1.0; three runs per Kappa-corpus entry.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.919
Intercept2.272
R^20.987
P-value1.42e-94
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.870
Mean absolute length residual2.790
Mean absolute percent error6.8%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.384, R^2 = 0.147, p = 8.00e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.420, R^2 = 0.177, p = 1.33e-05.
METEOR vs passage length
METEOR: significant negative; r = -0.436, R^2 = 0.190, p = 5.84e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.384, R^2 = 0.148, p = 7.87e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.286, R^2 = 0.082, p = 0.0039.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.311, R^2 = 0.097, p = 0.0016.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.299, R^2 = 0.089, p = 0.0025.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.311, R^2 = 0.097, p = 0.0016.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.384 0.147 8.00e-05 -0.00255 ok
chrF++ 100 significant negative -0.420 0.177 1.33e-05 -0.00175 ok
METEOR 100 significant negative -0.436 0.190 5.84e-06 -0.00194 ok
ROUGE-L 100 significant negative -0.384 0.148 7.87e-05 -0.00161 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.286 0.082 0.0039 -0.00215 ok
Trigram recall 100 significant negative -0.311 0.097 0.0016 -0.00227 ok
Trigram F1 100 significant negative -0.299 0.089 0.0025 -0.00221 ok
Trigram Jaccard 100 significant negative -0.311 0.097 0.0016 -0.00226 ok

Translation Similarity Metrics

Mean BLEU-451.9%
Mean chrF++72.7%
Mean METEOR77.0%
Mean ROUGE-L77.6%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU48.7%
Unigram F178.5%
Bigram F155.3%
Trigram precision42.7%
Trigram recall41.4%
Trigram F142.0%
Trigram Jaccard26.6%
4-gram F132.6%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 236 -14.492 38.8% 62.8% 59.2% 66.4% N/A N/A N/A reviewed/approved 14850 gpt-5.6-sol
2604 Καρχηδών 117 123 13.166 32.5% 63.5% 70.6% 68.6% N/A N/A N/A reviewed/approved 14895 gpt-5.6-sol
2342 Κάμιρος 54 65 13.084 43.3% 74.2% 69.6% 70.6% N/A N/A N/A reviewed/approved 14778 gpt-5.6-sol
2328 Καλλίπολις 43 54 12.196 37.8% 62.1% 66.6% 59.8% N/A N/A N/A reviewed/approved 14706 gpt-5.6-sol
2056 Καβειρία 112 95 -10.238 31.5% 67.3% 57.2% 64.7% N/A N/A N/A reviewed/approved 14427 gpt-5.6-sol
2116 Κάληρος 40 29 -10.046 19.2% 50.9% 54.9% 58.0% N/A N/A N/A reviewed/approved 14643 gpt-5.6-sol
2603 Κάρυστος 175 172 8.844 49.8% 69.8% 71.3% 73.6% N/A N/A N/A reviewed/approved 14886 gpt-5.6-sol
2605 Κάσιον 56 62 8.245 47.7% 72.3% 77.3% 71.2% N/A N/A N/A reviewed/approved 14904 gpt-5.6-sol
2626 Κατάνη 95 97 7.391 36.4% 60.1% 68.6% 66.7% N/A N/A N/A reviewed/approved 14994 gpt-5.6-sol
7260 Κυχρεῖος πάγος 61 65 6.648 42.1% 66.2% 78.0% 66.7% N/A N/A N/A reviewed/approved 15237 gpt-5.6-sol
2055 Καβασσός 109 96 -6.480 36.3% 59.7% 62.9% 66.3% N/A N/A N/A reviewed/approved 14418 gpt-5.6-sol
7259 Κύφος 54 58 6.084 45.6% 73.1% 85.0% 80.4% N/A N/A N/A reviewed/approved 15228 gpt-5.6-sol
2346 Κάναι 77 79 5.939 56.0% 70.8% 80.1% 80.8% N/A N/A N/A reviewed/approved 14796 gpt-5.6-sol
2468 Καπετώλιον 138 135 5.860 24.1% 50.8% 55.0% 57.1% N/A N/A N/A reviewed/approved 14823 gpt-5.6-sol
7255 Κυτέριον 22 28 5.502 52.4% 73.8% 72.8% 72.0% N/A N/A N/A reviewed/approved 15192 gpt-5.6-sol
2597 Καρπασία 112 101 -4.238 29.5% 58.5% 62.2% 63.8% N/A N/A N/A reviewed/approved 14859 gpt-5.6-sol
2470 Καππαδοκία 68 69 4.213 24.7% 58.3% 86.9% 83.2% N/A N/A N/A reviewed/approved 14832 gpt-5.6-sol
3530 Κριώα 26 22 -4.175 41.7% 65.7% 76.5% 79.2% N/A N/A N/A reviewed/approved 15066 gpt-5.6-sol
7258 Κύτωρος 30 34 4.148 49.3% 80.1% 86.3% 81.2% N/A N/A N/A reviewed/approved 15219 gpt-5.6-sol
2114 Κάλβιος 18 15 -3.820 23.8% 52.2% 59.2% 60.6% N/A N/A N/A reviewed/approved 14625 gpt-5.6-sol
7246 Κύρις 20 17 -3.659 67.0% 80.8% 85.2% 86.5% N/A N/A N/A reviewed/approved 15111 gpt-5.6-sol
3496 Κοτιάειον 71 64 -3.545 37.1% 57.1% 63.2% 63.7% N/A N/A N/A reviewed/approved 15048 gpt-5.6-sol
2326 Καλλίαρος 47 49 3.519 57.9% 80.2% 92.8% 89.6% N/A N/A N/A reviewed/approved 14688 gpt-5.6-sol
2078 Καινύς 22 19 -3.498 52.1% 66.4% 71.8% 68.3% N/A N/A N/A reviewed/approved 14526 gpt-5.6-sol
2455 Κάναστρον 60 54 -3.432 19.7% 51.9% 59.2% 63.2% N/A N/A N/A reviewed/approved 14805 gpt-5.6-sol
2083 Καλαμένθη 24 21 -3.336 29.9% 50.4% 42.7% 40.0% N/A N/A N/A reviewed/approved 14580 gpt-5.6-sol
7257 Κυτώνιον 25 22 -3.256 81.2% 83.8% 84.7% 89.4% N/A N/A N/A reviewed/approved 15210 gpt-5.6-sol
7247 Κύρνος 50 45 -3.239 35.7% 59.3% 67.9% 67.4% N/A N/A N/A reviewed/approved 15120 gpt-5.6-sol
7253 Κυρτώνιος 18 22 3.180 40.6% 69.3% 68.6% 65.0% N/A N/A N/A reviewed/approved 15174 gpt-5.6-sol
2075 Καινίνη 15 13 -3.062 79.5% 88.0% 87.7% 92.9% N/A N/A N/A reviewed/approved 14508 gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 1.425; cross-validated R^2: -0.136.

Positive termCoefficientDocsMean presentMean absent
απο 2.8504 27 1.869 -0.691
παιδος 2.6445 4 6.128 -0.255
ος 2.6316 6 4.808 -0.307
δε και 2.5243 8 1.992 -0.173
πολις 2.5007 68 0.300 -0.637
εν τη 2.3714 7 4.326 -0.326
πολιχνιον 2.2149 3 7.036 -0.218
του 2.0458 37 1.238 -0.727
εκληθη 1.9681 4 5.963 -0.248
εστι δε και 1.9535 3 5.321 -0.165
αι 1.8883 3 5.896 -0.182
ορος 1.8415 6 5.415 -0.346
δευτερα 1.8174 3 4.541 -0.140
και αλλη 1.8020 3 7.687 -0.238
οι 1.7279 11 2.542 -0.314
και θηλυκον 1.5853 3 7.079 -0.219
εκαλειτο δε 1.5349 3 2.506 -0.078
ουδετερως 1.4955 3 3.777 -0.117
τη 1.4355 15 1.613 -0.285
λεγεται και 1.3686 8 1.780 -0.155
Negative termCoefficientDocsMean presentMean absent
το -2.6203 70 -0.069 0.160
οικητωρ -2.1970 6 -4.284 0.273
οτι -2.1655 5 -0.855 0.045
καππαδοκιας -2.0150 3 -3.314 0.102
τινες -1.9506 6 -3.137 0.200
το θηλυκον -1.9416 6 -3.706 0.237
ουτως -1.6633 5 -4.021 0.212
εθνος -1.6204 8 -3.722 0.324
μεταξυ -1.4844 5 -2.187 0.115
εστι και -1.4818 20 -0.291 0.073
ει -1.3824 4 -4.213 0.176
το κτητικον -1.3243 7 -3.549 0.267
κτητικον -1.3243 7 -3.549 0.267
ης -1.3238 3 -3.513 0.109
φησιν -1.2989 10 -0.886 0.098
εκαλειτο -1.2929 7 -1.076 0.081
δε οτι -1.2914 3 -2.475 0.077
και του -1.2636 4 -3.150 0.131
ως το -1.2599 3 -4.555 0.141
τοπικα -1.2371 3 -3.335 0.103

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.025.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 1.7460 7 7.786 2.414
απο 1.7156 27 5.347 1.844
και 1.5422 66 3.431 1.546
δε 1.3625 46 3.647 2.060
την 1.1121 13 5.742 2.349
εκαλειτο δε 1.0911 3 12.167 2.500
αφ 1.0423 6 6.146 2.576
αφ ου 0.9268 5 6.907 2.573
οτι 0.9229 5 4.859 2.681
δια 0.8986 16 4.818 2.404
περι 0.8948 11 5.063 2.509
της 0.8765 35 4.259 1.999
κατα 0.8655 10 6.140 2.418
του 0.8483 37 4.111 2.014
δε και 0.8471 8 6.147 2.498
ος 0.8349 6 5.989 2.586
τω 0.8254 16 4.637 2.438
πολιχνιον 0.8227 3 7.036 2.659
παιδος 0.7682 4 6.128 2.651
τα 0.7583 12 5.833 2.375
Negative termCoefficientDocsMean presentMean absent
ως -1.0357 52 2.888 2.684
το εθνικον -0.9956 56 2.493 3.168
εθνικον -0.9956 56 2.493 3.168
και το -0.8049 9 3.057 2.764
προς τη -0.7833 4 1.190 2.857
στραβων -0.7649 15 2.758 2.796
κωμη -0.7566 3 0.735 2.854
νησος -0.7311 9 1.664 2.901
δευτερω -0.6691 3 0.963 2.847
τους -0.6029 4 1.552 2.842
του δε -0.5870 3 1.273 2.837
ειρηται -0.5119 3 2.071 2.812
μια -0.5054 3 0.990 2.846
τεταρτω -0.4851 3 1.014 2.845
φησι το -0.4832 3 0.451 2.862
εκ -0.4804 6 2.763 2.792
προς -0.4650 15 3.075 2.740
εδει -0.4415 3 1.235 2.838
εκαταιος ασια -0.4126 5 1.088 2.880
ασια -0.4126 5 1.088 2.880

AI English terms predicting large absolute residuals

Features: 315; ridge alpha: 2.894; cross-validated R^2: 0.037.

Positive termCoefficientDocsMean presentMean absent
was 2.0025 19 5.705 2.106
and 1.6489 54 3.763 1.648
was called 1.4626 5 10.166 2.402
after 1.2048 18 5.635 2.165
the 1.1762 93 2.875 1.667
on 1.0673 18 4.413 2.434
with 1.0446 15 4.684 2.456
says 1.0270 19 5.675 2.113
by 1.0060 15 4.897 2.418
most 0.9530 4 11.072 2.445
son 0.8583 16 5.789 2.219
son of 0.8583 16 5.789 2.219
on the 0.8147 7 4.810 2.638
his on 0.8140 3 9.914 2.570
usage 0.7912 3 8.016 2.628
called 0.7883 32 4.363 2.050
karia 0.7709 3 9.250 2.590
is the 0.7192 4 7.432 2.597
sicily 0.6920 3 7.695 2.638
diphthong 0.6896 3 8.197 2.623
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9952 50 2.272 3.308
the ethnonym -0.8970 55 2.428 3.232
ethnonym is -0.8808 51 2.429 3.166
as in -0.8791 35 2.392 3.004
ethnonym -0.7898 56 2.568 3.073
book -0.6696 39 2.330 3.084
is called -0.5689 5 1.608 2.852
strabo -0.5129 15 2.758 2.796
near -0.5054 15 2.505 2.840
it is also -0.4578 4 0.779 2.874
an -0.4493 22 2.748 2.802
island -0.4477 14 2.323 2.866
an island -0.4429 10 1.615 2.921
in book of -0.4360 16 2.248 2.893
book of his -0.4360 16 2.248 2.893
book of -0.4360 16 2.248 2.893
in his asia -0.4233 5 1.088 2.880
his asia -0.4233 5 1.088 2.880
one -0.4203 8 2.886 2.782
as -0.4193 53 3.013 2.539

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: 0.048.

Positive termCoefficientDocsMean presentMean absent
on 1.4053 16 4.633 2.439
and 1.2978 52 3.749 1.751
the 1.2701 90 2.912 1.693
was 1.2128 20 5.136 2.204
after 1.2073 22 5.290 2.085
used to be 1.1224 6 9.059 2.390
also 1.0212 43 3.827 2.008
is with 1.0094 4 9.335 2.517
used 0.9862 8 8.092 2.329
used to 0.9862 8 8.092 2.329
to be 0.8925 9 6.926 2.381
son 0.8875 17 6.050 2.122
son of 0.8875 17 6.050 2.122
are also 0.8604 4 7.399 2.598
after the 0.8560 6 7.339 2.500
his on 0.8317 3 9.914 2.570
says 0.8170 17 4.950 2.347
karia 0.8087 3 9.250 2.590
when 0.8052 7 6.145 2.537
to be called 0.7912 5 9.102 2.458
Negative termCoefficientDocsMean presentMean absent
as in -1.2687 29 1.982 3.120
the ethnonym is -1.0475 52 2.217 3.411
ethnonym is -1.0475 52 2.217 3.411
the ethnonym -0.8432 56 2.506 3.152
ethnonym -0.8024 57 2.575 3.075
an -0.7644 22 2.280 2.934
book -0.6035 39 2.330 3.084
and the -0.5267 15 3.089 2.737
of his -0.5110 18 2.195 2.921
it is -0.4961 24 3.117 2.687
strabo -0.4826 15 2.758 2.796
in his asia -0.4694 5 1.088 2.880
his asia -0.4694 5 1.088 2.880
book of his -0.4652 16 2.248 2.893
book of -0.4652 16 2.248 2.893
in book of -0.4514 14 1.526 2.996
an island -0.4122 9 1.323 2.935
but -0.3963 12 2.437 2.838
as -0.3865 60 2.918 2.599
written -0.3814 3 1.192 2.839

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.