paper_guidance_ablation_gpt56 v1 Evaluation

Generated: 2026-09-20 11:22:57 UTC

Pairs100
Distinct lemmas100
Slope0.919
R^20.987
BLEU-452.0%
chrF++72.8%
METEOR77.1%
ROUGE-L77.7%
BERTScore96.4%
COMET80.4%
BLEURT77.2%

Profile version id: 2586

First translation using this version: 2026-07-18

Total usable AI runs for this version: 300 runs across 100 lemmas

Created: 2026-07-18 21:14:55.545407+10:00

Active: True

Notes: Arm A: Stephanos v2 static prompt with no matched entry-specific guidance. Model gpt-5.6-sol; Responses API; medium reasoning; top_p 1.0; three runs per Kappa-corpus entry.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.919
Intercept2.290
R^20.987
P-value1.28e-94
Pearson r0.994
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.870
Mean absolute length residual2.783
Mean absolute percent error6.7%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.387, R^2 = 0.150, p = 6.89e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.424, R^2 = 0.180, p = 1.09e-05.
METEOR vs passage length
METEOR: significant negative; r = -0.447, R^2 = 0.199, p = 3.20e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.391, R^2 = 0.153, p = 5.67e-05.
BERTScore vs passage length
BERTScore: significant negative; r = -0.584, R^2 = 0.341, p = 1.85e-10.
COMET vs passage length
COMET: significant negative; r = -0.464, R^2 = 0.216, p = 1.14e-06.
BLEURT vs passage length
BLEURT: significant negative; r = -0.699, R^2 = 0.488, p = 6.40e-16.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.291, R^2 = 0.085, p = 0.0033.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.318, R^2 = 0.101, p = 0.0013.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.305, R^2 = 0.093, p = 0.0021.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.317, R^2 = 0.100, p = 0.0013.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.387 0.150 6.89e-05 -0.00258 ok
chrF++ 100 significant negative -0.424 0.180 1.09e-05 -0.00178 ok
METEOR 100 significant negative -0.447 0.199 3.20e-06 -0.00199 ok
ROUGE-L 100 significant negative -0.391 0.153 5.67e-05 -0.00165 ok
BERTScore 100 significant negative -0.584 0.341 1.85e-10 -0.00044 ok
COMET 100 significant negative -0.464 0.216 1.14e-06 -0.00089 ok
BLEURT 100 significant negative -0.699 0.488 6.40e-16 -0.00250 ok
Trigram precision 100 significant negative -0.291 0.085 0.0033 -0.00219 ok
Trigram recall 100 significant negative -0.318 0.101 0.0013 -0.00233 ok
Trigram F1 100 significant negative -0.305 0.093 0.0021 -0.00225 ok
Trigram Jaccard 100 significant negative -0.317 0.100 0.0013 -0.00230 ok

Translation Similarity Metrics

Mean BLEU-452.0%
Mean chrF++72.8%
Mean METEOR77.1%
Mean ROUGE-L77.7%
Mean BERTScore96.4%
Mean COMET80.4%
Mean BLEURT77.2%
Legacy smoothed corpus BLEU48.7%
Unigram F178.6%
Bigram F155.4%
Trigram precision42.7%
Trigram recall41.4%
Trigram F142.1%
Trigram Jaccard26.6%
4-gram F132.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 236 -14.459 38.8% 62.8% 59.2% 66.4% 92.1% 80.8% 52.4% reviewed/approved 14850 gpt-5.6-sol
2604 Καρχηδών 117 123 13.170 32.5% 63.5% 70.6% 68.6% 94.1% 78.8% 63.8% reviewed/approved 14895 gpt-5.6-sol
2342 Κάμιρος 54 65 13.076 43.3% 74.2% 69.6% 70.6% 94.8% 78.5% 68.8% reviewed/approved 14778 gpt-5.6-sol
2328 Καλλίπολις 43 54 12.187 37.8% 62.1% 66.6% 59.8% 93.6% 74.0% 64.7% reviewed/approved 14706 gpt-5.6-sol
2056 Καβειρία 112 95 -10.234 31.5% 67.3% 57.2% 64.7% 95.6% 79.7% 66.0% reviewed/approved 14427 gpt-5.6-sol
2116 Κάληρος 40 29 -10.056 19.2% 50.9% 54.9% 58.0% 93.0% 72.6% 68.4% reviewed/approved 14643 gpt-5.6-sol
2603 Κάρυστος 175 172 8.860 49.8% 69.8% 71.3% 73.6% 95.5% 79.3% 67.0% reviewed/approved 14886 gpt-5.6-sol
2605 Κάσιον 56 62 8.238 47.7% 72.3% 77.3% 71.2% 95.9% 77.7% 73.1% reviewed/approved 14904 gpt-5.6-sol
2626 Κατάνη 95 97 7.391 36.4% 60.1% 68.6% 66.7% 93.5% 73.9% 63.4% reviewed/approved 14994 gpt-5.6-sol
7260 Κυχρεῖος πάγος 61 65 6.642 42.1% 66.2% 78.0% 66.7% 95.7% 77.0% 64.0% reviewed/approved 15237 gpt-5.6-sol
2055 Καβασσός 109 96 -6.477 36.3% 59.7% 62.9% 66.3% 94.6% 77.1% 53.4% reviewed/approved 14418 gpt-5.6-sol
7259 Κύφος 54 58 6.076 45.6% 73.1% 85.0% 80.4% 95.9% 76.6% 72.8% reviewed/approved 15228 gpt-5.6-sol
2346 Κάναι 77 79 5.936 56.0% 70.8% 80.1% 80.8% 94.6% 76.1% 66.1% reviewed/approved 14796 gpt-5.6-sol
2468 Καπετώλιον 138 135 5.868 24.1% 50.8% 55.0% 57.1% 88.9% 68.4% 55.7% reviewed/approved 14823 gpt-5.6-sol
7255 Κυτέριον 22 28 5.489 52.4% 73.8% 72.8% 72.0% 96.4% 77.4% 76.3% reviewed/approved 15192 gpt-5.6-sol
2597 Καρπασία 112 101 -4.234 29.5% 58.5% 62.2% 63.8% 94.3% 75.3% 58.8% reviewed/approved 14859 gpt-5.6-sol
2470 Καππαδοκία 68 69 4.208 24.7% 58.3% 86.9% 83.2% 95.2% 76.0% 69.6% reviewed/approved 14832 gpt-5.6-sol
3530 Κριώα 26 22 -4.187 41.7% 65.7% 76.5% 79.2% 96.7% 77.6% 74.7% reviewed/approved 15066 gpt-5.6-sol
7258 Κύτωρος 30 34 4.136 49.3% 80.1% 86.3% 81.2% 97.4% 80.0% 82.3% reviewed/approved 15219 gpt-5.6-sol
2114 Κάλβιος 18 15 -3.834 23.8% 52.2% 59.2% 60.6% 92.9% 72.3% 71.3% reviewed/approved 14625 gpt-5.6-sol
7246 Κύρις 20 17 -3.673 67.0% 80.8% 85.2% 86.5% 97.4% 84.2% 77.9% reviewed/approved 15111 gpt-5.6-sol
3496 Κοτιάειον 71 64 -3.549 37.1% 57.1% 63.2% 63.7% 94.0% 73.7% 73.7% reviewed/approved 15048 gpt-5.6-sol
2078 Καινύς 22 19 -3.511 52.1% 66.4% 71.8% 68.3% 95.5% 83.6% 76.8% reviewed/approved 14526 gpt-5.6-sol
2326 Καλλίαρος 47 49 3.510 57.9% 80.2% 92.8% 89.6% 97.8% 77.8% 80.8% reviewed/approved 14688 gpt-5.6-sol
2455 Κάναστρον 60 54 -3.438 19.7% 51.9% 59.2% 63.2% 93.2% 68.3% 62.0% reviewed/approved 14805 gpt-5.6-sol
2083 Καλαμένθη 24 21 -3.349 29.9% 50.4% 42.7% 40.0% 93.3% 75.8% 69.9% reviewed/approved 14580 gpt-5.6-sol
7257 Κυτώνιον 25 22 -3.268 81.2% 83.8% 84.7% 89.4% 98.0% 81.5% 80.3% reviewed/approved 15210 gpt-5.6-sol
7247 Κύρνος 50 45 -3.247 35.7% 59.3% 67.9% 67.4% 96.0% 78.0% 78.2% reviewed/approved 15120 gpt-5.6-sol
7253 Κυρτώνιος 18 22 3.166 40.6% 69.3% 68.6% 65.0% 96.8% 77.2% 81.6% reviewed/approved 15174 gpt-5.6-sol
2075 Καινίνη 15 13 -3.077 79.5% 88.0% 87.7% 92.9% 99.3% 88.6% 92.5% reviewed/approved 14508 gpt-5.6-sol

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 1.425; cross-validated R^2: -0.139.

Positive termCoefficientDocsMean presentMean absent
απο 2.8527 27 1.867 -0.690
παιδος 2.6443 4 6.119 -0.255
ος 2.6287 6 4.807 -0.307
δε και 2.5229 8 1.994 -0.173
πολις 2.4791 68 0.291 -0.619
εν τη 2.3711 7 4.317 -0.325
πολιχνιον 2.2107 3 7.030 -0.217
του 2.0275 37 1.233 -0.724
εκληθη 1.9645 4 5.962 -0.248
εστι δε και 1.9510 3 5.320 -0.165
αι 1.8786 3 5.892 -0.182
ορος 1.8380 6 5.413 -0.345
δευτερα 1.8148 3 4.535 -0.140
και αλλη 1.8000 3 7.688 -0.238
οι 1.7198 11 2.537 -0.314
και θηλυκον 1.5855 3 7.078 -0.219
εκαλειτο δε 1.5378 3 2.523 -0.078
ουδετερως 1.4884 3 3.770 -0.117
τη 1.4321 15 1.609 -0.284
λεγεται και 1.3700 8 1.782 -0.155
Negative termCoefficientDocsMean presentMean absent
το -2.6389 70 -0.077 0.179
οικητωρ -2.2025 6 -4.288 0.274
οτι -2.1721 5 -0.858 0.045
καππαδοκιας -2.0182 3 -3.321 0.103
τινες -1.9539 6 -3.146 0.201
το θηλυκον -1.9436 6 -3.704 0.236
ουτως -1.6619 5 -4.015 0.211
εστι και -1.4992 20 -0.294 0.073
μεταξυ -1.4973 5 -2.194 0.115
εθνος -1.4663 8 -3.612 0.314
ει -1.3785 4 -4.209 0.175
ης -1.3274 3 -3.519 0.109
το κτητικον -1.3247 7 -3.548 0.267
κτητικον -1.3247 7 -3.548 0.267
φησιν -1.2996 10 -0.885 0.098
εκαλειτο -1.2984 7 -1.072 0.081
δε οτι -1.2965 3 -2.482 0.077
και του -1.2792 4 -3.159 0.132
ως το -1.2605 3 -4.555 0.141
τοπικα -1.2412 3 -3.341 0.103

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.029.

Positive termCoefficientDocsMean presentMean absent
εκαλειτο 1.7508 7 7.782 2.407
απο 1.7141 27 5.343 1.836
και 1.5623 66 3.431 1.524
δε 1.3713 46 3.646 2.048
την 1.1138 13 5.742 2.341
εκαλειτο δε 1.0918 3 12.163 2.493
αφ 1.0425 6 6.143 2.568
αφ ου 0.9275 5 6.906 2.566
οτι 0.9273 5 4.858 2.674
δια 0.9034 16 4.820 2.395
περι 0.8974 11 5.061 2.501
της 0.8895 35 4.260 1.987
κατα 0.8680 10 6.137 2.410
του 0.8601 37 4.112 2.002
δε και 0.8478 8 6.143 2.491
ος 0.8375 6 5.990 2.578
τω 0.8325 16 4.638 2.430
πολιχνιον 0.8237 3 7.030 2.652
παιδος 0.7660 4 6.119 2.644
τα 0.7627 12 5.832 2.367
Negative termCoefficientDocsMean presentMean absent
ως -1.0235 52 2.888 2.669
το εθνικον -0.9810 56 2.494 3.151
εθνικον -0.9810 56 2.494 3.151
και το -0.8050 9 3.055 2.756
προς τη -0.7788 4 1.189 2.849
στραβων -0.7550 15 2.761 2.787
κωμη -0.7543 3 0.735 2.846
νησος -0.7251 9 1.666 2.893
δευτερω -0.6657 3 0.957 2.839
τους -0.6006 4 1.550 2.834
του δε -0.5855 3 1.278 2.829
ειρηται -0.5127 3 2.065 2.805
μια -0.4992 3 0.989 2.838
φησι το -0.4824 3 0.453 2.855
εκ -0.4801 6 2.758 2.784
τεταρτω -0.4613 3 1.026 2.837
προς -0.4551 15 3.078 2.731
εδει -0.4436 3 1.228 2.831
εκαταιος ασια -0.3966 5 1.101 2.871
ασια -0.3966 5 1.101 2.871

AI English terms predicting large absolute residuals

Features: 315; ridge alpha: 2.894; cross-validated R^2: 0.039.

Positive termCoefficientDocsMean presentMean absent
was 2.0083 19 5.703 2.098
and 1.6663 54 3.764 1.631
was called 1.4634 5 10.161 2.395
after 1.2038 18 5.630 2.158
the 1.1808 93 2.866 1.674
on 1.0661 18 4.409 2.426
with 1.0476 15 4.684 2.447
says 1.0275 19 5.672 2.105
by 1.0148 15 4.900 2.409
most 0.9526 4 11.064 2.438
son 0.8566 16 5.784 2.211
son of 0.8566 16 5.784 2.211
on the 0.8156 7 4.810 2.630
his on 0.8124 3 9.898 2.563
called 0.7933 32 4.360 2.041
usage 0.7896 3 7.998 2.622
karia 0.7711 3 9.241 2.583
is the 0.7218 4 7.438 2.589
sicily 0.6956 3 7.696 2.631
diphthong 0.6886 3 8.187 2.616
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -0.9843 50 2.273 3.292
the ethnonym -0.8873 55 2.429 3.216
as in -0.8719 35 2.393 2.993
ethnonym is -0.8700 51 2.429 3.151
ethnonym -0.7802 56 2.568 3.056
book -0.6556 39 2.332 3.071
is called -0.5669 5 1.603 2.845
strabo -0.5070 15 2.761 2.787
an -0.4975 22 2.709 2.804
near -0.4950 15 2.508 2.831
it is also -0.4603 4 0.772 2.867
island -0.4393 14 2.322 2.858
an island -0.4338 10 1.618 2.912
in book of -0.4307 16 2.244 2.885
book of his -0.4307 16 2.244 2.885
book of -0.4307 16 2.244 2.885
one -0.4183 8 2.887 2.774
in his asia -0.4125 5 1.101 2.871
his asia -0.4125 5 1.101 2.871
as -0.4114 53 3.012 2.524

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: 0.050.

Positive termCoefficientDocsMean presentMean absent
on 1.4055 16 4.629 2.431
and 1.3250 52 3.748 1.737
the 1.2636 90 2.904 1.696
was 1.2432 20 5.138 2.194
after 1.2203 22 5.286 2.077
used to be 1.1258 6 9.057 2.382
also 1.0379 43 3.827 1.996
is with 1.0141 4 9.333 2.510
used 0.9889 8 8.089 2.322
used to 0.9889 8 8.089 2.322
son 0.8948 17 6.046 2.115
son of 0.8948 17 6.046 2.115
to be 0.8943 9 6.923 2.373
are also 0.8635 4 7.396 2.591
after the 0.8572 6 7.343 2.492
his on 0.8322 3 9.898 2.563
says 0.8239 17 4.951 2.339
karia 0.8119 3 9.241 2.583
when 0.8108 7 6.147 2.530
to be called 0.7937 5 9.096 2.451
Negative termCoefficientDocsMean presentMean absent
as in -1.2665 29 1.981 3.111
the ethnonym is -1.0336 52 2.218 3.395
ethnonym is -1.0336 52 2.218 3.395
the ethnonym -0.8305 56 2.507 3.135
an -0.8143 22 2.239 2.936
ethnonym -0.7896 57 2.576 3.057
book -0.5894 39 2.332 3.071
and the -0.5227 15 3.089 2.729
of his -0.5030 18 2.193 2.912
it is -0.4976 24 3.115 2.678
strabo -0.4676 15 2.761 2.787
book of his -0.4586 16 2.244 2.885
book of -0.4586 16 2.244 2.885
in his asia -0.4568 5 1.101 2.871
his asia -0.4568 5 1.101 2.871
in book of -0.4449 14 1.524 2.988
an island -0.4034 9 1.327 2.927
but -0.3962 12 2.434 2.831
written -0.3807 3 1.189 2.832
as -0.3721 60 2.918 2.580

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.