gpt-4-turbo-2024-04-09 v2 Evaluation

Generated: 2026-07-25 12:53:01 UTC

Pairs100
Distinct lemmas100
Slope0.888
R^20.982
BLEU-440.0%
chrF++64.5%
METEOR68.7%
ROUGE-L70.7%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2405

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4 Turbo (2024-04-09) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.888
Intercept3.076
R^20.982
P-value9.56e-87
Pearson r0.991
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.270
Mean absolute length residual3.356
Mean absolute percent error8.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.459, R^2 = 0.211, p = 1.53e-06.
chrF++ vs passage length
chrF++: significant negative; r = -0.438, R^2 = 0.192, p = 5.11e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.488, R^2 = 0.238, p = 2.66e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.489, R^2 = 0.239, p = 2.49e-07.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.401, R^2 = 0.161, p = 3.60e-05.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.413, R^2 = 0.171, p = 1.95e-05.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.408, R^2 = 0.166, p = 2.53e-05.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.392, R^2 = 0.154, p = 5.45e-05.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.459 0.211 1.53e-06 -0.00305 ok
chrF++ 100 significant negative -0.438 0.192 5.11e-06 -0.00193 ok
METEOR 100 significant negative -0.488 0.238 2.66e-07 -0.00277 ok
ROUGE-L 100 significant negative -0.489 0.239 2.49e-07 -0.00243 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.401 0.161 3.60e-05 -0.00319 ok
Trigram recall 100 significant negative -0.413 0.171 1.95e-05 -0.00328 ok
Trigram F1 100 significant negative -0.408 0.166 2.53e-05 -0.00323 ok
Trigram Jaccard 100 significant negative -0.392 0.154 5.45e-05 -0.00284 ok

Translation Similarity Metrics

Mean BLEU-440.0%
Mean chrF++64.5%
Mean METEOR68.7%
Mean ROUGE-L70.7%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU36.8%
Unigram F169.5%
Bigram F143.5%
Trigram precision31.0%
Trigram recall29.6%
Trigram F130.3%
Trigram Jaccard17.9%
4-gram F121.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 103 -22.558 7.9% 36.9% 31.3% 39.8% N/A N/A N/A reviewed/approved 12373 gpt-4-turbo-2024-04-09
3288 Κορώνεια 110 117 16.294 26.6% 60.4% 61.7% 61.4% N/A N/A N/A reviewed/approved 12445 gpt-4-turbo-2024-04-09
2484 Καρία 270 229 -13.714 20.3% 49.4% 44.4% 50.9% N/A N/A N/A reviewed/approved 12382 gpt-4-turbo-2024-04-09
2342 Κάμιρος 54 62 10.997 31.0% 62.0% 65.3% 63.8% N/A N/A N/A reviewed/approved 12358 gpt-4-turbo-2024-04-09
2054 Καβαλίς 65 70 9.233 42.9% 65.6% 63.2% 71.1% N/A N/A N/A reviewed/approved 12235 gpt-4-turbo-2024-04-09
2116 Κάληρος 40 30 -8.578 16.2% 45.7% 39.3% 45.7% N/A N/A N/A reviewed/approved 12313 gpt-4-turbo-2024-04-09
2465 Κάνωπος 83 85 8.258 25.4% 59.7% 66.6% 67.9% N/A N/A N/A reviewed/approved 12370 gpt-4-turbo-2024-04-09
2605 Κάσιον 56 61 8.221 38.2% 65.3% 71.0% 66.7% N/A N/A N/A reviewed/approved 12400 gpt-4-turbo-2024-04-09
7244 Κυρήνη 54 59 7.997 36.4% 61.4% 66.0% 65.5% N/A N/A N/A reviewed/approved 12463 gpt-4-turbo-2024-04-09
2626 Κατάνη 95 80 -7.393 26.1% 55.0% 48.5% 64.0% N/A N/A N/A reviewed/approved 12430 gpt-4-turbo-2024-04-09
7254 Κύτα 84 85 7.370 25.9% 55.9% 52.3% 59.2% N/A N/A N/A reviewed/approved 12493 gpt-4-turbo-2024-04-09
2604 Καρχηδών 117 114 7.081 17.4% 52.0% 49.7% 51.7% N/A N/A N/A reviewed/approved 12397 gpt-4-turbo-2024-04-09
2603 Κάρυστος 175 165 6.603 33.0% 61.4% 61.7% 65.3% N/A N/A N/A reviewed/approved 12394 gpt-4-turbo-2024-04-09
2470 Καππαδοκία 68 70 6.571 21.3% 57.2% 78.0% 71.0% N/A N/A N/A reviewed/approved 12376 gpt-4-turbo-2024-04-09
2455 Κάναστρον 60 50 -6.329 23.3% 56.3% 57.3% 58.2% N/A N/A N/A reviewed/approved 12367 gpt-4-turbo-2024-04-09
7264 Κωλιάς 61 51 -6.216 33.2% 62.0% 60.5% 71.4% N/A N/A N/A reviewed/approved 12523 gpt-4-turbo-2024-04-09
2609 Κάσπειρος 135 129 6.105 36.4% 61.8% 54.0% 63.6% N/A N/A N/A reviewed/approved 12406 gpt-4-turbo-2024-04-09
7255 Κυτέριον 22 28 5.398 28.6% 57.1% 59.3% 64.0% N/A N/A N/A reviewed/approved 12496 gpt-4-turbo-2024-04-09
2346 Κάναι 77 76 4.583 31.1% 54.6% 68.2% 70.1% N/A N/A N/A reviewed/approved 12364 gpt-4-turbo-2024-04-09
2057 Καβελλιών 34 29 -4.253 53.7% 75.1% 80.3% 85.7% N/A N/A N/A reviewed/approved 12244 gpt-4-turbo-2024-04-09
7267 Κωνώπη 64 64 4.121 42.3% 65.6% 66.5% 68.2% N/A N/A N/A reviewed/approved 12532 gpt-4-turbo-2024-04-09
7259 Κύφος 54 55 3.997 34.2% 64.4% 71.3% 71.6% N/A N/A N/A reviewed/approved 12508 gpt-4-turbo-2024-04-09
7252 Κύρτωνες 27 31 3.960 8.6% 47.7% 48.2% 48.3% N/A N/A N/A reviewed/approved 12487 gpt-4-turbo-2024-04-09
7253 Κυρτώνιος 18 23 3.948 50.2% 74.3% 84.2% 73.2% N/A N/A N/A reviewed/approved 12490 gpt-4-turbo-2024-04-09
2599 Καρπήσιοι 11 9 -3.839 10.2% 39.0% 41.3% 50.0% N/A N/A N/A reviewed/approved 12388 gpt-4-turbo-2024-04-09
7250 Κυρταία 40 35 -3.578 46.4% 64.6% 63.6% 66.7% N/A N/A N/A reviewed/approved 12481 gpt-4-turbo-2024-04-09
7248 Κύρου πόλις 32 35 3.523 21.3% 57.8% 63.2% 62.7% N/A N/A N/A reviewed/approved 12475 gpt-4-turbo-2024-04-09
7247 Κύρνος 50 44 -3.453 18.3% 51.3% 58.0% 55.3% N/A N/A N/A reviewed/approved 12472 gpt-4-turbo-2024-04-09
2081 Καλάθη 49 50 3.434 45.9% 71.2% 84.3% 82.8% N/A N/A N/A reviewed/approved 12286 gpt-4-turbo-2024-04-09
2075 Καινίνη 15 13 -3.389 79.5% 88.0% 87.7% 92.9% N/A N/A N/A reviewed/approved 12268 gpt-4-turbo-2024-04-09

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: 0.006.

Positive termCoefficientDocsMean presentMean absent
και 1.8651 66 0.676 -1.313
οι 1.3822 11 4.957 -0.613
απο 1.2674 27 2.314 -0.856
ου 1.2310 14 3.562 -0.580
βοιωτιας 1.0422 3 7.258 -0.224
πολιτης 1.0168 18 3.574 -0.784
τε 0.9875 5 4.001 -0.211
δε και 0.8019 8 2.869 -0.249
δυναται 0.7738 3 8.291 -0.256
πολις 0.7727 68 0.828 -1.759
παρα 0.7717 7 1.901 -0.143
παιδος 0.7528 4 5.263 -0.219
εστι 0.7308 24 1.696 -0.536
εστι δε και 0.6777 3 8.378 -0.259
εστι δε 0.6202 5 5.355 -0.282
την 0.6025 13 0.703 -0.105
τη 0.5901 15 1.636 -0.289
ιβηριας 0.5850 3 5.084 -0.157
πολις εν 0.5765 5 4.101 -0.216
εν τη 0.5590 7 2.909 -0.219
Negative termCoefficientDocsMean presentMean absent
γαρ -1.7863 9 -2.255 0.223
δια -1.2499 16 -1.171 0.223
περι -1.1195 11 -4.409 0.545
δια του -0.9935 10 -2.361 0.262
του -0.9213 37 0.834 -0.490
οτι -0.8074 5 -1.887 0.099
ηρωδιανος -0.7616 3 -12.920 0.400
οικητωρ -0.7567 6 -4.127 0.263
το -0.7211 70 0.030 -0.071
το εθνικον -0.7027 56 -0.641 0.816
εθνικον -0.7027 56 -0.641 0.816
ζευς -0.5910 3 -10.563 0.327
δε της -0.5727 6 -3.489 0.223
εθνος -0.5723 8 -2.013 0.175
θεοπομπος -0.5579 5 -2.346 0.123
ει -0.5529 4 -5.500 0.229
τα -0.5397 12 -1.796 0.245
φησιν -0.5014 10 -2.343 0.260
τινες -0.4905 6 -2.240 0.143
θρακης -0.4855 5 -2.210 0.116

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.226.

Positive termCoefficientDocsMean presentMean absent
δε 1.5837 46 4.824 2.105
γαρ 1.5606 9 7.451 2.951
την 1.3988 13 7.606 2.721
και 1.1435 66 4.054 2.002
εις 1.0004 7 10.955 2.784
δια του 0.9258 10 6.410 3.017
δια 0.8782 16 5.322 2.982
τα 0.7732 12 7.573 2.781
του 0.7431 37 4.586 2.634
απο 0.6845 27 5.306 2.635
ουτως 0.5859 5 8.809 3.069
βοιωτιας 0.5784 3 7.258 3.235
αι 0.5554 3 11.606 3.101
ζευς 0.5548 3 13.618 3.039
ηρωδιανος 0.5249 3 12.920 3.060
εν 0.4891 37 4.459 2.708
οτι 0.4772 5 5.494 3.244
εκαλειτο 0.4593 7 6.170 3.144
εστι 0.4546 24 4.492 2.998
ος 0.4542 6 8.076 3.055
Negative termCoefficientDocsMean presentMean absent
ως -0.7884 52 3.128 3.604
δευτερω -0.6411 3 0.648 3.440
το εθνικον -0.5148 56 3.483 3.195
εθνικον -0.5148 56 3.483 3.195
κωμη -0.4822 3 1.024 3.428
στραβων -0.4283 15 3.046 3.411
νησος -0.4252 9 2.495 3.441
μια -0.4136 3 1.145 3.425
καππαδοκιας -0.4072 3 1.623 3.410
τεταρτω -0.3443 3 1.341 3.418
εκ -0.3416 6 2.172 3.432
προς τη -0.3361 4 1.922 3.416
διονυσιος -0.3053 4 2.325 3.399
τους -0.3018 4 2.489 3.392
παρα το -0.2911 3 1.616 3.410
πλησιον -0.2797 8 2.982 3.389
ηροδοτος -0.2760 9 4.131 3.280
παυσανιας -0.2359 4 2.041 3.411
ης -0.2297 3 2.236 3.391
τον -0.2285 10 3.114 3.383

AI English terms predicting large absolute residuals

Features: 318; ridge alpha: 4.125; cross-validated R^2: -0.226.

Positive termCoefficientDocsMean presentMean absent
the 1.9499 91 3.463 2.275
for 1.8431 13 5.561 3.027
is 1.1647 89 3.462 2.502
with 1.1599 13 5.601 3.021
and 0.9579 56 4.471 1.938
is called 0.9030 7 8.118 2.998
it 0.8917 37 4.970 2.408
city is 0.8698 3 8.758 3.189
in this 0.8586 3 15.152 2.991
when 0.8418 5 11.319 2.937
zeus 0.8205 5 11.136 2.947
that 0.8169 11 6.867 2.922
called 0.8024 32 5.105 2.533
athens 0.7457 3 12.344 3.078
either 0.7290 4 10.491 3.059
is the 0.7067 3 12.473 3.074
the feminine 0.6725 14 6.112 2.907
herodian 0.6684 3 12.920 3.060
in 0.6261 85 3.474 2.689
by 0.5913 18 4.889 3.020
Negative termCoefficientDocsMean presentMean absent
per -0.6234 15 2.676 3.476
as per -0.6234 15 2.676 3.476
the ethnonym is -0.5980 54 3.406 3.298
ethnonym is -0.5980 54 3.406 3.298
his -0.5015 25 2.995 3.476
city of -0.4549 20 2.852 3.482
the ethnonym -0.4447 57 3.433 3.254
ethnonym -0.4447 57 3.433 3.254
island -0.4178 14 2.760 3.453
an island -0.4164 10 2.461 3.456
book -0.4134 38 2.928 3.618
as -0.4124 58 3.486 3.177
strabo -0.3982 15 3.046 3.411
are called -0.3952 5 3.314 3.358
a city of -0.3872 19 2.629 3.527
an -0.3718 27 3.917 3.149
a city -0.3702 64 3.504 3.093
name -0.3590 3 1.052 3.427
as in -0.3477 34 3.133 3.471
of his -0.3427 13 2.301 3.514

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 5.878; cross-validated R^2: -0.238.

Positive termCoefficientDocsMean presentMean absent
with 1.5977 16 5.175 3.010
the 1.4717 90 3.500 2.063
and 0.9004 52 4.470 2.149
is 0.7614 87 3.543 2.103
sanctuary 0.7601 3 13.012 3.058
on 0.6447 16 5.511 2.946
forms 0.6195 4 11.090 3.034
on the 0.6146 10 5.607 3.106
this 0.5951 15 7.284 2.663
there is also 0.5608 22 5.043 2.880
is with 0.5506 4 11.575 3.014
after 0.5036 22 5.333 2.798
there is 0.5023 24 4.725 2.924
is also 0.5018 31 4.466 2.858
when 0.4963 7 8.296 2.984
after the 0.4903 6 7.265 3.107
there 0.4799 34 4.318 2.861
however 0.4753 10 7.932 2.848
diphthong 0.4665 3 12.918 3.060
boiotia 0.4562 3 7.258 3.235
Negative termCoefficientDocsMean presentMean absent
an -0.5802 22 2.716 3.537
book -0.5459 39 2.856 3.676
as in -0.4444 29 3.197 3.421
the ethnonym is -0.4275 52 3.405 3.303
ethnonym is -0.4275 52 3.405 3.303
ethnonym -0.3701 57 3.474 3.199
the ethnonym -0.3668 56 3.480 3.199
per -0.3634 20 2.912 3.467
as per -0.3634 20 2.912 3.467
as -0.3620 60 3.331 3.393
near -0.3573 16 3.060 3.413
a city in -0.3144 40 3.349 3.361
of his -0.3100 18 2.843 3.469
strabo -0.3037 15 3.046 3.411
island -0.2898 14 2.760 3.453
dionysios -0.2762 4 2.325 3.399
book of his -0.2747 16 2.995 3.425
book of -0.2747 16 2.995 3.425
about -0.2692 5 1.702 3.443
an island -0.2663 9 2.458 3.445

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.