gpt-4o-2024-08-06 v2 Evaluation

Generated: 2026-07-25 12:52:45 UTC

Pairs100
Distinct lemmas100
Slope0.922
R^20.983
BLEU-438.7%
chrF++64.7%
METEOR70.8%
ROUGE-L72.3%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2411

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-08-06) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.922
Intercept2.240
R^20.983
P-value6.39e-89
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.960
Mean absolute length residual3.183
Mean absolute percent error7.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.340, R^2 = 0.115, p = 0.0005.
chrF++ vs passage length
chrF++: significant negative; r = -0.363, R^2 = 0.132, p = 0.0002.
METEOR vs passage length
METEOR: significant negative; r = -0.455, R^2 = 0.207, p = 1.96e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.445, R^2 = 0.198, p = 3.41e-06.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.325, R^2 = 0.106, p = 0.0010.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.325, R^2 = 0.106, p = 0.0010.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.326, R^2 = 0.106, p = 0.0009.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.322, R^2 = 0.103, p = 0.0011.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.340 0.115 0.0005 -0.00193 ok
chrF++ 100 significant negative -0.363 0.132 0.0002 -0.00137 ok
METEOR 100 significant negative -0.455 0.207 1.96e-06 -0.00224 ok
ROUGE-L 100 significant negative -0.445 0.198 3.41e-06 -0.00194 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.325 0.106 0.0010 -0.00222 ok
Trigram recall 100 significant negative -0.325 0.106 0.0010 -0.00223 ok
Trigram F1 100 significant negative -0.326 0.106 0.0009 -0.00222 ok
Trigram Jaccard 100 significant negative -0.322 0.103 0.0011 -0.00189 ok

Translation Similarity Metrics

Mean BLEU-438.7%
Mean chrF++64.7%
Mean METEOR70.8%
Mean ROUGE-L72.3%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU38.4%
Unigram F173.0%
Bigram F145.7%
Trigram precision31.6%
Trigram recall30.7%
Trigram F131.2%
Trigram Jaccard18.5%
4-gram F122.1%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
3288 Κορώνεια 110 120 16.329 25.4% 57.4% 61.2% 61.5% N/A N/A N/A reviewed/approved 12745 gpt-4o-2024-08-06
2604 Καρχηδών 117 125 14.874 36.6% 66.0% 66.1% 66.4% N/A N/A N/A reviewed/approved 12697 gpt-4o-2024-08-06
2055 Καβασσός 109 88 -14.749 21.0% 51.3% 47.8% 52.8% N/A N/A N/A reviewed/approved 12538 gpt-4o-2024-08-06
2056 Καβειρία 112 91 -14.515 15.4% 48.8% 45.9% 59.1% N/A N/A N/A reviewed/approved 12541 gpt-4o-2024-08-06
2465 Κάνωπος 83 91 12.226 37.5% 67.8% 83.6% 75.9% N/A N/A N/A reviewed/approved 12670 gpt-4o-2024-08-06
2484 Καρία 270 239 -12.207 23.1% 52.6% 43.1% 56.6% N/A N/A N/A reviewed/approved 12682 gpt-4o-2024-08-06
2328 Καλλίπολις 43 54 12.110 37.9% 63.8% 72.3% 66.0% N/A N/A N/A reviewed/approved 12634 gpt-4o-2024-08-06
2603 Κάρυστος 175 173 9.392 31.3% 60.4% 65.0% 63.2% N/A N/A N/A reviewed/approved 12694 gpt-4o-2024-08-06
7248 Κύρου πόλις 32 40 8.253 24.4% 65.9% 78.1% 72.2% N/A N/A N/A reviewed/approved 12775 gpt-4o-2024-08-06
2342 Κάμιρος 54 60 7.967 34.9% 64.5% 70.1% 68.4% N/A N/A N/A reviewed/approved 12658 gpt-4o-2024-08-06
2609 Κάσπειρος 135 119 -7.724 34.9% 56.8% 48.2% 63.0% N/A N/A N/A reviewed/approved 12706 gpt-4o-2024-08-06
7255 Κυτέριον 22 29 6.474 39.1% 68.2% 67.5% 70.6% N/A N/A N/A reviewed/approved 12796 gpt-4o-2024-08-06
7247 Κύρνος 50 42 -6.345 28.1% 58.1% 67.1% 67.4% N/A N/A N/A reviewed/approved 12772 gpt-4o-2024-08-06
2605 Κάσιον 56 60 6.122 46.9% 69.6% 73.5% 70.7% N/A N/A N/A reviewed/approved 12700 gpt-4o-2024-08-06
7259 Κύφος 54 58 5.967 43.0% 70.2% 73.8% 71.4% N/A N/A N/A reviewed/approved 12808 gpt-4o-2024-08-06
2116 Κάληρος 40 34 -5.124 17.5% 47.6% 50.3% 48.6% N/A N/A N/A reviewed/approved 12613 gpt-4o-2024-08-06
2470 Καππαδοκία 68 70 5.057 24.9% 59.6% 76.4% 73.9% N/A N/A N/A reviewed/approved 12676 gpt-4o-2024-08-06
2329 Κάλπη 57 50 -4.800 42.5% 65.2% 67.6% 77.8% N/A N/A N/A reviewed/approved 12637 gpt-4o-2024-08-06
2326 Καλλίαρος 47 50 4.421 41.0% 66.5% 77.9% 74.2% N/A N/A N/A reviewed/approved 12628 gpt-4o-2024-08-06
3530 Κριώα 26 22 -4.215 37.5% 63.4% 69.6% 75.0% N/A N/A N/A reviewed/approved 12754 gpt-4o-2024-08-06
2331 Κάλυδνα 43 46 4.110 37.5% 66.0% 80.5% 83.1% N/A N/A N/A reviewed/approved 12643 gpt-4o-2024-08-06
7251 Κύρτος 66 59 -4.099 24.4% 49.8% 74.1% 70.4% N/A N/A N/A reviewed/approved 12784 gpt-4o-2024-08-06
2119 Κάλλατις 65 66 3.823 34.8% 61.1% 70.3% 64.1% N/A N/A N/A reviewed/approved 12622 gpt-4o-2024-08-06
7267 Κωνώπη 64 65 3.746 44.8% 68.7% 68.5% 72.9% N/A N/A N/A reviewed/approved 12832 gpt-4o-2024-08-06
7261 Κύψελα 34 30 -3.591 51.9% 71.4% 78.9% 78.1% N/A N/A N/A reviewed/approved 12814 gpt-4o-2024-08-06
2597 Καρπασία 112 102 -3.515 30.5% 61.7% 53.5% 65.4% N/A N/A N/A reviewed/approved 12685 gpt-4o-2024-08-06
2057 Καβελλιών 34 37 3.409 33.7% 66.8% 68.9% 70.4% N/A N/A N/A reviewed/approved 12544 gpt-4o-2024-08-06
2325 Καλλίαι 24 21 -3.370 37.0% 65.6% 75.9% 80.0% N/A N/A N/A reviewed/approved 12625 gpt-4o-2024-08-06
2330 Καλύβη 24 21 -3.370 41.4% 62.9% 64.1% 80.0% N/A N/A N/A reviewed/approved 12640 gpt-4o-2024-08-06
2625 Κατακεκαυμένη 46 48 3.343 44.4% 67.3% 72.8% 70.2% N/A N/A N/A reviewed/approved 12724 gpt-4o-2024-08-06

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.061.

Positive termCoefficientDocsMean presentMean absent
πολις 1.6232 68 0.447 -0.949
απο 1.5763 27 2.165 -0.801
πολιτης 1.4693 18 3.635 -0.798
δε και 1.3900 8 3.861 -0.336
εν τη 1.3875 7 4.510 -0.339
εστι δε και 1.3383 3 10.438 -0.323
και 1.2890 66 0.265 -0.514
δευτερα 1.1459 3 9.075 -0.281
τριτη 1.0546 3 8.746 -0.270
παιδος 1.0506 4 5.035 -0.210
πολις προς 0.9616 3 6.576 -0.203
ος 0.9574 6 4.091 -0.261
και το 0.9571 9 2.675 -0.265
πολιχνιον 0.9303 3 6.230 -0.193
εστι δε 0.9231 5 3.909 -0.206
τη 0.8960 15 1.579 -0.279
στραβων 0.8572 15 2.534 -0.447
κατα 0.8539 10 2.191 -0.243
εις 0.8307 7 2.751 -0.207
εκαλειτο δε 0.7725 3 4.020 -0.124
Negative termCoefficientDocsMean presentMean absent
το -1.4473 70 -0.249 0.580
καππαδοκιας -1.4290 3 -6.107 0.189
ως το -1.2491 3 -11.121 0.344
εθνος -1.0139 8 -4.251 0.370
τοπικα -0.9554 3 -6.151 0.190
τα τοπικα -0.9554 3 -6.151 0.190
γαρ -0.9271 9 -2.478 0.245
παυσανιας -0.9178 4 -5.140 0.214
οικητωρ -0.8765 6 -3.785 0.242
λυκιας -0.8737 3 -5.986 0.185
και του -0.8361 4 -4.751 0.198
τε -0.7572 5 -0.507 0.027
οτι -0.7474 5 -1.429 0.075
ης -0.6983 3 -3.644 0.113
ειναι -0.6922 8 -1.086 0.094
φησιν -0.6664 10 -1.428 0.159
θρακης -0.6525 5 -2.301 0.121
πλησιον -0.6273 8 -0.653 0.057
ελλανικος -0.6267 4 -6.513 0.271
το θηλυκον -0.6232 6 -3.283 0.210

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: 0.031.

Positive termCoefficientDocsMean presentMean absent
την 2.2591 13 7.144 2.591
και 1.8974 66 3.800 1.986
δε 1.7911 46 4.483 2.076
δυναται 1.4222 3 12.517 2.894
εκαλειτο 1.4112 7 8.092 2.813
παρα 1.3562 7 7.725 2.841
εν 1.2492 37 4.512 2.402
απο 1.1966 27 5.419 2.356
κατα 1.1615 10 6.483 2.816
βοιωτιας 1.1487 3 10.660 2.952
εκαλειτο δε 1.1376 3 12.158 2.905
εστι δε 1.1344 5 9.715 2.839
τριτη 1.0366 3 10.213 2.965
ως το 1.0221 3 11.121 2.937
πολις εν 0.9809 5 7.607 2.950
εστι δε και 0.9808 3 10.438 2.959
δε και 0.9655 8 7.464 2.811
δευτερα 0.9649 3 9.884 2.976
εστι 0.9476 24 4.878 2.648
της 0.9196 35 4.591 2.425
Negative termCoefficientDocsMean presentMean absent
το εθνικον -1.4014 56 2.817 3.648
εθνικον -1.4014 56 2.817 3.648
δευτερω -0.9784 3 0.439 3.268
προς τη -0.7838 4 1.692 3.245
αυτην -0.7568 9 2.572 3.243
στραβων -0.7389 15 3.370 3.150
πλησιον -0.6885 8 1.928 3.292
του -0.6126 37 4.258 2.551
αυτην φησιν -0.6051 3 1.349 3.240
ηροδοτος -0.5958 9 3.297 3.172
ην -0.5722 13 3.197 3.181
εξ -0.5679 3 1.369 3.239
τεταρτω -0.5515 3 1.889 3.223
των -0.5371 17 4.142 2.986
κωμη -0.5195 3 1.980 3.220
μια -0.4922 3 1.274 3.242
ορος -0.4849 6 3.321 3.174
ως στραβων -0.4597 4 2.064 3.230
πορρω -0.4476 3 0.834 3.256
ου πορρω -0.4476 3 0.834 3.256

AI English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.894; cross-validated R^2: 0.025.

Positive termCoefficientDocsMean presentMean absent
and 2.1397 55 4.159 1.990
is a city 2.0903 3 13.028 2.878
of 1.3365 85 3.373 2.103
the 1.0518 94 3.247 2.171
according 1.0329 7 7.287 2.874
according to 1.0329 7 7.287 2.874
is 0.9699 90 3.328 1.874
is also a 0.9604 16 6.103 2.627
but 0.9564 10 6.592 2.804
also a 0.9365 17 5.985 2.609
is a 0.9228 30 4.309 2.700
whom 0.8773 5 7.273 2.968
from 0.8748 37 4.484 2.419
claims 0.8665 3 6.427 3.083
written 0.8405 3 9.728 2.980
written with 0.8405 3 9.728 2.980
was called 0.8249 4 9.315 2.927
places 0.8085 5 8.235 2.917
he 0.7841 3 9.864 2.976
also 0.7741 43 4.453 2.224
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.2592 53 2.499 3.955
the ethnonym -1.1562 56 2.631 3.885
ethnonym is -1.0858 54 2.721 3.725
as in -1.0336 37 2.632 3.507
near -0.7942 18 2.278 3.381
ethnonym -0.7893 58 3.045 3.373
there was -0.6067 4 1.013 3.273
because the -0.6009 3 0.781 3.257
it is -0.5691 27 3.767 2.967
with an -0.4744 4 1.723 3.244
strabo book -0.4521 4 1.364 3.259
the second book -0.4452 3 0.439 3.268
the second -0.4452 3 0.439 3.268
second book -0.4452 3 0.439 3.268
in the second -0.4452 3 0.439 3.268
another in -0.4407 3 1.490 3.235
from which -0.4124 8 3.443 3.160
strabo -0.4066 15 3.370 3.150
inhabitant is a -0.4028 6 1.983 3.259
of his -0.4014 15 2.745 3.260

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.894; cross-validated R^2: 0.029.

Positive termCoefficientDocsMean presentMean absent
the 1.9308 90 3.388 1.338
of 1.7054 66 3.894 1.803
and 1.6730 52 4.266 2.010
for 1.5324 11 7.890 2.601
also 1.2135 43 4.512 2.180
this 1.0957 15 7.700 2.386
all 1.0833 3 12.054 2.909
form 1.0109 11 5.476 2.900
boiotia 0.9948 3 10.660 2.952
who 0.9481 13 6.169 2.737
kappadokia 0.9169 3 7.059 3.063
there 0.9114 34 4.788 2.356
is from 0.8987 12 5.290 2.896
used to be 0.8899 6 9.425 2.785
is a city 0.8714 3 9.058 3.001
used 0.8613 8 8.895 2.686
used to 0.8613 8 8.895 2.686
from 0.8410 37 4.242 2.561
there are 0.7810 5 7.947 2.932
his on 0.7768 3 9.472 2.988
Negative termCoefficientDocsMean presentMean absent
the ethnonym is -1.2502 52 2.450 3.977
ethnonym is -1.2502 52 2.450 3.977
the ethnonym -0.9331 56 2.878 3.571
ethnonym -0.7910 57 3.086 3.311
as in -0.7895 29 2.931 3.286
name -0.7831 7 1.318 3.323
one of -0.7227 5 0.881 3.304
one of the -0.5800 4 0.996 3.274
which -0.5797 20 3.239 3.169
herodotos -0.5518 9 3.297 3.172
near -0.5187 16 2.637 3.287
city and -0.5131 4 0.882 3.279
a city and -0.5131 4 0.882 3.279
either -0.4673 6 4.153 3.121
says that it -0.4657 8 3.492 3.156
herodotos book -0.4599 5 2.186 3.235
with a -0.4410 4 1.802 3.240
book -0.4349 39 3.254 3.137
it -0.4334 37 4.048 2.675
was from -0.4108 4 1.659 3.246

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.