gpt-4o-2024-05-13 v2 Evaluation

Generated: 2026-07-25 12:52:38 UTC

Pairs100
Distinct lemmas100
Slope0.835
R^20.914
BLEU-439.7%
chrF++64.3%
METEOR69.5%
ROUGE-L71.8%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2408

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-05-13) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.835
Intercept4.383
R^20.914
P-value4.78e-54
Pearson r0.956
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words41.270
Mean absolute length residual4.523
Mean absolute percent error9.4%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.340, R^2 = 0.116, p = 0.0005.
chrF++ vs passage length
chrF++: significant negative; r = -0.365, R^2 = 0.133, p = 0.0002.
METEOR vs passage length
METEOR: significant negative; r = -0.428, R^2 = 0.183, p = 8.81e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.436, R^2 = 0.190, p = 5.73e-06.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.288, R^2 = 0.083, p = 0.0036.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.312, R^2 = 0.097, p = 0.0016.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.304, R^2 = 0.092, p = 0.0021.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.303, R^2 = 0.092, p = 0.0022.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.340 0.116 0.0005 -0.00200 ok
chrF++ 100 significant negative -0.365 0.133 0.0002 -0.00153 ok
METEOR 100 significant negative -0.428 0.183 8.81e-06 -0.00225 ok
ROUGE-L 100 significant negative -0.436 0.190 5.73e-06 -0.00207 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.288 0.083 0.0036 -0.00199 ok
Trigram recall 100 significant negative -0.312 0.097 0.0016 -0.00204 ok
Trigram F1 100 significant negative -0.304 0.092 0.0021 -0.00203 ok
Trigram Jaccard 100 significant negative -0.303 0.092 0.0022 -0.00173 ok

Translation Similarity Metrics

Mean BLEU-439.7%
Mean chrF++64.3%
Mean METEOR69.5%
Mean ROUGE-L71.8%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU38.6%
Unigram F172.3%
Bigram F146.3%
Trigram precision33.7%
Trigram recall31.4%
Trigram F132.5%
Trigram Jaccard19.4%
4-gram F123.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 32 -87.656 0.4% 13.9% 11.2% 22.4% N/A N/A N/A reviewed/approved 13733 gpt-4o-2024-05-13
3288 Κορώνεια 110 116 19.733 24.3% 58.5% 58.1% 64.9% N/A N/A N/A reviewed/approved 13644 gpt-4o-2024-05-13
2328 Καλλίπολις 43 55 14.699 26.1% 55.7% 57.6% 55.1% N/A N/A N/A reviewed/approved 13533 gpt-4o-2024-05-13
2604 Καρχηδών 117 116 13.886 28.7% 57.9% 63.4% 63.8% N/A N/A N/A reviewed/approved 13596 gpt-4o-2024-05-13
7260 Κυχρεῖος πάγος 61 68 12.663 35.2% 59.3% 64.8% 58.9% N/A N/A N/A reviewed/approved 13710 gpt-4o-2024-05-13
2346 Κάναι 77 80 11.298 45.0% 62.7% 73.5% 77.7% N/A N/A N/A reviewed/approved 13563 gpt-4o-2024-05-13
2116 Κάληρος 40 27 -10.795 10.4% 46.7% 44.6% 53.7% N/A N/A N/A reviewed/approved 13512 gpt-4o-2024-05-13
2626 Κατάνη 95 94 10.263 37.3% 61.9% 63.9% 63.5% N/A N/A N/A reviewed/approved 13629 gpt-4o-2024-05-13
2342 Κάμιρος 54 58 8.511 39.2% 66.6% 72.8% 73.2% N/A N/A N/A reviewed/approved 13557 gpt-4o-2024-05-13
7259 Κύφος 54 58 8.511 42.0% 70.4% 74.2% 73.2% N/A N/A N/A reviewed/approved 13707 gpt-4o-2024-05-13
2603 Κάρυστος 175 159 8.438 29.3% 57.8% 53.7% 59.3% N/A N/A N/A reviewed/approved 13593 gpt-4o-2024-05-13
2465 Κάνωπος 83 81 7.286 39.1% 67.0% 79.7% 78.0% N/A N/A N/A reviewed/approved 13569 gpt-4o-2024-05-13
7255 Κυτέριον 22 30 7.241 28.6% 57.2% 51.4% 53.8% N/A N/A N/A reviewed/approved 13695 gpt-4o-2024-05-13
2115 Καλὴ ἀκτή 64 65 7.157 27.0% 49.3% 54.8% 54.3% N/A N/A N/A reviewed/approved 13509 gpt-4o-2024-05-13
2597 Καρπασία 112 105 7.062 37.2% 64.5% 62.6% 68.2% N/A N/A N/A reviewed/approved 13584 gpt-4o-2024-05-13
2605 Κάσιον 56 58 6.840 45.3% 71.3% 79.9% 73.7% N/A N/A N/A reviewed/approved 13599 gpt-4o-2024-05-13
2455 Κάναστρον 60 48 -6.501 28.3% 55.8% 57.4% 68.5% N/A N/A N/A reviewed/approved 13566 gpt-4o-2024-05-13
2607 Κάσος 57 58 6.005 34.0% 64.8% 78.1% 76.5% N/A N/A N/A reviewed/approved 13602 gpt-4o-2024-05-13
2470 Καππαδοκία 68 67 5.816 27.9% 59.6% 80.4% 77.0% N/A N/A N/A reviewed/approved 13575 gpt-4o-2024-05-13
7266 Κώμη 79 76 5.628 12.0% 44.1% 42.0% 43.9% N/A N/A N/A reviewed/approved 13728 gpt-4o-2024-05-13
2599 Καρπήσιοι 11 8 -5.571 43.0% 58.7% 55.0% 63.2% N/A N/A N/A reviewed/approved 13587 gpt-4o-2024-05-13
2054 Καβαλίς 65 64 5.322 49.3% 65.5% 58.7% 71.3% N/A N/A N/A reviewed/approved 13434 gpt-4o-2024-05-13
2331 Κάλυδνα 43 45 4.699 57.8% 77.7% 83.3% 84.1% N/A N/A N/A reviewed/approved 13542 gpt-4o-2024-05-13
2083 Καλαμένθη 24 20 -4.430 35.1% 56.6% 60.3% 68.2% N/A N/A N/A reviewed/approved 13491 gpt-4o-2024-05-13
2326 Καλλίαρος 47 48 4.358 38.5% 65.5% 73.2% 71.6% N/A N/A N/A reviewed/approved 13527 gpt-4o-2024-05-13
7264 Κωλιάς 61 51 -4.337 28.2% 56.1% 55.4% 61.9% N/A N/A N/A reviewed/approved 13722 gpt-4o-2024-05-13
2119 Κάλλατις 65 63 4.322 36.4% 59.9% 60.1% 64.1% N/A N/A N/A reviewed/approved 13521 gpt-4o-2024-05-13
7245 Κύρης 13 11 -4.242 33.9% 57.6% 77.8% 83.3% N/A N/A N/A reviewed/approved 13665 gpt-4o-2024-05-13
3530 Κριώα 26 22 -4.101 38.3% 63.6% 69.9% 75.0% N/A N/A N/A reviewed/approved 13653 gpt-4o-2024-05-13
2075 Καινίνη 15 13 -3.912 68.9% 82.3% 87.7% 92.9% N/A N/A N/A reviewed/approved 13467 gpt-4o-2024-05-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.154.

Positive termCoefficientDocsMean presentMean absent
και 0.9159 66 0.491 -0.953
απο 0.7661 27 5.091 -1.883
εστι 0.5262 24 3.733 -1.179
δε και 0.4585 8 6.870 -0.597
πολις 0.4100 68 1.690 -3.591
ου 0.4063 14 4.615 -0.751
πολιτης 0.3814 18 4.478 -0.983
την 0.3794 13 -1.303 0.195
οι 0.3712 11 5.388 -0.666
παρα 0.3539 7 5.036 -0.379
των 0.3407 17 4.246 -0.870
εστι δε 0.3387 5 8.708 -0.458
δευτερα 0.3347 3 12.444 -0.385
τε 0.3259 5 5.338 -0.281
τη 0.3153 15 3.097 -0.547
εν τη 0.3085 7 5.117 -0.385
εστι και 0.3070 20 3.290 -0.822
βοιωτιας 0.2992 3 7.620 -0.236
εστι δε και 0.2946 3 12.106 -0.374
δυο 0.2891 4 7.604 -0.317
Negative termCoefficientDocsMean presentMean absent
γαρ -1.5212 9 -7.202 0.712
δια του -0.9952 10 -5.902 0.656
του -0.7775 37 1.242 -0.729
δια -0.6464 16 -1.113 0.212
ηρωδιανος -0.5786 3 -30.722 0.950
εξ -0.5252 3 -28.490 0.881
ζευς -0.4651 3 -26.091 0.807
ωστε -0.4468 3 -25.828 0.799
αι -0.4250 3 -26.137 0.808
το -0.4168 70 0.024 -0.056
το εθνικον -0.3738 56 -1.161 1.478
εθνικον -0.3738 56 -1.161 1.478
τα -0.3389 12 -5.008 0.683
φησιν -0.3125 10 -7.079 0.787
ειναι -0.2897 8 -8.957 0.779
δε της -0.2867 6 -9.788 0.625
περι -0.2476 11 -6.138 0.759
τινες -0.2147 6 -3.301 0.211
εις -0.1973 7 -7.704 0.580
εθνος -0.1969 8 -1.307 0.114

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.901.

Positive termCoefficientDocsMean presentMean absent
γαρ 5.3806 9 13.667 3.619
του 3.7053 37 7.325 2.878
δια του 3.5968 10 13.382 3.539
δια 3.0537 16 11.083 3.274
την 2.1928 13 12.654 3.308
ος 1.9524 6 21.027 3.470
ζευς 1.9455 3 33.624 3.623
δε 1.9322 46 6.339 2.977
εν 1.8072 37 6.993 3.073
αι 1.7371 3 32.300 3.664
εις 1.7049 7 18.401 3.479
ωστε 1.6419 3 32.609 3.655
δε της 1.6351 6 20.029 3.534
ηρωδιανος 1.6328 3 30.722 3.713
εξ 1.5262 3 30.791 3.711
τα 1.4454 12 12.411 3.448
φησιν 1.2215 10 13.085 3.572
περι 1.1295 11 12.533 3.533
ειναι 0.8526 8 13.275 3.762
τω 0.8439 16 9.882 3.503
Negative termCoefficientDocsMean presentMean absent
ως -1.5610 52 3.515 5.616
το εθνικον -0.8898 56 4.665 4.343
εθνικον -0.8898 56 4.665 4.343
χωρα -0.7178 6 2.120 4.677
λυκιας -0.7004 3 1.186 4.627
τους -0.6511 4 2.126 4.623
ης -0.6399 3 0.667 4.643
δευτερω -0.6253 3 2.190 4.596
εθνος -0.6066 8 2.369 4.711
ηροδοτος -0.5942 9 3.077 4.666
και του -0.5889 4 1.762 4.638
καππαδοκιας -0.5877 3 2.319 4.592
οικητωρ -0.5681 6 3.007 4.620
νησος -0.5676 9 3.099 4.664
πολις -0.5673 68 3.928 5.789
φασι -0.5567 8 5.086 4.474
ακροπολις -0.5546 3 1.285 4.624
εστι και -0.5520 20 4.742 4.469
προς τη -0.5324 4 2.597 4.604
φησι -0.5167 12 3.835 4.617

AI English terms predicting large absolute residuals

Features: 305; ridge alpha: 11.938; cross-validated R^2: -0.999.

Positive termCoefficientDocsMean presentMean absent
with 2.4167 17 9.705 3.462
with i' 1.9397 4 26.218 3.619
with an 1.8738 3 30.367 3.724
about 1.7990 4 24.004 3.712
it 1.7902 40 7.132 2.785
says it is 1.6107 6 17.629 3.687
i' 1.5511 7 15.985 3.661
says it 1.5101 7 15.463 3.700
called 1.1959 26 8.295 3.198
says 1.0786 17 9.462 3.512
of the 1.0090 37 7.483 2.785
it is 0.9410 28 8.225 3.084
an 0.8067 23 7.303 3.693
and 0.7465 53 6.160 2.678
a 0.7082 93 4.695 2.238
was called 0.3657 4 9.478 4.317
was 0.3246 17 5.299 4.364
of 0.3134 82 4.778 3.365
who 0.3122 8 6.894 4.317
the 0.2682 94 4.631 2.838
Negative termCoefficientDocsMean presentMean absent
as -0.7359 58 3.629 5.758
as in -0.7239 33 2.735 5.404
ethnonym -0.7001 57 3.116 6.389
the ethnonym -0.6891 56 3.171 6.245
ethnonym is -0.6651 54 3.198 6.080
the ethnonym is -0.6531 53 3.257 5.952
book -0.6332 38 2.962 5.480
a city -0.4848 64 3.861 5.700
a city of -0.4064 42 3.701 5.119
his -0.3392 35 3.772 4.928
book of his -0.3330 15 2.919 4.806
book of -0.3330 15 2.919 4.806
in -0.2878 83 4.856 2.897
name -0.2760 6 1.982 4.686
in book -0.2705 13 2.912 4.764
an island -0.2697 10 3.099 4.682
near -0.2684 20 4.756 4.465
also -0.2555 43 4.586 4.476
of his -0.2532 16 3.528 4.713
are -0.2468 24 4.827 4.428

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 4.125; cross-validated R^2: -0.875.

Positive termCoefficientDocsMean presentMean absent
with 8.0445 16 10.096 3.462
on the 3.7948 10 13.816 3.491
sanctuary 3.4610 3 33.927 3.614
on 3.2009 16 9.952 3.489
forms 3.0321 4 23.863 3.718
the 2.9170 90 4.755 2.435
and 2.1384 52 6.223 2.682
it 1.9700 37 6.974 3.084
zeus 1.6626 5 23.230 3.539
from 1.6088 37 6.382 3.432
was 1.6085 20 9.882 3.184
because 1.5736 12 13.185 3.342
long 1.5529 3 31.733 3.682
one 1.5069 12 12.443 3.443
herodian 1.5052 3 30.722 3.713
diphthong 1.4867 3 32.212 3.667
at 1.4619 6 20.116 3.528
with a 1.4264 4 25.447 3.652
that the 1.4073 6 19.304 3.580
it is with 1.3316 3 30.664 3.715
Negative termCoefficientDocsMean presentMean absent
as -1.7769 60 3.697 5.764
book -1.2721 39 2.952 5.528
as in -1.2393 29 2.981 5.153
ethnonym -1.0686 57 4.617 4.399
the ethnonym is -1.0684 52 4.713 4.317
ethnonym is -1.0684 52 4.713 4.317
the ethnonym -1.0254 56 4.690 4.312
are -0.9295 25 3.792 4.767
near -0.8754 16 3.958 4.631
also -0.8754 43 4.568 4.490
is a -0.8244 31 4.219 4.660
an -0.8148 22 2.598 5.067
which -0.8088 20 3.779 4.709
for -0.7309 11 3.888 4.602
sea -0.6975 9 3.121 4.662
about -0.6825 5 2.063 4.653
city -0.6739 70 3.920 5.932
among -0.6735 4 2.338 4.614
lykia -0.6604 3 1.186 4.627
in lykia -0.6604 3 1.186 4.627

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.