gpt-4o-2024-05-13 v2 Evaluation

Generated: 2026-09-20 11:20:14 UTC

Pairs100
Distinct lemmas100
Slope0.835
R^20.914
BLEU-439.8%
chrF++64.4%
METEOR69.7%
ROUGE-L72.0%
BERTScore95.2%
COMET77.4%
BLEURT71.2%

Profile version id: 2408

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4o (2024-05-13) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.835
Intercept4.398
R^20.914
P-value4.59e-54
Pearson r0.956
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words41.270
Mean absolute length residual4.517
Mean absolute percent error9.3%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.344, R^2 = 0.119, p = 0.0004.
chrF++ vs passage length
chrF++: significant negative; r = -0.371, R^2 = 0.138, p = 0.0001.
METEOR vs passage length
METEOR: significant negative; r = -0.440, R^2 = 0.194, p = 4.64e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.446, R^2 = 0.199, p = 3.37e-06.
BERTScore vs passage length
BERTScore: significant negative; r = -0.560, R^2 = 0.313, p = 1.40e-09.
COMET vs passage length
COMET: significant negative; r = -0.389, R^2 = 0.152, p = 6.20e-05.
BLEURT vs passage length
BLEURT: significant negative; r = -0.709, R^2 = 0.502, p = 1.57e-16.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.295, R^2 = 0.087, p = 0.0028.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.320, R^2 = 0.102, p = 0.0012.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.312, R^2 = 0.097, p = 0.0016.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.310, R^2 = 0.096, p = 0.0017.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.344 0.119 0.0004 -0.00203 ok
chrF++ 100 significant negative -0.371 0.138 0.0001 -0.00156 ok
METEOR 100 significant negative -0.440 0.194 4.64e-06 -0.00230 ok
ROUGE-L 100 significant negative -0.446 0.199 3.37e-06 -0.00212 ok
BERTScore 100 significant negative -0.560 0.313 1.40e-09 -0.00051 ok
COMET 100 significant negative -0.389 0.152 6.20e-05 -0.00084 ok
BLEURT 100 significant negative -0.709 0.502 1.57e-16 -0.00260 ok
Trigram precision 100 significant negative -0.295 0.087 0.0028 -0.00204 ok
Trigram recall 100 significant negative -0.320 0.102 0.0012 -0.00208 ok
Trigram F1 100 significant negative -0.312 0.097 0.0016 -0.00208 ok
Trigram Jaccard 100 significant negative -0.310 0.096 0.0017 -0.00176 ok

Translation Similarity Metrics

Mean BLEU-439.8%
Mean chrF++64.4%
Mean METEOR69.7%
Mean ROUGE-L72.0%
Mean BERTScore95.2%
Mean COMET77.4%
Mean BLEURT71.2%
Legacy smoothed corpus BLEU38.7%
Unigram F172.3%
Bigram F146.4%
Trigram precision33.7%
Trigram recall31.4%
Trigram F132.5%
Trigram Jaccard19.4%
4-gram F123.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 32 -87.650 0.4% 13.9% 11.2% 22.4% 81.4% 47.5% 32.1% reviewed/approved 13733 gpt-4o-2024-05-13
3288 Κορώνεια 110 116 19.734 24.3% 58.5% 58.1% 64.9% 93.3% 76.2% 57.2% reviewed/approved 13644 gpt-4o-2024-05-13
2328 Καλλίπολις 43 55 14.690 26.1% 55.7% 57.6% 55.1% 92.2% 73.7% 61.1% reviewed/approved 13533 gpt-4o-2024-05-13
2604 Καρχηδών 117 116 13.888 28.7% 57.9% 63.4% 63.8% 93.2% 77.9% 63.7% reviewed/approved 13596 gpt-4o-2024-05-13
7260 Κυχρεῖος πάγος 61 68 12.657 35.2% 59.3% 64.8% 58.9% 94.5% 71.8% 60.0% reviewed/approved 13710 gpt-4o-2024-05-13
2346 Κάναι 77 80 11.295 45.0% 62.7% 73.5% 77.7% 94.1% 76.2% 60.5% reviewed/approved 13563 gpt-4o-2024-05-13
2116 Κάληρος 40 27 -10.804 10.4% 46.7% 44.6% 53.7% 92.2% 72.0% 66.4% reviewed/approved 13512 gpt-4o-2024-05-13
2626 Κατάνη 95 94 10.262 37.3% 61.9% 63.9% 63.5% 93.9% 73.5% 62.8% reviewed/approved 13629 gpt-4o-2024-05-13
2342 Κάμιρος 54 58 8.504 39.2% 66.6% 72.8% 73.2% 95.6% 77.6% 63.4% reviewed/approved 13557 gpt-4o-2024-05-13
7259 Κύφος 54 58 8.504 42.0% 70.4% 74.2% 73.2% 95.5% 74.5% 67.1% reviewed/approved 13707 gpt-4o-2024-05-13
2603 Κάρυστος 175 159 8.449 29.3% 57.8% 53.7% 59.3% 93.8% 76.1% 61.8% reviewed/approved 13593 gpt-4o-2024-05-13
2465 Κάνωπος 83 81 7.284 39.1% 67.0% 79.7% 78.0% 94.5% 76.4% 63.8% reviewed/approved 13569 gpt-4o-2024-05-13
7255 Κυτέριον 22 30 7.229 28.6% 57.2% 51.4% 53.8% 94.9% 76.3% 72.3% reviewed/approved 13695 gpt-4o-2024-05-13
2115 Καλὴ ἀκτή 64 65 7.152 27.0% 49.3% 54.8% 54.3% 90.5% 67.4% 50.4% reviewed/approved 13509 gpt-4o-2024-05-13
2597 Καρπασία 112 105 7.064 37.2% 64.5% 62.6% 68.2% 94.8% 75.5% 58.2% reviewed/approved 13584 gpt-4o-2024-05-13
2605 Κάσιον 56 58 6.833 45.3% 71.3% 79.9% 73.7% 95.8% 76.9% 69.5% reviewed/approved 13599 gpt-4o-2024-05-13
2455 Κάναστρον 60 48 -6.507 28.3% 55.8% 57.4% 68.5% 94.0% 68.6% 62.0% reviewed/approved 13566 gpt-4o-2024-05-13
2607 Κάσος 57 58 5.998 34.0% 64.8% 78.1% 76.5% 95.9% 77.0% 77.5% reviewed/approved 13602 gpt-4o-2024-05-13
2470 Καππαδοκία 68 67 5.811 27.9% 59.6% 80.4% 77.0% 94.8% 74.6% 67.5% reviewed/approved 13575 gpt-4o-2024-05-13
7266 Κώμη 79 76 5.625 12.0% 44.1% 42.0% 43.9% 91.7% 72.0% 51.1% reviewed/approved 13728 gpt-4o-2024-05-13
2054 Καβαλίς 65 64 5.317 49.3% 65.5% 58.7% 71.3% 95.1% 74.7% 63.3% reviewed/approved 13434 gpt-4o-2024-05-13
2599 Καρπήσιοι 10 8 -4.749 53.9% 67.5% 70.6% 77.8% 97.7% 79.5% 84.0% reviewed/approved 13587 gpt-4o-2024-05-13
2331 Κάλυδνα 43 45 4.690 57.8% 77.7% 83.3% 84.1% 96.0% 80.9% 75.6% reviewed/approved 13542 gpt-4o-2024-05-13
2083 Καλαμένθη 24 20 -4.441 35.1% 56.6% 60.3% 68.2% 93.6% 74.7% 67.5% reviewed/approved 13491 gpt-4o-2024-05-13
2326 Καλλίαρος 47 48 4.350 38.5% 65.5% 73.2% 71.6% 96.0% 69.5% 64.6% reviewed/approved 13527 gpt-4o-2024-05-13
7264 Κωλιάς 61 51 -4.343 28.2% 56.1% 55.4% 61.9% 94.3% 71.9% 54.2% reviewed/approved 13722 gpt-4o-2024-05-13
2119 Κάλλατις 65 63 4.317 36.4% 59.9% 60.1% 64.1% 94.4% 69.1% 64.3% reviewed/approved 13521 gpt-4o-2024-05-13
7245 Κύρης 13 11 -4.255 33.9% 57.6% 77.8% 83.3% 93.7% 70.0% 74.1% reviewed/approved 13665 gpt-4o-2024-05-13
3530 Κριώα 26 22 -4.112 38.3% 63.6% 69.9% 75.0% 95.6% 75.4% 69.7% reviewed/approved 13653 gpt-4o-2024-05-13
2075 Καινίνη 15 13 -3.925 68.9% 82.3% 87.7% 92.9% 98.8% 87.5% 88.7% reviewed/approved 13467 gpt-4o-2024-05-13

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 24.245; cross-validated R^2: -0.155.

Positive termCoefficientDocsMean presentMean absent
και 0.9118 66 0.484 -0.940
απο 0.7654 27 5.088 -1.882
εστι 0.5255 24 3.730 -1.178
δε και 0.4583 8 6.870 -0.597
πολις 0.4064 68 1.682 -3.575
ου 0.4057 14 4.612 -0.751
πολιτης 0.3802 18 4.472 -0.982
την 0.3795 13 -1.302 0.195
οι 0.3705 11 5.384 -0.665
των 0.3543 17 4.292 -0.879
παρα 0.3537 7 5.036 -0.379
εστι δε 0.3386 5 8.706 -0.458
δευτερα 0.3345 3 12.438 -0.385
τε 0.3258 5 5.334 -0.281
τη 0.3146 15 3.093 -0.546
εν τη 0.3081 7 5.110 -0.385
εστι και 0.3062 20 3.286 -0.822
βοιωτιας 0.2991 3 7.617 -0.236
εστι δε και 0.2945 3 12.104 -0.374
δυο 0.2889 4 7.600 -0.317
Negative termCoefficientDocsMean presentMean absent
γαρ -1.5209 9 -7.198 0.712
δια του -0.9955 10 -5.904 0.656
του -0.7793 37 1.237 -0.726
δια -0.6467 16 -1.115 0.212
ηρωδιανος -0.5787 3 -30.716 0.950
εξ -0.5253 3 -28.493 0.881
ζευς -0.4651 3 -26.082 0.807
ωστε -0.4470 3 -25.826 0.799
αι -0.4253 3 -26.141 0.808
το -0.4200 70 0.017 -0.039
το εθνικον -0.3771 56 -1.169 1.488
εθνικον -0.3771 56 -1.169 1.488
τα -0.3391 12 -5.007 0.683
φησιν -0.3127 10 -7.079 0.787
ειναι -0.2900 8 -8.962 0.779
δε της -0.2868 6 -9.789 0.625
περι -0.2481 11 -6.142 0.759
τινες -0.2153 6 -3.310 0.211
εις -0.1973 7 -7.700 0.580
εθνος -0.1799 8 -1.208 0.105

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.885.

Positive termCoefficientDocsMean presentMean absent
γαρ 5.3808 9 13.665 3.613
του 3.7111 37 7.324 2.869
δια του 3.5973 10 13.379 3.533
δια 3.0543 16 11.081 3.267
την 2.1933 13 12.652 3.302
ος 1.9531 6 21.025 3.464
ζευς 1.9455 3 33.612 3.618
δε 1.9365 46 6.338 2.966
εν 1.8155 37 6.993 3.063
αι 1.7376 3 32.292 3.658
εις 1.7052 7 18.398 3.473
ωστε 1.6420 3 32.607 3.649
δε της 1.6360 6 20.029 3.527
ηρωδιανος 1.6334 3 30.716 3.707
εξ 1.5272 3 30.792 3.705
τα 1.4477 12 12.410 3.441
φησιν 1.2235 10 13.084 3.566
περι 1.1317 11 12.532 3.527
ειναι 0.8537 8 13.273 3.756
τω 0.8458 16 9.880 3.496
Negative termCoefficientDocsMean presentMean absent
ως -1.5513 52 3.516 5.602
το εθνικον -0.8796 56 4.667 4.327
εθνικον -0.8796 56 4.667 4.327
χωρα -0.7164 6 2.115 4.671
λυκιας -0.7003 3 1.186 4.620
εθνος -0.6630 8 2.267 4.713
τους -0.6502 4 2.126 4.617
ης -0.6401 3 0.661 4.637
δευτερω -0.6221 3 2.186 4.590
ηροδοτος -0.5910 9 3.074 4.660
καππαδοκιας -0.5854 3 2.327 4.585
και του -0.5846 4 1.770 4.632
οικητωρ -0.5652 6 3.006 4.614
νησος -0.5646 9 3.102 4.657
πολις -0.5581 68 3.928 5.769
ακροπολις -0.5555 3 1.270 4.618
φασι -0.5537 8 5.091 4.468
εστι και -0.5483 20 4.741 4.461
προς τη -0.5292 4 2.603 4.597
φησι -0.5151 12 3.834 4.611

AI English terms predicting large absolute residuals

Features: 305; ridge alpha: 11.938; cross-validated R^2: -0.985.

Positive termCoefficientDocsMean presentMean absent
with 2.4181 17 9.705 3.455
with i' 1.9403 4 26.217 3.613
with an 1.8738 3 30.358 3.718
about 1.8000 4 24.006 3.705
it 1.7926 40 7.132 2.775
says it is 1.6113 6 17.628 3.681
i' 1.5514 7 15.980 3.655
says it 1.5109 7 15.463 3.694
called 1.1969 26 8.291 3.191
says 1.0800 17 9.462 3.505
of the 1.0112 37 7.482 2.776
it is 0.9423 28 8.223 3.077
an 0.7853 23 7.270 3.695
and 0.7513 53 6.160 2.665
a 0.7152 93 4.697 2.127
was called 0.3666 4 9.480 4.311
was 0.3260 17 5.298 4.358
of 0.3192 82 4.779 3.325
who 0.3125 8 6.893 4.311
the 0.2665 94 4.624 2.850
Negative termCoefficientDocsMean presentMean absent
as -0.7319 58 3.630 5.742
as in -0.7203 33 2.738 5.394
ethnonym -0.6957 57 3.119 6.372
the ethnonym -0.6847 56 3.173 6.228
ethnonym is -0.6606 54 3.200 6.064
the ethnonym is -0.6486 53 3.259 5.936
book -0.6298 38 2.963 5.470
a city -0.4812 64 3.862 5.683
a city of -0.4031 42 3.703 5.108
his -0.3365 35 3.771 4.919
book of his -0.3308 15 2.922 4.799
book of -0.3308 15 2.922 4.799
in -0.2814 83 4.858 2.856
name -0.2758 6 1.979 4.679
in book -0.2690 13 2.914 4.757
an island -0.2677 10 3.103 4.675
near -0.2664 20 4.758 4.457
also -0.2536 43 4.585 4.467
of his -0.2510 16 3.530 4.705
are -0.2453 24 4.826 4.420

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 4.125; cross-validated R^2: -0.860.

Positive termCoefficientDocsMean presentMean absent
with 8.0428 16 10.094 3.455
on the 3.7977 10 13.818 3.484
sanctuary 3.4607 3 33.922 3.608
on 3.2016 16 9.951 3.482
forms 3.0319 4 23.854 3.712
the 2.9107 90 4.748 2.444
and 2.1638 52 6.223 2.669
it 1.9712 37 6.976 3.073
zeus 1.6620 5 23.224 3.533
was 1.6491 20 9.882 3.176
from 1.6123 37 6.383 3.422
because 1.5971 12 13.185 3.335
long 1.5520 3 31.722 3.676
one 1.5123 12 12.442 3.437
herodian 1.5060 3 30.716 3.707
diphthong 1.4852 3 32.202 3.661
at 1.4757 6 20.115 3.522
with a 1.4274 4 25.448 3.645
that the 1.4084 6 19.308 3.573
it is with 1.3323 3 30.657 3.709
Negative termCoefficientDocsMean presentMean absent
as -1.7633 60 3.698 5.746
book -1.2635 39 2.953 5.518
as in -1.2358 29 2.982 5.145
ethnonym -1.0543 57 4.619 4.382
the ethnonym is -1.0530 52 4.715 4.303
ethnonym is -1.0530 52 4.715 4.303
the ethnonym -1.0105 56 4.692 4.295
are -0.9255 25 3.790 4.760
also -0.8628 43 4.567 4.480
near -0.8561 16 3.960 4.624
an -0.8516 22 2.562 5.069
is a -0.8217 31 4.219 4.652
which -0.7999 20 3.782 4.701
for -0.7328 11 3.886 4.595
sea -0.6895 9 3.129 4.655
about -0.6795 5 2.069 4.646
among -0.6752 4 2.329 4.609
city -0.6679 70 3.920 5.911
lykia -0.6601 3 1.186 4.620
in lykia -0.6601 3 1.186 4.620

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.