gpt-4-turbo-2024-04-09 v2 Evaluation

Generated: 2026-09-20 11:20:51 UTC

Pairs100
Distinct lemmas100
Slope0.887
R^20.982
BLEU-440.1%
chrF++64.5%
METEOR68.9%
ROUGE-L70.8%
BERTScore94.9%
COMET77.8%
BLEURT71.4%

Profile version id: 2405

First translation using this version: 2026-07-16

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-16 06:54:03.500756+10:00

Active: True

Notes: Model timeline evaluation: GPT-4 Turbo (2024-04-09) using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.887
Intercept3.092
R^20.982
P-value8.41e-87
Pearson r0.991
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.270
Mean absolute length residual3.349
Mean absolute percent error7.9%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.460, R^2 = 0.211, p = 1.50e-06.
chrF++ vs passage length
chrF++: significant negative; r = -0.445, R^2 = 0.198, p = 3.49e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.502, R^2 = 0.252, p = 1.03e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.501, R^2 = 0.251, p = 1.09e-07.
BERTScore vs passage length
BERTScore: significant negative; r = -0.617, R^2 = 0.380, p = 8.38e-12.
COMET vs passage length
COMET: significant negative; r = -0.464, R^2 = 0.215, p = 1.19e-06.
BLEURT vs passage length
BLEURT: significant negative; r = -0.758, R^2 = 0.575, p = 6.48e-20.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.410, R^2 = 0.168, p = 2.25e-05.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.421, R^2 = 0.178, p = 1.26e-05.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.417, R^2 = 0.174, p = 1.61e-05.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.397, R^2 = 0.158, p = 4.35e-05.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.460 0.211 1.50e-06 -0.00306 ok
chrF++ 100 significant negative -0.445 0.198 3.49e-06 -0.00195 ok
METEOR 100 significant negative -0.502 0.252 1.03e-07 -0.00282 ok
ROUGE-L 100 significant negative -0.501 0.251 1.09e-07 -0.00247 ok
BERTScore 100 significant negative -0.617 0.380 8.38e-12 -0.00055 ok
COMET 100 significant negative -0.464 0.215 1.19e-06 -0.00090 ok
BLEURT 100 significant negative -0.758 0.575 6.48e-20 -0.00300 ok
Trigram precision 100 significant negative -0.410 0.168 2.25e-05 -0.00324 ok
Trigram recall 100 significant negative -0.421 0.178 1.26e-05 -0.00332 ok
Trigram F1 100 significant negative -0.417 0.174 1.61e-05 -0.00327 ok
Trigram Jaccard 100 significant negative -0.397 0.158 4.35e-05 -0.00286 ok

Translation Similarity Metrics

Mean BLEU-440.1%
Mean chrF++64.5%
Mean METEOR68.9%
Mean ROUGE-L70.8%
Mean BERTScore94.9%
Mean COMET77.8%
Mean BLEURT71.4%
Legacy smoothed corpus BLEU36.9%
Unigram F169.5%
Bigram F143.6%
Trigram precision31.1%
Trigram recall29.7%
Trigram F130.4%
Trigram Jaccard17.9%
4-gram F121.9%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2468 Καπετώλιον 138 103 -22.550 7.9% 36.9% 31.3% 39.8% 86.3% 67.9% 47.8% reviewed/approved 12373 gpt-4-turbo-2024-04-09
3288 Κορώνεια 110 117 16.296 26.6% 60.4% 61.7% 61.4% 93.6% 74.6% 63.0% reviewed/approved 12445 gpt-4-turbo-2024-04-09
2484 Καρία 270 229 -13.684 20.3% 49.4% 44.4% 50.9% 88.4% 76.9% 38.0% reviewed/approved 12382 gpt-4-turbo-2024-04-09
2342 Κάμιρος 54 62 10.989 31.0% 62.0% 65.3% 63.8% 95.1% 76.9% 62.8% reviewed/approved 12358 gpt-4-turbo-2024-04-09
2054 Καβαλίς 65 70 9.228 42.9% 65.6% 63.2% 71.1% 93.6% 74.4% 62.7% reviewed/approved 12235 gpt-4-turbo-2024-04-09
2116 Κάληρος 40 30 -8.587 16.2% 45.7% 39.3% 45.7% 91.1% 68.4% 52.9% reviewed/approved 12313 gpt-4-turbo-2024-04-09
2465 Κάνωπος 83 85 8.255 25.4% 59.7% 66.6% 67.9% 93.5% 73.7% 65.3% reviewed/approved 12370 gpt-4-turbo-2024-04-09
2605 Κάσιον 56 61 8.215 38.2% 65.3% 71.0% 66.7% 94.6% 73.5% 67.1% reviewed/approved 12400 gpt-4-turbo-2024-04-09
7244 Κυρήνη 54 59 7.989 36.4% 61.4% 66.0% 65.5% 93.2% 76.3% 70.7% reviewed/approved 12463 gpt-4-turbo-2024-04-09
2626 Κατάνη 95 80 -7.393 26.1% 55.0% 48.5% 64.0% 92.8% 72.6% 57.5% reviewed/approved 12430 gpt-4-turbo-2024-04-09
7254 Κύτα 84 85 7.368 25.9% 56.1% 53.1% 60.4% 92.1% 75.1% 57.3% reviewed/approved 12493 gpt-4-turbo-2024-04-09
2604 Καρχηδών 117 114 7.085 17.4% 52.0% 49.7% 51.7% 92.6% 75.7% 55.0% reviewed/approved 12397 gpt-4-turbo-2024-04-09
2603 Κάρυστος 175 165 6.617 33.0% 61.4% 61.7% 65.3% 93.4% 75.5% 60.3% reviewed/approved 12394 gpt-4-turbo-2024-04-09
2470 Καππαδοκία 68 70 6.566 21.3% 57.2% 78.0% 71.0% 95.3% 75.7% 66.9% reviewed/approved 12376 gpt-4-turbo-2024-04-09
2455 Κάναστρον 60 50 -6.335 23.3% 56.3% 57.3% 58.2% 94.5% 74.3% 67.7% reviewed/approved 12367 gpt-4-turbo-2024-04-09
7264 Κωλιάς 61 51 -6.222 33.2% 62.0% 60.5% 71.4% 95.6% 72.2% 60.9% reviewed/approved 12523 gpt-4-turbo-2024-04-09
2609 Κάσπειρος 135 129 6.112 36.4% 61.8% 54.0% 63.6% 89.6% 72.2% 64.4% reviewed/approved 12406 gpt-4-turbo-2024-04-09
7255 Κυτέριον 22 28 5.385 28.6% 57.1% 59.3% 64.0% 94.0% 72.8% 71.2% reviewed/approved 12496 gpt-4-turbo-2024-04-09
2346 Κάναι 77 76 4.580 31.1% 54.6% 68.2% 70.1% 93.6% 74.5% 62.6% reviewed/approved 12364 gpt-4-turbo-2024-04-09
2057 Καβελλιών 34 29 -4.263 53.7% 75.1% 80.3% 85.7% 97.5% 80.4% 78.9% reviewed/approved 12244 gpt-4-turbo-2024-04-09
7267 Κωνώπη 64 64 4.116 42.3% 65.6% 66.5% 68.2% 95.7% 74.8% 69.3% reviewed/approved 12532 gpt-4-turbo-2024-04-09
7259 Κύφος 54 55 3.989 34.2% 64.4% 71.3% 71.6% 95.5% 73.9% 64.6% reviewed/approved 12508 gpt-4-turbo-2024-04-09
7252 Κύρτωνες 27 31 3.949 8.6% 47.7% 48.2% 48.3% 92.6% 76.3% 64.8% reviewed/approved 12487 gpt-4-turbo-2024-04-09
7253 Κυρτώνιος 18 23 3.935 50.2% 74.3% 84.2% 73.2% 97.1% 82.1% 84.2% reviewed/approved 12490 gpt-4-turbo-2024-04-09
7250 Κυρταία 40 35 -3.587 46.4% 64.6% 63.6% 66.7% 95.8% 71.9% 67.6% reviewed/approved 12481 gpt-4-turbo-2024-04-09
7248 Κύρου πόλις 32 35 3.512 21.3% 57.8% 63.2% 62.7% 94.6% 77.2% 74.2% reviewed/approved 12475 gpt-4-turbo-2024-04-09
7247 Κύρνος 50 44 -3.461 18.3% 51.3% 58.0% 55.3% 93.6% 74.6% 69.9% reviewed/approved 12472 gpt-4-turbo-2024-04-09
2081 Καλάθη 49 50 3.426 45.9% 71.2% 84.3% 82.8% 95.3% 74.8% 73.5% reviewed/approved 12286 gpt-4-turbo-2024-04-09
2075 Καινίνη 15 13 -3.403 79.5% 88.0% 87.7% 92.9% 99.3% 88.6% 92.5% reviewed/approved 12268 gpt-4-turbo-2024-04-09
2802 Κάδρεμα 24 21 -3.389 37.2% 58.7% 61.3% 66.7% 96.0% 75.1% 74.5% reviewed/approved 12259 gpt-4-turbo-2024-04-09

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: 0.006.

Positive termCoefficientDocsMean presentMean absent
και 1.8525 66 0.670 -1.300
οι 1.3795 11 4.953 -0.612
απο 1.2659 27 2.311 -0.855
ου 1.2285 14 3.559 -0.579
βοιωτιας 1.0421 3 7.255 -0.224
πολιτης 1.0117 18 3.567 -0.783
τε 0.9872 5 3.997 -0.210
δε και 0.8012 8 2.870 -0.250
δυναται 0.7735 3 8.288 -0.256
παρα 0.7713 7 1.902 -0.143
πολις 0.7612 68 0.820 -1.742
παιδος 0.7519 4 5.254 -0.219
εστι 0.7272 24 1.693 -0.535
εστι δε και 0.6774 3 8.377 -0.259
εστι δε 0.6195 5 5.354 -0.282
την 0.6035 13 0.705 -0.105
τη 0.5878 15 1.632 -0.288
ιβηριας 0.5843 3 5.078 -0.157
πολις εν 0.5760 5 4.096 -0.216
εν τη 0.5577 7 2.901 -0.218
Negative termCoefficientDocsMean presentMean absent
γαρ -1.7846 9 -2.251 0.223
δια -1.2507 16 -1.173 0.223
περι -1.1211 11 -4.413 0.545
δια του -0.9942 10 -2.363 0.263
του -0.9278 37 0.829 -0.487
οτι -0.8090 5 -1.891 0.100
ηρωδιανος -0.7616 3 -12.912 0.399
οικητωρ -0.7594 6 -4.131 0.264
το -0.7309 70 0.023 -0.053
το εθνικον -0.7130 56 -0.649 0.827
εθνικον -0.7130 56 -0.649 0.827
ζευς -0.5907 3 -10.552 0.326
δε της -0.5729 6 -3.489 0.223
θεοπομπος -0.5600 5 -2.354 0.124
ει -0.5526 4 -5.497 0.229
τα -0.5399 12 -1.794 0.245
εθνος -0.5113 8 -1.907 0.166
φησιν -0.5021 10 -2.343 0.260
τινες -0.4924 6 -2.250 0.144
θρακης -0.4898 5 -2.221 0.117

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.218.

Positive termCoefficientDocsMean presentMean absent
δε 1.5886 46 4.823 2.094
γαρ 1.5610 9 7.450 2.944
την 1.3999 13 7.604 2.713
και 1.1550 66 4.054 1.982
εις 1.0010 7 10.953 2.777
δια του 0.9261 10 6.406 3.010
δια 0.8795 16 5.321 2.974
τα 0.7749 12 7.570 2.774
του 0.7510 37 4.585 2.623
απο 0.6872 27 5.305 2.626
ουτως 0.5870 5 8.806 3.062
βοιωτιας 0.5783 3 7.255 3.228
αι 0.5573 3 11.605 3.094
ζευς 0.5547 3 13.605 3.032
ηρωδιανος 0.5256 3 12.912 3.053
εν 0.4977 37 4.458 2.698
οτι 0.4792 5 5.493 3.236
εκαλειτο 0.4630 7 6.171 3.137
εστι 0.4582 24 4.489 2.989
ος 0.4552 6 8.074 3.048
Negative termCoefficientDocsMean presentMean absent
ως -0.7790 52 3.128 3.589
δευτερω -0.6413 3 0.635 3.433
το εθνικον -0.5037 56 3.484 3.177
εθνικον -0.5037 56 3.484 3.177
κωμη -0.4814 3 1.020 3.421
στραβων -0.4233 15 3.048 3.402
νησος -0.4223 9 2.496 3.434
μια -0.4112 3 1.142 3.417
καππαδοκιας -0.4053 3 1.624 3.403
εκ -0.3400 6 2.170 3.424
τεταρτω -0.3349 3 1.349 3.411
προς τη -0.3334 4 1.922 3.409
διονυσιος -0.3062 4 2.324 3.392
τους -0.3014 4 2.484 3.385
παρα το -0.2908 3 1.613 3.403
πλησιον -0.2758 8 2.981 3.381
ηροδοτος -0.2715 9 4.131 3.272
παυσανιας -0.2349 4 2.040 3.404
ης -0.2289 3 2.235 3.384
τον -0.2252 10 3.116 3.375

AI English terms predicting large absolute residuals

Features: 318; ridge alpha: 5.878; cross-validated R^2: -0.220.

Positive termCoefficientDocsMean presentMean absent
the 1.5375 91 3.455 2.282
for 1.4056 13 5.558 3.019
is 0.9008 89 3.463 2.428
with 0.8866 13 5.597 3.013
and 0.8376 56 4.470 1.922
it 0.7213 37 4.968 2.399
is called 0.6815 7 8.118 2.990
city is 0.6467 3 8.767 3.182
when 0.6404 5 11.316 2.930
in this 0.6390 3 15.154 2.984
that 0.6358 11 6.866 2.914
called 0.6310 32 5.103 2.524
zeus 0.6109 5 11.129 2.940
athens 0.5578 3 12.343 3.071
either 0.5430 4 10.490 3.052
is the 0.5296 3 12.474 3.067
the feminine 0.5184 14 6.111 2.900
herodian 0.4891 3 12.912 3.053
there 0.4770 30 4.764 2.743
in 0.4661 85 3.475 2.637
Negative termCoefficientDocsMean presentMean absent
per -0.4743 15 2.671 3.469
as per -0.4743 15 2.671 3.469
the ethnonym is -0.4639 54 3.407 3.281
ethnonym is -0.4639 54 3.407 3.281
his -0.3972 25 2.995 3.467
city of -0.3681 20 2.854 3.473
the ethnonym -0.3551 57 3.435 3.236
ethnonym -0.3551 57 3.435 3.236
book -0.3536 38 2.929 3.606
an -0.3224 27 3.887 3.150
as -0.3213 58 3.486 3.160
a city of -0.3178 19 2.631 3.518
an island -0.3093 10 2.463 3.448
island -0.3080 14 2.761 3.445
a city -0.2972 64 3.506 3.071
as in -0.2954 34 3.134 3.460
of his -0.2878 13 2.300 3.506
strabo -0.2864 15 3.048 3.402
are called -0.2835 5 3.313 3.351
an inhabitant is -0.2633 7 1.888 3.459

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 5.878; cross-validated R^2: -0.227.

Positive termCoefficientDocsMean presentMean absent
with 1.5995 16 5.174 3.002
the 1.4885 90 3.491 2.068
and 0.9082 52 4.469 2.136
is 0.7722 87 3.545 2.039
sanctuary 0.7599 3 13.007 3.050
on 0.6471 16 5.511 2.937
forms 0.6206 4 11.081 3.027
on the 0.6162 10 5.608 3.098
this 0.5995 15 7.284 2.655
there is also 0.5626 22 5.042 2.872
is with 0.5511 4 11.564 3.007
there is 0.5041 24 4.724 2.915
is also 0.5037 31 4.465 2.848
after 0.5028 22 5.331 2.790
when 0.4980 7 8.294 2.977
after the 0.4910 6 7.265 3.099
there 0.4835 34 4.317 2.851
however 0.4757 10 7.928 2.840
diphthong 0.4667 3 12.904 3.054
it 0.4604 37 4.837 2.475
Negative termCoefficientDocsMean presentMean absent
an -0.6113 22 2.678 3.538
book -0.5387 39 2.857 3.664
as in -0.4416 29 3.196 3.412
the ethnonym is -0.4199 52 3.407 3.287
ethnonym is -0.4199 52 3.407 3.287
ethnonym -0.3627 57 3.476 3.181
per -0.3614 20 2.911 3.459
as per -0.3614 20 2.911 3.459
the ethnonym -0.3594 56 3.481 3.181
as -0.3554 60 3.332 3.375
near -0.3535 16 3.061 3.404
a city in -0.3090 40 3.351 3.348
of his -0.3050 18 2.844 3.460
strabo -0.3006 15 3.048 3.402
island -0.2875 14 2.761 3.445
dionysios -0.2762 4 2.324 3.392
book of his -0.2706 16 2.995 3.417
book of -0.2706 16 2.995 3.417
about -0.2666 5 1.707 3.436
an island -0.2623 9 2.461 3.437

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.