gpt-5 v2 Evaluation

Generated: 2026-07-25 12:53:09 UTC

Pairs100
Distinct lemmas100
Slope0.821
R^20.968
BLEU-447.7%
chrF++70.0%
METEOR74.5%
ROUGE-L76.2%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2344

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 13:49:25.227272+10:00

Active: True

Notes: Model timeline evaluation: GPT-5 using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.821
Intercept4.469
R^20.968
P-value2.60e-75
Pearson r0.984
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words40.740
Mean absolute length residual3.641
Mean absolute percent error8.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.414, R^2 = 0.171, p = 1.88e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.458, R^2 = 0.210, p = 1.62e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.472, R^2 = 0.223, p = 7.23e-07.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.422, R^2 = 0.178, p = 1.22e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.294, R^2 = 0.087, p = 0.0030.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.330, R^2 = 0.109, p = 0.0008.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.315, R^2 = 0.099, p = 0.0014.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.312, R^2 = 0.098, p = 0.0016.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.414 0.171 1.88e-05 -0.00265 ok
chrF++ 100 significant negative -0.458 0.210 1.62e-06 -0.00180 ok
METEOR 100 significant negative -0.472 0.223 7.23e-07 -0.00223 ok
ROUGE-L 100 significant negative -0.422 0.178 1.22e-05 -0.00168 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.294 0.087 0.0030 -0.00214 ok
Trigram recall 100 significant negative -0.330 0.109 0.0008 -0.00241 ok
Trigram F1 100 significant negative -0.315 0.099 0.0014 -0.00229 ok
Trigram Jaccard 100 significant negative -0.312 0.098 0.0016 -0.00220 ok

Translation Similarity Metrics

Mean BLEU-447.7%
Mean chrF++70.0%
Mean METEOR74.5%
Mean ROUGE-L76.2%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU44.6%
Unigram F177.3%
Bigram F152.9%
Trigram precision40.4%
Trigram recall37.2%
Trigram F138.7%
Trigram Jaccard24.0%
4-gram F129.4%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 189 -37.236 27.0% 51.6% 47.4% 61.4% N/A N/A N/A reviewed/approved 11781 gpt-5-2025-08-07
2603 Κάρυστος 175 169 20.793 37.5% 63.7% 64.1% 68.6% N/A N/A N/A reviewed/approved 11793 gpt-5-2025-08-07
2609 Κάσπειρος 135 127 11.647 42.8% 66.0% 68.5% 66.4% N/A N/A N/A reviewed/approved 11805 gpt-5-2025-08-07
2604 Καρχηδών 117 112 11.432 32.3% 61.4% 62.5% 66.7% N/A N/A N/A reviewed/approved 11796 gpt-5-2025-08-07
2465 Κάνωπος 83 83 10.358 39.0% 65.4% 82.0% 78.3% N/A N/A N/A reviewed/approved 11769 gpt-5-2025-08-07
2056 Καβειρία 112 87 -9.461 28.3% 63.0% 52.4% 62.3% N/A N/A N/A reviewed/approved 11640 gpt-5-2025-08-07
7254 Κύτα 84 82 8.537 45.1% 66.0% 64.8% 72.3% N/A N/A N/A reviewed/approved 11892 gpt-5-2025-08-07
2605 Κάσιον 56 59 8.535 42.1% 68.1% 73.9% 69.6% N/A N/A N/A reviewed/approved 11799 gpt-5-2025-08-07
2470 Καππαδοκία 68 68 7.679 28.7% 61.1% 77.1% 76.5% N/A N/A N/A reviewed/approved 11775 gpt-5-2025-08-07
7266 Κώμη 79 77 7.644 20.4% 48.6% 48.6% 48.7% N/A N/A N/A reviewed/approved 11928 gpt-5-2025-08-07
2079 Καισάρεια 39 29 -7.502 29.1% 51.4% 55.2% 64.7% N/A N/A N/A reviewed/approved 11679 gpt-5-2025-08-07
2083 Καλαμένθη 24 17 -7.181 32.1% 54.9% 49.1% 63.4% N/A N/A N/A reviewed/approved 11691 gpt-5-2025-08-07
2330 Καλύβη 24 17 -7.181 38.6% 58.3% 57.5% 68.3% N/A N/A N/A reviewed/approved 11739 gpt-5-2025-08-07
2119 Κάλλατις 65 65 7.143 48.0% 69.0% 73.2% 70.8% N/A N/A N/A reviewed/approved 11721 gpt-5-2025-08-07
2342 Κάμιρος 54 55 6.178 55.5% 74.7% 79.7% 80.7% N/A N/A N/A reviewed/approved 11757 gpt-5-2025-08-07
2599 Καρπήσιοι 11 8 -5.504 32.2% 65.5% 55.0% 63.2% N/A N/A N/A reviewed/approved 11787 gpt-5-2025-08-07
7255 Κυτέριον 22 28 5.461 46.7% 74.2% 64.6% 72.0% N/A N/A N/A reviewed/approved 11895 gpt-5-2025-08-07
7260 Κυχρεῖος πάγος 61 60 5.428 42.6% 65.5% 72.7% 67.8% N/A N/A N/A reviewed/approved 11910 gpt-5-2025-08-07
2116 Κάληρος 40 32 -5.323 20.4% 51.9% 55.1% 58.3% N/A N/A N/A reviewed/approved 11712 gpt-5-2025-08-07
2346 Κάναι 77 73 5.286 58.3% 74.8% 85.7% 86.7% N/A N/A N/A reviewed/approved 11763 gpt-5-2025-08-07
2328 Καλλίπολις 43 45 5.213 50.0% 68.7% 73.8% 68.2% N/A N/A N/A reviewed/approved 11733 gpt-5-2025-08-07
3288 Κορώνεια 110 100 5.182 32.1% 62.8% 62.3% 71.1% N/A N/A N/A reviewed/approved 11844 gpt-5-2025-08-07
7259 Κύφος 54 54 5.178 42.5% 69.1% 77.1% 70.4% N/A N/A N/A reviewed/approved 11907 gpt-5-2025-08-07
2114 Κάλβιος 18 15 -4.253 30.9% 58.2% 65.3% 66.7% N/A N/A N/A reviewed/approved 11706 gpt-5-2025-08-07
2628 Καταονία 24 20 -4.181 42.8% 65.4% 61.5% 68.2% N/A N/A N/A reviewed/approved 11835 gpt-5-2025-08-07
2054 Καβαλίς 65 62 4.143 61.7% 75.0% 75.8% 85.0% N/A N/A N/A reviewed/approved 11634 gpt-5-2025-08-07
2059 Καδμεία 20 25 4.104 38.1% 70.5% 71.7% 75.6% N/A N/A N/A reviewed/approved 11649 gpt-5-2025-08-07
2622 Καστωλοῦ πεδίον 42 43 4.034 44.2% 68.9% 82.2% 77.6% N/A N/A N/A reviewed/approved 11814 gpt-5-2025-08-07
7246 Κύρις 20 17 -3.896 67.0% 80.8% 85.2% 86.5% N/A N/A N/A reviewed/approved 11868 gpt-5-2025-08-07
3496 Κοτιάειον 71 59 -3.785 29.1% 51.1% 55.8% 58.5% N/A N/A N/A reviewed/approved 11847 gpt-5-2025-08-07

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 2.894; cross-validated R^2: -0.121.

Positive termCoefficientDocsMean presentMean absent
οι 2.7018 11 5.822 -0.720
απο 2.4809 27 2.550 -0.943
και 2.4596 66 0.750 -1.457
τε 2.3690 5 6.079 -0.320
ου 2.1563 14 4.065 -0.662
και το 2.1039 9 5.853 -0.579
αφ ου 1.6411 5 5.510 -0.290
εκληθη 1.5976 4 9.698 -0.404
ος 1.5555 6 4.579 -0.292
εν τη 1.5423 7 4.560 -0.343
εν 1.4800 37 1.797 -1.055
αφ 1.4627 6 5.275 -0.337
τον 1.3484 10 3.022 -0.336
τη 1.3158 15 1.684 -0.297
εστι δε και 1.2297 3 6.275 -0.194
υπο 1.2114 4 6.322 -0.263
εκληθη δε 1.2084 3 11.121 -0.344
δυο 1.1875 4 5.241 -0.218
πολιτης 1.1500 18 3.065 -0.673
απο του 1.1069 9 4.064 -0.402
Negative termCoefficientDocsMean presentMean absent
δια -1.8963 16 0.483 -0.092
εθνος -1.8876 8 -6.330 0.550
το -1.6691 70 0.305 -0.711
περι -1.6336 11 -3.081 0.381
οικητωρ -1.4018 6 -6.244 0.399
το θηλυκον -1.3981 6 -5.272 0.336
τω περι -1.3167 4 -8.034 0.335
εν τω περι -1.3167 4 -8.034 0.335
θρακης -1.2479 5 -3.045 0.160
καππαδοκιας -1.1087 3 -3.227 0.100
ηρωδιανος -1.0859 3 -13.804 0.427
το εθνικον -1.0355 56 -0.423 0.539
εθνικον -1.0355 56 -0.423 0.539
πολις θρακης -0.9398 4 -2.869 0.120
τινες -0.9343 6 -2.454 0.157
εστι και -0.9332 20 0.581 -0.145
ως ελλανικος -0.9250 3 -12.257 0.379
κατα -0.9173 10 -1.710 0.190
φησιν -0.9158 10 -2.484 0.276
προς τη -0.9143 4 -2.387 0.099

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.238.

Positive termCoefficientDocsMean presentMean absent
και 1.6063 66 4.249 2.460
δια 1.4054 16 7.518 2.903
την 1.2685 13 8.489 2.917
τη 1.2578 15 8.174 2.841
απο 1.2266 27 6.697 2.511
απο της 1.0983 6 13.553 3.008
δε 1.0800 46 4.835 2.624
γαρ 1.0665 9 10.452 2.967
εστι 1.0203 24 5.850 2.944
εκαλειτο δε 0.9781 3 23.154 3.038
αφ 0.8555 6 8.429 3.335
τω 0.8501 16 6.468 3.103
εκαλειτο δε και 0.8396 3 23.154 3.038
αφ ου 0.8041 5 9.294 3.344
εν 0.7937 37 4.804 2.958
περι 0.7892 11 6.736 3.259
δε και 0.7822 8 11.294 2.976
εις 0.7644 7 11.795 3.027
θηλυκον 0.7392 12 9.267 2.874
λεγεται και 0.7179 8 11.450 2.962
Negative termCoefficientDocsMean presentMean absent
το εθνικον -0.6176 56 3.848 3.378
εθνικον -0.6176 56 3.848 3.378
τον -0.4915 10 3.658 3.639
δευτερω -0.4904 3 1.299 3.714
νησος -0.4714 9 2.703 3.734
πολις -0.4472 68 3.915 3.059
πλησιον -0.4459 8 2.577 3.734
πολιτης -0.4231 18 3.860 3.593
του δε -0.4063 3 1.321 3.713
τους -0.3926 4 2.383 3.694
προς τη -0.3873 4 2.671 3.682
μια -0.3813 3 1.714 3.701
ειναι -0.3483 8 2.656 3.727
ουδετερως -0.3408 3 0.764 3.730
ην -0.3347 13 3.176 3.711
προς -0.3095 15 4.140 3.553
εκ -0.3055 6 3.137 3.673
πολυβιος -0.3024 5 1.715 3.742
πορρω -0.2913 3 1.205 3.716
ου πορρω -0.2913 3 1.205 3.716

AI English terms predicting large absolute residuals

Features: 272; ridge alpha: 11.938; cross-validated R^2: -0.329.

Positive termCoefficientDocsMean presentMean absent
and 0.7877 56 4.549 2.486
the 0.7607 92 3.685 3.133
also 0.7262 40 5.077 2.684
with 0.6846 16 6.842 3.031
there 0.6224 33 5.690 2.632
by 0.5733 14 7.848 2.956
was 0.5408 18 6.789 2.950
on 0.5212 16 6.151 3.163
says with 0.5076 3 13.494 3.336
was called 0.4386 5 12.285 3.186
and the 0.4327 16 7.824 2.844
for 0.4222 14 8.283 2.885
when 0.4154 4 16.544 3.103
is also 0.4020 26 5.187 3.098
there is 0.3968 23 5.740 3.014
there is also 0.3944 22 5.910 3.001
his on 0.3832 3 14.924 3.292
usage 0.3817 3 15.698 3.268
is 0.3748 89 3.621 3.805
zeus 0.3611 4 17.963 3.044
Negative termCoefficientDocsMean presentMean absent
the ethnonym -0.3059 59 3.757 3.475
ethnonym -0.3059 59 3.757 3.475
the ethnonym is -0.2825 56 3.849 3.376
ethnonym is -0.2825 56 3.849 3.376
an -0.2310 23 3.129 3.794
a city -0.2216 65 3.986 3.001
city of -0.2172 47 3.261 3.978
a city of -0.2172 47 3.261 3.978
in book -0.2166 17 4.247 3.517
book -0.1917 39 4.051 3.379
an island -0.1881 10 2.715 3.744
one -0.1770 7 2.322 3.740
city -0.1701 68 3.915 3.059
in book of -0.1689 16 4.423 3.492
book says -0.1658 3 0.310 3.744
according to -0.1523 5 2.299 3.712
according -0.1523 5 2.299 3.712
neuter -0.1509 4 1.180 3.744
as in -0.1466 34 3.730 3.595
book of his -0.1429 15 4.677 3.458

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 5.878; cross-validated R^2: -0.222.

Positive termCoefficientDocsMean presentMean absent
the 1.5310 90 3.745 2.703
for 1.1724 11 9.409 2.928
also 1.1500 43 5.013 2.606
is with 1.0692 4 13.685 3.223
there 1.0171 34 5.429 2.720
with 0.9140 16 6.612 3.075
is also 0.8595 31 5.654 2.737
is 0.7776 87 3.705 3.215
usage 0.7640 4 11.692 3.306
when 0.7617 7 10.188 3.148
there is 0.7525 24 5.730 2.982
and 0.7250 52 4.676 2.520
to 0.7175 31 5.374 2.863
used to be 0.6939 6 13.250 3.028
feminine is 0.6903 12 9.741 2.809
with the 0.6893 5 11.005 3.254
according 0.6875 5 12.685 3.165
according to 0.6875 5 12.685 3.165
after 0.6724 22 6.570 2.815
named 0.6704 5 9.023 3.358
Negative termCoefficientDocsMean presentMean absent
as in -0.6932 29 3.641 3.641
the ethnonym is -0.5890 52 3.468 3.828
ethnonym is -0.5890 52 3.468 3.828
ethnonym -0.4707 57 3.837 3.382
the ethnonym -0.4559 56 3.869 3.351
a city -0.4533 64 3.911 3.161
a -0.4240 91 3.714 2.903
in book -0.3924 16 4.320 3.512
in book of -0.3780 14 2.255 3.867
one of -0.3491 5 1.472 3.755
near -0.3327 16 4.084 3.557
a city in -0.3309 40 3.972 3.420
territory -0.3229 5 1.945 3.730
one of the -0.3126 4 1.411 3.734
book -0.3111 39 4.051 3.379
island -0.2929 14 3.139 3.723
is a -0.2881 31 5.075 2.997
of his -0.2871 18 4.146 3.530
city in -0.2759 41 4.084 3.334
was from -0.2699 4 1.652 3.724

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.