gpt-5.3-chat-latest v2 Evaluation

Generated: 2026-07-25 12:53:33 UTC

Pairs100
Distinct lemmas100
Slope0.889
R^20.984
BLEU-448.1%
chrF++69.8%
METEOR75.2%
ROUGE-L76.2%
BERTScoreN/A
COMETN/A
BLEURTN/A

Profile version id: 2335

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 12:28:27.643960+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.3 Chat using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.889
Intercept3.029
R^20.984
P-value5.01e-90
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.160
Mean AI words42.280
Mean absolute length residual3.182
Mean absolute percent error7.1%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.391, R^2 = 0.152, p = 5.90e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.431, R^2 = 0.186, p = 7.63e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.453, R^2 = 0.205, p = 2.18e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.403, R^2 = 0.162, p = 3.29e-05.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.316, R^2 = 0.100, p = 0.0014.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.346, R^2 = 0.120, p = 0.0004.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.332, R^2 = 0.110, p = 0.0007.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.336, R^2 = 0.113, p = 0.0006.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.391 0.152 5.90e-05 -0.00247 ok
chrF++ 100 significant negative -0.431 0.186 7.63e-06 -0.00171 ok
METEOR 100 significant negative -0.453 0.205 2.18e-06 -0.00206 ok
ROUGE-L 100 significant negative -0.403 0.162 3.29e-05 -0.00164 ok
BERTScore 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
COMET 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
BLEURT 0 not enough data N/A N/A N/A N/A needs at least two rows with metric scores
Trigram precision 100 significant negative -0.316 0.100 0.0014 -0.00220 ok
Trigram recall 100 significant negative -0.346 0.120 0.0004 -0.00238 ok
Trigram F1 100 significant negative -0.332 0.110 0.0007 -0.00229 ok
Trigram Jaccard 100 significant negative -0.336 0.113 0.0006 -0.00219 ok

Translation Similarity Metrics

Mean BLEU-448.1%
Mean chrF++69.8%
Mean METEOR75.2%
Mean ROUGE-L76.2%
Mean BERTScoreN/A
Mean COMETN/A
Mean BLEURTN/A
Legacy smoothed corpus BLEU46.4%
Unigram F177.1%
Bigram F153.4%
Trigram precision40.8%
Trigram recall39.0%
Trigram F139.9%
Trigram Jaccard24.9%
4-gram F130.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 225 -18.015 29.8% 56.4% 53.0% 63.4% N/A N/A N/A reviewed/approved 11184 gpt-5.3-chat-latest
2604 Καρχηδών 117 121 13.977 36.7% 66.7% 66.4% 65.0% N/A N/A N/A reviewed/approved 11199 gpt-5.3-chat-latest
2603 Κάρυστος 175 172 13.425 47.1% 68.3% 69.1% 70.3% N/A N/A N/A reviewed/approved 11195 gpt-5.3-chat-latest
2468 Καπετώλιον 138 115 -10.688 11.6% 41.9% 50.3% 53.8% N/A N/A N/A reviewed/approved 11175 gpt-5.3-chat-latest
7267 Κωνώπη 64 70 10.086 53.5% 70.3% 69.7% 68.7% N/A N/A N/A reviewed/approved 11332 gpt-5.3-chat-latest
2342 Κάμιρος 54 61 9.974 43.4% 72.0% 77.0% 76.5% N/A N/A N/A reviewed/approved 11155 gpt-5.3-chat-latest
2056 Καβειρία 112 93 -9.579 23.3% 56.2% 55.5% 68.3% N/A N/A N/A reviewed/approved 11040 gpt-5.3-chat-latest
2328 Καλλίπολις 43 49 7.751 45.0% 66.3% 68.8% 69.6% N/A N/A N/A reviewed/approved 11132 gpt-5.3-chat-latest
2470 Καππαδοκία 68 71 7.530 17.1% 56.2% 84.9% 79.1% N/A N/A N/A reviewed/approved 11178 gpt-5.3-chat-latest
7254 Κύτα 84 85 7.309 41.9% 66.0% 63.1% 68.6% N/A N/A N/A reviewed/approved 11295 gpt-5.3-chat-latest
2465 Κάνωπος 83 84 7.198 53.5% 70.9% 83.4% 83.8% N/A N/A N/A reviewed/approved 11169 gpt-5.3-chat-latest
7255 Κυτέριον 22 29 6.417 29.5% 65.8% 58.1% 62.7% N/A N/A N/A reviewed/approved 11297 gpt-5.3-chat-latest
2609 Κάσπειρος 135 129 5.978 48.3% 69.7% 63.6% 74.2% N/A N/A N/A reviewed/approved 11206 gpt-5.3-chat-latest
7260 Κυχρεῖος πάγος 61 63 5.752 46.7% 67.2% 77.2% 69.4% N/A N/A N/A reviewed/approved 11315 gpt-5.3-chat-latest
2331 Κάλυδνα 43 47 5.751 59.9% 82.1% 88.0% 86.7% N/A N/A N/A reviewed/approved 11142 gpt-5.3-chat-latest
2116 Κάληρος 40 33 -5.582 20.7% 51.6% 55.0% 54.8% N/A N/A N/A reviewed/approved 11112 gpt-5.3-chat-latest
2083 Καλαμένθη 24 19 -5.361 46.8% 64.0% 59.6% 74.4% N/A N/A N/A reviewed/approved 11089 gpt-5.3-chat-latest
2455 Κάναστρον 60 51 -5.359 29.6% 61.2% 57.1% 66.7% N/A N/A N/A reviewed/approved 11166 gpt-5.3-chat-latest
7264 Κωλιάς 61 53 -4.248 33.4% 58.9% 61.3% 70.2% N/A N/A N/A reviewed/approved 11326 gpt-5.3-chat-latest
2115 Καλὴ ἀκτή 64 64 4.086 34.6% 54.1% 64.9% 65.6% N/A N/A N/A reviewed/approved 11109 gpt-5.3-chat-latest
7244 Κυρήνη 54 55 3.974 29.0% 60.1% 57.9% 56.9% N/A N/A N/A reviewed/approved 11262 gpt-5.3-chat-latest
2599 Καρπήσιοι 11 9 -3.806 28.2% 52.6% 58.8% 60.0% N/A N/A N/A reviewed/approved 11191 gpt-5.3-chat-latest
7246 Κύρις 20 17 -3.806 69.5% 80.1% 83.3% 86.5% N/A N/A N/A reviewed/approved 11268 gpt-5.3-chat-latest
7266 Κώμη 79 77 3.753 19.1% 44.0% 37.9% 42.3% N/A N/A N/A reviewed/approved 11330 gpt-5.3-chat-latest
2079 Καισάρεια 39 34 -3.694 47.9% 70.2% 79.8% 84.9% N/A N/A N/A reviewed/approved 11078 gpt-5.3-chat-latest
2622 Καστωλοῦ πεδίον 42 44 3.640 54.9% 73.6% 84.8% 83.7% N/A N/A N/A reviewed/approved 11215 gpt-5.3-chat-latest
2346 Κάναι 77 75 3.531 60.7% 73.3% 82.7% 85.5% N/A N/A N/A reviewed/approved 11161 gpt-5.3-chat-latest
7247 Κύρνος 50 44 -3.471 34.1% 62.4% 73.3% 68.1% N/A N/A N/A reviewed/approved 11272 gpt-5.3-chat-latest
2330 Καλύβη 24 21 -3.361 44.3% 63.1% 58.1% 75.6% N/A N/A N/A reviewed/approved 11138 gpt-5.3-chat-latest
2628 Καταονία 24 21 -3.361 34.8% 64.6% 75.8% 75.6% N/A N/A N/A reviewed/approved 11235 gpt-5.3-chat-latest

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.169.

Positive termCoefficientDocsMean presentMean absent
και 1.6563 66 0.378 -0.734
απο 1.0263 27 1.704 -0.630
δε και 0.9976 8 2.407 -0.209
οι 0.9793 11 3.641 -0.450
λεγεται 0.8560 9 3.408 -0.337
τε 0.8282 5 4.153 -0.219
λεγεται και 0.8098 8 3.490 -0.304
τους 0.8065 4 2.751 -0.115
ος 0.7840 6 3.624 -0.231
τη 0.7576 15 1.590 -0.281
πολις 0.7340 68 0.665 -1.413
εστι δε και 0.6926 3 6.309 -0.195
εκληθη 0.6714 4 6.503 -0.271
εν τη 0.6192 7 2.751 -0.207
ου 0.5959 14 1.793 -0.292
πολιτης 0.5903 18 2.381 -0.523
εστι δε 0.5851 5 2.687 -0.141
εκαλειτο δε 0.5574 3 3.129 -0.097
παιδος 0.5554 4 4.251 -0.177
των 0.5406 17 1.518 -0.311
Negative termCoefficientDocsMean presentMean absent
το -0.9008 70 0.098 -0.228
του -0.8235 37 0.497 -0.292
γαρ -0.8038 9 -0.035 0.003
εθνος -0.7220 8 -3.803 0.331
θρακης -0.6568 5 -2.805 0.148
τινες -0.6035 6 -2.582 0.165
φησιν -0.5631 10 -1.924 0.214
καππαδοκιας -0.5370 3 -2.989 0.092
μεν -0.5128 4 -7.078 0.295
το εθνικον -0.4568 56 -0.388 0.493
εθνικον -0.4568 56 -0.388 0.493
δια -0.4513 16 0.955 -0.182
ποταμου -0.4385 3 -1.175 0.036
πολις θρακης -0.4360 4 -2.167 0.090
μεταξυ -0.4329 5 -1.448 0.076
θεοπομπος -0.4116 5 -1.960 0.103
τις -0.4060 3 -1.767 0.055
δεκατω -0.4050 3 -2.433 0.075
ζευς -0.4047 3 -8.391 0.260
οικητωρ -0.3861 6 -3.007 0.192

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.277.

Positive termCoefficientDocsMean presentMean absent
και 2.2114 66 3.823 1.938
δε 1.3878 46 4.359 2.180
γαρ 1.3749 9 7.713 2.734
δια 1.3267 16 6.222 2.603
την 1.2734 13 7.844 2.486
τη 1.0000 15 5.550 2.764
εκαλειτο δε 0.9927 3 15.139 2.813
εκαλειτο 0.9802 7 8.043 2.817
δια του 0.9309 10 6.894 2.770
εστι 0.8797 24 5.123 2.570
δε και 0.8687 8 7.834 2.778
εις 0.8531 7 8.821 2.758
λεγεται και 0.8507 8 8.445 2.725
ος 0.8328 6 7.899 2.881
απο 0.8268 27 5.213 2.431
λεγεται 0.7773 9 7.812 2.724
εκαλειτο δε και 0.7231 3 15.139 2.813
τα 0.6904 12 6.647 2.710
τω 0.6698 16 5.086 2.820
το θηλυκον 0.6421 6 7.618 2.899
Negative termCoefficientDocsMean presentMean absent
πολιτης -0.5090 18 3.414 3.131
πολις -0.5027 68 3.360 2.804
δευτερω -0.5008 3 0.935 3.252
προς -0.5006 15 3.036 3.208
ως -0.4989 52 3.379 2.969
εκ -0.4850 6 2.241 3.242
αιγυπτου -0.4745 3 1.307 3.240
μια -0.4703 3 1.268 3.242
πολις και -0.4608 6 3.575 3.157
τον -0.4285 10 3.394 3.159
στραβων -0.4173 15 3.499 3.126
ουδετερως -0.3957 3 1.193 3.244
το εθνικον -0.3892 56 3.474 2.811
εθνικον -0.3892 56 3.474 2.811
τοπικα -0.3693 3 2.340 3.208
τα τοπικα -0.3693 3 2.340 3.208
του δε -0.3669 3 1.324 3.240
πλησιον -0.3565 8 2.466 3.245
προς τη -0.3110 4 2.514 3.210
οικητωρ -0.2810 6 4.878 3.074

AI English terms predicting large absolute residuals

Features: 321; ridge alpha: 2.894; cross-validated R^2: -0.237.

Positive termCoefficientDocsMean presentMean absent
and 2.0576 56 3.986 2.159
the 1.8853 91 3.282 2.170
also 1.5771 47 4.480 2.031
was 1.4442 19 5.347 2.675
with 1.3060 17 5.338 2.741
for 1.3054 13 6.246 2.725
there 1.1227 32 5.033 2.312
is also 1.1175 29 4.860 2.497
diphthong 0.9118 3 10.427 2.958
on 0.9116 14 5.150 2.862
by 0.8809 13 5.951 2.769
there is 0.8587 25 5.063 2.555
it is also 0.8522 8 8.334 2.734
also said 0.8448 4 12.181 2.807
is also said 0.8448 4 12.181 2.807
was called 0.8043 5 8.130 2.922
after 0.7745 15 5.890 2.704
said 0.7457 5 10.495 2.797
there is also 0.7404 22 4.821 2.720
he 0.7387 4 9.707 2.910
Negative termCoefficientDocsMean presentMean absent
is a -0.7631 28 3.474 3.069
a -0.6456 90 3.301 2.111
a city -0.6360 66 3.361 2.835
which the -0.5926 5 1.427 3.275
one of -0.5708 4 1.207 3.265
the ethnonym is -0.5285 52 3.184 3.181
ethnonym is -0.5285 52 3.184 3.181
of egypt -0.5158 3 1.307 3.240
egypt -0.5158 3 1.307 3.240
city -0.4913 68 3.360 2.804
strabo book -0.4666 10 2.545 3.253
one of the -0.4582 3 1.268 3.242
island -0.4535 14 2.924 3.224
as in -0.4441 38 3.274 3.126
the ethnonym -0.4326 58 3.392 2.892
ethnonym -0.4326 58 3.392 2.892
one -0.4158 10 2.706 3.235
an -0.4090 26 2.926 3.273
book -0.4087 41 3.268 3.123
city and -0.3806 4 1.944 3.234

Human English terms predicting large absolute residuals

Features: 336; ridge alpha: 2.031; cross-validated R^2: -0.152.

Positive termCoefficientDocsMean presentMean absent
the 2.4037 90 3.316 1.984
with 2.0590 16 5.309 2.777
and 1.8280 52 4.097 2.191
on 1.8255 16 5.288 2.781
also 1.5119 43 4.382 2.277
is with 1.3715 4 11.177 2.849
to 1.3199 31 4.766 2.471
too 1.1793 3 9.901 2.975
there 1.1537 34 4.671 2.415
when 1.1376 7 8.423 2.788
also said 1.1161 6 10.001 2.747
said 1.1161 6 10.001 2.747
is also 1.0743 31 4.746 2.480
usage 1.0619 4 7.496 3.003
their 0.9904 3 15.139 2.813
forms 0.9761 4 11.453 2.838
who 0.9105 13 5.525 2.832
used to be 0.9032 6 9.101 2.805
he 0.8942 3 10.358 2.960
into 0.8926 3 7.718 3.042
Negative termCoefficientDocsMean presentMean absent
as in -1.3724 29 2.671 3.391
one -0.8050 12 3.110 3.192
the ethnonym is -0.7099 52 3.132 3.237
ethnonym is -0.7099 52 3.132 3.237
egypt -0.7039 3 1.307 3.240
is a -0.6960 31 3.675 2.961
territory -0.6804 5 1.373 3.278
one of -0.6321 5 1.571 3.267
a city -0.6265 64 3.405 2.787
city in -0.6173 41 3.381 3.044
a city in -0.6055 40 3.386 3.047
in the -0.5849 25 3.738 2.997
city -0.5799 70 3.363 2.761
one of the -0.5635 4 1.568 3.250
a -0.5455 91 3.224 2.756
herodotos book -0.5435 5 2.118 3.238
strabo -0.5395 15 3.499 3.126
was from -0.5293 4 2.125 3.226
name -0.5245 7 3.288 3.174
an -0.5038 22 3.050 3.220

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.