gpt-5.3-chat-latest v2 Evaluation

Generated: 2026-09-20 11:21:42 UTC

Pairs100
Distinct lemmas100
Slope0.889
R^20.984
BLEU-448.1%
chrF++69.8%
METEOR75.3%
ROUGE-L76.2%
BERTScore95.9%
COMET79.0%
BLEURT74.3%

Profile version id: 2335

First translation using this version: 2026-07-15

Total usable AI runs for this version: 100 runs across 100 lemmas

Created: 2026-07-15 12:28:27.643960+10:00

Active: True

Notes: Model timeline evaluation: GPT-5.3 Chat using prompt text from gpt-5.5 v2; approved-human-only for the 100-row Kappa corpus.

Prompt length scatter plot

Length Regression

FormulaAI words = intercept + slope * human words
Slope0.889
Intercept3.045
R^20.984
P-value4.31e-90
Pearson r0.992
Mean source words30.400
Length fallback rows0
Mean human words44.150
Mean AI words42.280
Mean absolute length residual3.176
Mean absolute percent error7.0%

Metric vs Passage Length

These plots test whether score changes with source passage length for this prompt version. The dashed line is an ordinary least-squares fit; the annotation reports n, r, R^2, p-value, and slope.

BLEU-4 vs passage length
BLEU-4: significant negative; r = -0.391, R^2 = 0.152, p = 5.90e-05.
chrF++ vs passage length
chrF++: significant negative; r = -0.432, R^2 = 0.187, p = 7.09e-06.
METEOR vs passage length
METEOR: significant negative; r = -0.457, R^2 = 0.209, p = 1.72e-06.
ROUGE-L vs passage length
ROUGE-L: significant negative; r = -0.406, R^2 = 0.165, p = 2.82e-05.
BERTScore vs passage length
BERTScore: significant negative; r = -0.579, R^2 = 0.335, p = 2.89e-10.
COMET vs passage length
COMET: significant negative; r = -0.473, R^2 = 0.224, p = 6.57e-07.
BLEURT vs passage length
BLEURT: significant negative; r = -0.741, R^2 = 0.550, p = 1.12e-18.
Trigram precision vs passage length
Trigram precision: significant negative; r = -0.316, R^2 = 0.100, p = 0.0014.
Trigram recall vs passage length
Trigram recall: significant negative; r = -0.347, R^2 = 0.121, p = 0.0004.
Trigram F1 vs passage length
Trigram F1: significant negative; r = -0.333, R^2 = 0.111, p = 0.0007.
Trigram Jaccard vs passage length
Trigram Jaccard: significant negative; r = -0.336, R^2 = 0.113, p = 0.0006.
Metric Rows Pattern Pearson r R^2 P-value Slope Status
BLEU-4 100 significant negative -0.391 0.152 5.90e-05 -0.00247 ok
chrF++ 100 significant negative -0.432 0.187 7.09e-06 -0.00171 ok
METEOR 100 significant negative -0.457 0.209 1.72e-06 -0.00206 ok
ROUGE-L 100 significant negative -0.406 0.165 2.82e-05 -0.00165 ok
BERTScore 100 significant negative -0.579 0.335 2.89e-10 -0.00047 ok
COMET 100 significant negative -0.473 0.224 6.57e-07 -0.00091 ok
BLEURT 100 significant negative -0.741 0.550 1.12e-18 -0.00273 ok
Trigram precision 100 significant negative -0.316 0.100 0.0014 -0.00220 ok
Trigram recall 100 significant negative -0.347 0.121 0.0004 -0.00238 ok
Trigram F1 100 significant negative -0.333 0.111 0.0007 -0.00229 ok
Trigram Jaccard 100 significant negative -0.336 0.113 0.0006 -0.00219 ok

Translation Similarity Metrics

Mean BLEU-448.1%
Mean chrF++69.8%
Mean METEOR75.3%
Mean ROUGE-L76.2%
Mean BERTScore95.9%
Mean COMET79.0%
Mean BLEURT74.3%
Legacy smoothed corpus BLEU46.4%
Unigram F177.1%
Bigram F153.4%
Trigram precision40.8%
Trigram recall39.0%
Trigram F139.9%
Trigram Jaccard24.9%
4-gram F130.7%

Largest Length Residuals

Lemma ID Headword Human words AI words Residual BLEU-4 chrF++ METEOR ROUGE-L BERTScore COMET BLEURT Human Run Model
2484 Καρία 270 225 -17.985 29.8% 56.4% 53.0% 63.4% 90.2% 77.5% 42.9% reviewed/approved 11184 gpt-5.3-chat-latest
2604 Καρχηδών 117 121 13.981 36.7% 66.7% 66.4% 65.0% 94.3% 77.5% 61.8% reviewed/approved 11199 gpt-5.3-chat-latest
2603 Κάρυστος 175 172 13.438 47.1% 68.3% 69.1% 70.3% 94.8% 78.7% 67.1% reviewed/approved 11195 gpt-5.3-chat-latest
2468 Καπετώλιον 138 115 -10.681 11.6% 41.9% 50.3% 53.8% 87.1% 69.2% 49.4% reviewed/approved 11175 gpt-5.3-chat-latest
7267 Κωνώπη 64 70 10.080 53.5% 70.3% 69.7% 68.7% 94.8% 72.6% 67.4% reviewed/approved 11332 gpt-5.3-chat-latest
2342 Κάμιρος 54 61 9.967 43.4% 72.0% 77.0% 76.5% 96.2% 79.0% 69.1% reviewed/approved 11155 gpt-5.3-chat-latest
2056 Καβειρία 112 93 -9.576 23.3% 56.2% 55.5% 68.3% 94.4% 75.5% 56.6% reviewed/approved 11040 gpt-5.3-chat-latest
2328 Καλλίπολις 43 49 7.742 45.0% 66.3% 68.8% 69.6% 94.7% 76.4% 72.2% reviewed/approved 11132 gpt-5.3-chat-latest
2470 Καππαδοκία 68 71 7.525 17.1% 56.2% 84.9% 79.1% 95.0% 77.2% 73.4% reviewed/approved 11178 gpt-5.3-chat-latest
7254 Κύτα 84 85 7.307 41.9% 66.0% 66.7% 69.8% 94.4% 77.4% 63.4% reviewed/approved 11295 gpt-5.3-chat-latest
2465 Κάνωπος 83 84 7.195 53.5% 70.9% 83.4% 83.8% 95.3% 76.5% 71.9% reviewed/approved 11169 gpt-5.3-chat-latest
7255 Κυτέριον 22 29 6.404 29.5% 65.8% 58.1% 62.7% 95.5% 76.7% 75.7% reviewed/approved 11297 gpt-5.3-chat-latest
2609 Κάσπειρος 135 129 5.985 48.3% 69.7% 63.6% 74.2% 90.7% 73.6% 68.4% reviewed/approved 11206 gpt-5.3-chat-latest
7260 Κυχρεῖος πάγος 61 63 5.746 46.7% 67.2% 77.2% 69.4% 95.9% 75.7% 60.2% reviewed/approved 11315 gpt-5.3-chat-latest
2331 Κάλυδνα 43 47 5.742 59.9% 82.1% 88.0% 86.7% 95.9% 80.4% 78.5% reviewed/approved 11142 gpt-5.3-chat-latest
2116 Κάληρος 40 33 -5.592 20.7% 51.6% 55.0% 54.8% 93.4% 73.6% 69.0% reviewed/approved 11112 gpt-5.3-chat-latest
2083 Καλαμένθη 24 19 -5.373 46.8% 64.0% 59.6% 74.4% 94.5% 75.8% 71.8% reviewed/approved 11089 gpt-5.3-chat-latest
2455 Κάναστρον 60 51 -5.365 29.6% 61.2% 57.1% 66.7% 94.8% 71.9% 60.0% reviewed/approved 11166 gpt-5.3-chat-latest
7264 Κωλιάς 61 53 -4.254 33.4% 58.9% 61.3% 70.2% 95.1% 71.7% 58.4% reviewed/approved 11326 gpt-5.3-chat-latest
2115 Καλὴ ἀκτή 64 64 4.080 34.6% 54.1% 64.9% 65.6% 93.3% 66.2% 59.2% reviewed/approved 11109 gpt-5.3-chat-latest
7244 Κυρήνη 54 55 3.967 29.0% 60.1% 57.9% 56.9% 93.5% 75.4% 70.0% reviewed/approved 11262 gpt-5.3-chat-latest
7246 Κύρις 20 17 -3.819 69.5% 80.1% 83.3% 86.5% 97.3% 82.8% 76.7% reviewed/approved 11268 gpt-5.3-chat-latest
7266 Κώμη 79 77 3.750 19.1% 44.0% 37.9% 42.3% 92.7% 72.3% 52.5% reviewed/approved 11330 gpt-5.3-chat-latest
2079 Καισάρεια 39 34 -3.703 47.9% 70.2% 79.8% 84.9% 95.4% 78.9% 75.2% reviewed/approved 11078 gpt-5.3-chat-latest
2622 Καστωλοῦ πεδίον 42 44 3.631 54.9% 73.6% 84.8% 83.7% 96.4% 76.6% 70.3% reviewed/approved 11215 gpt-5.3-chat-latest
2346 Κάναι 77 75 3.527 60.7% 73.3% 82.7% 85.5% 96.1% 77.8% 66.9% reviewed/approved 11161 gpt-5.3-chat-latest
7247 Κύρνος 50 44 -3.479 34.1% 62.4% 73.3% 68.1% 96.1% 75.8% 75.9% reviewed/approved 11272 gpt-5.3-chat-latest
2330 Καλύβη 24 21 -3.373 44.3% 63.1% 58.1% 75.6% 96.0% 79.6% 70.4% reviewed/approved 11138 gpt-5.3-chat-latest
2628 Καταονία 24 21 -3.373 34.8% 64.6% 75.8% 75.6% 96.1% 79.0% 71.7% reviewed/approved 11235 gpt-5.3-chat-latest
7257 Κυτώνιον 25 22 -3.262 72.6% 80.5% 75.7% 80.9% 98.7% 85.4% 84.4% reviewed/approved 11303 gpt-5.3-chat-latest

Residual Predictors

Positive signed-residual terms are associated with AI translations longer than expected from the human length; negative signed-residual terms are associated with shorter AI translations. Absolute-residual models identify terms associated with larger AI-human length divergence.

Greek source terms predicting signed length residuals

Features: 176; ridge alpha: 5.878; cross-validated R^2: -0.173.

Positive termCoefficientDocsMean presentMean absent
και 1.6438 66 0.371 -0.721
απο 1.0249 27 1.701 -0.629
δε και 0.9969 8 2.408 -0.209
οι 0.9765 11 3.637 -0.450
λεγεται 0.8553 9 3.408 -0.337
τε 0.8279 5 4.149 -0.218
λεγεται και 0.8097 8 3.491 -0.304
τους 0.8043 4 2.742 -0.114
ος 0.7836 6 3.623 -0.231
τη 0.7553 15 1.586 -0.280
πολις 0.7224 68 0.657 -1.395
εστι δε και 0.6922 3 6.308 -0.195
εκληθη 0.6707 4 6.501 -0.271
εν τη 0.6179 7 2.743 -0.206
ου 0.5934 14 1.790 -0.291
των 0.5908 17 1.567 -0.321
πολιτης 0.5852 18 2.375 -0.521
εστι δε 0.5845 5 2.686 -0.141
εκαλειτο δε 0.5583 3 3.145 -0.097
παιδος 0.5544 4 4.242 -0.177
Negative termCoefficientDocsMean presentMean absent
το -0.9106 70 0.090 -0.209
του -0.8300 37 0.492 -0.289
γαρ -0.8020 9 -0.030 0.003
θρακης -0.6611 5 -2.816 0.148
εθνος -0.6609 8 -3.697 0.321
τινες -0.6053 6 -2.591 0.165
φησιν -0.5638 10 -1.923 0.214
καππαδοκιας -0.5382 3 -2.996 0.093
μεν -0.5127 4 -7.073 0.295
το εθνικον -0.4671 56 -0.396 0.504
εθνικον -0.4671 56 -0.396 0.504
δια -0.4521 16 0.953 -0.181
πολις θρακης -0.4404 4 -2.179 0.091
μεταξυ -0.4362 5 -1.455 0.077
θεοπομπος -0.4137 5 -1.968 0.104
τις -0.4074 3 -1.774 0.055
δεκατω -0.4060 3 -2.440 0.075
ζευς -0.4044 3 -8.380 0.259
οικητωρ -0.3888 6 -3.011 0.192
ηρωδιανος -0.3805 3 -9.384 0.290

Greek source terms predicting large absolute residuals

Features: 176; ridge alpha: 4.125; cross-validated R^2: -0.268.

Positive termCoefficientDocsMean presentMean absent
και 2.2259 66 3.824 1.918
δε 1.3949 46 4.359 2.168
γαρ 1.3747 9 7.710 2.727
δια 1.3269 16 6.220 2.596
την 1.2740 13 7.842 2.479
τη 1.0034 15 5.550 2.757
εκαλειτο δε 0.9932 3 15.135 2.806
εκαλειτο 0.9852 7 8.044 2.809
δια του 0.9308 10 6.890 2.763
εστι 0.8872 24 5.123 2.561
δε και 0.8690 8 7.830 2.771
εις 0.8535 7 8.819 2.751
λεγεται και 0.8507 8 8.441 2.718
ος 0.8340 6 7.897 2.875
απο 0.8292 27 5.212 2.423
λεγεται 0.7776 9 7.808 2.718
εκαλειτο δε και 0.7233 3 15.135 2.806
τα 0.6921 12 6.644 2.703
τω 0.6743 16 5.086 2.812
το θηλυκον 0.6413 6 7.608 2.893
Negative termCoefficientDocsMean presentMean absent
πολιτης -0.5032 18 3.414 3.124
δευτερω -0.4936 3 0.940 3.245
προς -0.4919 15 3.041 3.200
ως -0.4910 52 3.379 2.956
πολις -0.4892 68 3.361 2.783
εκ -0.4819 6 2.243 3.235
αιγυπτου -0.4754 3 1.298 3.234
μια -0.4662 3 1.270 3.235
πολις και -0.4582 6 3.579 3.150
τον -0.4291 10 3.390 3.152
στραβων -0.4102 15 3.503 3.118
ουδετερως -0.3952 3 1.191 3.237
το εθνικον -0.3759 56 3.476 2.794
εθνικον -0.3759 56 3.476 2.794
τοπικα -0.3689 3 2.338 3.202
τα τοπικα -0.3689 3 2.338 3.202
του δε -0.3677 3 1.321 3.233
πλησιον -0.3513 8 2.466 3.238
προς τη -0.3067 4 2.519 3.203
οικητωρ -0.2768 6 4.877 3.067

AI English terms predicting large absolute residuals

Features: 321; ridge alpha: 2.894; cross-validated R^2: -0.227.

Positive termCoefficientDocsMean presentMean absent
and 2.0733 56 3.986 2.144
the 1.8773 91 3.274 2.180
also 1.5868 47 4.479 2.020
was 1.4490 19 5.345 2.667
with 1.3082 17 5.336 2.733
for 1.3063 13 6.244 2.717
there 1.1299 32 5.033 2.302
is also 1.1228 29 4.859 2.488
on 0.9134 14 5.150 2.855
diphthong 0.9097 3 10.414 2.952
by 0.8847 13 5.952 2.761
there is 0.8626 25 5.063 2.547
it is also 0.8527 8 8.328 2.728
also said 0.8448 4 12.174 2.801
is also said 0.8448 4 12.174 2.801
was called 0.8076 5 8.133 2.915
after 0.7743 15 5.886 2.697
said 0.7461 5 10.489 2.791
there is also 0.7455 22 4.822 2.712
he 0.7372 4 9.695 2.904
Negative termCoefficientDocsMean presentMean absent
is a -0.7594 28 3.473 3.060
a city -0.6284 66 3.362 2.815
a -0.6269 90 3.303 2.032
which the -0.5888 5 1.429 3.268
one of -0.5689 4 1.210 3.258
the ethnonym is -0.5178 52 3.186 3.165
ethnonym is -0.5178 52 3.186 3.165
of egypt -0.5160 3 1.298 3.234
egypt -0.5160 3 1.298 3.234
city -0.4831 68 3.361 2.783
an -0.4723 26 2.896 3.274
strabo book -0.4617 10 2.548 3.246
one of the -0.4589 3 1.270 3.235
island -0.4441 14 2.929 3.216
as in -0.4382 38 3.274 3.116
the ethnonym -0.4219 58 3.394 2.874
ethnonym -0.4219 58 3.394 2.874
one -0.4131 10 2.708 3.228
book -0.3973 41 3.268 3.112
city and -0.3792 4 1.944 3.227

Human English terms predicting large absolute residuals

Features: 335; ridge alpha: 2.031; cross-validated R^2: -0.140.

Positive termCoefficientDocsMean presentMean absent
the 2.4265 90 3.307 1.992
with 2.0627 16 5.307 2.770
and 1.8571 52 4.098 2.177
on 1.8307 16 5.288 2.774
also 1.5239 43 4.382 2.266
is with 1.3797 4 11.172 2.843
to 1.3273 31 4.767 2.461
too 1.1747 3 9.896 2.968
there 1.1583 34 4.672 2.405
when 1.1401 7 8.417 2.781
also said 1.1151 6 9.995 2.741
said 1.1151 6 9.995 2.741
is also 1.0776 31 4.747 2.470
usage 1.0654 4 7.485 2.996
their 0.9906 3 15.135 2.806
forms 0.9787 4 11.442 2.831
who 0.9158 13 5.526 2.825
used to be 0.9067 6 9.101 2.798
into 0.8948 3 7.722 3.035
diphthong 0.8876 3 10.414 2.952
Negative termCoefficientDocsMean presentMean absent
as in -1.3719 29 2.669 3.383
one -0.7987 12 3.111 3.185
egypt -0.7054 3 1.298 3.234
is a -0.6922 31 3.675 2.952
the ethnonym is -0.6905 52 3.134 3.221
ethnonym is -0.6905 52 3.134 3.221
territory -0.6793 5 1.373 3.271
one of -0.6311 5 1.573 3.260
a city -0.6232 64 3.405 2.768
city in -0.6121 41 3.382 3.033
a city in -0.5994 40 3.387 3.035
in the -0.5809 25 3.740 2.988
city -0.5764 70 3.363 2.739
an -0.5653 22 3.012 3.222
one of the -0.5624 4 1.569 3.243
herodotos book -0.5423 5 2.114 3.232
was from -0.5338 4 2.119 3.220
a -0.5269 91 3.226 2.665
name -0.5249 7 3.285 3.168
strabo -0.5215 15 3.503 3.118

Prompt Text

Show prompt text
You are an expert classical philologist and translator specialising in Stephanos of Byzantium's Ethnika.

Goal
- Produce a clear, scholarly English translation in the established reviewed house style.

Output rules (required)
- Respond ONLY by calling the submit_translation tool with a single string field: {"translation": "..."}.
- The translation text must contain only the translation (no analysis, no commentary, no multiple options).

A) Formatting + spelling
- Use Australasian spelling and punctuation conventions.
- Preserve paragraphing/line-breaks of the Greek source:
  - Do not introduce new paragraphs unless the Greek has them.
  - If the Greek includes a poetic quotation with line breaks, format the English quotation with the same line breaks.
- Use single quotes for quoted forms/snippets: '...'. Avoid double quotes.
- Use *italics* (asterisks) for titles of ancient works: e.g., *Cypriaka*.

B) Opening / structure
- Begin with the headword transliterated into Latin letters (no Greek diacritics), then a short definition.
  - Typically: Headword: ...
  - Appositive openings like 'Karia, the country.' are acceptable when the entry is of that type.
- Keep enumerations of homonymous places as inline numbered items like (2) ... (3) ..., matching the Greek's structure.

C) Transliteration + naming
- Do NOT use macrons/acute accents in transliteration: Karystos (not Kárystos), Kaspeiros (not Káspeiros).
- Prefer Greek-form transliteration with kappa = k; avoid Latinised exonyms when Stephanos is discussing the Greek form:
  - Kapetolion (not Capitolium)
  - Karchedon (not Carthage)
  - Chalkedon (not Chalcedon)
- Use conventional English names for major places/regions when standard (Rome, Cyprus, Egypt, Syria, India).
- Translate Πόντος as 'the Black Sea' when it is clearly the sea/region reference.

D) Citations (authors/works/books)
- Convert Greek book numerals to Arabic digits.
- Keep citations compact, mirroring the source's incompleteness:
  - Author + work title: 'Hellanikos in his *Cypriaka*'
  - Author + book: 'Strabo, book 12: ...' / 'Herodotus, book 3.'
  - Author + work + book: 'Dionysios in book 3 of his *Bassarika*: ...'
- Ignore modern/editorial locator codes and apparatus-like add-ons:
  - omit RE/SH numbers, (GG ...), (FGrHist ...), chapter/section locators like (12.8.12), [C 576.21], Il. 2.676, etc.
  - keep only the author/work/book level that Stephanos is using.

E) Fixed formulae (be consistent)
- τὸ ἐθνικόν X -> 'The ethnonym is 'X'.'
- ὁ πολίτης X -> 'A citizen is a 'X'.'
- τὸ θηλυκόν / καὶ θηλυκόν X -> 'In the feminine 'X'.'
- Keep (as needed): 'The possessive is '...'.' / 'An inhabitant is a '...'.' / 'A deme-member is a '...'.'
- For τὰ τοπικά (place-adverb forms), use 'The locatives are ...' and list forms in single quotes.

F) Philological/orthographic discussion
- When the entry discusses spelling/letters/diphthongs, preserve Greek letters in Greek script (e.g., ι, ει, οι).
- Prefer transliteration (not full Greek script) for cited alternative spellings unless the point requires showing a specific Greek letter/feature.

G) Comparisons / derivational analogies (ὡς + X)
- For ὡς + X where X is an etymologically related noun in the nominative, use this fixed English pattern:
  - (as in 'X')
- Keep it exactly: parentheses + single quotes (no 'like X', no bare 'as X').

H) ἀφ' οὗ
- If it has a masculine antecedent referring to a person (typically the eponym) in a naming context (implicit/explicit καλεῖται/ἐκαλεῖτο/κέκληται), translate as 'after whom' (i.e., 'named after whom').
- If it has an identifiable explicit neuter antecedent, translate as 'from which'.
- If it has a dropped neuter antecedent and functions adverbially, handle cautiously:
  - often 'hence' / 'thence'
  - in Stephanos' idiosyncratic 'example-marker' usage, 'as per' can be acceptable when it clearly introduces an ethnonym/person-as-example.

I) Morphology scaffolding (only when needed)
- If Stephanos is citing an author specifically to indicate grammatical case/gender/number, and that feature would otherwise be unclear in English, add brief tags after the relevant form:
  - case: (nom.), (acc.), (gen.), (dat.), (voc.)
  - gender: (m.), (f.), (n.)
  - number: (sing.), (pl.)

J) Fidelity + restraint
- Translate directly; do not add background explanation or modern bibliography.
- Avoid gratuitous adversatives ('however') for δέ when it is merely continuative.
- Do not add glosses like (synoikia), (kalathos) unless Stephanos explicitly defines a term.
- Preserve uncertainty when the Greek is uncertain/corrupt; do not invent.

Now translate the provided entry accordingly, and submit it via submit_translation.