Translation Similarity by Model Release Date

Generated: 2026-09-20 11:23:15 UTC

Back to translation prompt evaluation.

This page tracks automated reference-similarity metrics over model releases while keeping the Stephanos prompt version fixed. The overview includes twelve OpenAI releases and available Claude observations on the same 100-entry Kappa corpus.

Benchmark Overview

Translation similarity by model release date for three Stephanos prompt versions
Mean of BLEU-4, chrF++, METEOR, and ROUGE-L across the same 100 Kappa entries. Circles are OpenAI model-prompt observations; dashed lines are prompt-specific OLS fits. Claude observations are shown as diamonds and excluded from the fitted trends. These are reference-similarity scores, not calibrated measures of philological correctness or human equivalence. Download the vector PDF.

Human-equivalent dates below use a provisional automated-score target of 90.0%. Replace this target with a second-human baseline when the project has one; until then, the projection is a planning proxy, not a claim about human adequacy.

OpenAI Planning Charts

Mean automated metric
Mean automated metric.
ROUGE-L
ROUGE-L.
Same-length mean automated metric
Same-length mean automated metric.

Human-Equivalent Projection

Prompt v. Metric Target Completed releases Latest release Latest score Annualized slope Projected target date Steady check Projection status
1 Mean automated metric 90.0% 12 2026-07-09 59.4% +1.2 pp 2050-01-20 not steady linear projection from completed releases
1 ROUGE-L 90.0% 12 2026-07-09 57.7% +2.1 pp 2040-08-09 not steady linear projection from completed releases
1 Same-length mean automated metric 90.0% 12 2026-07-09 50.6% +4.6 pp 2034-04-27 not steady linear projection from completed releases
2 Mean automated metric 90.0% 12 2026-07-09 76.4% +2.9 pp 2031-07-12 not steady linear projection from completed releases
2 ROUGE-L 90.0% 12 2026-07-09 77.6% +3.0 pp 2030-07-26 not steady linear projection from completed releases
2 Same-length mean automated metric 90.0% 12 2026-07-09 42.8% +3.4 pp 2040-03-20 not steady linear projection from completed releases
3 Mean automated metric 90.0% 12 2026-07-09 78.5% +3.6 pp 2029-11-18 not steady linear projection from completed releases
3 ROUGE-L 90.0% 12 2026-07-09 80.6% +3.9 pp 2028-11-27 not steady linear projection from completed releases
3 Same-length mean automated metric 90.0% 12 2026-07-09 44.3% +4.6 pp 2035-10-16 not steady linear projection from completed releases

Model Timeline Table

Prompt v. Model Release API date Pairs Mean metric Delta vs prior ROUGE-L ROUGE-L delta Status BLEU-4 chrF++ METEOR BERTScore COMET BLEURT Same-length mean Same-length delta Same-length vs Mix. Same-length vs Comp. Length slope Actual run models Source
1 GPT-4 Turbo (2024-04-09) / v1 2024-04-09 2024-04-09 100 56.7% N/A 53.8% N/A complete 17.5% 47.9% 53.0% 91.6% 71.1% 62.3% 47.3% N/A -11.7 pp -5.0 pp 0.898 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
1 GPT-4o (2024-05-13) / v1 2024-05-13 2024-05-13 100 58.9% +2.2 pp 57.2% +3.4 pp complete 19.8% 50.2% 55.3% 92.3% 72.4% 65.4% 38.6% -8.7 pp -20.4 pp -13.8 pp 0.637 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
1 GPT-4o (2024-08-06) / v1 2024-08-06 2024-08-06 100 58.3% -0.6 pp 56.1% -1.1 pp complete 18.4% 49.8% 54.3% 92.1% 72.9% 64.5% 46.3% +7.7 pp -12.7 pp -6.1 pp 0.965 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
1 GPT-4o (2024-11-20) / v1 2024-11-20 2024-11-20 100 57.7% -0.6 pp 54.6% -1.5 pp complete 16.5% 49.0% 54.7% 91.9% 72.7% 64.7% 44.5% -1.8 pp -14.5 pp -7.9 pp 1.050 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
1 GPT-4.1 (2025-04-14) / v1 2025-04-14 2025-04-14 100 59.5% +1.8 pp 58.7% +4.1 pp complete 19.6% 51.2% 57.0% 92.3% 72.7% 65.2% 52.4% +7.9 pp -6.6 pp +0.1 pp 0.999 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
1 GPT-5 / v1 2025-08-07 2025-08-07 100 60.4% +0.8 pp 61.8% +3.1 pp complete 20.7% 51.9% 60.5% 92.4% 71.9% 63.4% 53.0% +0.6 pp -6.0 pp +0.6 pp 0.981 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
1 GPT-5.1 / v1 2025-11-13 2025-11-13 100 60.0% -0.4 pp 60.2% -1.6 pp complete 20.3% 51.9% 59.5% 92.2% 71.9% 63.9% 54.6% +1.6 pp -4.4 pp +2.2 pp 1.130 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
1 GPT-5.2 / v1 2025-12-11 2025-12-11 100 60.2% +0.2 pp 58.5% -1.7 pp complete 23.4% 53.9% 58.8% 91.9% 71.5% 63.6% 48.9% -5.8 pp -10.1 pp -3.5 pp 1.010 gpt-5.2: 3; gpt-5.2-2025-12-11: 97 OpenAI GPT-5.2 model docs
1 GPT-5.3 Chat / v1 2026-03-03 2026-03-03 100 61.1% +0.8 pp 61.2% +2.7 pp complete 23.1% 52.9% 59.6% 92.6% 72.7% 65.6% 53.1% +4.2 pp -5.9 pp +0.7 pp 0.987 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
1 GPT-5.4 / v1 2026-03-05 2026-03-05 100 60.1% -1.0 pp 60.1% -1.0 pp complete 23.2% 52.7% 57.6% 92.4% 71.5% 63.0% 54.0% +0.9 pp -5.0 pp +1.6 pp 0.977 gpt-5.4: 100 OpenAI API changelog
1 GPT-5.5 / v1 2026-04-23 2026-04-24 100 60.2% +0.1 pp 58.4% -1.7 pp complete 25.1% 53.8% 56.1% 92.3% 72.2% 63.6% 53.2% -0.8 pp -5.8 pp +0.8 pp 1.003 gpt-5.5: 100 OpenAI API changelog
1 GPT-5.6 Sol / v1 2026-07-09 2026-07-09 100 59.4% -0.8 pp 57.7% -0.7 pp complete 23.3% 51.8% 56.1% 92.2% 71.6% 63.1% 50.6% -2.6 pp -8.4 pp -1.8 pp 0.998 gpt-5.6-sol: 100 OpenAI API changelog
2 GPT-4 Turbo (2024-04-09) / v2 2024-04-09 2024-04-09 100 69.8% N/A 70.8% N/A complete 40.1% 64.5% 68.9% 94.9% 77.8% 71.4% 29.7% N/A -29.3 pp -22.6 pp 0.887 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
2 GPT-4o (2024-05-13) / v2 2024-05-13 2024-05-13 100 69.9% +0.2 pp 72.0% +1.1 pp complete 39.8% 64.4% 69.7% 95.2% 77.4% 71.2% 37.4% +7.7 pp -21.6 pp -14.9 pp 0.835 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
2 GPT-4o (2024-08-06) / v2 2024-08-06 2024-08-06 100 70.1% +0.2 pp 72.4% +0.4 pp complete 38.8% 64.8% 70.8% 95.3% 77.7% 71.1% 38.8% +1.4 pp -20.2 pp -13.6 pp 0.922 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
2 GPT-4o (2024-11-20) / v2 2024-11-20 2024-11-20 100 70.8% +0.7 pp 72.5% +0.1 pp complete 40.3% 65.9% 71.4% 95.4% 78.2% 72.0% 40.0% +1.1 pp -19.0 pp -12.4 pp 0.900 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
2 GPT-4.1 (2025-04-14) / v2 2025-04-14 2025-04-14 100 71.2% +0.4 pp 73.1% +0.6 pp complete 41.3% 66.1% 72.4% 95.4% 78.2% 72.2% 44.0% +4.1 pp -15.0 pp -8.4 pp 0.948 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
2 GPT-5 / v2 2025-08-07 2025-08-07 100 73.9% +2.7 pp 76.4% +3.3 pp complete 47.8% 70.1% 74.7% 96.1% 78.8% 73.6% 37.8% -6.2 pp -21.2 pp -14.5 pp 0.821 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
2 GPT-5.1 / v2 2025-11-13 2025-11-13 100 72.9% -1.0 pp 74.7% -1.7 pp complete 45.4% 68.7% 73.6% 95.7% 78.8% 73.3% 44.2% +6.3 pp -14.8 pp -8.2 pp 0.966 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
2 GPT-5.2 / v2 2025-12-11 2025-12-11 100 74.3% +1.4 pp 76.1% +1.5 pp complete 48.2% 70.5% 75.7% 96.0% 79.0% 74.3% 38.6% -5.5 pp -20.4 pp -13.8 pp 0.921 gpt-5.2-2025-12-11: 100 OpenAI GPT-5.2 model docs
2 GPT-5.3 Chat / v2 2026-03-03 2026-03-03 100 74.1% -0.2 pp 76.2% +0.1 pp complete 48.1% 69.8% 75.3% 95.9% 79.0% 74.3% 41.5% +2.9 pp -17.5 pp -10.9 pp 0.889 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
2 GPT-5.4 / v2 2026-03-05 2026-03-05 100 74.2% +0.1 pp 77.4% +1.1 pp complete 46.8% 69.8% 76.7% 96.1% 79.1% 73.7% 42.9% +1.4 pp -16.1 pp -9.5 pp 0.939 gpt-5.4: 100 OpenAI API changelog
2 GPT-5.5 / v2 2026-04-23 2026-04-24 100 76.1% +1.9 pp 78.1% +0.8 pp complete 52.0% 72.5% 77.4% 96.4% 80.0% 76.1% 41.8% -1.0 pp -17.2 pp -10.6 pp 0.888 gpt-5.5: 100 OpenAI API changelog
2 GPT-5.6 Sol / v2 2026-07-09 2026-07-09 100 76.4% +0.3 pp 77.6% -0.6 pp complete 52.4% 72.9% 77.5% 96.5% 80.6% 77.1% 42.8% +1.0 pp -16.2 pp -9.6 pp 0.925 gpt-5.6-sol: 100 OpenAI API changelog
3 GPT-4 Turbo (2024-04-09) / v3 2024-04-09 2024-04-09 100 68.2% N/A 68.8% N/A complete 35.8% 63.1% 68.3% 94.4% 77.0% 70.1% 38.2% N/A -20.8 pp -14.2 pp 0.921 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
3 GPT-4o (2024-05-13) / v3 2024-05-13 2024-05-13 100 71.6% +3.4 pp 74.5% +5.7 pp complete 41.6% 67.0% 72.9% 95.3% 78.1% 71.9% 36.4% -1.8 pp -22.6 pp -16.0 pp 0.880 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
3 GPT-4o (2024-08-06) / v3 2024-08-06 2024-08-06 100 71.9% +0.3 pp 74.8% +0.3 pp complete 42.5% 67.3% 73.7% 95.4% 78.0% 71.6% 38.7% +2.3 pp -20.3 pp -13.7 pp 0.939 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
3 GPT-4o (2024-11-20) / v3 2024-11-20 2024-11-20 100 72.0% +0.1 pp 74.0% -0.8 pp complete 42.5% 67.4% 73.6% 95.4% 78.4% 72.6% 40.3% +1.6 pp -18.7 pp -12.0 pp 0.981 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
3 GPT-4.1 (2025-04-14) / v3 2025-04-14 2025-04-14 100 73.0% +1.0 pp 75.9% +1.9 pp complete 43.7% 68.4% 75.8% 95.6% 78.7% 73.0% 41.6% +1.2 pp -17.4 pp -10.8 pp 0.976 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
3 GPT-5 / v3 2025-08-07 2025-08-07 100 75.8% +2.8 pp 79.3% +3.4 pp complete 49.1% 72.3% 78.3% 96.2% 79.9% 75.2% 42.2% +0.6 pp -16.8 pp -10.2 pp 0.902 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
3 GPT-5.1 / v3 2025-11-13 2025-11-13 100 74.1% -1.6 pp 76.9% -2.5 pp complete 46.3% 70.7% 76.9% 95.7% 79.0% 73.5% 44.5% +2.4 pp -14.5 pp -7.8 pp 1.005 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
3 GPT-5.2 / v3 2025-12-11 2025-12-11 100 74.2% +0.1 pp 77.5% +0.6 pp complete 46.6% 69.9% 77.2% 95.7% 79.0% 73.6% 46.8% +2.2 pp -12.2 pp -5.6 pp 0.979 gpt-5.2-2025-12-11: 100 OpenAI GPT-5.2 model docs
3 GPT-5.3 Chat / v3 2026-03-03 2026-03-03 100 74.7% +0.5 pp 77.6% +0.2 pp complete 48.4% 70.2% 77.0% 95.8% 79.3% 74.5% 45.5% -1.3 pp -13.5 pp -6.9 pp 0.932 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
3 GPT-5.4 / v3 2026-03-05 2026-03-05 100 77.1% +2.4 pp 79.8% +2.2 pp complete 52.8% 74.2% 80.1% 96.6% 80.1% 76.0% 45.7% +0.2 pp -13.3 pp -6.7 pp 0.963 gpt-5.4: 100 OpenAI API changelog
3 GPT-5.5 / v3 2026-04-23 2026-04-24 100 79.6% +2.5 pp 81.6% +1.9 pp complete 58.6% 77.8% 81.2% 97.1% 81.6% 79.2% 50.3% +4.6 pp -8.7 pp -2.1 pp 0.929 gpt-5.5: 100 OpenAI API changelog
3 GPT-5.6 Sol / v3 2026-07-09 2026-07-09 100 78.5% -1.1 pp 80.6% -1.0 pp complete 55.6% 75.5% 80.6% 96.8% 81.4% 78.9% 44.3% -6.0 pp -14.7 pp -8.1 pp 0.924 gpt-5.6-sol: 100 OpenAI API changelog

Downloadable Table

Generated by generate_translation_prompt_evaluation.py.