Translation Similarity by Model Release Date

Generated: 2026-07-25 12:54:32 UTC

Back to translation prompt evaluation.

This page tracks automated reference-similarity metrics over model releases while keeping the Stephanos prompt version fixed. The overview includes twelve OpenAI releases and available Claude observations on the same 100-entry Kappa corpus.

Benchmark Overview

Translation similarity by model release date for three Stephanos prompt versions
Mean of BLEU-4, chrF++, METEOR, and ROUGE-L across the same 100 Kappa entries. Circles are OpenAI model-prompt observations; dashed lines are prompt-specific OLS fits. Claude observations are shown as diamonds and excluded from the fitted trends. These are reference-similarity scores, not calibrated measures of philological correctness or human equivalence. Download the vector PDF.

Human-equivalent dates below use a provisional automated-score target of 90.0%. Replace this target with a second-human baseline when the project has one; until then, the projection is a planning proxy, not a claim about human adequacy.

OpenAI Planning Charts

Mean automated metric
Mean automated metric.
ROUGE-L
ROUGE-L.
Same-length mean automated metric
Same-length mean automated metric.

Human-Equivalent Projection

Prompt v. Metric Target Completed releases Latest release Latest score Annualized slope Projected target date Steady check Projection status
1 Mean automated metric 90.0% 12 2026-07-09 47.2% +2.2 pp 2044-10-22 not steady linear projection from completed releases
1 ROUGE-L 90.0% 12 2026-07-09 57.7% +2.1 pp 2040-08-20 not steady linear projection from completed releases
1 Same-length mean automated metric 90.0% 12 2026-07-09 41.0% +6.6 pp 2032-12-03 not steady linear projection from completed releases
2 Mean automated metric 90.0% 12 2026-07-09 70.0% +4.0 pp 2031-08-20 not steady linear projection from completed releases
2 ROUGE-L 90.0% 12 2026-07-09 77.4% +3.0 pp 2030-08-08 not steady linear projection from completed releases
2 Same-length mean automated metric 90.0% 12 2026-07-09 31.7% +5.2 pp 2037-07-28 not steady linear projection from completed releases
3 Mean automated metric 90.0% 12 2026-07-09 72.9% +4.9 pp 2030-02-02 not steady linear projection from completed releases
3 ROUGE-L 90.0% 12 2026-07-09 80.4% +3.9 pp 2028-12-22 not steady linear projection from completed releases
3 Same-length mean automated metric 90.0% 12 2026-07-09 31.6% +6.2 pp 2034-12-31 not steady linear projection from completed releases

Model Timeline Table

Prompt v. Model Release API date Pairs Mean metric Delta vs prior ROUGE-L ROUGE-L delta Status BLEU-4 chrF++ METEOR BERTScore COMET BLEURT Same-length mean Same-length delta Same-length vs Mix. Same-length vs Comp. Length slope Actual run models Source
1 GPT-4 Turbo (2024-04-09) / v1 2024-04-09 2024-04-09 100 43.0% N/A 53.8% N/A complete 17.5% 47.9% 53.0% N/A N/A N/A 38.5% N/A -9.3 pp -0.5 pp 0.898 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
1 GPT-4o (2024-05-13) / v1 2024-05-13 2024-05-13 100 45.6% +2.6 pp 57.2% +3.3 pp complete 19.8% 50.2% 55.3% N/A N/A N/A 25.9% -12.6 pp -21.8 pp -13.1 pp 0.637 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
1 GPT-4o (2024-08-06) / v1 2024-08-06 2024-08-06 100 44.6% -1.0 pp 56.1% -1.1 pp complete 18.4% 49.7% 54.3% N/A N/A N/A 36.3% +10.4 pp -11.5 pp -2.7 pp 0.965 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
1 GPT-4o (2024-11-20) / v1 2024-11-20 2024-11-20 100 43.7% -0.9 pp 54.6% -1.5 pp complete 16.5% 49.0% 54.7% N/A N/A N/A 31.6% -4.6 pp -16.1 pp -7.4 pp 1.050 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
1 GPT-4.1 (2025-04-14) / v1 2025-04-14 2025-04-14 100 46.6% +2.9 pp 58.7% +4.1 pp complete 19.6% 51.2% 56.9% N/A N/A N/A 44.3% +12.6 pp -3.5 pp +5.3 pp 0.999 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
1 GPT-5 / v1 2025-08-07 2025-08-07 100 48.7% +2.1 pp 61.8% +3.1 pp complete 20.7% 51.9% 60.4% N/A N/A N/A 45.6% +1.3 pp -2.2 pp +6.6 pp 0.981 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
1 GPT-5.1 / v1 2025-11-13 2025-11-13 100 48.0% -0.7 pp 60.2% -1.6 pp complete 20.3% 51.9% 59.5% N/A N/A N/A 48.1% +2.6 pp +0.4 pp +9.1 pp 1.131 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
1 GPT-5.2 / v1 2025-12-11 2025-12-11 100 48.6% +0.7 pp 58.5% -1.7 pp complete 23.4% 53.9% 58.7% N/A N/A N/A 39.1% -9.1 pp -8.7 pp +0.1 pp 1.011 gpt-5.2: 3; gpt-5.2-2025-12-11: 97 OpenAI GPT-5.2 model docs
1 GPT-5.3 Chat / v1 2026-03-03 2026-03-03 100 49.1% +0.5 pp 61.1% +2.7 pp complete 23.0% 52.8% 59.5% N/A N/A N/A 48.3% +9.3 pp +0.6 pp +9.3 pp 0.987 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
1 GPT-5.4 / v1 2026-03-05 2026-03-05 100 48.4% -0.8 pp 60.1% -1.0 pp complete 23.2% 52.7% 57.5% N/A N/A N/A 47.6% -0.7 pp -0.1 pp +8.6 pp 0.978 gpt-5.4: 100 OpenAI API changelog
1 GPT-5.5 / v1 2026-04-23 2026-04-24 100 48.3% -0.1 pp 58.4% -1.7 pp complete 25.1% 53.7% 56.0% N/A N/A N/A 46.8% -0.8 pp -0.9 pp +7.8 pp 1.003 gpt-5.5: 100 OpenAI API changelog
1 GPT-5.6 Sol / v1 2026-07-09 2026-07-09 100 47.2% -1.1 pp 57.7% -0.7 pp complete 23.3% 51.8% 56.0% N/A N/A N/A 41.0% -5.8 pp -6.7 pp +2.0 pp 0.998 gpt-5.6-sol: 100 OpenAI API changelog
2 GPT-4 Turbo (2024-04-09) / v2 2024-04-09 2024-04-09 100 61.0% N/A 70.7% N/A complete 40.0% 64.5% 68.7% N/A N/A N/A 12.5% N/A -35.2 pp -26.5 pp 0.888 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
2 GPT-4o (2024-05-13) / v2 2024-05-13 2024-05-13 100 61.3% +0.4 pp 71.8% +1.1 pp complete 39.7% 64.3% 69.5% N/A N/A N/A 24.0% +11.4 pp -23.8 pp -15.0 pp 0.835 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
2 GPT-4o (2024-08-06) / v2 2024-08-06 2024-08-06 100 61.6% +0.3 pp 72.3% +0.5 pp complete 38.7% 64.7% 70.8% N/A N/A N/A 26.1% +2.1 pp -21.6 pp -12.9 pp 0.922 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
2 GPT-4o (2024-11-20) / v2 2024-11-20 2024-11-20 100 62.4% +0.8 pp 72.4% +0.0 pp complete 40.2% 65.8% 71.4% N/A N/A N/A 26.7% +0.6 pp -21.0 pp -12.3 pp 0.900 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
2 GPT-4.1 (2025-04-14) / v2 2025-04-14 2025-04-14 100 63.2% +0.7 pp 73.1% +0.7 pp complete 41.2% 66.0% 72.4% N/A N/A N/A 32.2% +5.5 pp -15.6 pp -6.8 pp 0.948 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
2 GPT-5 / v2 2025-08-07 2025-08-07 100 67.1% +3.9 pp 76.2% +3.2 pp complete 47.7% 70.0% 74.5% N/A N/A N/A 27.4% -4.8 pp -20.4 pp -11.6 pp 0.821 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
2 GPT-5.1 / v2 2025-11-13 2025-11-13 100 65.5% -1.6 pp 74.6% -1.6 pp complete 45.3% 68.7% 73.5% N/A N/A N/A 33.7% +6.4 pp -14.0 pp -5.3 pp 0.966 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
2 GPT-5.2 / v2 2025-12-11 2025-12-11 100 67.5% +2.0 pp 76.0% +1.4 pp complete 48.1% 70.4% 75.6% N/A N/A N/A 26.9% -6.8 pp -20.8 pp -12.1 pp 0.922 gpt-5.2-2025-12-11: 100 OpenAI GPT-5.2 model docs
2 GPT-5.3 Chat / v2 2026-03-03 2026-03-03 100 67.3% -0.2 pp 76.2% +0.2 pp complete 48.1% 69.8% 75.2% N/A N/A N/A 29.9% +2.9 pp -17.9 pp -9.1 pp 0.889 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
2 GPT-5.4 / v2 2026-03-05 2026-03-05 100 67.6% +0.2 pp 77.2% +1.0 pp complete 46.8% 69.7% 76.5% N/A N/A N/A 32.1% +2.2 pp -15.7 pp -6.9 pp 0.940 gpt-5.4: 100 OpenAI API changelog
2 GPT-5.5 / v2 2026-04-23 2026-04-24 100 69.9% +2.3 pp 78.0% +0.8 pp complete 51.9% 72.4% 77.3% N/A N/A N/A 29.8% -2.3 pp -17.9 pp -9.2 pp 0.888 gpt-5.5: 100 OpenAI API changelog
2 GPT-5.6 Sol / v2 2026-07-09 2026-07-09 100 70.0% +0.1 pp 77.4% -0.6 pp complete 52.3% 72.9% 77.4% N/A N/A N/A 31.7% +1.9 pp -16.1 pp -7.3 pp 0.925 gpt-5.6-sol: 100 OpenAI API changelog
3 GPT-4 Turbo (2024-04-09) / v3 2024-04-09 2024-04-09 100 58.9% N/A 68.6% N/A complete 35.8% 63.0% 68.1% N/A N/A N/A 23.3% N/A -24.4 pp -15.7 pp 0.921 gpt-4-turbo-2024-04-09: 100 OpenAI GPT-4 Turbo model docs
3 GPT-4o (2024-05-13) / v3 2024-05-13 2024-05-13 100 63.9% +5.0 pp 74.3% +5.7 pp complete 41.5% 66.9% 72.7% N/A N/A N/A 22.4% -1.0 pp -25.4 pp -16.6 pp 0.880 gpt-4o-2024-05-13: 100 OpenAI GPT-4o model docs
3 GPT-4o (2024-08-06) / v3 2024-08-06 2024-08-06 100 64.4% +0.6 pp 74.6% +0.3 pp complete 42.4% 67.2% 73.5% N/A N/A N/A 24.8% +2.4 pp -23.0 pp -14.2 pp 0.939 gpt-4o-2024-08-06: 100 OpenAI GPT-4o model docs
3 GPT-4o (2024-11-20) / v3 2024-11-20 2024-11-20 100 64.3% -0.2 pp 73.9% -0.8 pp complete 42.5% 67.3% 73.4% N/A N/A N/A 26.8% +2.0 pp -21.0 pp -12.2 pp 0.982 gpt-4o-2024-11-20: 100 OpenAI GPT-4o model docs
3 GPT-4.1 (2025-04-14) / v3 2025-04-14 2025-04-14 100 65.8% +1.6 pp 75.7% +1.9 pp complete 43.7% 68.4% 75.6% N/A N/A N/A 29.7% +3.0 pp -18.0 pp -9.3 pp 0.977 gpt-4.1-2025-04-14: 100 OpenAI GPT-4.1 model docs
3 GPT-5 / v3 2025-08-07 2025-08-07 100 69.7% +3.9 pp 79.2% +3.5 pp complete 49.1% 72.3% 78.2% N/A N/A N/A 29.5% -0.2 pp -18.3 pp -9.5 pp 0.902 gpt-5-2025-08-07: 100 OpenAI GPT-5 model docs
3 GPT-5.1 / v3 2025-11-13 2025-11-13 100 67.6% -2.1 pp 76.7% -2.6 pp complete 46.2% 70.7% 76.8% N/A N/A N/A 33.3% +3.8 pp -14.5 pp -5.7 pp 1.005 gpt-5.1-2025-11-13: 100 OpenAI GPT-5.1 model docs
3 GPT-5.2 / v3 2025-12-11 2025-12-11 100 67.7% +0.1 pp 77.3% +0.6 pp complete 46.6% 69.9% 77.1% N/A N/A N/A 37.5% +4.2 pp -10.2 pp -1.5 pp 0.980 gpt-5.2-2025-12-11: 100 OpenAI GPT-5.2 model docs
3 GPT-5.3 Chat / v3 2026-03-03 2026-03-03 100 68.1% +0.4 pp 77.4% +0.1 pp complete 48.3% 70.1% 76.7% N/A N/A N/A 35.0% -2.6 pp -12.8 pp -4.0 pp 0.932 gpt-5.3-chat-latest: 100 OpenAI GPT-5.3 Instant release
3 GPT-5.4 / v3 2026-03-05 2026-03-05 100 71.6% +3.4 pp 79.6% +2.2 pp complete 52.7% 74.1% 79.9% N/A N/A N/A 35.2% +0.3 pp -12.5 pp -3.8 pp 0.963 gpt-5.4: 100 OpenAI API changelog
3 GPT-5.5 / v3 2026-04-23 2026-04-24 100 74.2% +2.7 pp 81.2% +1.6 pp complete 57.6% 77.2% 80.8% N/A N/A N/A 38.7% +3.5 pp -9.0 pp -0.3 pp 0.929 gpt-5.5: 100 OpenAI API changelog
3 GPT-5.6 Sol / v3 2026-07-09 2026-07-09 100 72.9% -1.3 pp 80.4% -0.8 pp complete 55.5% 75.4% 80.4% N/A N/A N/A 31.6% -7.2 pp -16.2 pp -7.4 pp 0.925 gpt-5.6-sol: 100 OpenAI API changelog

Downloadable Table

Generated by generate_translation_prompt_evaluation.py.