New Paper on Transformer Limits in Metocean Forecasting

Our team has published a new paper in Journal of Computational Science on when transformer models work well for metocean forecasting, and when simpler convolutional models remain a better choice.

The article, “Exploring the limitations of transformer models for metocean forecasting,” was authored by Julia Borisova, Mikhail Borisov, Stanislava Vostrikova, Viktor Golikov, Andrey Kuznetsov, Gleb Solovev, Alexander A. Stepanets, Mikhail Krinitskiy, and Nikolay O. Nikitin.

The paper studies three metocean forecasting settings: long-term Arctic sea ice forecasting, atmospheric bias correction, and ocean forecasting. The authors compare transformer-based architectures with convolutional neural networks and show that transformer limitations are conditional rather than universal.

For Arctic sea ice forecasting, transformer models such as TimeSformer and SwinLSTM struggled with annual dynamics, including summer melt, while lightweight CNN baselines performed better and improved error metrics by up to 30% against existing state-of-the-art systems. In atmospheric bias correction, CNNs also outperformed transformer-based alternatives, reducing errors in Global Forecast System fields by about 20% relative to transformers.

The ocean forecasting case produced a different result: transformer models with contrastive pre-training achieved stronger performance across ocean variables, including a 40% reduction in mixed layer depth error. The study therefore offers a practical message for metocean machine learning: start with CNNs when data are limited or fine spatial structure is critical, and use transformers when representation learning and pre-training can support the task.

Read the paper