To prevent the gradient norm from blowing up during the end of training, the paper introduces AdamC — it has a corrected weight decay where the weight decay is tied to the square of the learning rate:
$$\frac{\gamma_t^2}{\gamma_{\max}} \lambda$$
This is also now used in Thinky's model Inkling — https://thinkingmachines.ai/news/introducing-inkling/