The Math Behind Keras 3 Optimizers: Deep Understanding and Software | by Peng Qian | Aug, 2024

It is a bit completely different from what the books say.

Optimizers are an important a part of everybody working in machine studying.

Everyone knows optimizers decide how the mannequin will converge the loss perform throughout gradient descent. Thus, utilizing the best optimizer can enhance the efficiency and the effectivity of mannequin coaching.

Moreover basic papers, many books clarify the rules behind optimizers in easy phrases.

Nevertheless, I just lately discovered that the efficiency of Keras 3 optimizers doesn’t fairly match the mathematical algorithms described in these books, which made me a bit anxious. I fearful about misunderstanding one thing or about updates within the newest model of Keras affecting the optimizers.

So, I reviewed the source code of a number of frequent optimizers in Keras 3 and revisited their use circumstances. Now I wish to share this information to save lots of you time and assist you grasp Keras 3 optimizers extra shortly.

For those who’re not very accustomed to the newest adjustments in Keras 3, right here’s a fast rundown: Keras 3 integrates TensorFlow, PyTorch, and JAX, permitting us to make use of cutting-edge deep studying frameworks simply via Keras APIs.