Authors

• Tae-Geun Kim (Yonsei University, South Korea)

Abstract

This study proposes two novel learning rate schedulers: the Hyperbolic Learning Rate Scheduler (HyperbolicLR) and the Exponential Hyperbolic Learning Rate Scheduler (ExpHyperbolicLR). These schedulers attempt to address the inconsistent learning curves often observed in conventional schedulers when adjusting the number of epochs. By leveraging the asymptotic behavior of hyperbolic curves, the proposed schedulers maintain more consistent learning curves across varying epoch settings. Experimental results on various deep learning tasks and architectures demonstrate that both HyperbolicLR and ExpHyperbolicLR maintain more consistent performance improvements compared to conventional schedulers as the number of epochs increases. These findings suggest that our hyperbolic-based learning rate schedulers offer a more robust and efficient approach to training deep neural networks, especially in scenarios where computational resources or time constraints limit extensive hyperparameter searches.

Important links

• arXiv: 2407.15200
• GitHub: Axect/HyperbolicLR

Citation

@misc{kim2024hyperboliclr,
    title={HyperbolicLR: Epoch insensitive learning rate scheduler}, 
    author={Tae-Geun Kim},
    year={2024},
    eprint={2407.15200},
    archivePrefix={arXiv},
    primaryClass={cs.LG},
    url={https://arxiv.org/abs/2407.15200}, 
}