2023-09-28
Shrink-Perturb improves architecture mixing during Population Based Training for Neural Architecture Search
Publication
Publication
In this work, we show that simultaneously training and mixing neural networks is a promising way to conduct Neural Architecture Search (NAS). For hyperparameter optimization, reusing the partially trained weights allows for efficient search, as was previously demonstrated by the Population Based Training (PBT) algorithm. We propose PBT-NAS, an adaptation of PBT to NAS where architectures are improved during training by replacing poorly-performing networks in a population with the result of mixing well-performing ones and inheriting the weights using the shrink-perturb technique. After PBT-NAS terminates, the created networks can be directly used without retraining. PBT-NAS is highly parallelizable and effective: on challenging tasks (image generation and reinforcement learning) PBT-NAS achieves superior performance compared to baselines (random search and mutation-based PBT).
| Additional Metadata | |
|---|---|
| doi.org/10.3233/FAIA230294 | |
| Frontiers in artificial intelligence and applicationsFAIA | |
| Distributed and Automated Evolutionary Deep Architecture Learning with Unprecedented Scalability , Optimization for and with Machine Learning | |
| 26th European Conference on Artificial Intelligence, ECAI 2023 | |
| , | |
| Organisation | Centrum Wiskunde & Informatica, Amsterdam (CWI), The Netherlands |
|
Chebykin, A., Dushatskiy, A., Alderliesten, T., & Bosman, P. (2023). Shrink-Perturb improves architecture mixing during Population Based Training for Neural Architecture Search. In Proceedings of the 26th European Conference on Artificial Intelligence, ECAI 2023 (pp. 381–388). doi:10.3233/FAIA230294 |
|
