[Test refactor 3/5] Notification service improvement (#15727)

* Per-folder tests reorganization

* Review comments

Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Stas Bekman <stas@stason.org>

Author: 3 people