Neural networks are becoming massive monsters that are hard to train (without the "regular" 12 last-generation GPUs).

Let me introduce you to Zero-Cost proxies

References

1. https://www.technologyreview.com/2022/08/05/1056814/automation-ai-machine-learning-automl/
2. https://iclr-blog-track.github.io/2022/03/25/zero-cost-proxies/