This book is excellent. Understanding Cryptography by Christof Paar and Jan Petzel. This is where I learned the math from first principles

https://link.springer.com/book/10.1007/978-3-662-69007-9 

If you’re looking for something more hands on, Jimmy Song’s Programming Bitcoin is a great Python resource 

https://github.com/jimmysong/programmingbitcoin