Have you checked which program / package is consuming RAM?  

 I can't ssh in and view anything it's basically frozen when it happens 

 You should enable a log to monitor RAM consumption 

 I would also install zram to handle slight RAM overcommit on virtual nachine better. 

 Verify that you are leaving free ram for the proxmox hypervisor.

Also install the driver package in the virtual machine (virtio). 

 What system do you use? 
Big-little architecture? If yes, make sure you are up to date with the latest microcode on the host. 

https://tteck.github.io/Proxmox/ 

 Might also want to watch disk utilization. You might be having iowait throttle the rest of the system  

 never seen that on VMware ESXi or Xen hypervisors 

 Is your VM's CPU tyoe set to "host"? 

 This issue could be because of low disk space on the vm. Also had this problem due to a bug in kernel with respect to specific cpu model(was resolved by proxmox upgrade).  

 try turn off mem baloon, or ksm