I check for opengraph's info for preview cards in every url. Thus, I end up hitting the server even for images and videos. When that happens, you also receive the content-type of the file in the HTTP header. I use that to determine how to render the media. 

I still check extensions first, but that has proven to be problematic (some image servers serve different formats, other urls have query params that affect the discovery and many don't actually offer extensions at all).