Then it's kinda like, No. I don't want to go into details.

But it really depends on the situation. I feel like most people don't get mad, just pushy and annoying asking why you said no. 

If they ask a second time, I'd probably say, Please stop asking, I already said no.