i collect vinyl records and i had a similar situation like this happen to me about a year ago over some pretty big $$$ 7"s.
i never gave the dude his deposit back and after i dug up some dirt on him ripping people off he never harassed me again.
but yeah! a deposit is a deposit. it's his lose. not yours. dude shouldn't have messed around if he was just gonna back out.