I guess it all depends on what feels worse, knowing you took a pass on a big one, or knowing he's out there some where dead and rotting.
I've never had the misfortune of a non-recovery, but there's only been one time I second guessed taking a pass and that feeling only lasted a few hours. My first year of hunting on my own(rifle) I had a huge buck cross behind me and had an excellent 40 yd broadside shot. Gun raised, already thinking of the trip to the taxidermist with my first buck, I saw the yellow of a school bus passing trough the trees behind him(as he was between me and the highway). That was that, I just couldn't bring myself to shoot. I waited for him to move along the trail and give me a good shot but he never did. At the end of the day, I told my dad what happened. I thought he'd be disappointed, but he wasn't. I think he was more proud of me passing on a dangerous shot than if I would have downed that monster. He even went as far as to repeat the story to all his friends after we got home. Ever since then, I too take pride in passing an iffy/unsafe shot. I believe it's one of the main characteristics of a true hunter/sportsman.
On the flip side, I have a super lucky buddy who seems to rattle/grunt in a nice buck every couple years. He shoots all of them. He's recovered about half. Don't get me wrong he's a good dude, and would never intentionally take a bad shot or wound a deer. However, buck fever does strange things to one's judgement. It can make a bad shot look good and post-shot wait time seem like days. He has gotten much better and the his last three bucks have been recovered(not the case for the first three). Hopefully, he's gotten over the effects of buck fever.
For what it's worth, I'm proud of you for taking a pass. He'll be bigger next year anyway.