Nope, don't use rubber gloves. But I rarely touch anything with my hunting gloves on if I can help it. My decoy is only a 2D view of a doe feeding. You can see her full rear and the tail is the dominating feature. I put one of my VS-1 bombs behind her. I think the issue I had is that the bucks spooked when expecting some movement. The one stomped a couple times and got noticeably confused when it don't move.
I had 2 reasons for trying the decoy. One to keep the heat off me while in stand due to being generally low in trees and sometimes limited cover. The other is b/c in years past I was able to grunt up some bucks but they would look around then turn and walk away or skirt around the edge, just out of range.
My days of decoy hunting may not be over, but I was successful without it this year.