I had a neighbor like that once. I wish I had just talked to him about his weirdness. Instead, I let him take me to my breaking point and I went off on him. I didn't hit him or anything, I just verbally assaulted him to the point he literally pissed himself. He eventually divorced and moved away. I never had a problem with him after our "chat".
I don't recommend that approach. I like neighborhoods to have a positive vibe. Mine definitely didn't for a while. Felt more awkward than having him inserting himself everywhere.
I hate to say it but this is about your only answer, or to move. You must be very blunt and direct and make sure he knows you don't like his little snide remarks in front of your wife. The whole let him down gently over a beer approach will not work. People like this can't take a hint which is why you are in the situation to begin with. I had a really bad neighbor like this once and unfortunately the only thing that solved the problem was moving.
Or you can just burn his house down.