Don't worry about it. Dicks are weird like that, and you had a lot going on. Drinking, first experience, watching your wife get naked with another man in front of you for the first time, and then for him to be not just large, but comically large. You enjoyed it, your wife's lover enjoyed it, and your wife enjoyed it.
Now that you're more aware of the sort of thing to expect, in terms of your feelings, that the other guy might be huge (or will be, if it's the same guy), then try again. No pressure. It doesn't have to be a "threesome", which can be kind of over-rated anyway, IMO. When I've had the opportunity to be with two women at once, it's pretty hard not to focus on one at a time, even though the fact of the other one being there is hot. Don't drink next time. Enjoy watching her fuck, whether you get hard or not. You probably will, and then after he pulls out, you can get a feel of your wife's freshly fucked, stretched pussy. You'll probably cum in about 5 seconds. Again, so what? Everyone's having fun, right?