To expand on below
To make this more personal, if we assume Kavanaugh is a rapist based on the evidence at hand, this is the message, I as a man, will receive:
How you behave doesn't matter. It doesn't matter what you've accomplished or how you've treated women. If one woman comes forward and makes an accusation against you, no matter how vague or difficult to corroborate, we will believe her, and assume you are a rapist, assaulter, or abuser. There is no amount of goodwill you can bank that will protect you from this.
Maybe it's worth it. Maybe this is better or a just comeuppance for the fear women have had to live with.
But it is a real cost. And not all men will respond to this by being super-careful. We are removing an incentive to behave well. If I'm going to be considered a rapist anyway, why not?
I'm not saying this is how I would personally respond, how how men should respond. Basic human decency and compassion should lead to men treating women with respect. But obviously that hasn't worked for everyone, and for the type of man on the margin of treating women badly, we're removing a reason to exercise restraint.
Again, I'm not saying this is the only or most important concern. But it is a real concern