The Constitution didn’t establish a right to vote for men in general or any men in particular. It left the question of which citizens were allowed to vote fully up to the states.

Or to go deeper: The Declaration of Independence limited voting to landowners. The Constitution set no regulations whatsoever for which citizens could vote, leaving it wholly up to the states. There are various trends in state laws over time but nothing federal regarding who can vote (other than various immigration laws about who can be naturalized). Until the 15th Amendment: “The right of citizens of the United States to vote shall not be denied or abridged by the United States or by any state on account of race, color, or previous condition of servitude.”

Technically, men did not have a federally protected right to vote until women did, the 19th amendment. Though state laws had expanded to give essentially all free white men the vote in every state shortly before the Civil War, but that’s not from that federal point of view you’re so worried about.