I see what you mean and when I first heard that saying I thought so too. That’s why I developed a bit after. In my opinion it does not mean “if you’re not with us you’re against us in an irreversible manner” at all (I don’t fight against individuals, I fight against a toxic ideology). To me, it mostly aims to denounce the fact that centrists are actually right wing people who don’t admit it. And, at least in France, the situation tends to confirm that : Macron was officially “neither on the left nor on the right” and it has become clearer and clearer that he is, in fact, completely on the right end of the political spectrum (if not worse for some of our ministers like Retailleau who clearly carries a far right ideology).

In France we have a saying “Si t’es pas de gauche, t’es de droite” i.e “if you’re not a leftist, then you’re on the right wing” meaning that if you do not explicitly reject the paradigm supported by the right then you’re implicitly accepting it and thus indirectly support right wing and conservatism.