The Negative-Positive Dichotomy is just a Mathematical Abstraction

In mathematics, negative and positive numbers exist as mirror images in infinite continua. This abstraction is generally convenient, although it can cause difficulties, such as the need for "imaginary" or "complex" numbers in the solution of differential equations. However, physicists in the last couple of centuries have come more and more to apply this negative-positive dichotomy as an absolute rule and guide to understanding the nature of reality itself -- matter vs. antimatter, negative and positive quarks, possibly, even the concepts of electrons and protons. Is it at all possible that physicists are over-applying the mathematical abstraction of negative and positive numbers, and that this abstract dichotomy is actually confusing and distorting our perceptions of physical reality?