It’s racist because it’s racist. It has historically been used in a racist manner, and that’s socially recognized. So, if you do it now, it’s interpreted as racist.
(Much like the swastika is a hate symbol because the swastika is a hate symbol. It has been used as such and now it’s socially recognized as such (even though there’s nothing inherently bad about a bunch of geometric lines (and, in fact, was a positive symbol for the first 5,000 years of its use)).)
https://xkcd.com/2929/