Most factory alloy wheels are painted (with a layer of clear coat) like the rest of the car.
Which means it's equally vulnerable to scratches and swirls if not cleaned with proper equipment, and fading.
So, I tend to avoid using harsh cleaners or brush.
Waxing the wheels will help in the cleaning process, as brake dust and dirt will be less clingy.
But the wax will wear off quickly due to the heat... so, don't expect long lasting effect.
Just depends on how much effort and fussy you are with them.