I mean, they do make your device slow. That why tools like InSpectre exists. For some old cpu’s like my notebooks one it can be up to 20% performance impact, so if you not planning to use it with internet (or at least as main access point via browser) ever again, why not get yourself free performance?
Also, have in mind that main memory shared with igpu, so if we talking about memory heavy tasks like 8k video render with masks, effects and stuff, then this memory will be eaten not only by app, but also with gpu encoder. 64gb would be more close to “bearable minimum” but oh well, magic apple ram I guess.