I think it’s both the button and the grip texture around it cause it’s the contact point where I press the hardest with my hand
The plastic button has some sharp edges. You could try one these anodized aluminum mag release.
1❤️1