There are a few annoying pen/pencil habits already so I thought I'd add to the problem. If you have the protected bearings like the one's in the picture, you'll want to take off the covers, clean out the grease and use an alternative lube. I use PTFE lubricant that I already have for my printer. There are plenty of tutorials online for how to do this and get your bearings spinning with less friction.
The tip of the topper and the bearing outers are designed to be a fairly tight friction fit.
Happy fidgeting!