If it hurts, you're doing it wrong. Sounds like neither is the right size. your small one is too small, and your big one too big. A properly sized one may occasionally fall off but it should not be immediate.
Also, falling off is only an issue if you're either wearing boxers or going commando. It doesn't happen often, but I can sense when my ring is about to fall off. I will stop and pretend to tie my shoe, and let it drop down my pant leg, and tuck it into my sock, and nobody else even knows what just went on. I can then either leave the ring in my sock, or go to the restroom and fix things.