I always got the feeling that Matt and Simon just might have a "secret" open relationship. Or that the two of them don't mind bringing in another guy into their married sex life. Not just some random guy, but someone they both know personally and trust.
I say secret because of Matt's celebrity and his status, including Simon's as well. But it wouldn't surprise me one bit if one day it ever came out that they have an open relationship, or have had a threesome with another guy, like Jonathan Bailey. That's just the vibe I get from those two.