So you admit that the knowledge of YOUR actions (the two men) would destroy the family. Removing either you or the husband from it would prevent that. But you're certain you aren't as complicit. And somehow this lets you sleep at night.
My hope was that just the thought of having to face the person YOU are hurting would be enough to wake you up to the reality and level of hurt you are causing.
I have the advantage of perspective in this. I've unfortunately been EVERY player in this game...the third-party, the cheating spouse, and the scorned spouse. I was shit. It's a wrong that can't be righted, ever. I was THAT hurt and fucked in the head by my wife's cheating. But eventually I got my head wrapped around it, and have tried to pay that forward by warning off others, as I and others here have tried to do. I only wish I had that benefit of counsel.
Do with this as you will. Your life, your destiny, your conscience.