If they won't obey their marriage vows, how can they be trusted to obey any vows or oaths they take upon entering their leadership role?
This was almost an exact quote of Harry Truman's so you're in good company in your opinion.
I remember hearing something several years ago about how Truman was the only president (that we know of) to have never cheated on his wife. That, coupled with what I already knew about him, made me respect him more than I already did.
One website on US Presidents said that, actually, most have been faithful to their wives--only 5 Presidents seem to have been known to definitely be unfaithful--Harding, FDR, JFK, Lyndon Johnson and Clinton. Some examples of more modern Presidents:
Teddy Roosevelt (faithful to both wives, being widowed and remarried before becoming President)
Ronald Reagan (at least to Nancy, I'm not sure about his whole relationship with first wife Jane Wyman)
Gerald Ford
Richard Nixon (I think, not absolutely positive)
George Bush Sr (a whisper about one lady, Jennifer Fitzgerald, angrily denied by all, no solid evidence)
George Bush Jr
Some of our most popular Presidents have been unfaithful, though it wasn't widely known during their Presidency--Dwight Eisenhower (a relationship during WW2), JFK, FDR, Bill Clinton. Of course, on our most disastrous (and corrupt) Presidents, Harding, was a big philanderer.
It used to be considered (if known and especially if the press didn't like the candidate) a deal-breaker--witness Gary Hart. Post-Clinton, I don't think it's as important. Three of the top candidates for the Republican nomination--McCain (first wife after his release from being a POW), Giuliani and Newt Gingrich (blech) have all admitted to affairs. Don't know about the Democrats since all the media attention focuses on Hilary Clinton.