A wife held him back from doing the things he really wanted to.

This, of course, refers back to the actual ball and chain, which was a heavy metal ball secured to a prisoner's leg by means of a chain and manacle. The ball and chain was in use in both Britain and the USA by the early 19th century (and possibly much earlier). The earliest citation in print is from The Times,