From An Inquiry Into the Nature and Causes of the Wealth of Nations [emphasis mine]:

“To restrain private people, it may be said, from receiving in payment the promissory notes of a banker for any sum, whether great or small, when they themselves are willing to receive them ; or, to restrain a banker from issuing such notes, when all his neighbours are willing to accept of them, is a manifest violation of that natural liberty, which it is the proper business of law not to infringe, but to support.

Such regulations may, no doubt, be considered as in some respects a violation of natural liberty. But those exertions of the natural liberty of a few individuals, which might endanger the security of the whole society, are, and ought to be, restrained by the laws of all governments; of the most free, as well as of the most despotical. The obligation of building party walls, in order to prevent the communication of fire, is a violation of natural liberty, exactly of the same kind with the regulations of the banking trade which are here proposed.”