Was James Madison Against Separation of Church and State?

Most likely James Madison changed his view on this subject, however he appears to be in-line with the other Founding Fathers while a public servant. Did Madison change his views after he retired, rejecting Thomas Jefferson's belief that religion is left to the States?

If there were a majority of one sect, a bill of rights would be a poor protection for liberty. Happily for the states, they enjoy the utmost freedom of religion...Fortunately for this commonwealth, a majority of the people are decidedly against any exclusive establishment. There is not a shadow of right in the general government to intermeddle with religion. Its least interference with it would be a most flagrant usurpation. I can appeal to my uniform conduct on this subject, that I have warmly supported religious freedom. It is better that this security should be depended upon from the general legislature, than from one particular state. A particular state might concur in one religious project. But the United States abound in such a variety of sects, that it is a strong security against religious persecution; and it is sufficient to authorize a conclusion, that no one sect will ever be able to outnumber or depress the rest.

--James Madison, June 12, 1788. Elliot's Debates In the Several State Conventions on the Adoption of the Federal Constitution(Virginia)