Published in 1842, the Vestiarium Scoticum: From the Manuscript formerly in the library of the Scots College at Douay; with an Introduction and Notes By John Sobieski Stuart has stirred up more controversy than any other book concerning the origins of tartans. As stated in the Vestiarium Scoticum, John  wrote its title and notes, while Charles produced the images of the tartans. Nearly all of the seventy-five tartans that were prescribed to both highland and lowland clans were previously unknown to the manufacturer's, yet, after the production of this book many clans adopted the tartans prescribed to them and has influenced the production of tartans ever since.

The brothers claimed to have in their possession three manuscripts dating from the sixteenth to eighteenth centuries that describe the patterns of clan tartans. Only one of these manuscripts has ever been seen, making it easier for critics to disclaim its authenticity, as well as the improbability that the only three manuscripts known to have described clan tartans are all in the possession of one family.

The brothers claimed that one of the manuscripts was signed by John Leslie, Bishop of Ross. The manuscript was dated to 1571, and was among papers belonging to Mary, Queen of Scots. These papers were transferred to various people and finally arrived in the possession of Charles and John Stuart. The authenticity of the document was disputed by Sir Walter Scott, who argued that neither historical nor linguistic evidence supported the notion that Lowlanders possessed old, traditional tartans. Scott's derision of the notion is particularly clear in his statement "I would rather suppose the author to have been some tartan weaver zealous for his craft, who wished to extend the use of tartans over the whole kingdom.”

Another factor regarding the authenticity of their book is identity of the author of the manuscript cited in the book. The author  named was Sir Richard Urquhart of the Cromarty family, is not found in that family’s genealogy records. Additionally, the Stuart brothers claimed that his purpose of documenting the clan tartans was to salvage their history, but there should not have been a need for such a document in the sixteenth century as there no particular threat to the tartan or related identities at that time. Furthermore,  the language used is inconsistent with the supposed date of creation, as well as internally inconsistent within the document,. For example in one sentence we see the spelling  ‘princyppal’, but later we see the spelling ‘principal.’ When testing for authenticity of the paper and ink in 1894, Dr. Stevenson Macadam,  a lecturer of Chemistry at the Analytical Laboratory, Surgeons Hall, Edinburgh, stated, “I am of opinion that the document bears evidence of having been treated with chemical agents in order to give the writing a more aged appearance than is entitled to.”

The authenticity has never been fully solved, and the manuscripts have yet to be found.