Prior to Vatican II, were Protestant hymns allowed in Catholic Masses?
Define "Protestant Hymns."
It's no so easy with a generic label like that.
For example ... Bach (a Protestant) wrote the chorale O Haupt voll Blut und Wunden for his "St. Matthew's Passion." It was based on a Latin hymn written in the 13th cenutry by a Catholic. It has since come down to us as "O Sacred Head Surrounded," or, in Protestant churches, "O Sacred Head now Wounded" (which is a much more faithful translation).
So is that a Catholic or a Protestant hymn?
Here's another one: St. Francis wrote the "Canticle of the Sun." It was later translated into the hymn we know as "All Creatures of Our God and King." Ralph Vaughan Williams arranged the harmony found in most of our hymnals. In Protestant churches, it is sung under this title, but it's in the SSPX hymnal under the title "Rejoice O Mary, Heavenly Queen," with a slightly bastardized harmonization.
Is that a Catholic or a Protestant hymn?
One more: there is a beautiful Latin poem by St. Bernard of Clairvaux known as Iesu dulcis memoria - Protestant hymnals have an absolutely beautiful translation and harmonization of this hymn, sung under the title "Jesus the Very Thought of Thee."
Is that a Catholic or Protestant hymn?
"Lead, Kindly Light" is a heart-wrenching and soul-searching piece penned by John Henry Newman, a decade before he converted to the Catholic Church. It has been mostly commandeered by the Protestant churches. Should it be surrendered to them just because they use it the most, and because it was written by an Anglican, even though it's content is about Newman's dawning realization that he needed to convert?
The question is not so easy.