There are apparently 2 contenders to the title of being the 1st scientific journal. The Royal Society claims that their Philosophical Transactions, the 1st issue of which came out on 6 March 1665, is "the world's oldest scientific journal in continuous publication". The very 1st scientific journal may, however, have been the French Journal des Sçavans; its 1st issue was published only 2 months earlier on January 5, 1665. The time was certainly ripe for regular and serial publications of scientific ideas.
According to this brief Wikipedia article, the Journal des Sçavans was published with interruptions until 1816 when it "became more of a literary journal, and ceased to carry significant scientific material". Hence, PT's claim to be the oldest continuously published scientific journal.
Although the Royal Society won't let you read PT unless you pay them a sum enough to ransom a kidnapped queen, the Google Books has fortunately scanned the very 1st volume of PT and has it available for free here (do you hear that Royal Society?). Here is the title page of volume 1:
This brings us to the 2nd question: What was the first scientific paper ever published on mollusks? Now, I don't know if the very 1st numbers of JS are available on the Internet, but, even if they were, I wouldn't be able to decipher that peculiar language the French use. So, I will instead look for an answer to our question in PT.
Going down the table of contents of volume 1 of PT, I come to a paper titled "An Extract of a Letter, Written from Holland, about Preserving of Ships from Being Worm-Eaten" that was published on p. 190. (Curiously, PT didn't give the authors' names with their letters until near the end of the 1st volume.)
Here is an extract from that letter*:
What were those wood-eating "worms" that the writer wrote about? Why, they were mollusks, of course; specifically, bivalves of the family Teredinidae that are commonly known as shipworms. And the funny thing about this report is that, according to its author, it was an extract of an article that had come out in the 15 February 1666 issue of the Journal des Sçavans!
Although the article was more about how to prevent the infestations of ships with shipworms than about the animals themselves, I will argue that it was probably the 1st article about mollusks published in a scientific journal.
*The articles published in PT back then were actually "letters" sent to the editor.