The earliest journals were multidisciplinary, they were published by academies since
the second half of
17th century. Before that time, communication was only by books and letters. Letters sometimes were copied and circulated. The first French, English and German journals were:
Journal des sçavans, 1665 (published papers in all sciences and humanities)
Philosophical transactions of the Royal Society, 1665 (mainly physics and mathematics)
Acta eruditorum, 1682 (Started by Leibniz, published in physics and mathematics).
A curious example of multidisciplinary journal of 18th century was
Ladies' Diary started in 1704, where many mathematical papers were published, for example the
seminal papers of Landen on "Landen's transform".
Specialized mathematical journals appear in early 19 century.
The oldest specialized mathematical journal which still continues is the Journal fur die reine und angewandte Mathematik (a. k. a. Crelle's journal), started in 1826. A French analog (with identical title) Journal de Mathematiques pures et appliquee (Liouville's journal) started in 1836.