Here is Wilkins's translation of Riemann's paper. Remarks at the start of the paper show that Riemann was well aware of the convergence issues:"For this investigation my point of departure is provided by the observation of Euler that the product $\prod\frac{1}{1-\frac{1}{p^{s}}}=\sum\frac{1}{n^{s}}$ if one substitutes for $p$ all prime numbers, and for $n$ all whole numbers.
The function of the complex variable $s$ which is represented by these two expressions, wherever they converge, I denote by $\zeta(s)$. Both expressions converge only when the real part of $s$ is greater than $1$; at the same time an expression for the function can easily be found which always remains valid". The reference here is not to Euler's 1737 "factorization" of the harmonic series but to 1748 Introductio in Analysin Infinitorum, where the identity for $s>1$ appears. Although Euler did not work with convergence in the modern sense he already knew the difference between $s=1$ and $s>1$ cases since he summed the Basel series in 1735 ($s=2$, later all even $s$), and used unlimited growth of harmonic series to prove infinitude of primes in that 1737 paper.
Cauchy addressed convergence of the $p$-series for real $s$ in his texts, and the infinite product reduces to a sum by logarithming, a trick used by Riemann later in the paper, and by others before him. In fact, Dirichlet used analogous identity for $L$ functions, with a character $\chi$ on top of the fractions, in his famous 1837 paper on primes in arithmetic progressions. And in 1848 Chebychev presented to St. Petersburg Academy his results, published in early 1850s, on the asymptotics of the count of primes (a version of the "prime number theorem"), where he logarithms the Euler product for real $s>1$, and derives an identity for $\zeta(s)$ (not under this name, of course). In 1859, when the paper was published, Riemann was familiar with the work of Dirichlet and likely Chebychev (who is mentioned in his unpublished notes), so he saw replacing real $s$ with complex one, and rephrasing the convergence in terms of real parts, as a minor issue. It appears that Kronecker simply wrote down what was a well known folklore.
The main insight of the paper is a close relation discovered by Riemann between specific traits of the distribution of primes and locations of the zeros of the $\zeta$ function, which lead to strengthened formulations of the prime number theorem. The problem was of high interest to contemporary mathematicians, with Gauss and Bertrand in addition to Dirichlet and Chebychev making important contributions prior to Riemann's work. The paper redirected the further work on the prime number theorem into analytic vein leading to Hadamard's and Vallee Poussin's 1896 results on $\zeta$ zeros. See Bombieri's exposition in the millenium problem description.
This being said, the standards of rigor weren't modern ones even after Gauss. In another paper Riemann used the Dirichlet principle to prove existence of a holomorphic function with specified poles on a Riemann surface, the kind of variational existence argument later famously criticized by Weierstrass. According to Klein's Development of Mathematics in the 19th Century, this did not prompt anyone, Weierstrass included, to doubt his results. To paraphrase, rigor is a friend but insight is a better friend. This hasn't changed if Witten's Fields medal and the vast literature on mirror symmetry are any indications. Perelman left out many technical steps and conspicuously declined to publish his proof of geometrization in refereed journals, and so on.
