TL; DR: the "ganzen Zahlen" refers to positive integers. Gauss, Dedekind and Cantor were reifying new mathematical objects obtained by explicit or implicit procedures, and then forgetting the procedures. In contrast, Kronecker took the given (by God or nature) objects as fixed once and for all, and then tracked the procedures performed on them.
Around the same time as the quote that Weber ascribed to Kronecker the latter wrote a paper Über den Zahlbegriff explaining what he meant. The initial version appeared in 1887, and is translated in the sourcebook From Kant to Hilbert. The full version was published the same year in the Crelle journal, and the added part is translated by Dean. For context and commentary see Boniface, The Concept of Number from Gauss to Kronecker.
According to Kronecker (and Cayley, Sylvester, etc., it was the dominant view in the 19th century), mathematics is a natural science, and must be treated as such:
"Arithmetic and algebra belong to a well defined domain; the positive integers, the systems of integers represented by integral functions with positive integer coefficients,
are considered there as motion in kinematics and as matter in natural sciences."
Its basic objects must not only be free of contradiction, but also come from experience, and have procedural definitions that unambiguously decide in each instance what fits. Kronecker's definition of the "whole number" is similar to Frege's and Cantor's, and relies on the equipotence relation:
"Each ordinal number characterises a determined class of
equivalent collections... Thus, for instance, “three fingers” is a characteristic invariant of the class formed with the collections of three objects."
However, Kronecker opposed the platonist tendencies of Dedekind and Cantor (traceable back to Gauss) of reifying numbers and other objects beyond that, and "criticized Peirce and Peano, who thought they could extend mathematics beyond its frontiers, and dominate “all the spiritual realm”", as Boniface puts it. Furthermore:
"We have seen that for Gauss the
successive widenings of the concept of number were the most important feature of
the development of pure mathematics. Thus, he originally limited arithmetic to the
positive integers alone, but associated to this position a concept of number which
includes the rational, irrational, and complex numbers, and which therefore concerns
not only arithmetic, but also algebra and analysis. For Kronecker, on the contrary,
the concept of number must be strictly limited to positive integers, while arithmetic
is extended to algebra and to analysis.
It is very important for Kronecker to keep
the initial meanings of the concept of number and, more generally, of fundamental
concepts, because by widening these concepts to adapt them to other scientific areas,
their meanings risk being diluted. Thus fundamental mathematical concepts have
fixed meanings, determined by experience, and must not be changed. A generalization of the concept of number and, therefore, a development of mathematics through
an extension of the domain of its objects are thus forbidden."
How Kronecker planned to do away with the "forbidden objects" of algebra and analysis without discarding the results (seemingly) about them is also described in the paper. Here is Dean's summary:
"In the 1887 article “Über den Zahlbegriff” [4] Leopold Kronecker set himself the task (among other things)
of demonstrating that talk of algebraic numbers is unnecessary, in the sense that it can be eliminated in
favor of (somewhat elaborate) talk of natural numbers...
Kronecker’s approach allows us to avoid conceiving of negative numbers as a new sort of object that
enlarges our domain. Using Gauss’ modular arithmetic notion of congruence, Kronecker recasts the preceding
equation [$7-9=3-5$] as
$7 + 9x ≡ 3 + 5x \pmod{x + 1}$,
where $x$ is indeterminate. We can recapture the other way of thinking, namely that we are enlarging our
domain, if we conceive of the indeterminate $x$ as an actual element determined by the equation $x + 1 = 0$,
but from Kronecker’s perspective it is important that we need not do so.
This earlier version of the article abruptly ends at that point with a concluding paragraph, leaving
Kronecker’s stated goal of discarding the concept of irrational algebraic numbers untouched. However, an
extended version of “Über den Zahlbegriff” appeared in Crelle’s journal in 1887 [4]. Therein, Kronecker
expands §5 by inserting, before the concluding paragraph, first a quick demonstration that rational numbers
can be eliminated via indeterminates, and second the promised elimination of algebraic numbers in general".
