The motivation for applying derivatives to polynomials over general fields is their use in detecting multiple roots: if $K$ is a general field, a polynomial $f(x)$ in $K[x]$ has no repeated roots (edit: in a field extension of $K$ over which $f(x)$ splits completely) if and only if $f(x)$ is relatively prime to $f'(x)$ in $K[x]$.
Formal derivatives on polynomials in $K[x]$ for a general field $K$ were introduced by Steinitz in Section 9 of his big 1910 paper on the algebraic theory of fields and used in Section 10 in order to determine when a polynomial (especially an irreducible polynomial) over a general field has a multiple root. (Perfect fields are defined in Section 11 and separable field extensions are defined in Section 13 as extensions of the "first kind".)
Bourbaki writes in the Historical Note after the section Commutative Fields in his book Algebra II that in the 19th century, Dedekind "sensed" the general notion of a derivation (look at his paper here and see the footnote by Emmy Noether on the 2nd page) and that Galois "is fully aware of the formal character" of differentiation of polynomials and used this to show an irreducible polynomial in $\mathbf F_p[x]$ has no multiple roots in his development of finite fields. So if you are looking for the very first use of derivatives outside the setting of real or complex polynomials, then it goes back to Galois for the case of polynomials over a finite field.
Going back even farther, the use of derivatives as a formal operation (no limits) to detect multiple roots of real polynomials in some sense precedes calculus by Newton and Leibniz! This is called Hudde’s rule and you can read about it here.