The turn to operators occurred in Heisenberg's 1925 paper, and it was a compromise between positions taken by Bohr and Born. Bohr wanted minimal modifications to classical mechanics, like his quantization rules for the hydrogen atom, while Born wanted a new discrete mechanics governed by difference equations. Heisenberg's idea was to keep equations of motion more or less classical, but to reinterpret symbols in them. Those symbols represent observables (position, momentum, energy, etc.), and in Hamiltonian dynamics they are functions of positions $x$ and momenta $p$, which form the phase space. Applying this idea Heisenberg discovered that his symbols would not commute, for example $xp-px=i\hbar$ with the Planck constant $\hbar$, so they could not be numbers. A mathematician friend of his pointed out that matrices do not commute, and could satisfy such relations, and this gave Heisenberg's proposal its name, matrix mechanics. Except that since electrons in an atom have infinitely many energy levels these "matrices" had to be of infinite size.
At about the same time Schrodinger was working from a different perspective, he wanted to reduce quantum effects to wave dynamics. So he represented states of quantum systems by wave functions $\psi$ on the phase space and looked for equations of motion. His idea was that quantum particles are wave packets, and he originally thought of $|\psi|^2$ as charge density, only later Born identified it with probability density. After matrix mechanics came out Schrodinger realized that he needed to introduce observables into his picture, but they could not be mere symbols. Since states are wave functions on the phase space rather than its points, observables can not be functions of them, they have to act on wave functions like matrices act on vectors. After some experimenting in 1926 he came up with representing position by a multiplication operator $\psi\mapsto x\psi$, and momentum by differentiation $\psi\mapsto-i\hbar\frac{\partial \psi}{\partial x}$. The big clue was that if we think of $x,p$ as these operators then $(xp-px)\psi=i\hbar\,\psi$. Then we can form other observables from these ones. Classical kinetic energy is $p^2/2m$, replacing $p$ by its operator we get $-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x^2}$ from the Schrodinger equation. The total energy of a classical oscillator is $p^2/2m+kx^2/2$, replacing gives the Hamiltonian of the quantum oscillator $-\frac{\hbar^2}{2m}\frac{\partial^2}{\partial x^2}+\frac{kx^2}2$. And so on.
In 1930 Dirac conceptualized this picture in his textbook Principles of Quantum Mechanics (1930), where he introduced the bra-ket notation among other things. Then von Neumann gave an axiomatic formulation based on the notion of abstract Hilbert space that he introduced, and established that matrix and wave mechanics were talking about the same thing in two different ways. In Schrodinger's picture the states became elements of a Hilbert space, namely the space $L^2$ of square integrable (wave) functions on the position space. The observables became self-adjoint operators on it. Upon choice of an orthonormal basis in this space however the states become (infinite) vectors, and the operators become (infinite) matrices. This is the Heisenberg's picture.
There is a relevant thread on Physics SE How did the operators come about? Landsman gives a nice short survey of historical developments in Between Classical and Quantum. More comprehensive references are Kragh's Quantum Generations, and Jammer's classic Conceptual Development of Quantum Mechanics.