Specifically, is there any research paper or record of where someone renowned in their field performed experiments by using the properties of physics, engineering or mathematics to develop a musical instrument that created the best sound and had negligible chance to to go off key or to lose its pitch? The piano and violin are two instruments that come to mind for this question. If there are any papers that speak of efforts such as this, please post an answer below. Any help is tremendously appreciated. Thank you.
There is probably nothing "flawless" in this world. But mathematicians and physicists indeed took part in the improvement of musical instruments. See, for example Wikipedia article Well temperament which mentions the invention of the musical scale for modern keyboard instruments. Many people, scientists and musicians participated in this development. Some historians credit the invention to Simon Stevin (physicist), others to Zhu Zaiyu (mathematician). Anyway, a modern grand piano is probably a "perfect instrument", and several mathematicians and physicists took active part in the development of the theory on which its design is based. French Wikipedia gives much more detail on this history and mentions d'Alembert (mathematician) in this connection. Russian Wikipedia mentions mathematicians Henricus Grammateus and Marin Mersenne who contributed.
Let me add that the very first person who investigated the musical scale from the point of view of mathematics was Pythagoras himself. See Pythagorean tuning. So applications to music are as old as mathematics itself.
Leon Theremin, born in St. Petersburg, Russia in 1896, was a youthful enthusiast of both music and physics. As a young man working at the Physical Technical Institute in Petrograd, he noticed that something odd happened when he hooked up audio circuits to an electrical device called an oscillator in a certain configuration. The oscillator produced an audible tone when he held his hands near it, and he could shift the tone just by waving his hands back and forth. He conceived the idea of an electronic musical instrument: not just an acoustic instrument embellished or amplified electronically, but an instrument that would produce purely electronic music.
While adapting the dielectric device by adding circuitry to generate an audio tone, Theremin noticed that the pitch changed when his hand moved around. In October 1920 he first demonstrated this to Ioffe who called in other professors and students to hear. Theremin recalled trying to find the notes for tunes he remembered from when he played the cello, such as the Swan by Saint-Saëns. By November 1920 Theremin had given his first public concert with the instrument, now modified with a horizontal volume antenna replacing the earlier foot-operated volume control. He named it the "etherphone", to be known as the Терменвокс (Termenvox) in the Soviet Union, as the Thereminvox in Germany, and later as the "theremin" in the United States.
Theremin’s invention is still the only musical instrument, which is played without any physical contact. The sound it can produce runs from a drone to a whine, but is always somewhat otherworldly, and is best recognised for its prominence in the soundtrack to various early Hollywood sci-fi movies, like “The Day the Earth Stood Still.” However, the theremin does make a memorable pop music appearance in the background of the Beach Boys’ “Good Good.”
Maybe it does not have a direct relationship to your question, but I remember having a teacher in college for a Mechanics course, which developed a musical instrument himself in order to have more bass-sounds in a folk Catalan musical ensemble. Unfortunately, I haven't found any documentation related to it in English. The only information I have found is in Catalan in this page:
The abstract of the project seems semi-related to your question, as basically it is the design of a new instrument with all the physics in mind:
Design of a xeremia barítona (barytone), non-existent until now, which is a type of wind instrument that has been designed to perform Catalan chamber music. This musical instrument will be used with other traditional Catalan instruments such as the tenora and the tible (type of shawm).
Hope it helps!