Did Newton view mathematics only as a tool for natural philosophy?

Question

Or did he think it was also important to explore the structure of mathematics?

In what works of Newton can I verify this?

Answer 1

Certainly no. If you look at his mathematical papers, you see that he was also interested in "pure mathematics". For example, his great paper on classification of real cubic curves. It has no relation to "natural philosophy". Or his method of expansion of an algebraic function into a power series (which is called the Puiseux series nowadays). There was no problem of physics at that time related to this.

Answer 2

Q: "Did Newton view mathematics only as a tool for natural philosophy?" 
"Did he think it was also important to explore the structure of mathematics?" 

The answer by Alexandre Eremenko mentions Newton's work on some purely mathematical problems, e.g. in geometry, including problems that had no counterpart in natural-philosophy (physics). Hence his answer 'no' to the first part of the present question.

But the answer to the other part of the present question may be different. There is material in Newton's (mostly later) writing that deals with foundational matters. The present question includes the idea of "exploring the structure of mathematics", and this arguably includes topics such as mathematical certainty and mathematical foundations. Newton's Principia considers at least two problems of mathematical foundations, and proposes solutions (see below).

Also, a recent book -- Niccolò Guicciardini's (2009) Isaac Newton on Mathematical Certainty and Method (MIT Press) (preview at https://books.google.com/books?id=U4I82SJKqAIC ) -- deals with a variety of foundational topics in Newton's work and is also a useful guide to numerous places in Newton that illustrate points relevant to the present question.

As the title of Guicciardini's book implies, Newton gave particular attention to problems of mathematical certainty, and among them the logical problems of newer mathematical methods related to 'calculus', notably (in the 1680s) Cavalieri's 'indivisibles', now seen as related also to the later infinitesimals, insomuch as similar results could be obtained by either method. Entities such as 'indivisibles' were in doubt because of their seeming ambiguity (was their measure zero or non-zero?).

Newton compared these uncertainties to methods regarded by himself (along with some others of his time and later) as a 'gold standard' of certainty in mathematical reasoning: the object of Newton's admiration was the example of reasoning and proof contained in Euclid's Elements of geometry. Newton tried to offer geometrical justifications, up to Greek standards, of a method that solves the same problems as the indivisibles and their relatives. (Newton in the later part of his career gave warm praises of Greek geometry and of Euclid : Guicciardini (2009) quotes at least one of them, and Newton's early biographer Pemberton (editor of the 3rd edition of Principia) told of another.)

On this first foundational point, Newton's proposed solution to the problem of the indivisibles was his theory of limits ('first and last ratios'), described in Principia Book 1, section 1 (p.41 of the 1729 English translation, available at https://archive.org/details/bub_gb_Tm0FAAAAQAAJ/page/n84/mode/1up ). Newton's theory of limits was well appreciated by some early savants, for example it was relied on and much cited by Pierre Varignon from about 1701, at the Paris Academy of Sciences, when he was defending the (Leibnizian) calculus against sharp criticisms made by Michel Rolle. Newton's theory of limits was also later explained at (much) greater length in Colin Maclaurin's (1742) 'Theory of Fluxions'. That work in turn provided the basis cited by Jean-Étienne Montucla in his well-known 'Histoire des mathématiques' (vol.3:110-119) where he stated his belief that the calculus and the method of fluxions were shown to be valid by Maclaurin's demonstrations -- and the Leibnizian calculus and Newton's method of fluxions had to be equally as certain as each other, since they are fundamentally the same, and Maclaurin had demonstrated Newton's method without any assumption of infinitesimals or anything else that could lend itself to dispute. In this way a line of literature begun by Newton's theory of limits had by the end of the 18th century attained considerable respect.

The other foundational question to mention here is about boundaries of mathematical certainty: this arises mainly implicitly in Book 3 of the Principia. Newton portrayed physical reasoning as subject to a set of rules that contrast with conditions for mathematical reasoning (which he did not describe, probably assuming familiarity on the part of his readers). His 'Rules of reasoning in philosophy' (1729 English translation at https://archive.org/details/bub_gb_6EqxPav3vIsC/page/202/mode/1up ) include both a principle of induction from phenomena (Rules I to III) and also a caution that conclusions gained from that process may be 'subject to exceptions' (Rule IV, at https://archive.org/details/bub_gb_6EqxPav3vIsC/page/205/mode/1up ). Thus the rules clearly carry an implication that conclusions of physical reasoning possess a degree of assurance less than the mathematical certainty that may be attained by consideration of the mathematical models and other similar matters described in book 1. Thus Newton appears to deal with a fundamental boundary between matters capable of mathematical certainty and other areas of enquiry.

In sum, Newton does appear to have been concerned at least with some isolated though important topics of mathematical structure or foundations, even though it has also been said, perhaps on a broad view, that he "wrote little on the foundations of mathematics, as now understood".

Answer 3

Newton came at the end of a long line of people who believed in the tradition of magic, alchemy, and cabala - as rooted in Jewish mysticism. Newton writings strongly suggest he believed that the universe in general, and the bible in particular is a cryptogram set by the Almighty.

As such, he also saw mathematics (and possible his four rules) as a tool to be used in his hermeneutic investigations. For example, on the basis of his interpretation of the Book of Revelations he calculated that the world would not end before 2060 CE. He also wrote a book about Solomon's Temple, interpreting it as a source of mathematical knowledge.