Is there any evidence that Fermat wrote on the margin of a book "I have discovered a truly marvelous demonstration of this proposition that this margin is too narrow to contain." Everyone repeats this, but is there an image of the note? And did people write on margins of books in the 17th century when books must have been truly expensive? Has any author looked at these questions?
Yes, it is true. Fermat's own copy was used in the publication of Diophantus by Fermat's son Samuel, and he included Fermat's notes. The original with Fermat's handwriting is lost. https://www.joh.cam.ac.uk/library/special_collections/early_books/fermat.htm#:~:text=When%20reviewing%20his%20copy%20of,to%20fit%20in%20the%20margin. (The page with his most famous marginal note is reproduced, "Observatio Domini Petri de Fermat").
Fermat made many remarks on the margin of Diophantus. These remarks inspired Euler and others, and eventually all his statements were proved, though the proof of the last one took a while.
For the general question: yes people wrote on margins. I guess this was the original purpose of margins. Many old books were even bound with several blank pages in the beginning or more frequently at the end, to give more space for people to write their notes. Yes, books were expensive. But a book, especially a scientific one, was considered not a decoration object but rather a tool for work. And once one had some substantial thoughts or remarks when reading the text, the most natural place to record them was the margins, or sheets attached to the book. Please notice that paper for writing was also expensive.