Lewis Carroll, a logician, wrote Alice in Wonderland. Probably the best selling work of fiction by a mathematician ever.
Sofya Kovalevskaya wrote a Russian Childhood where she mentions how she was inspired to learn mathematics by the wallpaper in her bedroom. It had been papered over by the pages of a textbook. She also mentions how she was annoyed by Dostoyevsky.
The Irish mathematician and physicist, Hamilton wrote poetry, but was advised by his friend, Wordsworth, to stick to maths and physics.
Mathilde Marcolli who has worked with Alain Connes on Non-Commutative Geometry has written a series of anarchist/dada inspired texts. She's also working on a book investigating the notions of spatiality in mathematics and art.
David Brin who trained as an astrophysicist wrote The Postman, a science-fiction classic but which was made into a terrible film.