It goes back at least to Aristotle's De Anima, Book II, ch. 7-11 (these five chapters being respectively devoted to sight, hearing, smell, taste, and touch). This is perhaps where it started, since Aristotle was an incorrigible cataloguer of all human experience, be it either sensory or intellectual.
That there are no more than five senses was subsequently argued by Aristotle in the beginning of Book III.
Perhaps it is also interesting to notice that in Buddhist psychology, there are not the five senses, but there are the six saḷāyatana, which comprise the five
'Aristotelian' senses as well as "mind", whose sense objects include thoughts, feelings, etc. See the page for Ayatana on Wikipedia.
Since this question has been revived, I want to add an interesting perspective found in the book Inner Christianity by Richard Smoley, a respected author on topics such as religion and esotericism. He mentions the possibility of interpreting the story of Jesus and the Samaritan woman at the well as an allusion to the five senses.
One episode in John’s Gospel illuminates with special clarity the relation between the “I” and the world. Christ meets a woman of Samaria at a well and asks her for water. She refuses on the grounds that he is a Jew, and Jews have no dealings with Samaritans. In response he offers her living water and tells her to go and bring her husband back with her. She replies: “Sir, I have no husband. Jesus said unto her, Thou hast well said, I have no husband: for thou hast had five husbands; and he whom thou hast is not thy husband: in that thou sayest truly” (John 4:17–18).
From earliest times the woman of Samaria has been understood as a
personification of the soul. She has had five husbands, meaning the five senses (the number five in esoteric terms often has this connotation). She is living with another man, who is “not her husband.” Maurice Nicoll suggests that this indicates a state where the soul has begun to detach itself from the external world; “at this point the ‘soul’ vaguely turns to other interests — perhaps to some sort of philosophy or to different forms of so-called occultism, to opinions, theories and imagination and so on, in an endeavour to satisfy its thirst with truth other than the truth of the senses.”
From Inner Christianity, ch. 6.
