The earliest reference I can find to "banana brackets" is in:
G. Malcolm. Data structures and program transformation. Science of Computer Programming, 14(2-3):255-280, October 1990.
Where they are clearly crescent-moon/banana shaped symbols: ⦅...⦆.
The later style using $($ and $|$ seems to be a typographic practicality, and is used by ...