Lucky you! (or me :-) ). This question was answered a while back on Math.SE
I found his original thoughts in the translated version of
"Institutiones calculi differentialis cum eius usu in analysi
finitorum ac doctrina serierum, volume 1", chapter 7. The translation
is called "Foundations of Differential Calculus" and a link is found
The earliest reference I can find to "banana brackets" is in:
G. Malcolm. Data structures and program transformation. Science of Computer Programming, 14(2-3):255-280, October 1990.
Where they are clearly crescent-moon/banana shaped symbols: ⦅...⦆.
The later style using $($ and $|$ seems to be a typographic practicality, and is used by ...
On page 10 of that book the author wrote
The most important example of a ground field is the field of common rational numbers for which I use the freely invented symbol 9...
where he uses the symbol in question instead of the 9 that I used above.