Hi MarkoS,
First let me mention that since Hylocereus sp. hybridize so easily, you may never get an exact identification as to which species you have. Authors of books and scholarly articles cannot even agree on the number of species in the genus, which some sources indicating as few as 14 and others listing 25 or more. Having said that, below are some of the characteristics used:
A common error is to identify the species by fruit flesh color, ie. all white-fleshed fruit are H. undatus, all magenta-fleshed species are H. polyrhizus. None of the original descriptions include flesh color. So what are some of the identifying characteristics? As you allude to, flower characteristics are commonly used, including flower color; length of the flower; size and distance relationship between the stigma and the anthers, color of the stigma, etc. Stem characteristics are also utilized, including stem color (many are bright green and others have a bluish tint); distance between areoli, the shape of the area between two areoli; the number of spines on each areola (usually 0-8), etc. This is by no means an exhaustive list; there are many more but I don't bore everyone most than I already have.
As to the four-sided stem, this is often seen on immature seedlings. Eventually as the plant matures, the new growth switches to the three-sided stem. Incidentally 5-sided stem formations have been reported, and occassionally poly-sided growth occurs.