It tastes good but not really good. There's something intriguing and tropical about the flavor that compels one to want to eat more. But at the same time, the flavor is a little watery, slightly insipid, kind of sour and not very sweet. I think most people would agree it's an okay fruit but not a really good one.
The shape of the fruits are very ornamental and decorative though. It's the type of thing that makes a good garnish on a salad.
You are right though, a full ripened fresh one tastes noticeably and substantially better than the typical one found at a supermarket.
Maybe the typical supermarket one is a 6 or 6.5 out of 10, while a fresh fully ripe one is 7.5 , maybe approaching 8.
Fresh has more and better flavor than supermarket one. If you don't feel like the flavor of the supermarket fruit is anything special, that might change if you try a good one.
This is a comment I found left by somebody else:
"A starfruit picked fresh off the tree tastes substantially different than all the ones I have tasted from supermarkets. True, it was kind of bland, a bit watery, sour, and very slightly astringent, but it also had a very compelling tropical-like flavor to it that made me keep eating."