Yah, some trees require several years before a judgement call can be made. The glenn is famously a workhorse mango, yet I know at least 2 folks who've had to wait several years for theirs to bear. If you had a 1/10 acre lot and you only had space for 2 or 3 trees, then I could see why you would be getting antsy. But I remember you having a pretty good sized lot, so removing the tree this early in the game doesn't seem like it's going to serve much of a purpose.