Well it all depends on you, and how frequently you intend to pick the fruits. If you're really going to go out there every single day to pick fruits, then maybe it would be ok. I wouldn't put a starfruit tree close to my house. They just produce too fast and drop too fast. Also lots of little bugs around dropped fruits.
Other bad trees around the house: java plum, mulberry, and watermelon tree. Just kidding on the last one.
If you have the room, it's best to put flowering (ornamental) plants closest to house and fruiting trees a bit further back.