You want to wait at the very least till it starts getting that yellow/orange blush. Depending on how many fruits your tree holds, you could try letting it almost fully ripen on the tree. I would not pick it green. One caveat, when the fruit gets to a certain ripeness stage it turns very mushy, like a Carrie or even worse.