First, if it's starting to soften at all it's definitely ready. Next test, I poke the skin with a fingernail and often pick if no latex fills the hole (we don't market our fruit so I don't worry about aesthetics, and the little nail marks don't affect eating quality - don't know what I'd do if we cared about looks!). I still might not pick if the fruit seems really hard, I try to wait for at least a slight change in hardness. Basically, I've learned of in doubt, wait. Picked too early they never ripen well, start to wrinkle without completely losing the latex, flesh stays a pale tan color. Ripened correctly, they soften without wrinkling, lose all the latex, flesh turns the color of dark brown sugar (which pretty much describes the flavor as well). Huge difference.
As for scratching the skin looking for a color change, ours don't turn tan until they're softening so that doesn't work for us.
Truly Tropical has a couple of videos about saps that talk about when to pick.