Yes, they can be difficult to time. I usually start taking one off every couple of days starting when they are full size - usually about now. I let them ripen on the counter, which can take some time. When they start tasting “right”, I begin harvesting them in greater number. In my yard, they are typically optimal to be picked green around In early July. That timing will differ slightly annually and obviously may be different in your location.
They are not bad when let ripen on the tree- just not quite as good as those picked mature green.