Granite should be ok. It is ok for Aquariums also, which should be an extra indication of suitability.
It is inert, but unlike clay particles, perlite etc, doesn't absorb water. That could either be an advantage or disadvantage, depending on requirements.
Because of its density, it is not prone to float onto the surface like some other light components.
Very fine milled granite is a source of K, potassium in organic farming. Silica is also present, which is a mineral often ignored in fertiliser applications.