The reason disposing of used citrus solvent is illegal is not just because of the solvent, but because of the chemicals the solvent dissolves. Even if the carrier is safe, what it carries is not.
Also, I doubt you drink much in your orange juice. iirc, the solvent's in the rinds--not the pulp. That's akin to saying that syanide is safe as you eaten lots of cherries and peaches. (Cyanide may extracted from the stones of peaches and cherries.)