This usually results from Orca3D not being able to find an equilibrium. Often the cause is a surface edge becoming submerged, such as the deck edge when the vessel heels. If this is the case, add a deck (or other surface) to your model to seal it. If you do not expect an open edge to become submerged, you should check your VCG to be sure that it is correct.