I don't think it's the seed oils, I think it's the extraction and refining process.
Sugar is something you can easily refine in your own kitchen. Anything you can refine yourself is generally going to be less toxic than stuff that requires an entire factory and chemistry lab