Your opinion is your opinion. You can write it. It does mean you don’t really care about the non binary person that uses those pronouns. Refusing they/them erases the visibility of non binary people. But nobody said you had to care either if it comes down to it. But don’t expect people who use those pronouns to put up with your erasure.
People with learning disabilities don’t get to dictate a non binary person’s pronouns even if it’s hard either. People with learning disabilities to get to dictate how they will accept being talked about but this really isn’t the argument you think it is.
Erase people’s identities if you must, but don’t expect them to put up with it.