I am not sure, if “Safely removing” your external drive really helps, but thinking what’s the loss, I always try to remove the drive safely.
Recently I had mounted the external drive to use with WSL(Windows Subsystem Linux). After doing all the work, I carefully closed all the open programs using the drive and tried doing the unmount. It didn’t work.
Frustrated, I had been pulling the drive out without safely removing it for the last couple of times. Just to realize later that my WSL working directory was of the external drive.
So the solution was just to “cd” out of any directory which is inside the external drive and then run the “umount” command.
As simple as that(but then there are some days when you are not thinking well 🙂 )