Just as how there is a configuration for My Drives and one for Team Drives, there should also be a separated configuration for Shared with me files.
Currently it is very anoying that if you have the shared with me foles synced, they show in the same location as your own files.