When you do the sync process, you cannot link existing elements clients to existing xero clients. you can only import xero clients that are not in elements.
You should be able to 'match' the client in xero with the one in elements. see how other platforms achieve this syncing.
Thanks for logging this Louis, we certainly want to upgrade the Xero sync to allow for existing clients to be connected / mapped. It's not something we're actively working on but let's see how many votes this gets.