I did some testing that can work.
Delete the switch , or delete the cable itself. After next topology poll, the port number on the cable then updated.
In the EventLog, there is something can show connection changed, but in map itself, it doesn't update automatically. We will change the switches location in production periodically, that can give us trouble.
Please advice. Thanks!