The reason for that is that SSdb stores the information about how the topology looks like. Normaly, during a SS save, the FT server is synched with the primary. This is the reason for example for having a 12 hour time between configured SSdb saves.
If you change the layout on the FT server, before starting again the primary server, you'd have to load the a fresh SSdbsave from the FT server.
It works, I did it several times. That's the only problem.