No, provided there is never a moment where the ESXi host IPs are no longer allowed.
If you first add explicit ALLOW rules for every ESXi management IP (or the required /32 addresses), save the configuration, verify the rules are in place, and only then remove the broader /24 rule, the hosts should continue communicating with vCenter without disconnecting. The firewall is evaluated based on the active rule set, so as long as each host still matches an allow rule, connectivity is maintained.
The only situation that would cause hosts to disconnect is if you accidentally leave an ESXi management IP uncovered, or if the rule order causes a reject rule to be evaluated before the corresponding allow rule. Rule ordering is critical, especially when using explicit reject entries.
Before making the change, I'd still recommend taking an offline snapshot or ensuring you have a rollback plan, particularly if the vCenter is production, since an incorrect firewall rule can lock you out of the appliance itself.