However, my questions:
Q1: Port 8507 is picked up automatically and is not configured or active on TEST INSTANCE, so how can the agent trigger the job on TEST when the port 8507 port is not open?
A1: The System Agent is listening on port 7520 and the scheduler on port 8507
When the scheduler triggers a job on the System Agent, it sends an AFM to the System Agent on port 7520
Then the System Agent sends the job status to the scheduler on port 8507
Q2: Can an agent communicate with both the instances with port 7520 simultaneously?
A2: Yes there is a single receiving port on the System Agent and all your schedulers are going to send AFMs to this port to trigger jobs. If for any reason, you prefer to have separate receiving ports on the System Agent for instances TEST and DEV, you can install another System Agent on the same machine listening to another port (for example 7530)
Q3: I was under assumption that for agent to communicate with server/scheduler, both should have same crypt key.
Here in my case, ABC agent is already having one crypt key from DEV, so it means that DEV scheduler and ABC agent has right crypt key in place, but I didn’t copy the crypt key from TEST scheduler to ABC agent install directory, then
how and why the communication with ABC agent to TEST SCHEDULER worked?
A3: There is a single cryptkey.txt file on the System Agent and no way to have different keys for 2 two different instances.
It works because you are either using the Default AES encryption key or the SAME encryption key in the machine definition of this System Agent on these both instances
Q4: Can an Agent install directory holds 2 crypt keys related to 2 different instance? If yes, then how the security between agent and scheduler be maintained?
A4: No, this is not possible , see A3