I'm assuming that 8 GB of RAM is for the server and not allocated to the JVM.
It all depends on how much the heap is allocated to the application that the agent is monitoring. The reduceagentmemoryoverhead will free up heap usage but take on more CPU usage. We don't have any actual statistics on the reduction though.
The performance gain will be through heap but the performance loss will be on the CPU.