If no response is being received *without* specifying a selector query, then adding one won't help. The JMS selector query can only filter *out* messages on the queue/topic that don't match the query.
If you are using JMS Topics then they differ from Queues in one important way: You have to be subscribed to a topic *at the time a message is sent to it* in order to receive it. Unless you use a "durable" subscription, messages will not wait on a Topic for a client to pick up later.
From your description, it sounds like you may have two separate JMS steps, one that sends a request followed by one that receives the response. It's likely that the response is being sent before the second step can actually open its subscriber, so it's being missed.
You have a couple of options. The easiest is to just do it all in one step. When both the send and receive sides on the JMS step are enabled then it automatically opens the response queue subscriber *before* sending the request, avoiding any race conditions with the response. If they *have* to be two separate steps then you will actually need *three* steps: First the response step with its type changed to an Async listener, which starts the subscriber in the background. Second is the request send step. And third is an async consumer step that references the async key defined in the first step and actually receives the message.
In 8.0 you can also use the new JMS Send Receive step, which can also do either of the above patterns and is a bit less clunky to use.