Hm, that's a bit more tricky and I couldn't find (fast) a way to 'correctly' do this. What works is this: create your query in the query designer, then Edit it and pus 2 extra conditions:
(chg.close_date > (PdmToday(-1) + 60*60*6) ) and ( chg.close_date < (PdmToday(-1), 60*60*18) )
This should give you only the records where the Close Date is between 6 and 18. Unfortunately PdmToday has no syntax for specifying an hour adjustment.
Now, I say this is not a 'correct' way - there should be a possibility to add a new Filter in the Change Filters which would take into account the formula above - you can try to do that and you can then you can get rid of the custom query editing and just drop the Filter in Filter Panel in WebI designer.