It is possible to implement an "Invalid Password Dictionary" in APS, however it relies on a plain text password list.
Invalid Password Dictionary - CA Single Sign-On - 12.7 - CA Technologies Documentation
Utilizing a SHA1 hash list would request custom work or an Enhancement Request to be opened.
You would have to take the user's proposed password and generate a SHA1 hash. Then perform a look up of this hash against the downloaded list.
An additional concern would be performance depending on if you are performing the lookup only on Password changes or on every login for every user. Plus given the size of the database, a robust repository would be needed.
As for excluding characters, this can also be achieved via APS using Disallowed Characters.
Password Content Settings - CA Single Sign-On - 12.7 - CA Technologies Documentation
Hope this helps!