First, you indicated that you dropped the JAR into the LIB directory. Access the Workstation.log, Registry.log, VSE.log, etc. and check the class loader displays.
Do you see the JAR file being loaded? If so, there could be an issue with the .lisaextensions file configurations.
Did you add your JAR under a 'new' directory in LIB such as LIB/myJars (replace myJars with whatever)? I have seen this cause issues.
Second, I would suggest that you remove the JARs from LIB and drop them in hotdeploy. About the only JARs we recommend placing in LIB are JDBC and MOM-based JARs.
Third, are you using DevTest 10.2 AND the issue occurs when you open or attempt to add the asset using DT Workstation? DT 10.2 has a new copy feature that might be causing the JAR to be missing. Look at:
c:/users/<yourUserID>/DevTest/lib and /hotdeploy. Do you see <yourCustomJar>.jar in one of these directories? If not, place the JAR in hotdeploy and restart Workstation.
Report back what you find out given the above and we'll keep trying to help you solve the issue.