Generally speaking, JavaScript resources are meant to return data, for instance from a non-standard data source, whereas custom endpoints are meant to return other things, like binary data, HTML, or whatever.
JS resources are always authenticated, whereas custom endpoints don't have to be.
JS resources must return JSON, whereas custom endpoints can return anything.
JS resources can be attached to a parent resource, thereby enabling you to perform a join from a standard data source to a non-standard data source (like an LDAP directory, for instance). Custom endpoints are always top-level, and their URL differs from resources because they contain "http" instead of "rest".
If this is not enough, perhaps you could describe your specific use case, and we could discuss what makes sense for you?