There isn't any real documentation on it, unfortunately, since although I've certainly always wanted to have an external API I've never had the time to properly write the server implementation of a full object model for it let alone write API documentation.
Do post on the list though, with the specifics of the methods you're having failures calling and I'll try and help you out. It's best to keep that over there since it's easy to lose information on these forums.
There are a couple of pitfalls in the COM API because of limitations in the Windows Script Host; the COM APIs are designed around the capabilities of full C++ clients (or full Visual Basic) but the Windows Script Host interpreter has some rather particular (and not all that well-documented) limitations with the method argument lists that it works with, and that the ATL COM implementation of IDispatch in the server-side code doesn't do a lot to help with either.
So, some of the problems may be due to parameter type issues that you can't really work around except by using a more capable engine than Windows Script Host (e.g. a full implementation of Visual Basic or any .NET language, which includes the PowerShell environment).
There are also a couple of methods there that are stubbed out because I defined them to fit with capabilities of the console UI that were never implemented and so I never filled out my side. Mostly they return reasonably noticeable error codes to that effect (i.e., E_NOTIMPL). It certainly would be good to create a list of those, and we can work through them over a little while on the list.
Just so you know, there's a holiday weekend (the Queen's Birthday) coming up here in New Zealand and I expect that I won't have internet connectivity for much of it so I may be a couple days late responding.