One way I've found to handle 'fallthrough' logic if you don't use a public/private split (i.e., public goes through Web Agent "Pub" and private through Web Agent "Priv" therefore just assign different auth schemes) is below. I'm sure there are other ways, but this has worked pretty well for my setup that uses a centralized credential collector for all apps.
The pop-up itself is really browser behavior more so than SiteMinder. Since it sends WWW-Authenticate:Negotiate and if proper credentials aren't automatically negotiated browser prompts such that user 'can enter valid creds' -- which of course means nothing if you have IIS setup to do only "Negotiate:Kerberos" for example since no matter what you put in that popup it will never pass .
------------------
Pre-setup: Win 2012 R2 and IIS 8.5. Configured for Negotiate:Kerberos provider only (cause NTLM is just...bad).
1 - Create a centralized "redirect.jsp" (or whatever you want...aspx anything really) that takes an 'originaltarget' parameter to send the user back to what they originally requested (be sure to validate redirect domain so someone can't bounce things off it to trick users).
1a - Create application object to protect /redirect.jsp?auth=forms&originaltarget=* protected with forms auth
2 - Create a custom error page - for example formredirect.html - which is nothing but a simple redirect that snags the original target URL from the request (i.e., the app or page they were going to) and redirects to /redirect.jsp?auth=forms&originaltarget=[the URL you snagged]
3 -
Go to the folder location you specified for the .ntc (or if using inline credentials whatever folder is doing the windows auth). Set error mode to 'custom' and configure
401 = File and point to your redirect html file
401.1 = Execute URL and point to the same file location
Note: This step can adjust if you have anything going on in IIS. I specifically have some other filters in play that requires I set custom error only for 401 and 401.1 to be consistent across IE, Chrome and Firefox. If I do things with 401.2 it hoses it up. In testing I've also found differences between browsers when doing "File" vs "Redirect" vs "Execute URL"....So YMMV depending on specific setup.
You may have to play with adding in 401.2 possibly.
------------------
Now when a user hits the IWA protected page, if they are on domain and it works then they'll just passthrough normally. If they're off domain, or don't have IWA properly enabled, it will invoke the error file, redirect them to the "forms" protected resource and upon successful log in they'll get the right auth level (IWA scheme must be same or lower than Forms of course) and redirected back to their original target location without being prompted again for creds.
At least from my testing, when I hit it from a different domain or off the domain I don't get that popup using the 401 + 401.1 combination.
Of course the specific configs will depend on your overall app structure. Hopefully has some pointers that might be helpful to figuring it out for your setup..