DX Infrastructure Manager

Expand all | Collapse all

E2E Appmon: Clicking inside an application in a RDP session

  • 1.  E2E Appmon: Clicking inside an application in a RDP session

    Posted 04-12-2018 07:41 AM



    I have to write an E2E script for a customer.


    Workflow is as follows:

    1. Nimrecorder on windows 7 box, running VNC

    2. Start flow: Log in to RDP farm

    3. Open MS Dynamics AX client

    4. Do X number of manipulations on different functions

    5. Close MS Dynamics AX client

    6. Log off RDP farm

    7. End flow


    Point 2, 3, 5 and 6 are working fine.

    But point 4 is just failing whatever I try.


    I'm using Bitmaps to capture the mouseclicks because I'm running in a RDP window, plus it's resolution independent.


    Seems that within the dynamics client, he won't mouseclick properly on the bitmap.


    Example: I have a shortcut in the client called Stock Management. The bitmap has been saved and if I set the function: if ClickOnBitmap("C:\Program Files (x86)\NimSoft\e2e_scripting\Scripts\Bitmaps\AX_Stock_Management.bmp",left,single) = 0 then MsgBox ("Bitmap found") endif , it will trigger, so he recognizes the bitmap on the screen.


    I see the focus on the window (application) changing color, so he does seem to click inside the application, however, he does NOT click on the bitmap itself because I'm not getting the button press and the required screen.


    This seems to be an issue in every application I open inside the RDP window. As long as I stay in the RDP window, I can launch applications using the ClickOnBitmap function. If I go into the application itself, it seems like the function just clicks on the window of the application, instead of the bitmap of the application.


    Hope my example makes sense.


    I cannot record my actions directly because the application in question always opens up on a different location, so absolute mouseclicks are out of the question.  I cannot set it maximized because I'm not administrator on the RDP server.


    Question now is: Has anyone any experience with this? Is this possible at all or am I asking too much of the nimrecorder/wintask?





  • 2.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Broadcom Employee
    Posted 04-12-2018 08:05 AM

    Could you run the script on the server you are RDP'ing to?

    You've got the access to logon, which is what the e2e app mon needs, just need a robot on that server/host.

    Or maybe put the Dynamics client on another host which runs the script?


    If you can't maximise the window and control the position of the bitmap, then you are always going to struggle with that.

    I don't understand why maximising is an administrator function..

  • 3.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Broadcom Employee
    Posted 04-12-2018 08:41 AM

    Before you try to force it to to click on the image, use the mouse to click on the window is resides within for focus.


    Access ClickOnBitmap using the Help window->functions on the right side of the e2e scripting window. If you don't see it I think you have to press F4? try using that via the Wizard - double click to start the Wizard.


    Make sure when you save off the image you dont capture any extraneous area outside the image, just use the inner area of the bitmap and avoid using the area outside of it.


    Save off the .bmp file and make sure you use "Somewhere in the Window" and extend the coordinates manually in the script if you need to just by editing the coordinates in the script file if needed. Increase x, y coordinates by 10 or so each.


    Basically with ClickOnBitmap you're forcing the mouse to click on image/window where for example you can continue the script of enter credentials into a window or close a window etc.


    Watch the progress when re-running the script, via VNC viewer and adjust again if needed after noticing where the mouse has moved.


    Also, the window should be able to be maximized regardless of whether or not youre an administrator but e2e requires either creds of a local admin on the machine or domain admin.


    See also:

    e2e_appmon Best Practices checklist - CA Knowledge 


    Are you seeing any particular errors/error codes when the script runs and fails?



  • 4.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Posted 04-12-2018 09:21 AM

    Hi everyone


    Thanks for the quick replies.


    In the meantime I found the issue: my bitmap precision was set too low for these specific actions, so he didn't recognize them.

    #BitmapPrecision=100 was needed once I started the application. Everything I did before required #BitmapPrecision=80.


    So it took me the better part of the day to realize this. Once it's set back to 100, he recognizes the bitmaps and the clicks are now registering correctly.




  • 5.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Broadcom Employee
    Posted 04-12-2018 10:08 AM

    Hi Mark,


    Why was the #BitmapPrecision set to a value lower than the default?


    #BitmapPrecision - Specifies how closely the found image must match the input image in Pause on Bitmap and ClickOnBitmap functions


    In bitmap/image wizards, a confidence recognition scale was added, the default is 100% (exact match).



  • 6.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Posted 04-16-2018 03:14 AM

    Hey Steve,


    Dependent of what the script needs doing, bitmap precision needs to be modified. When I'm doing manipulations inside the AX window, it needs to be 100 or the bitmaps aren't recognized. Outside the AX window it needs to be lowered to 80 or he won't recognize them.


    I've tried recreating the bitmaps, but the result is the same. It's good to know anyway and it works like this, but it's a very delicate system. Once you're inside a RDP session, a lot of functions won't work anymore (like sendkeys for example), so you can only rely on bitmaps.


    I wonder if CA will use some other tech in the future for all this, wintask is rather old. I know there are some other software tools that work better, like ServiceTrace for example.




  • 7.  Re: E2E Appmon: Clicking inside an application in a RDP session

    Broadcom Employee
    Posted 04-16-2018 05:24 PM

    e2e (WinTask) offers this approach for determining the bitmap precision value:


    For getting a reasonable value of #BitmapPrecision, you need to go to the Image synchronization wizard, which can be opened by clicking the Wait Image toolbar button. Then, follow these steps:


     - Press Capture, then select the bitmap of interest, with the minimum rectangle such as it will remain unique in its surrounding area.

    Surrounding area = area around the bitmap of interest in which the bitmap may be found, depending on UI resizing and formatting. For example, a bitmap inside a website may move around depending on scroll, website formatting and font size. 

    Avoid capturing matte color backgrounds surrounding the essential bitmap content. For example, if you want to capture the Windows Start button,  select the smallest rectangle touching the Windows logo, do not capture the button background around it.

     - Select Somewhere in a defined area and click Define area. This is the surrounding area that I mentioned above, so the smallest rectangle in which the bitmap can be found between different runs of the script.

     - Decrease the Bitmap precision, then click Check. If the bitmap is not found at the original location any more, it means that you went too far with decreasing the precision. Repeat this until you find the minimum value ensuring that the bitmap can be found.


    You can use the #BitmapPrecision value obtained this way, together with the captured bitmap and the defined surrounding area, for ALL bitmap lookup functions (ClickOnBitmap, PauseBitmap etc.).