DX NetOps

 View Only
Expand all | Collapse all

F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

  • 1.  F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 16, 2019 04:18 AM

    Recently upgraded from 10.3.1 to 10.4 and have got problems with my F5 loadbalancer setup in front of both tomcat and webtomcat.

    Onelick Java (oneclick.jnlp)
    Before upgrade urls in client session was dynamic. Users accessed https://spectrum.x.x and and the loadbalancera forwarded the session to vlpspec001 or vlpspec001. All communication worked as the jnlp file contained dynamic urls (https://spectrum.x.x). But in 10.4 the downloaded onclick.jnlp contain static url's that point to either of the two answering oneclick servers, which the client do not have access to - so communication can't continue.

    OnclickWebApp (tomcat & webtomcat)
    In 10.3.1, you could access the oneclickwebapp directly on port 9443 (if configured), which worked fine. But in 10.3.2/10.4 there is a new way to setup the client session. When clicking the oneclickwebapp url a new browser opens which download index.jsp from the tomcat, which instructs the browser to wait for data from the webtomcat. So there is two different tomcats involved and the webtomcat session isn't established from the client. The loadbalancer therefor can't connect the server dataflow with the client.

    Anyoneelse haveing a loadbalancer in front on oneclick/webapponeclick and got it working ?

    /Roberth

    P.s. Everything is working inside firewalls without loadbalancing. I have created to tickets on this, but support/SE seems to be very slow in understanding the problem - as its working without loadbalancing :(


  • 2.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 16, 2019 07:51 AM
    Did you check if something need to be changed at F5 load balancer configuration? That is, mapping of the load balanced URL to individual node URLs.

    ------------------------------
    Thank you.
    Rajashekar
    ------------------------------



  • 3.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 16, 2019 08:13 AM
    If you mean oneclickwebapp, then thats one of my tickets at customer support. I have a working solution if I do not go thru loadbalancer. The loadbalancer have problem to handle the client session downloading index.jsp from tomcat and then wait for data flow from webtomcat from a different port without client asking for it. If Engineering could describe in details how this works, my network team can most likely configure loadbalance correct.

    If you talking about oneclick.jnlp, then nothing has changed in ports or access. Its just that sessions setup details forwarded from server to client include static url information - which make the client try to communicate directly to server instead of using loadbalancer.

    Two different problems introduced for environments with loadbalanceing in front of oneclick servers in spectrum 1.3.2 or 1.4 (I upgraded from 1.3.1 to 1.4, when I got the problem).


  • 4.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 16, 2019 10:55 AM
    Edited by Rajashekar Allala Sep 16, 2019 10:55 AM
    Regarding oneclick.jnlp, we may need to deal with the file oneclick.jnlp located at $SPECROOT/tomcat/webapps/spectrum/ as it is pointing to base as below.
    Not sure, how it is picking up the base URL (Ex: http://<serverName>:8080/spectrum).

    I am not aware of OneClick WebApp.


    ------------------------------
    Thank you.
    Rajashekar
    ------------------------------



  • 5.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 17, 2019 03:40 AM
    Even if I edit the oneclick.jnlp file to include loadbalancer url, the continuing communication seems hardcoded to server - so it can't continue :(


  • 6.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 18, 2019 10:59 AM
    We discovered with support's help that you must modify both the oneclick.jnlp AND the crypto.jnlp on the server and place your load balanced name in place of the $$href.  When we discovered the issue, our java client was going directly to the server and bypassing the load balancer.  We like to maintain (for our own testing) connections directly to the oneclick servers, but ask our users to use the LB URL we provide.  This obviously breaks.  We are told engineering will provide a patch to restore the previous functionality.


  • 7.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted Sep 19, 2019 02:15 AM
    Are you using the webapp ? if, have you got it working through LB?


  • 8.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    Best Answer

    Broadcom Employee
    Posted Sep 24, 2019 11:08 AM
    Regret for the delay, We have identified the issue and released a patch which fixed the issue.

    Spectrum_10.04.00.PTF_10.4.002
    Symptom : Spectrum-10.4 oneclick.jnlp href generation changed. It is creating url href with static FQDN of the webserver. 
    Resolution: OneClick.jnlp href generation will use the Hostname from request(Host Header) instead of static FQDN of webserver.


  • 9.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted May 08, 2022 02:51 AM
    can any of you actually show working examples of this working??  Im on 21.2.10 and my load balancing team is not getting how this works, gave them the link to https://knowledge.broadcom.com/external/article/212236/unable-to-access-oneclick-webapp-when-us.html  but that is vague also. 

    If any one actally has this working in an F5,  please share actual config minus your specific ips or whatever. If there is something that needs to be done in the webtomcat, can that be spelled out also. Pretend i am really ****.  cause i am. 

    thanks.


  • 10.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted May 08, 2022 02:53 AM
    you ever get this working. I'm lost with this and my load balancer team is even more lost.


  • 11.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Broadcom Employee
    Posted May 09, 2022 10:57 AM
    Edited by karen_brooks May 09, 2022 10:57 AM
    You can try modifying the oneclick.jnlp so that it references the URL from the LB to see if this helps

    <!-- JNLP File for Session Client -->
    <jnlp spec="1.0+" codebase="$$codebase"
    href="$$href">

    where codebase="http[s]://spectrum.acme.com[:port]/spectrum"

    If you are integrated with CABI, then you will probably need to modify the crytpoj.jnlp as well (same edit).


  • 12.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Broadcom Employee
    Posted May 09, 2022 08:01 PM
    Hello,

    I don't have an F5 example but I have customers that successfully use a load balancer (Avi, if that makes a difference) for OneClick and Webapp.  For the most part, they followed what was listed here:
    To achieve load balancing, identically configured OneClick web servers are accessed through an external load balancing device that employs host/session persistence and any load-balancing mode.  
    Check OneClick Web Server Status
    You can configure your load balancer to check the status of each OneClick server by using the following HTTP GET statement during periodic server health checks:
    http://<hostname>:<portnumber>/spectrum/stable
    A successful GET returns the contents of the "stable" file. The presence of the stable file indicates that the SpectrumTomcat process is in a stable state. Failure to retrieve the file indicates that the SpectrumTomcat process is not running or is unstable.

    They're using the stable check for both the regular Tomcat and WebTomcat configuration, ports 8443 and 9443 respectively.  That does leave the possibility that WebTomcat could be having an issue while Tomcat is fine so we're looking to configure an additional health check specifically for WebTomcat.

    -Rob



  • 13.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted May 10, 2022 12:11 AM
    Server:port/spectrum/stable looks to only work with port 80




  • 14.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Broadcom Employee
    Posted May 16, 2022 09:00 AM
    Not sure I understand.  It works on port 8080:



    And even for SSL/TLS:

    Are you saying this only works for Tomcat which you have configured for port 80 but not for WebTomcat, which defaults to port 9443?  If so, that is correct.  The customer example I mentioned before, they use the same check for both ports (if stable on 8443 comes up, then forward to 8443 and 9443 on that server) which won't catch the case where 8443 is up but 9443 is having problems. 

    The "stable" file is produced by a Java class on the Tomcat server that looks for error conditions and either writes or deletes the file based on what it finds.  We looked at copying that class over to the Webtomcat server but it's not something that we can do in the field so we're looking at a script to watch the catalina.out file for errors and write/delete a "stable" file so we can have a consistent check between the two ports.


  • 15.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted May 10, 2022 10:34 AM
    I need you to call me for the directions on disabling report manager on a oc server. I cant find that note.




  • 16.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Broadcom Employee
    Posted May 16, 2022 08:59 AM
    Dick I sent you an invite for 3pm EST today lets please still go ahead with that.

    Brian Flad
    Principal Support Engineer
    Agile Operations Division
    Melville, NY 11747
    Office: (631) 327-6846
    Brian.Flad@broadcom.com

    Sign up for Proactive Notifications to receive emails regarding important notifications, updates and release information regarding your NetOps and other Broadcom Software

    Free online classes at our new Training Academy. Learn more about your DX NetOps implementation:  Click Here for Training Academy

    Interested in having support available during your weekend upgrade?  Register for our next Designated Weekend Upgrade Program?:  Register Here
    On May 10, 2022, 8:55 AM -0400, Mishra, Nitish11 <nitish11.mishra@nttdata.com>, wrote:
    > Have opened a case with F5 to check on resets, will schedule a call once they respond. Will also go through this document you shared.
    >
    > Regards
    > Nitish Mishra | Network Advisor| | nitish11.mishra@nttdata.com
    > Mobile +91 9582598948 | nttdata.com/Americas
    > LB team DL for prompt response :- DL_Leveraged_Load_Balancing@nttdata.com
    > NTT DATA Services, LLC
    > Consulting | Industry Solutions | Digital | Application & Infrastructure Services | Cloud | BPO
    >
    > From: Baker, Dick <dick.baker@nttdata.com>
    > Sent: 10 May 2022 09:41
    > To: BROADCOM-dxnetopsmanager@ConnectedCommunity.org
    > Cc: Brian Flad <brian.flad@broadcom.com>; Mishra, Nitish11 <nitish11.mishra@nttdata.com>
    > Subject: RE: DX NetOps : F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    >
    > Server:port/spectrum/stable  looks to only work with port 80
    >
    > From: Robert Kettles via Broadcom <mail@connectedcommunity.org>
    > Sent: Monday, May 9, 2022 7:03 PM
    > To: Baker, Dick <dick.baker@nttdata.com>
    > Subject: RE: DX NetOps : F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    >
    > Hello, I don't have an F5 example but I have customers that successfully use a load balancer (Avi, if that makes a difference) for OneClick and...
    > DX NetOps
    > Post New Message
    >
    > Re: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    > Reply to Group
    > Reply to Sender
    > May 9, 2022 8:01 PM
    > Robert Kettles
    > Hello,
    >
    > I don't have an F5 example but I have customers that successfully use a load balancer (Avi, if that makes a difference) for OneClick and Webapp.  For the most part, they followed what was listed here:
    > https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/spectrum/21-2/administrating/oneclick-administration/oneclick-server-communications-and-network-configuration/load-balancers.html
    > To achieve load balancing, identically configured OneClick web servers are accessed through an external load balancing device that employs host/session persistence and any load-balancing mode.
    > Check OneClick Web Server Status
    > You can configure your load balancer to check the status of each OneClick server by using the following HTTP GET statement during periodic server health checks:
    > http://<hostname>:<portnumber>/spectrum/stable
    > A successful GET returns the contents of the "stable" file. The presence of the stable file indicates that the SpectrumTomcat process is in a stable state. Failure to retrieve the file indicates that the SpectrumTomcat process is not running or is unstable.
    >
    > They're using the stable check for both the regular Tomcat and WebTomcat configuration, ports 8443 and 9443 respectively. That does leave the possibility that WebTomcat could be having an issue while Tomcat is fine so we're looking to configure an additional health check specifically for WebTomcat.
    >
    > -Rob
    >   Reply to Group Online   Reply to Group via Email   View Thread   Recommend   Forward   Flag as Inappropriate
    > -------------------------------------------
    > Original Message:
    > Sent: May 08, 2022 02:52 AM
    > From: Dick Baker
    > Subject: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    >
    > you ever get this working. I'm lost with this and my load balancer team is even more lost.
    > Original Message:
    > Sent: Sep 16, 2019 04:18 AM
    > From: Robert Edberg
    > Subject: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?
    >
    >
    > Recently upgraded from 10.3.1 to 10.4 and have got problems with my F5 loadbalancer setup in front of both tomcat and webtomcat.
    >
    > Onelick Java (oneclick.jnlp)
    > Before upgrade urls in client session was dynamic. Users accessed https://spectrum.x.x and and the loadbalancera forwarded the session to vlpspec001 or vlpspec001. All communication worked as the jnlp file contained dynamic urls (https://spectrum.x.x). But in 10.4 the downloaded onclick.jnlp contain static url's that point to either of the two answering oneclick servers, which the client do not have access to - so communication can't continue.
    >
    > OnclickWebApp (tomcat & webtomcat)
    > In 10.3.1, you could access the oneclickwebapp directly on port 9443 (if configured), which worked fine. But in 10.3.2/10.4 there is a new way to setup the client session. When clicking the oneclickwebapp url a new browser opens which download index.jsp from the tomcat, which instructs the browser to wait for data from the webtomcat. So there is two different tomcats involved and the webtomcat session isn't established from the client. The loadbalancer therefor can't connect the server dataflow with the client.
    >
    > Anyoneelse haveing a loadbalancer in front on oneclick/webapponeclick and got it working ?
    >
    > /Roberth
    >
    > P.s. Everything is working inside firewalls without loadbalancing. I have created to tickets on this, but support/SE seems to be very slow in understanding the problem - as its working without loadbalancing :(
    >
    >
    >
    > You are receiving this notification because you followed the 'F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?' message thread. If you do not wish to follow this, please click here.
    > Update your email preferences to choose the types of email you receive
    > Unsubscribe from all participation emails
    > Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.

    --
    This electronic communication and the information and any files transmitted
    with it, or attached to it, are confidential and are intended solely for
    the use of the individual or entity to whom it is addressed and may contain
    information that is confidential, legally privileged, protected by privacy
    laws, or otherwise restricted from disclosure to anyone else. If you are
    not the intended recipient or the person responsible for delivering the
    e-mail to the intended recipient, you are hereby notified that any use,
    copying, distributing, dissemination, forwarding, printing, or copying of
    this e-mail is strictly prohibited. If you received this e-mail in error,
    please return the e-mail to the sender, delete it from your computer, and
    destroy any printed copy of it.




  • 17.  RE: F5 Loadbalancer not working in front of oneclick in 10.3.2/10.4 ?

    Posted May 10, 2022 11:48 AM
    Hi @Robert Edberg, all,

    we came across some issues with running loadbalancers in front of OneClick servers recently as well. There are numerous tec doc articles etc. that can be found or are referred to by support with all of them basically being outdated or non-helpful. So basically I found the following:
    Starting with some 21.2.x version, once the WebApp tomcat is launching a new client session, it's passing a OC URL to be connected to towards the client session, that seems to come from a client (browser) referrer. Since that is typically pointing to the "external" resource (the load-balanced one), that will not be reachable from the WebApp tomcat server normally. I modified the webswing.config by adding some parameters in between "compress 9" and "${customArgs}":

          "launcherConfig" : {

            "args" : "-compress 9 -host localhost -port 8443 -ssl true ${customArgs}",
    This makes the WebApp tomcat connect to the OC tomcat on the same server instead of trying to reach the external (load-balanced) URL.
    We now can launch OneClick WebApp through the client browser as intended and do see the bottom status bar showing "You are logged in as ... on https://localhost:8443".
    Support did not came up with an alternative solution and stating similar to "if it works you can do so".
    Of course there is more things to consider like which ports to use, how to setup the virtual servers/pools/members on the loadbalancer, etc. A key thing to remember also is, if you're using TLS (HTTPS), you need to activate that on both tomcat servers.
    The below picture I used to illustrate the issue and my proposed solution during the support case.

    OneClick WebApp loadbalancing
    hope this helps a little bit
    regards,
    Raphael