Sorry for the half-rant (or free consultancy, what ever your angle may be).
This seems like a revisit of sorts of my experience with CA's Salesforce era. Back then, as I mentioned before, the pages the server dished out had so much Java Script and external references, our corporate proxy totally choked on them. Forms, knowledge bases etc. would be unreliable, broken, unresponsive or required repeat reloads. The pages had so many analysis JS in it (which is also GDPR-problematic) it boggled the mind. And I'd bet a fiver most of the calls and frameworks weren't even actively used, but put in there by the marketing department at some point, then abandoned. I've seen this too many times not to bet on it.
For some reason, corporations like CA and Broadcom don't seem to see the value in slick, yet functional working websites, instead opting for "moar cloud, moar bling, moar analysis". Maybe customer isn't king, but reasonable company goal should also be to enable the enterprise clients to USE a web site that works with the enterprise client's ugly neccessities like proxies without constantly generating these exact (yet hard to pin or meassure) frustrations. Not just making a web presence that looks shiny and responsive with a single digit ping.
Now, I am no fan of the proxy server here (nobody but the auditors are) but these things are a fact of life for enterprise customers, and the killer fact is this: the thing worked for 99.9% of the web sites it allowed me to go to. Only CA's SalesForce presence was a jumbled mess when requested through that proxy. So there's certainly two sides, but only one who stands to lose business.
And rest assured, while Michael and I report these things, I'd once again bet another (fresh) fiver, 99 out of a 100 people just walk away. Same for the editor issues and other stuff, simple as that. Nobody has the patience to contantly deal with that, however the frustrations of clients getting the wrong redirect or a broken Ideation site sadly can't be meassured in hard figures very easily.
As for the user agent filter, I have done some more testing out of plain curiosity. You seem to be blocking wget exclusively. While blocking UA based on strings is not per se forbidden by any RFC, wrong HTTP status codes are frowned upon, and would merely help against the most lamest among lame script kiddies.
Gets better though. The custom error page for the 404 delivers a link "back to home page". That link goes to community.broadcom.com, but if called with the same wget string, generates the same 404 page again. Thus, without any additional request throttling (and if one has any, then why do the wget block in the first place), I recon an attempt to spider the broadcom site would generate MORE traffic than leaving the conditional "filter" away would do.
The only acceptable (yet futile and so easily to circumvent it's frankly useless) way would be to throw either HTTP/403 "forbidden", or HTTP/429 "too many requests", with proper server side throttling. That way, one could throttle the nice players, and the not so nice players ... well, they'll impersonate GoogleBot in their crawler's UA anyway.
Best regards,
Carsten
p.s. Michael: if you do happen to figure out what property makes it behave wrong, I'm very curious. Can you try without a proxy?
Original Message:
Sent: 09-17-2019 11:21 AM
From: Michael Lowry
Subject: CA Communities links no longer redirecting
I just checked on my phone. There it works. Our corporate web filter is very strict, so perhaps @Carsten Schmitz's findings are relevant.
Original Message:
Sent: 09-17-2019 10:51 AM
From: Carsten Schmitz
Subject: CA Communities links no longer redirecting
Well ...
It is true that the given URL does actually redirect to the correct page for me, too, at this time.
I also notice during some quick experiment, however, that calling the link with for example wget, and specifying nothing more as the user agent (i.e. using the default UA string including the word "wget"), the link produces a seemingly entriely false HTTP/404.
When I submit the exact _same_ request with a deliberately empty user agent string, it fetches the correct document. I can currently reproduce this reliably, and I am not using a proxy server.
I have thus downloaded a plugin and set my browser's UA to the string "wget" and I am getting a HTTP/404 with a custom handler page there as well.
Barring that I am possibly still mistaken and some other weird thing is at play (but I don't think so), I'm going to say this regardless: Once a system has gone off of that cliff of serving up blatantly false HTTP status codes in what must be assumed some attempt at blocking certain user agents, any analysis by way of comparision of what content (or redirects) gets delivered to any external client becomes utterly pointless in my eyes; that is, unless we fully know the code and the web server logs. One can in no way assume that Michael gets the same result as any other person due to demonstrated factors of uncertainty, as nobody on the outside can reasonably assess how many other standards the serving site willingly violates.
Sorry, but that's my $0.02.
Original Message:
Sent: 09-17-2019 10:00 AM
From: Michael Lowry
Subject: CA Communities links no longer redirecting
No. It redirects to:
https://community.broadcom.com/home
Original Message:
Sent: 09-17-2019 09:53 AM
From: Jason McClellan
Subject: CA Communities links no longer redirecting
What's wrong with it? It is landing on the correct page correct? ~jm
------------------------------
Thank you
Jason
Community Platform Owner, IT
Original Message:
Sent: 09-17-2019 04:07 AM
From: Michael Lowry
Subject: CA Communities links no longer redirecting
I am referring to this URL:
https://communities.ca.com/message/242065181-documentation-for-ae-messages-db-schema-java-apis-etc
Original Message:
Sent: 09-16-2019 09:16 AM
From: Jason McClellan
Subject: CA Communities links no longer redirecting
CA.com main URL is set to be decommissioned and will redirect to Broadcom.com. The legacy ca.com content is being migrated broadcom.com. You should see the changes soon.
------------------------------
Thank you
Jason
Community Platform Owner, IT
Original Message:
Sent: 09-16-2019 02:46 AM
From: Michael Lowry
Subject: CA Communities links no longer redirecting
The CA.com URL still does not work.
Google's indexes do not come into play for the problem I described. I'm not talking about Google searches. We have CA Communities deep links in our internal documentation.
Original Message:
Sent: 09-13-2019 03:19 PM
From: Jason McClellan
Subject: CA Communities links no longer redirecting
@Michael A. Lowry
The link is working again.
old URL
https://communities.ca.com/message/242065181-documentation-for-ae-messages-db-schema-java-apis-etc
new URL
https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=1435&MID=783045&CommunityKey=2e1b01c9-f310-4635-829f-aead2f6587c4&tab=digestviewer#bm289b2ec9-2e06-49d1-b410-e96e5ed83821
------------------------------
Thank you
Jason
Community Platform Owner, IT
Original Message:
Sent: 09-13-2019 10:11 AM
From: Michael Lowry
Subject: CA Communities links no longer redirecting
Today, another piece of fallout from the Broadcom acquisition of CA fell in my lap.
Before today, this link pointed to the discussion thread Documentation for AE, Messages, DB schema, Java APIs, etc.:
https://communities.ca.com/message/242065181-documentation-for-ae-messages-db-schema-java-apis-etc
Starting today, this link stopped working. Now it simply redirects to the main Broadcom communities page.
The new Broadcom URL of the message thread is:
https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=1435&MID=783045&CommunityKey=2e1b01c9-f310-4635-829f-aead2f6587c4&tab=digestviewer#bm289b2ec9-2e06-49d1-b410-e96e5ed83821
I gather that the mapping has been permanently turned off. If so, this means that all CA communities deep links will remain broken.
We have hundreds of these links in our internal documentation.