Rally Software

 View Only
  • 1.  Webhooks Update

    Broadcom Employee
    Posted Aug 20, 2020 07:16 PM

    To our valued Rally Customers, 


    We appreciate all of your feedback on this issue and understand how this is impacting your business. As you're aware over the past week we have seen an issue with webhook reliability, and have made significant improvements to address those issues. Webhook rules are now triggering and we are receiving successful status back for the majority of users. 


    Currently, we are seeing a larger volume of outbound messages to our customers leading to a delay in webhook delivery. We have also noticed an uptick in non-response replies when trying to deliver to a 3rd party consumer of our webhooks that may be causing these delay issues and working with them to determine the root cause of a dramatic increase in volume. 


    Our Rally engineering team is hard at work to resolve and improve this experience. And on behalf of your Rally team, we thank you for taking the time to read this update and continue to engage with us. We'll be updating this thread regularly as we drive to conclude this issue.



  • 2.  RE: Webhooks Update

    Posted Aug 21, 2020 05:38 AM
    Thank you for the update, Michele.

    We do observe some improvements and delays in webhook message delivery is now much shorter: 3h instead of 26h.

    With that, 

    we see the webhooks still do not respect the defined condition (e.g. limit on specific workspace or value in the custom field) and thus firing on every single work item, instead of a limited set of them.

    This may explain the uptick in the number of fires.

    Appreciate the meaningful update here as support team apparently does not have an access to this information and can not provide anything besides the reassuring vanilla  message.

    Querying for webhooks via REST API still fails with 'bad message' error.

    Thank you,
    Inna


  • 3.  RE: Webhooks Update

    Broadcom Employee
    Posted Aug 21, 2020 06:55 PM
    Hi Inna, 

    Thank you for the additional feedback,  which we have shared with our engineering team. 

    We have continued to investigate the root cause of the webhooks issue today, which we have little to provide an update on. Our teams are still looking into the two issues as mentioned previously.


    Currently, we are seeing a larger volume of outbound messages to our customers leading to a delay in webhook delivery. We have also noticed an uptick in non-response replies when trying to deliver to a 3rd party consumer of our webhooks that may be causing these delay issues and working with them to determine the root cause of a dramatic increase in volume. 


    We are truly sorry for the delay in webhook delivery and the inconvenience this has caused to your business. We will continue to provide updates daily as we narrow down the issue.