Webhooks do not respect the conditions

[Inna Shulman]

Hi,

for some time now we are having trouble with webhooks: they fire indiscriminately on all US and DE, not respecting the limiting query in the definition.

And now even querying for the list of webhooks (https://rally1.rallydev.com/apps/pigeon/api/v2/webhook) fails with '400 Bad request'.

Anyone else is seeing this problem?

Thank you,
Inna

[Caroline Cobert]

Yes! We are experiencing this too. It's a huge problem because the misfiring causes downstream alerts and now we can't get any listeners to fire at all to diagnose the issue. Hopefully Rally gets back to us on this!

[Inna Shulman]

We now have even more severe problem: some webhooks do not fire at all.

I now see we had it in and out for several days, but today it is totally broken.

Some changes trigger triple fire of the webhook. This does not do harm in our case. But 'your millage may vary'. 

[Andrea Schilde]

Hi @Inna Shulman and @Caroline Cobert,

thanks for reaching out and I am sorry to hear you are experiencing issues with Webhooks.
If you have not already, please open a support case. Our Support Team is best equipped to help debug this issue and find a resolution.

Best,

Andrea

------------------------------
Product Manager
Rally Software
------------------------------

[Inna Shulman]

Hi Andrea, 

we have the support case opened for this for several days now. 
They keep saying the problem was fixed and should go away really soon, but this does not happen.

As a Product Manager for Rally Software, can you shed some light on the general take of webhooks and where this capability is heading?
It is a great tool to compensate for the lack of the custom process definition and provides reasonable flexibility to enforce rules and otherwise shape the process.

It also used to work really good. But this epic failure is troubling.

What are the plans in this area?

Thank you!

[Inna Shulman]

One week later and we still experience huge delays with webhooks message delivery: yesterday we saw 26h, this morning i see 12h.

In addition to ignoring the custom field filter in the webhook definition, they now ignore the workspace filter too. So webhooks fire indiscriminately on all work items, creating unnecessary flood of the messages towards our application.

Support don't seem to be able to do anything about this.

Formal communication incorrectly stated that the only affected service was Lookback API and that the problem has been fixed. 
It might be, for a brief period, but then again, things became worse.

Question:
what alternative to the webhooks service do you use to establish the 'integration' between Rally and other systems, different workspaces of Rally or to enforce certain process rules?

Thank you,
Inna