Cloud Workload Protection

Expand all | Collapse all

CWP for Storage CloudFormation Setup Issue

CraigEV11-07-2019 11:03 AM

  • 1.  CWP for Storage CloudFormation Setup Issue

    Posted 08-29-2019 07:07 AM
      |   view attached

    Hi there,

    I'm facing an issue with CWP for Storage CloudFormation template. When I tried to deploy it, it's endup with below error. Is there way to sort it out. 

     

    CREATE_FAILED AWS::AutoScaling::AutoScalingGroup ControllerGroupPolicy Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

     

    Thank you!



  • 2.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 08-30-2019 02:50 AM

    That appears to be an issue on AWS's platform itself. I'd recommend opening up a support call with them. It hasn't got anything to do with CWP itself.

    Thanks!



  • 3.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-07-2019 08:11 AM

    Hi Raj

     

    I am facing the same issue. Did you receive any information from AWS?

     

    Thanks,

    Kjersti



  • 4.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-07-2019 11:03 AM

    The OP never responded at all.



  • 5.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-07-2019 02:44 PM

    This is really weird. The resource that the template seems blocked on is created and looks fine to me. I installed a stack from a template in late september, which worked fine. Has there been any updates to the template recently?

     

    Thanks,

    Kjersti



  • 6.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-07-2019 03:16 PM

    I tried installing an old version of the template, and it stops on the same resource creation. Is it possible for you to convey this information to some technical people? Could it be that AWS has changed something in the creation of these resources? Or, could they provide some tips on how to debug the template? The template is too large and complex for me to debug it without any more details on how to go about it.

     

    Thanks again,

    Kjersti



  • 7.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-08-2019 02:56 AM

    Actually no, it did eventually complete. It did stall for a long time on the same step, but it completed successfully eventually.

    Some guidelines on how to debug such issues would be greatly appreciated.



  • 8.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 11-21-2019 11:51 AM

    I am also currently having this issue. This is my first time installing th CWP stack. Any info would be very helpful



  • 9.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-05-2019 02:17 PM

    I got this reply from my Amazon ticket. It is clear this is a Symantec issue can someone please help. 

     

    Hey there, Greetings for the day! Thank you for your continued patience and understanding with respect to this case. I would like to inform you that we have performed additional testing to the Symantec CWP for Storage template and following were the observations - -> Since the error faced was 'Received 0 SUCCESS signal(s) out of 1’, the initial try was to increase the resource timeout defined in the Creation Policy for the ‘ControllerGroupPolicy’ Autoscaling Group. going through the template, it was observed that the current timeout is ‘PT20M’. So, we tried changing it to ‘PT40M’ to check if the application and the underlying resources need more time to provision.

    When launching the stack with this configuration, it was observed that the stack failed after 40 minutes. Therefore, we can rule out timeout as the reason for this issue. -> We then SSH’d into the back-end instance and confirmed if the ‘cfn-signal’ script is installed or not. If the script would have been missing, it could be the reason behind the autoscaling group not getting the success signal from the EC2 instance. However, upon running the command 'sudo find / -name cfn-signal’ on the instance, it was confirmed that the script is indeed present. These checks can be performed by you as well, following the article [1]. -> Further, having found no indication of the probable cause of the error in the logs provided by you as well as the logs generated while doing the replication of the issue at our end, we went through the Symantec CWP template, and it was noticed that the AutoScaling Group ‘ControllerGroupPolicy’ has an If condition defined for the Launch Template to be used, wherein, if while launching the Cloud Formation stack, the value for the parameter ‘CombinedCUPU’ is specified as Yes, the Launch template ‘AllInOneControllerLaunchTemplate’ will be used, else the Launch template ‘ControllerLaunchTemplate’ will be used. Going by the description of the parameter, when specified as Yes, the CU and PU instances will be deployed on the same EC2 instances. Thus, the next test performed was to launch two separate Cloud Formation stacks with ‘CombinedCUPU’ as ‘Yes’ and ’No’ and note the difference. However, in both cases the same error was received. Additionally, we also observed in the logs the script that was getting stuck

    - ========= With CombinedCUPU as ‘Yes’ - Launch Template -> AllInOneControllerLaunchTemplate Last script visible in the /var/log/cfn-init.log -> Running command 24-caf-start No test for command 24-caf-start Command 24-caf-start succeeded Command 24-caf-start output: Running command 25-enroll-check No test for command 25-enroll-check ——————— With CombinedCUPU as ’No’ - Launch Template -> ControllerLaunchTemplate Last script visible in the /var/log/cfn-init.log -> Running command 16-caf-start No test for command 16-caf-start Command 16-caf-start succeeded Command 16-caf-start output: Running command 17-enroll-check No test for command 17-enroll-check ========= Going through the aforementioned logs, it seems that in both the cases the command ‘enroll-check’ is where the operation gets stuck. Further going through the template, I checked the two launch templates, and I noticed that for the ‘enroll-check’ command, following script has been defined - ========= /usr/local/symantec/spe/settings/enroll_check.sh ========= Previously, when we had SSH’d into the instance and ran the script manually, we received the '/opt/Symantec/cafagent/bin/CAFStorage.ini not found’ error, while the file seemed to be present. For testing purpose, the template was further modified to remove the ‘enroll_check’ commands from the two launch template resources and the commands in queue to be executed after that. Once done, the Cloud Formation stack was launched again twice, once with ‘CombinedCUPU’ as ‘Yes’ and then as ’No’. This time, it was noticed that the autoscaling group ‘ControllerGroupPolicy' received the success signal from the cfn-signal script and the stack creation was successful. ========= Log Snippet for reference when the stack was created successfully -> 2019-12-04 09:34:35,550 [DEBUG] Running command 16-caf-start 2019-12-04 09:34:35,550 [DEBUG] No test for command 16-caf-start 2019-12-04 09:34:35,659 [INFO] Command 16-caf-start succeeded 2019-12-04 09:34:35,660 [DEBUG] Command 16-caf-start output: 2019-12-04 09:34:35,660 [DEBUG] No services specified 2019-12-04 09:34:35,662 [INFO] ConfigSets completed 2019-12-04 09:34:35,662 [DEBUG] Not clearing reboot trigger as scheduling support is not available 2019-12-04 09:34:35,662 [INFO] -----------------------Build complete----------------------- 2019-12-04 09:34:36,350 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.us-east-1.amazonaws.com 2019-12-04 09:34:36,350 [DEBUG] Signaling resource ControllerGroupPolicy in stack cwp-demo-stack with unique ID i-05b730f4e000000 and status SUCCESS =========

    Therefore, I believe that the issue is with the ‘enroll-check’ command defined in the Cloud Formation template provided by Symantec. You can test this out at your end as well by removing the enroll-check command and the ones after that from the two launch template resources. The stack creation would be successful. Since the operation is getting stuck at the enroll-check command, cfn-signal script is unable to signal back to the ‘ControllerGroupPolicy’ auto scaling group with any success/failure code, and so the command times out after 20 minutes. Even after increasing the timeout to 40 minutes, in order to allow more time for the command to complete its execution, it still timed out. This means that the issue is with the script itself.

    Further, the template being used is provided by Symantec and so we do not have much information as to what operation the enroll-check command is expected to perform and what other commands are dependant on it. We could however test the configuration from CloudFormation’s end by removing the command altogether from the template. This led to the successful creation of the stack, which means that all the network configuration and the required cfn-signal scripts, along with the resource timeout defined in the Creation Policy has been correctly defined. Any changes or modifications that we make to the template might cause some unforeseeable issues in the future, which is why it is not recommended to change anything in the template which is provided from an external source, unless we are sure about the expected outcome. Since the template is provided and maintained by Symantec CWP, we will not be able to troubleshoot this further and I would suggest you to contact the Symantec support team with our findings. They will be able to provide much detailed insight into the command and what possible reasons could be there behind it getting stuck and causing the resource creation to fail. In the end, please accept my most sincere apologies for the delay in getting back to you. I understand that it might have hampered your workflow but it took us some time to finish the tests at our end and to go through the template provided by Symantec. Once again, I thank you for your understanding and cooperation. Please do get back to me if you have any questions or concerns regarding the same and we will be delighted to assist. Wish you a great day ahead, take care! :)

    REFERENCES: [1] https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-failed-signal/

     



  • 10.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-05-2019 02:54 PM

    Thanks for posting the feedback from AWS, it is informative.

    I had sucess using an old template instead of the current, but that is only a short term fix.

    It would be nice if someone from Symantec could step in here and inform us that they are working on fixing the template.

     

    Thanks again!

     

    Kjersti



  • 11.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 12:04 AM

    You guys should raise a support call with Symantec if you have that option. Then post back with the update.



  • 12.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 06:26 AM

    Kjersti, would you be able to link to the template you are using?

     

    I have been trying to get in contact with Symentac support on and off for over a week with no luck at all either by chat, phone or support ticket.



  • 13.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 06:49 AM

    Just out of curiosity, how are you guys deploying the CFT file? Are you manually uploading the template? Or Are you using the CFT supplied via the S3 bucket during the setup wizard?

    It might be worth attempting to deploy from the CFT supplied by S3 bucket during the setup wizard and see if the issues presist from there. 



  • 14.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 06:58 AM

    I have tried both. I tried the setup wizard first, and only when that did not suceed did I do it manually.

    As for sending a support ticket, I tried that the last time, and it took me a week to get approved as a submitter of requests, I haven't found the time to go back and resubmit the issue, since I got the old template working. But of course, I should send a ticket.



  • 15.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 07:30 AM

    Interesting. 

    I'm somewhat inclined to think that this might be related to permissions, rather than a fault with the template. 

    I know as part of the CFT, if no IAM roles are specified during the integrations setup, then a new set of IAM roles are generated, based on permissions already established by the users AWS account profile. 

    I've got a sneaking suspicion that either, the AWS account you're using might not have enough privileges to execute the enroll_script, or rather, generate the CAFStorage.ini file. Or, that the IAM role specified is lacking those privileges. 

    I know during setup, a lot of the CAF files are either pushed to instances/resources via S3 buckets, or downloaded from Symantec servers. So there are a number of things that could lead into this being permissions. Either that for example, the IAM role being used to generate the stack has limited permissions to the VPC it sits on. Or, might not have the permissions required to execute scripts, etc. 

    Although, this is speculative based on scenarios I have seen in the past where I've encountered issues deployed the stack. 

    Is it possible you could also share the CFT that you're using?

    Thanks. 



  • 16.  RE: CWP for Storage CloudFormation Setup Issue

    Posted 12-06-2019 09:42 AM

    I had been deploying it with my user account which has admin (allow *) access in AWS. So i do not think it wouldve been user level permissions. Also if that was the case I would imagine (hope) that one of the steps in the stack wouldve failed with a permissions issue.

     

    I was able to open a case after waiting on the phone with Symantec. Case # 31173399. I hope they can help.

     

    I have also just tried to do another stack deployment after deleteing my previous stack and the link in the quick setup guide is now broken.