Release Automation

Release Automation KB: Artifact Distribution and Cleanup 

12-28-2014 05:40 AM

The process of artifact distribution and cleanup has several stages, which we will outline below.


Artifact NES Distribution

  1. Artifacts are retrieved by retrieval agent and downloaded to <Retrieval_Agent_Root>/files/ART<Release_ID>
  2. They are moved to <Retrieval_Agent_Root>\files_registry\<File_MD5>
    • This sub-stage is sometimes referred to as the "Retrieval Stage". While the files are being moved, there could be a point where the file exists on the agent at both locations at the same time (thus peaking at two times its size). After the move finishes, they exist only in 'files_registry'.
  3. They are copied to the NES at <NES_Root>\artifact_store\releaseId_<Release_ID >
    • Each NES will request and get only the artifacts that are needed by the agents that are connected to it.
  4. They are deleted from <Retrieval_Agent_Root>\files_registry\<File_MD5>.


The files now reside exclusively on <NES_Root>\artifact_store\releaseId_<Release_ID>

 

Artifact Agent Distribution

  1. All the release artifacts are now copied to the NES, at <NES_Root>\files_registry\<File_MD5>
    • At this point the file exists on both locations in the NES: 'files_registry' and 'artifact_store'.
  2. Every agent requests the artifacts it needs from its NES and downloads them to a folder named: <Agent_Root>\artifact_store\releaseId_<Release_ID >
    • The artifact will remain available there during the release processes runs.
  3. After all the agents have finished downloading the artifacts from their NESs, the artifacts will be deleted from <NES_Root>\files_registry\<File_MD5>

 

The files now reside on two locations:

<NES_Root>\artifact_store\releaseId_<Release_ID>

<Agent_Root>\artifact_store\releaseId_<Release_ID>

 

Artifact Cleanup

When the release reaches a final state ("failed" or "finished"), the NAC sends an order to all the NESs and agents participating in the release to delete the folders named with the release ID under the ‘artifact_store’ folder.

This should clear all the disk space retained by the release artifacts.

However, it is possible that some of the NESs/agents weren't reachable at the end of the release and have missed the cleanup command sent from the NAC.

In order to avoid keeping obsolete artifacts, once a day each NES and agent checks if all the release IDs under the ‘artifact_store’ folder are still relevant (i.e. the release in not in a final state) and deletes all the obsolete ones.

 

The release is finished and a day has passed.

The files now will not not exist on the retrieval agent, NES or target agent machines.

 

Artifact Caching

Every time a node transfers a file using an agent, it will also be stored in this node cache and used later for optimizations.

From the RA perspective the disk space allocated for caching (1 GB by default) is always considered as used by the system (even if it can be sometimes be free).

The cache works in LRU mode by default and will discard the least recently used file first, when it reaches the size limit.

Statistics
0 Favorited
2 Views
0 Files
0 Shares
0 Downloads

Tags and Keywords

Comments

03-14-2016 06:53 AM

"In order to avoid keeping obsolete artifacts, once a day each NES and agent checks if all the release IDs under the ‘artifact_store’ folder are still relevant (i.e. the release in not in a final state) and deletes all the obsolete ones."

Does "once a day" mean within 24 hours after a certain deployment or just somewhen at night each night?

And is it possible to change "once a day" to once every 12 hours?

 

 

02-10-2016 12:21 PM

Both NES and Agent servers storing files/artifacts in their local cache

Jacky

02-10-2016 05:14 AM

Hi,

One question.. in the below statement when you say Node, Is it Execution Server only?

 

Artifact Caching

Every time a node transfers a file using an agent, it will also be stored in this node cache and used later for optimizations.

01-09-2015 05:54 PM

Thank you Beerit for sharing this with the community!

Mary

Related Entries and Links

No Related Resource entered.