The process of artifact distribution and cleanup has several stages, which we will outline below.
Artifact NES Distribution
- Artifacts are retrieved by retrieval agent and downloaded to <Retrieval_Agent_Root>/files/ART<Release_ID>
- They are moved to <Retrieval_Agent_Root>\files_registry\<File_MD5>
- This sub-stage is sometimes referred to as the "Retrieval Stage". While the files are being moved, there could be a point where the file exists on the agent at both locations at the same time (thus peaking at two times its size). After the move finishes, they exist only in 'files_registry'.
- They are copied to the NES at <NES_Root>\artifact_store\releaseId_<Release_ID >
- Each NES will request and get only the artifacts that are needed by the agents that are connected to it.
- They are deleted from <Retrieval_Agent_Root>\files_registry\<File_MD5>.
The files now reside exclusively on <NES_Root>\artifact_store\releaseId_<Release_ID>
Artifact Agent Distribution
- All the release artifacts are now copied to the NES, at <NES_Root>\files_registry\<File_MD5>
- At this point the file exists on both locations in the NES: 'files_registry' and 'artifact_store'.
- Every agent requests the artifacts it needs from its NES and downloads them to a folder named: <Agent_Root>\artifact_store\releaseId_<Release_ID >
- The artifact will remain available there during the release processes runs.
- After all the agents have finished downloading the artifacts from their NESs, the artifacts will be deleted from <NES_Root>\files_registry\<File_MD5>
The files now reside on two locations:
When the release reaches a final state ("failed" or "finished"), the NAC sends an order to all the NESs and agents participating in the release to delete the folders named with the release ID under the ‘artifact_store’ folder.
This should clear all the disk space retained by the release artifacts.
However, it is possible that some of the NESs/agents weren't reachable at the end of the release and have missed the cleanup command sent from the NAC.
In order to avoid keeping obsolete artifacts, once a day each NES and agent checks if all the release IDs under the ‘artifact_store’ folder are still relevant (i.e. the release in not in a final state) and deletes all the obsolete ones.
The release is finished and a day has passed.
The files now will not not exist on the retrieval agent, NES or target agent machines.
Every time a node transfers a file using an agent, it will also be stored in this node cache and used later for optimizations.
From the RA perspective the disk space allocated for caching (1 GB by default) is always considered as used by the system (even if it can be sometimes be free).
The cache works in LRU mode by default and will discard the least recently used file first, when it reaches the size limit.