Containerized Linux x64 v24.2 Agent error : U02000186 Report file '/opt/agents/linux/temp/OAADYJIX.TXT' for Job 'JOBS.UNIX.TEST-AGENT (1799067)' cannot be opened. Error: 'java.io.IOException Broken pipe'.

[Marc Monlouis]

Hi community 

While i already succeded to contenerized agents WS REST, SQL, IG FTP, i meet some trouble to do so with Linux x64 v24.2 agent.
I refered to Broadcom doc Installing Containerized Java Agents to work.

Here are the main agents settings

[TCP/IP]
connection=myEngine.myDomain8443

[AUTHORIZATION]
keypassword=changeit

[VARIABLES]
UC_EX_IP_ADDR=p1docker.bo.itsgroup.com
UC_EX_IP_port=2300
UC_EX_JOB_MD_IP_ADDR=localhost
...

Container ports mapped  -> 2300:2300


I can see my agent in AWI an all seem to be Ok.
But all jobs executed on this agent failed with error
U02000186 Report file '/opt/agents/linux/temp/xxxxxx' for Job 'JOBS.UNIX.TEST-AGENT (1799067)' cannot be opened. Error: 'java.io.IOException Broken pipe'.
Agent is running as root in container.
I tried to execute in anonymous mode (i.e jobs executes as root) to avoid rights problems. I still have this error.

Any help is welcome

Thanks in advance 

[Oana Botez]

It seems that the name of the variable in the ini file you use to set the address of the job messenger is not correct. You can find the right name in the screenshot from the documentation below.

Hope this helps,

Oana

[Marc Monlouis]

Hi Oana, thanks for your answer

I fact the error was in this text not in ini file.
I corrected the tread text.

[Oana Botez]

Thanks for the update. The error is most likely caused by the Job messenger not being able to connect to the agent main process. Can you check if port 2300 is free in the container? Otherwise you could configure a different port for the Job messenger using the UC_EX_JOB_MD_IP_PORT setting.

BR,
Oana

[Marc Monlouis]

Hello Oana, 
Thanks for yuur answer 

please find below some more test.


The container is listening port 2300

Here is ss command execute from inside container:

bash-5.1# ss -tlpn
State                 Recv-Q                Send-Q                               Local Address:Port                               Peer Address:Port                Process
LISTEN                0                     50                                               *:2300                                          *:*                    users:(("java",pid=34,fd=23))


bash-5.1# ps -f -p  34
UID          PID    PPID  C STIME TTY          TIME CMD
root          34       1  0 14:12 ?        00:00:11 java -Xrs -Xmx1024M -jar ./ucxjoss.jar


The agent (process pid =34) is listening port 2300

Here is a check of port from outside of container 

openssl s_client  -connect localhost:2300 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = AE Agent Certificate
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = AE Agent Certificate
verify return:1
depth=0 C = US, OU = AE_LNX, CN = LINUX-1
verify return:1
139701829392192:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42
---
Certificate chain
 0 s:C = US, OU = AE_LNX, CN = LINUX-1
   i:CN = AE Agent Certificate
-----BEGIN CERTIFICATE-----

suppressed part

-----END CERTIFICATE-----
---
Server certificate
subject=C = US, OU = AE_LNX, CN = LINUX-1

issuer=CN = AE Agent Certificate

---
Acceptable client certificate CA names
CN = AE Agent Certificate
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3318 bytes and written 394 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: AE140F208013EEC45F880404E83A9E5076958E06AA67DD73A17E94ECACA79B1E
    Session-ID-ctx:
    Master-Key: 538A08C23EF32D56C0DE81B1D04B088F265E7E251B6169867892C56777A1B9508AEF8BC7F58109A77D7EEE47DFBA67EB
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1739803330
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: yes
---

So agent communicate ok on port 2300.  It uses self signed certificat,so we have an openssl return code 19. But it not a problem.

So it seems commication is ok on port 2300. 

Any idea ?

[Oana Botez]

Could you share the logs and/or trace files from the temp folder?

[Michael Dolinek]

Hi @Marc Monlouis

please make sure that all users have write permissions for the Agent's default folders such as temp, resources, backup. Either let the Agent create them at startup or use chmod in your Docker-file

e.g.

FROM ubuntu:20.04

 

RUN mkdir /opt/agent-unix

COPY ucxjlx6.tar.gz /opt/agent-unix

 

WORKDIR /opt/agent-unix

RUN tar -xzf ucxjlx6.tar.gz

RUN rmdir out & rmdir temp

RUN chmod 777 backup & chmod 777 resources 

 

WORKDIR /opt/agent-unix/bin

COPY --chmod=0755 run.sh .

 

ENTRYPOINT [ "/opt/agent-unix/bin/run.sh" ]

Michael

[Marc Monlouis]

Hi 

I finaly found the problem.
The library ibcrypt.so.1 is missing for process user_service_pkg.

I use Rocky linux 9.3 minimal image to build docker container.
Since RHEL 8 libcrypt.so.1 library is no more provides.
So i installed the package libxcrypt-compat-4.4.18-3.el9.x86_64.rpm to solve the problem.

I discovered that java agent V24.2.0 still uses C program. 
According to my understanding, the program user_service_pkg is the Job Messenger which is in charge of executing  the job and respond results to engine
If someone can confirm or correct it will be appreciate. 

Thanks for your help

[Adrian Fresno Menendez]

Hello Marc, your assumption is correct.

We had recently experienced the same on Amazon Linux 2023 as this library is no longer being delivered by default on the recent Linux OS Distributions and our agents (both c-based and java based) use it.

This is described in this article:

https://broadcomcms-software.wolkenservicedesk.com/external/article?articleNumber=388012

I will check with documentation team if we could add this prerequisite/remark when installing on recent OS distributions like RHEL9 or AL2023, thanks.

Adrian

[Marc Monlouis]

Hello Adrian, 

Many thanks for your confirmation.
Thanks Broadcom team for your reactivity.