New & changed ucsrv.ini options in v12.3

[Michael A. Lowry]

A few days ago, Automic Automation V12.3 was released. There are numerous differences between the v12.2 ucsrv.ini file and the v12.3 ucsrv.ini file. In particular, there were many changes to the options in the [PORTS], [REST], and [OBJECTSEARCH] sections.

Ports

The [PORTS] section is completely different in v12.3 compared to earlier versions. Whereas in v12.2 it was possible to specify ports for individual AE server processes, with v12.3, this does not seem to be possible. There are three options related to AE server ports in v12.3

[PORTS]
CP.PORTS=2217-2221
JCP.PORTS=2317
JWP.SYNC.PORTS=2271-2279

First of all, why its the option for specifying JWP ports named differently from the other options? (What's the meaning of SYNC in this context?)

Second, is it no longer possible to specify the listening ports for normal (non-Java) work processes?

Object search

The 12.3 documentation lists only one [OBJECTSEARCH] option:

[OBJECTSEARCH]
;
; index.cache.folder: Path to the folder where the search index wil be stored.
;
index.cache.folder=indexCache

I have an older version of the ucsrv.ini file that includes some additional options under [OBJECTSEARCH] that are not in the documentation.

[OBJECTSEARCH]
;
; enabled: Enable or disable the search index updater
;
; parameter enables or disables updates of the search index.
; Use it to temporarily disable the indexer if the JWP has stalled.
; Without index updates, the object search returns no or outdated results!
; 0 - Index Search is disabled.
; 1 - Index Search is enabled (default)
;
enabled=1
;
; update.delay.min: Minimum delay between index updates
;
; parameter configures the minimum delay (seconds) between index updates.
; If the JWP receives an OHC kick during
; period, its processing will be postponed after the delay.
;
update.delay.min=10
;
; update.delay.max: Maximum delay between index updates
;
; parameter configures the maximum delay (seconds) between index updates.
; The index will be updated after
; time has elapsed, OHC kicks received between min and max delay will be processed immediately.
;
update.delay.max=300
;
; update.batch.size: Number of indexed objects per batch
;
; parameter configures the number of objects that are indexed at once.
; If the number of objects exceeds the batch size, the indexer processes them in multiple batches with delays in between.
; The provided batch size must be supported by your database's limit on IN() queries.
;
update.batch.size=1000
;
; update.batch.delay: Delay between index batches
;
; parameter configures the delay between index update batches (seconds)
; The index updater will wait
; amount of seconds between index batches.
; Lower batch sizes and higher batch delays reduce the DB load during indexing,
; but also means slower index updates and a higher probability for outdated search results.
;
update.batch.delay=0

Do these options still work?

[Frank Muffke]

Hi

regarding WP specification in ini file: I heard from an Automic guy that the old port specification (WP1=.. WP2=... WP3=...) can be used but thr logic was moved to DB with version 12.3.The Port specification was only used to detect if a WP was running or not.

I changed it to th e new scheme but had a logical issue.

I started some WPs and they used randomly free ports.
When I started an agent (which runs on AE) i got the "address already in use" - error.

So either I have to start the agent priot to starting the WPs or I use the old scheme...

cheers, Wolfgang

------------------------------
I know I do really know it!
------------------------------

[Carsten Schmitz]

Hey ​Wolfgang,

> I heard from an Automic guy that the old port specification (WP1=.. WP2=... WP3=...) can be used
> but thr logic was moved to DB with version 12.3.

Can you elaborate, what does this mean (especially the "logic moved to DB" part). I do not quite understand this.

Thanks!

[Carsten Schmitz]

Very confusing, this.

A colleague of mine updated a sandbox recently. He only specified the CP range.

This is the smgr picture to go with that installation:

​
On the server, I see this port usage:

# netstat -anp | grep -i listen | grep ucsrvwp
tcp6       0      0 :::2270                 :::*                    LISTEN      10071/ucsrvwp

... so that's only the PWP using a TCP port. But I have many more than one WP:

# pgrep ucsrvwp
10071
10080
10136
10137
10140
10141
10145
30246
30250

Paging anyone from Automic for a statement.

Is this actually correct? Only the PWP using a TCP port now!? Do the workers not communicate not over TCP anymore, and if so - that's a major change in how everything works, or would I be mistaken?

We have announced an update of the first real installation soon, and this uncertainty is bad.

Thanks.

[Carsten Schmitz]

The WP ports in AWI, with the exception of the PWP also say "0" now:

But the additional workers seem to process test jobs, or at least the logs say so.

Did they change the WP to some other form of IPC that isn't TCP?

I also opened a ticket now to hopefully get confirmation if this setup is now correct.

[Michael A. Lowry]

The diagram on the AE v12.3 Server Processes page has not been updated, and still shows communication links between all CPs and WPs. The page Types of Server Process, new in v12.3, provides additional detail:

The primary work process (PWP) is the only work process (WP) that has a port assigned. This port is specified in the pwpPort= parameter in the [TCP/IP] section of the INI file of the Automation Engine. For more information, see Automation Engine.
The rest of the work processes (WPs) connect to the primary work process (PWP) and to the communication processes (CPs). Therefore, they do not need a port assigned and establish the connection through the port assigned to the process (PWP or CP) they connect to. 

This suggests that while the non-primary WPs do not open a listening socket, they do establish outgoing connections to the PWP and CPs.

See also my article on inter-process communication in AE 12.2.

[Carsten Schmitz]

G​ood information, thanks.

One would wish Automic would point that out in the changelogs more clearly. Not just the new syntax requirement, but also what they (must have) changed in the workings.

This could have easily broken some customer's monitoring, for example. Luckily, we only explicitly monitor the CP by connecting to their TCP ports, not the WP.

[Michael A. Lowry]

We run two nodes for each AE system. We run odd-number WPs & CPs on node A, and even-numbered WPs & CPs on node B.

Is this no longer possible in v12.3?

[Carsten Schmitz]

​We do the same (i.e. odd and even number convention), so we're slightly concerned about this, too.

But I don't (yet) see why this should not be possible anymore? Don't the WP get their names from the .smd file?

[Michael A. Lowry]

If you use the service manager, you can see the SMD names (the ones specified in the .smd file) in the undocumented fixServicename column, but these SMD names appear to have no effect on the names assigned to the processes when they start up.

The WPs & CPs get their names from the order in which they start. They get their ports from the next one in the ranges specified in the ucsrv.ini file.

Wolfgang indicated that the old way of assigning specific ports to specific AE server processes by name might still work. I have not tested this yet.

[Carsten Schmitz]

​Thanks.

> Wolfgang indicated that the old way of assigning specific ports to specific AE server processes by name might still work.

We haven't tested it either. My colleague says he has read that the backwards compatibility is only for JWP, JCP, but at the moment that's all hear-say. We'll be upgrading our first multi-homed engine this afternoon, and we prepared the .ini files according to the new syntax. Perhaps I'll be wiser then :)

[Carsten Schmitz]

So then, report from the 12.3 update:

As you know we don't touch ZDU at this time anymore, so classic update. Worked fine in the grand scheme of things.

We had to restart the engine once more after bringing it down, because despite stopping all clients and killing off all processes beforehand (verified), the update utility claimed "the system is still running". After cycling the engine on and off again, the DB update worked fine. Chalk one down to "have you tried turning it on and off again".

After the update, we couldn't get the JWP and JCP (the same Java process really) to work on one machine and only got a network exception. Turns out someone (*cough* me) had used the wrong commenting character ( # instead of ; ) further down in the file. This confused the ini file parser, but not enough to print a meaningful error, it passed it down to Java and the JWP/JCP then died with a network-centric Java exception. I accept half of the blame, but put 50% of the blame on really shoddy parser coding. Debugging this took two hours.

We now have the JWP and JCP assign themselves random ports, which works. Which makes me wonder: Is the AWI truly just trying all the ports to find the JCP? If so, why did we ever use well-known ports? Wouldn't it be cool if SSH was always running on a random port and we'd just try ports until we found an SSH banner? *

On the ticket and ports front, Automic has gotten back to me and written that they are gathering the requested information. I think we can establish by now that the non-PWP workers in 12.3 do not allocate TCP ports for LISTENing anymore. I'm keeping the ticket open since I'd really like to learn how much of a paradigm shift has actually happened with this between 12.2 and 12.3.

Br,



*) no

[Michael A. Lowry]

@Carsten Schmitz wrote:

We now have the JWP and JCP assign themselves random ports, which works. Which makes me wonder: Is the AWI truly just trying all the ports to find the JCP? If so, why did we ever use well-known ports?

In my system, the JCPs opened a listening socket on the port specified in JCP.PORTS.

[Carsten Schmitz]

Sure, that works. We simply did forgo the JCP.PORTS and JWP.SYNC.PORTS statements, having them pick random ports. Pinning them to specific ports is probably nice to have when you have a firewall between the machines.

But since the AWI appears to talk to those processes (at least the JCP) via TCP, and I can not see where the chosen port would be configured on the AWI side, I suspect the AWI still simply tries all the ports even when you pin them on the engine side. I don't have proof, but this isn't without precendence either: some years ago I posted an strace where some Automic process essentially did:

for i in 1-65535 do some tcp stuff

So my feeling of potential bewilderment remains.

[Michael A. Lowry]

Good lord, I hope it's not doing that. I see some outgoing connections from the JCP to the AWI, so perhaps the JCP initiates the connections, or uses those to tell the AWI its port number. I posted a reply to the IPC thread, asking for a better description of IPC in v12.3.

[Michael A. Lowry]

I think I found the answer in an earlier discussion between me and @Robert Thullner:
​

"When a JCP is starting up, it will register itself in one of the MQ tables (I do not know the exact one right now). This table contains all running AWA components. When a user logs in to the AWI, the CP will ask for available JCP endpoints and return it to the AWI. The information is retrieved from the above mentioned database table."

It's easy to forget that a lot of the IPC is done through the MQ tables.

[Carsten Schmitz]

​Thanks, again: good info!

So that must also be what Wolfgang Brueckler meant when he said that the logic moved to the DB (or something like that).

[Laura Albrecht]

@Carsten Schmitz - what did you mean about "we don't touch ZDU at this time anymore"? ​

------------------------------
Enterprise Scheduling Lead
Takeda
------------------------------

[Carsten Schmitz]

> ​what did you mean about "we don't touch ZDU at this time anymore"?

Hm, how can I phrase this without getting in trouble?

We've performed a number of ZDU attempts over the last two years. Some worked wonderfully. Then there was the other kind. We had issues with ZDU where we couldn't finish it and also not roll back to the previous version (at least not without a db restore and accompanying data losses from backups) because that's the rarely mentioned inherit nature of the ZDU: once you commit to it, you need to finish it. Not today, but eventually.

When we further inquired into the various issues we encountered, we initially got responses about a supposedly correct procedure that didn't match up with the other materials published by Automic. For the record, this part should at least be rectified by now because I have been assured in the aftermath of it the support engineers have now received additional briefings on the procedures by the development, but I'm not in a position to attest to either.

Memory of what all the issues were exactly is rather hazy. You might find some threads here where I reported or inquired on some issues, it's quite likely actually :)

I do distinctly remember that at the core of one failed ZDU, we tried to get information on UC4 objects that supposedly had (or had not) to be deactivated before our ZDU could be finalized, and that we couldn't finalize it for weeks because of an error message about some table not being empty. We were fully commited to get to the bottom of this, but the issue (and any change of the support call) was eventually resolved when the monthly patch cycle rebooted the AE servers, thus cycling the whole thing at once - the exact avoidance of which is, imho, the point of the ZDU :)

So all things considered, and even with presumably issues fixed and processes improved since the early days of ZDU (please don't ban me from the forums just yet, Broadcom, mkay?), the lost trust in the mechanism has led us to conclude that when waging our options, we will only perform "classical" updates for the forseeable future. While that does incur about 1-2 hours of downtime for us, the impact of the downtime has been judged as more acceptable over the uncertainty.

Best regards,
Carsten

[Laura Albrecht]

Ugh.  I don't want to get off track in this thread, so I'll start searching for your other ZDU threads, but this is not what I wanted to hear.  I attempted ZDU  myself several times and it was.... not good.  :-)  I too decided to stay with classic upgrades, but I thought that by now - 12.3 - those issues would be resolved, etc. Thanks for your honestly / info.  You may see another post from me in your ZDU thread.  :-)

------------------------------
Enterprise Scheduling Lead
Takeda
------------------------------

[Scott Hughes]

I found some occurrences where the version tables were getting out of synch.  My first attempt at ZDU at early release had unidentified open WP connections that just plain got lost but I have not seen those issues since AWI was rolled out.  There were some issues with Period objects refreshing during a ZDU, I dont recall if this was resolved but it is a rather old issue and I want to say it is no longer problematic.

ZDU adds a table UC_ZDU, which appears to store reference data for the ZDU process.  This table contains 2 rows during ZDU and 1 during normal operations.

I would try it out in sandbox a few times before dev.  My last encounter with ZDU about 6 months ago went very smooth.

-- 

~~~~~~~~~~~~~~~~~~~~~~~~
Scott Hughes
Verizon  [phone]
[e-mail] 
~~~~~~~~~~~~~~~~~~~~~~~~

[René Kappel]

Hi Michael,

this is still possible, but you would have to start one WP on node A and one on Node B and again one on Node A and so on.

It is no longer possible to assign the WPs to certain nodes by their number.

However, you'd still know easily which WP runs on which node by checking the 'host' column in the Adminstration view (f.k.a. System Overview)

Overall, I think this change is a good thing as it makes the configuration of large multi-node environments with up to 100 WPs easier as it is much easier to keep the ucsrv.ini files on alln nodes aligned.

just my two cents.

René

[Michael A. Lowry]

@René Kappel:
That's what I was afraid of. Coordinating the timing of starting AE server processes between multiple nodes is not practicable. It would be hard enough to arrange this during normal startup; it would be impossible after processes had stopped and restarted unexpectedly.

We appreciate the idea of minimizing the number of different ucsrv.ini files needed, but this implementation doesn't help us much.

• First, because we want to be able to tell at a glance which node a log file came from, we will have to update the log & trace file options to explicitly include the node name.
• Second, because we use net areas, we already need different config files for that.

What would be more useful would be the option to set AE server options via some combination of

• environment variables, and
• an external configuration server (something like Spring Cloud Config).
  

This would go a long way toward making the AE cloud-ready too.​​

[Michael A. Lowry]

While I'm making requests, why not add these capabilities?

1. Define variables at the top of the ucsrv.ini file, optionally based on the values of environment variables, and then use these variables in specifications elsewhere in the file. (Similar to the VAR declaration in .smd files)
2. Include the specific process type in the log & trace file names (I.e., distinguish WP from JWP and CP from JCP.)
3. Specify a range of CP numbers assigned to each net area, and to automatically include the net area in the log & trace file names.

This would make it possible for us to use just one ucsrv.ini file for all server processes.

[Scott Hughes]

The one thing I noted about the ucsrv.ini settings was that I would no longer have to worry about contention between the JCP and CP ports.  I used to set the JCP with a huge delay as it starts up lightning fast and if allowed can disrupt agent communication.

Looking at most of the changes I suspect this is preemptive to contanerization of the AE.

------------------------------
Scott Hughes
Senior Automation Engineer
Verizon
Albuquerque, NM
------------------------------

[Michael A. Lowry]

I also insert substantial delays in the SMC file, to help ensure that the process numbers in the Service Manager line up (1=1, 2=2, and so on). It's a bit silly that a better system has not been devised.

[Carsten Schmitz]

... especially when you consider that delays and the order in smc are short lived.

We had a consultant here and painstakingly ordered all of our processes into the most "correct" start order with delays. Some weeks and a few rewrites of the file by Service Manager later, it's all ordered differently again.

So ... make a backup while you have it nicely set up :)

[Rafael Lubera]

So do the WP's still need to be assigned or is there a wp.ports statement that can be used as well ?

 

Rafael Lubera
Technology Specialist Automation Engineering

T [phone]
C [phone]
[e-mail]

CME Group
20 South Wacker Drive
Chicago, Illinois 60606
www.cmegroup.com

 

---

NOTICE: This message, and any attachments, are for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at E-Communication Disclaimer. If you are not the intended recipient, please delete this message. CME Group and its subsidiaries reserve the right to monitor all email communications that occur on CME Group information systems.

[Robert Thullner]

No, the only WP port which is required in the ucsrv.ini is the pwpPort. 

The configured wp*=<port> were only used for naming WPs. The port was opened just to find out if the name is already taken. They were never used for any other operational purposes.