Automic Workload Automation

Expand all | Collapse all

How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

  • 1.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-07-2017 07:45 AM
    How to executes DB Load in parallel with batch mode :

    ucybdbun -BTRANSPORTALL -C1000 -XI:\UC4\Utility\db\uc_data1000.txt.

    Because the file ucybdbld_log_00 is in use.


    ERROR: Permission denied, can't create logfile ..\TEMP\ucybdbun_log_00.txt




  • 2.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-07-2017 08:13 AM
    ERROR: Permission denied, can't create logfile ..\TEMP\ucybdbun_log_00.txt

    How to change the name of the log file ?


  • 3.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-07-2017 08:32 AM
    While any unload is running the logfile *_00.txt is locked by the unload tool - as ist currently running...

    You could install a second instance (directory) of utilities or DB unload but I am not sure if this prevents you from any DB locks...

    Would it be feasible for you starting one unload after the other?


  • 4.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-07-2017 04:18 PM
    It really ought to be possible to instruct the AE utility programs (ucybdbun, ucybchng, ucybdbld, etc) to send logging information to standard output and tracing information to standard error.


  • 5.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-09-2017 08:19 AM
    Karina_Mankauskas_553

    I suspect this is on Windows? You could try the same thing with AE utilities on Linux/UNIX, those should not block their logfile for writing (disclaimer: did not try it myself).

    Michael A. Lowry

    It really ought to be possible to instruct the AE utility programs (ucybdbun, ucybchng, ucybdbld, etc) to send logging information to standard output and tracing information to standard error.
    But you can, at least on UNIX/Linux  :)

    Just set "logging=" to /dev/stdout and the traces to /dev/stderr in the ini file, works like a dream - just tried it.

    Not sure if the same thing can be done on Windows, since I don't have a Windows installation. There is still a device called CON on Windows with heritage back to CP/M, and there is SCREEN$ which goes back to OS/2 IIRC, one of them might work the same way. But probably not  :p




  • 6.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-09-2017 08:28 AM
    Karina_Mankauskas_553 , once more:

    It appears that you can override certain ini options on the command line. See the "Start Parameters" paragraph of this page. Then you could use separate log files for your parallel sessions, which would not conflict.

    It is, however, unclear to me how exactly one would go about that, I find the respective Automic documentation confusing. You could give it a try, or invoke the professionals via a ticket if it can't be figured out.


  • 7.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-09-2017 08:32 AM
    Hi Carsten_Schmitz_7883 , ok thank you. Just I need know what options I must insert in command line for not show conflicts.
    I'ill do tests.





  • 8.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-09-2017 10:12 AM
     
    Just set "logging=" to /dev/stdout and the traces to /dev/stderr in the ini file, works like a dream - just tried it.
    Brilliant! If this works as you say, I’m going to roll this change out. Finally, a way to log & trace individual runs of the utilities.


  • 9.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-09-2017 02:03 PM
    For Windows its possible as well with a little scripting Trick   :p

    2coyc93574e3.jpghttps://us.v-cdn.net/5019921/uploads/editor/8y/2coyc93574e3.jpg" width="516">


    [EDIT]

    Precondition: call the unload utility in Batch Mode

    Step1: do a COPY of the file ucybdbun.ini to a temporary & unique filename containing the RUN ID
    Step2: within the new INI file change entry ucybdbun_log_##.txt to a unique entry containing the RUN ID
    (I use a Powershell command for this)
    Step3: execute Unload Utility in Batch mode with the temporary new INI file

    voila....





  • 10.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-10-2017 04:44 AM
    With all this said though, I'd probably still follow FrankMuffke 's advise and try to do one after another.

    Doing those in parallel - well, I don't (yet) see the advantage, and you may be encountering database deadlocks. Any time savings from parallel operation are probably rather small, at least unless your DB server is purely built on a solid state array ...


  • 11.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-10-2017 04:55 AM
    Carsten_Schmitz_7883

    Tssssss and this negative comment after I spent about 200 hours and 30.000 Euros for a new invention.....
    Was just brainstorming for a cool name for this tool... - wcambldblt
    Wolfgangs cool automatic multiple batch loading DB load tool
    :p

    BTW: technically you are absolute right, for the sake of safety I would do all Loads and unloads one after the other to prevent any DB issues as well.



  • 12.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-10-2017 05:07 AM
    Sorry FrankMuffke, but I thought it had to be said. Thought I'd redeemed myself by crediting your initial, and very appropriate wisdom though  ;)

    As for the names, I don't care for the Automic convention of naming tools that much. Yes, it's systematic, but it looks like the fallout from the recently exploded ASCII factory and it's not catchy at all.

    I think "Automatic Wolfgang" has a nice ring to it for your complete tools suite, or possibly "Wolfgang Alpha". (No, baaaad Google, I did not mean Wolfram Alpha!").

    But you need to change your name every couple of months, that's just how the Automic ecosystem rolls ...  :D


  • 13.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-10-2017 05:22 AM
    Simultaneous DB unloads usually do not make sense, because there is only one transport case per AE client. Simultaneous DB loads to the same client do work, but they can cause performance degradation, particularly for connected GUI & Java API users.

    Therefore:
    • We use a SYNC object to prevent simultaneous unloads from the same AE system (we use only one client per system). This ensures that each unload has exclusive access to the transport case.
    • We serialize DB loads. This helps prevent the aforementioned performance problems.


  • 14.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-10-2017 05:31 AM
    Simultaneous DB unloads usually do not make sense, because there is only one transport case per AE client.
    The only realistic situation I could imagine would be:
    * performing multiple DB unloads of different clients
    * performing multiple DB unloads of one or more clients while a "system" unload is running (TRANSPORTALL, etc..)

    But honestly I think thats not a very common issue...


  • 15.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-13-2017 03:43 PM
    If you want the Sourcecode (just for fun or study) let me know :-)


  • 16.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 02:58 AM

    I would not recommend to run simultaneous DB loads (transport case) to the same client. This may cause inconsistent version control (version management) objects.

    This should be double checked with support / product development to avoid problems in that area in the future.



  • 17.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 03:07 AM
    Josef Scharl said:

    I would not recommend to run simultaneous DB loads (transport case) to the same client. This may cause inconsistent version control (version management) objects.

    I suggested this possibility in the other discussion thread. I have not tested this scenario yet, but I agree that it could lead to inconsistencies.

    We would also be interested to know whether simultaneous XML imports are similarly discouraged.


  • 18.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 04:22 AM

    Michael A. Lowry

    said:

    We would also be interested to know whether simultaneous XML imports are similarly discouraged.

    Well, simultaneous XML imports will work, because the version control object is crated within the import transaction ("importobject") directly. This transaction locks the object for update – so it’s save, no inconsistences to VC can happen.



  • 19.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 06:51 AM
    Josef Scharl said:

    I would not recommend to run simultaneous DB loads (transport case) to the same client. This may cause inconsistent version control (version management) objects.

    This should be double checked with support / product development to avoid problems in that area in the future.

    Fully agree!

    What about Unloads? save?


  • 20.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 07:35 AM

    I would not recommend to run simultaneous DB loads (transport case) to the same client. This may cause inconsistent version control (version management) objects.

    This should be double checked with support / product development to avoid problems in that area in the future.


    Hi Josef_Scharl_103 ,

    Ok, but If I need to run simultaneous DB loads , but for the differents clients?  for example, from file ucdata1000.txt, ucdata2000.txt, ucdata3000.txt,  for clients 1000 /2000/3000 , with command :

    :set &hnd=prep_process_var(VARA.CLIENTS)
    :process &hnd
    :  set &client=get_process_line(&hnd,1)
    :  p &client

    UCYBDBld -B -C&client AY -MY -XC:\Automic_v12\Utility\DB\uc_data&client.TXT -EREPLACE
    :endprocess


    Only one file finished with success.
    It is possible?
    Thanks
    Karina


  • 21.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-14-2017 08:06 AM
    Karina_Mankauskas_553

    Doesn't your code example in fact do that one after another, and not in parallel? Yes, it's one loop, but it should wait for ucybdbld to return before starting the next iteration.

    (Disclaimer: I just tried this with a "sleep 10" in a prep_process_var loop like yours, and in that case it does wait for the sleep to finish before starting the next. It could in theory be different for ucybdbld, but I don't think so ... edit: yeah, I'm pretty quite certain now it's not different)

    In other news, I'm curious: Why does this need to be run in parallel?


  • 22.  How to executes DB Load in parallel? Because the file ucybdbld_log_00 is in use.

    Posted 11-15-2017 09:52 AM
    Hi Karina_Mankauskas_553

    In case the transport cases are loaded into different client, I would not expect any site effect to the version management.

     KR, Josef