VMmark

 View Only
Back to discussions
Expand all | Collapse all

ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

weiweiliang

weiweiliangJul 31, 2012 06:41 PM

Thanks Joshua. Attached are the files.

  • 1.  ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 27, 2012 11:30 PM

    Hi,

    I'm  having an issue with running VMMark 2.1. Each time I launch the STAX  job, ds2webdriver.exe stops working, but if I run the command line  "ds2webdriver.exe --target=DS2WEBA0" separately in a cmd window, it  works fine. Looks likes an issue with STAF and STAX, so I reinstalled  both and reconfigured STAF.cfg according to the benchmarking guide, but  it still doesn't work.

    I  've gone through all the threads I can find about this topic (e.g  fixing /etc/hosts file), but got no luck in fixing this problem. 

    PLEASE HELP! Any suggestion is recommended. Thanks.

    STAF version: 3.4.10

    STAX version: 3.4.5

    Log from DS2WEBA0

    <!--
    * DVD Store Common Include PHP Page - dscommon.inc
    *
    * Copyright (C) 2005 Dell, Inc. <dave_jaffe@dell.com> and <todd_muirhead@dell.com>
    *
    * Used by DVD Store php pages
    *
    * Last Updated 5/13/05
    *
    *  This program is free software; you can redistribute it and/or modify
    *  it under the terms of the GNU General Public License as published by
    *  the Free Software Foundation; either version 2 of the License, or
    *  (at your option) any later version.
    *
    *  This program is distributed in the hope that it will be useful,
    *  but WITHOUT ANY WARRANTY; without even the implied warranty of
    *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    *  GNU General Public License for more details.
    *
    *  You should have received a copy of the GNU General Public License
    *  along with this program; if not, write to the Free Software
    *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
    -->


    <HTML>
    <HEAD><TITLE>DVD Store Login Page</TITLE></HEAD>
    <BODY>
    <FONT FACE="Arial" COLOR="#0000FF">
    <H1 ALIGN=CENTER>DVD Store</H1>
    Can't connect to MySQL server on 'DS2-DB' (4)

    Thread 5: Error in Login for User user23891095, thread exiting
    Thread 9: Login: Username/password incorrect; str_acc length: 1116  str_acc:


    <!--
    * DVD Store Common Include PHP Page - dscommon.inc
    *
    * Copyright (C) 2005 Dell, Inc. <dave_jaffe@dell.com> and <todd_muirhead@dell.com>
    *
    * Used by DVD Store php pages
    *
    * Last Updated 5/13/05
    *
    *  This program is free software; you can redistribute it and/or modify
    *  it under the terms of the GNU General Public License as published by
    *  the Free Software Foundation; either version 2 of the License, or
    *  (at your option) any later version.
    *
    *  This program is distributed in the hope that it will be useful,
    *  but WITHOUT ANY WARRANTY; without even the implied warranty of
    *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    *  GNU General Public License for more details.
    *
    *  You should have received a copy of the GNU General Public License
    *  along with this program; if not, write to the Free Software
    *  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
    -->


    <HTML>
    <HEAD><TITLE>DVD Store Login Page</TITLE></HEAD>
    <BODY>
    <FONT FACE="Arial" COLOR="#0000FF">
    <H1 ALIGN=CENTER>DVD Store</H1>
    Can't connect to MySQL server on 'DS2-DB' (4)

    Thread 9: Error in Login for User user6002984, thread exiting

    et=    70.0 n_overall=119 opm=101 rt_tot_lastn_max_msec=55469  rt_tot_avg_msec=1830 rt_tot_sampled=-2147483648 rollbacks: n=2 %=  1.7
    threads[5].ThreadState= Stopped
    threads[9].ThreadState= Stopped
    Error  in parsing neworderid: Input string was not in a correct format.;  ind_b= 14  ind_e= 81  str_acc[]= Store Common Include PHP Page -  dscommon.inc
    *
    * Copyright (C) 2005 Dell, Inc.
    Thread 0: Error in Purchase, thread exiting

    et=    80.0 n_overall=119 opm=89 rt_tot_lastn_max_msec=55469  rt_tot_avg_msec=1830 rt_tot_sampled=-2147483648 rollbacks: n=2 %=  1.7
    threads[0].ThreadState= Stopped
    threads[5].ThreadState= Stopped
    threads[9].ThreadState= Stopped
    Thread 3:  error in parsing response from browse request
    Thread 3: Error in Browse by title, thread exiting



  • 2.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 27, 2012 11:40 PM

    SSH into  ds2weba0.  Once there run "ping ds2-db".  What does it return.

    What is in the restore file for your ds2db0 "ConfighDS2DB_0.txt?  This is found in your latest results folder.



  • 3.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 28, 2012 12:07 AM

    Thanks for your reply.

    1) From ds2webA0, I can ping ds2-db without problem.

    2) ConfigDS2DB_0.txt shows this:

    Prepping DS2DB
    Shutting down service MySQL ..done
    Starting service MySQL ..done
    Reloading DS2 DB's database
    DS2DB Reload Completed
    Fri Jul 27 14:57:56 PDT 2012



  • 4.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 30, 2012 04:01 PM

    Could you also check that mysql is actually able to restart after the restore process?  From the output you provided, it seems that it's unable to connect to it. 



  • 5.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 30, 2012 05:03 PM

    Yes, I tried the following two commands on DS2DB0, both ran ok.

    /etc/init.d/mysql stop

    /etc/init.d/mysql start



  • 6.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 30, 2012 05:17 PM

    Also, one thing to note is that after database is restored on DS2DB0, although connectting to mysql within STAX job failed, I could run this command successfully: C:\vclient0\ds2webA>ds2webdriver.exe --target=ds2weba0

    Log is attached.



  • 7.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 30, 2012 05:49 PM

    Well your command isn't exactly what is going to be run by the harness.  The main thing being the size of the db.

    Can you manually try the following:

    ds2webdriver.exe --think_time=0.085 --target=ds2weba0 --db_size=20gb --warmup_time=0 --n_threads=10 --run_time=300



  • 8.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 30, 2012 06:03 PM

    Thanks Joshua.

    Yes, you are right. The command gave me the same error :smileysad: Please see the attachment for log.



  • 9.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 30, 2012 06:10 PM

    Are you sure you followed all of the instructions for setting up the DS2DB?  The section "Prepare the DVD Store 2 Database" should be completed prior to running.



  • 10.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 30, 2012 06:17 PM

    I think so.

    We imporeted the VM from the pre-built template first and then follow steps 4 - 9 of section "Prepare the DVD Store 2 Database" on page 138 to finish the configuration.

    Is there any command I can run from DS2WebA0 to verify the connection to mysql db on DS2DB0? Thanks.



  • 11.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 31, 2012 05:42 PM

    I actually rebuilt DS2DB0 VM yesterday, but still couldn't make ds2webdriver.exe work. DS2WebA0.wrf shows the following error:

    ****************************************************************

    ...

    <HTML>
    <HEAD><TITLE>DVD Store Login Page</TITLE></HEAD>
    <BODY>
    <FONT FACE="Arial" COLOR="#0000FF">
    <H1 ALIGN=CENTER>DVD Store</H1>
    Can't connect to MySQL server on 'DS2-DB' (4)

    Thread 4: Error in Login for User user7363948, thread exiting
    Thread 9: Login: Username/password incorrect; str_acc length: 1116  str_acc:

    ...

    *******************************************************************

    To verifiy the above action manually, I launched this page manually on client0 by opening http://DS2WebA0/ds2/index.html.

    The page opened asked for a username and password. I then gave "web" and "web", it gave the same error saying "username/password incorrect".

    I am just wondering does this page work for you guys if you enter "web" web" there? 

    Running "mysql -h localhost -u web -pweb" doesn't give me any error on DS2DB0.

    Any suggestion is appreciated! Thanks!



  • 12.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 31, 2012 06:21 PM

    Can you post your /etc/hosts files from both DS2WebA0 and DS2DB0?

    Also, let's check the size of your DS2 database.

    1.SSH into your DS2DB.

    2.  Create a new file CheckSize.sql

    3.  In the file add the following:

    SELECT table_schema "*", sum( data_length + index_length ) / 1024 / 1024 "DB (MB)" 
    FROM information_schema.TABLES GROUP BY table_schema ;
    4. In the ssh console run "mysql < CheckSize.sql"
    5. Post output here.



  • 13.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 31, 2012 06:41 PM

    Thanks Joshua.

    Attached are the files.



  • 14.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 31, 2012 07:56 PM

    To test the DB connection you can copy the attached file to /srv/www/htdocs/ds2/ on your ds2web vm. Make sure it is readable by everyone (chmod +r test-connection.php)

    open a browser from the client and open the page:

    http://ds2weba0/ds2/test-connection.php

    If you get "Connected to MySQL" then your DB connection is working.



  • 15.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 31, 2012 09:05 PM

    Thanks for the script.

    I got "Connected to MySQL", so the connection should be ok.

    Anything wrong with the hosts files and db size? What else should I check for? Thanks.



  • 16.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 31, 2012 09:16 PM

    Your database connection is fine.

    It looks like the database may not have fully loaded.  you are going to have to go back and redo the steps under "Prepare the DVD Store 2 Database".



  • 17.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 31, 2012 09:32 PM

    When I prepared the DVD store 2 Database, I think I strictly followed the steps given in the benchmark :smileysad:  and I didn't run into any errors.

    Just curious, in case I re-create the db, what should be the approximately correct size if DB loading has no issue?



  • 18.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Jul 31, 2012 10:13 PM

    According to the errors you are getting all the users are not being  loaded into the DB.  After you run the Install_DVDStore_vmmark.pl script  you will have 2 csv files in ds2/data_files/cust which contain all the  users.  Each file should be ~3GB in size.   There should be 3 csv files for each month of the year in the ds2/data_files/orders directory.  These files should be sized ~280MB, ~230MB, and ~90MB each.   Finally there will be 2 csv files in ds2/data_files/prod directory, 3M and 10M.

    Incidentally the usernames are user1 - user40000000.  the password is 'password'.  You could manually try logging in with one of these.  web/web is only used on the backend.



  • 19.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Jul 31, 2012 10:23 PM

    Thanks for the info. I will double check these files.



  • 20.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 01, 2012 06:15 AM

    I re-created DS2DB0 vm and checked all the .csv files. All the file size are correct. Reloading didn't give me any error either.

    But ds2webdriver.exe still stops working. Funny thing is that when I used the user name "user2737279" mentioned below in page http://ds2webb0/ds2/index.html, there is no issue for the user to login and it even shows this user's previous purchases. What can I do? Should I upload my virtual machine files?

    <HTML>
    <HEAD><TITLE>DVD Store Login Page</TITLE></HEAD>
    <BODY>
    <FONT FACE="Arial" COLOR="#0000FF">
    <H1 ALIGN=CENTER>DVD Store</H1>
    Can't connect to MySQL server on 'DS2-DB' (4)

    Thread 8: Error in Login for User user2737279, thread exiting
    Thread 7: Login: Username/password incorrect; str_acc length: 1116  str_acc:



  • 21.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 01, 2012 04:23 PM

    The only thing I can think of is that your host is bottlenecking on storage, as a result it is not able to process the login requests in a timely manner and you are getting the errors.  Your opm and response times in your wrf file are very bad, you should be getting 2k-4k opm with a response time of 100-200.

    What is your storage system?



  • 22.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 01, 2012 04:53 PM

    IC. We are actually in short of machines and therefore I put client0 on a virtual machine :smileysad: Probably that's why.

    Let me try to find a physical machine and then get back to you on the results. Thanks a lot for the help!



  • 23.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 01, 2012 04:56 PM

    I don't think it is the client, rather the dvdstore DB VM which is having disk bottleneck.



  • 24.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 01, 2012 06:00 PM

    The storage we use is LSI CTS 2600 with 24 disks.

    Also one thing that's worth noting is that I ran the following command manually on client0 by specifying only 1 thread instead of 10 threads and so far it's still running. I wonder whether there is a way to tune this n_threads number in my STAX job to let it going?

    ds2webdriver.exe --think_time=0.085 --target=ds2weba0 --db_size=20gb --warmup_time=0 --n_threads=1 --run_time=300

    Attached is the log from running above command.



  • 25.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 01, 2012 10:48 PM

    I moved DS2DB0 to another storage system but still got the same error.

    Looks like n_threads= 7 is a threadhold my system can handle. In other words, if I run command "ds2webdriver.exe --think_time=0.085 --target=ds2weba0 --db_size=20gb --warmup_time=0 --n_threads=7 --run_time=300", system runs ok. But if I change the n_threads to "8", it starts to fail.

    I would like to know other than the usual suspect, what other factors could play a role in restricting the number of threads it could handle correctly?



  • 26.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 12:04 AM

    Is your client still a virtual machine?  Seems like you are having bad clock drift, the elapsed time should be going up by 10s each, but your numbers are a bit variable.



  • 27.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 12:24 AM

    Yes, client0 is still on a vm at this moment. But I don't see a big time different between these vms because if there is, time sync check will fail when I run STAF. Or did I misunderstand your question?



  • 28.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 12:29 AM

    In your wrf file, the et= line should go up by 10s each line, but there is some variance in the amount that yours are increasing.  Also, it is curious that the first few samples have such high response times.  Could be an issue on the client system, possibly insufficient CPU available?



  • 29.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 05:10 PM

    I increased both CPU and memory on client0, but few threads of ds2webdriver.exe still died :smileysad:



  • 30.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 05:14 PM

    What is the configuration of your client system?  how many cpu what is the memory? Are you using 1Gb network?



  • 31.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 05:24 PM

    I'm still using a virtual machine as the client. CPU is 6GB and memory is 8GB.



  • 32.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 05:27 PM

    What do you mean 6GB cpu?

    Can you verify that you DO NOT have a proxy set in you internet options on the client?

    I would strongly suggest getting just the DS2 driver installed on a physical system so we can narrow down which componant is failing.



  • 33.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 05:39 PM

    1) Sorry I meant to say I use 6 cores for CPU.

    2) I double checked and didn't see proxy server set on client0

    3) I'm not sure what are the steps necessary for installing DS2 driver only on a physical server? Could you please let me know?

    Thanks!



  • 34.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 05:52 PM

    You should be able to just copy the ds2webdriver.exe file to any windows system that has .net 2 framework installed on it.



  • 35.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 06:16 PM

    Thanks, I will try it and get back to you.



  • 36.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 09:08 PM

    I set up a new client0 on a physical machine which has 16GB memory, 4 core CPU and 1 Gb network just for ds2webdrive testing, but still get the same error! Does this prove client0 is not the bottleneck?

    Is it possible for any of you to provide customer support on site?



  • 37.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 10:05 PM

    Can you post the log files from your ds2web system, they are in /var/log/apache2



  • 38.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 10:15 PM

    Attached are the logs. Thanks.



  • 39.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 10:20 PM

    Sorry, forgot to attach the logs.



  • 40.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 11:11 PM

    Is the ds2db and ds2web vm on the same vswitch?  The logs indicate that the ds2web is not able to connect to ds2db.  It seems to happen intermitantly.  I am guessing there is some sort of network issue between these VMs.

    Are there any errors, dropped, or overrun packets showing in ifconfig on the ds2db and ds2web vm (jus run ifconfig on each)?



  • 41.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 02, 2012 11:20 PM

    Yes, all the vmmark VMs are on the same vswitch.  And by running ifconfig on both DS2DB0 and DS2WebA0, I didn't see any errors or dropped or overruns.

    Please do remember that the sql connection error only happens when ds2webdriver.exe is run with more than 7 threads, but doesn't get any error when I run with less than 7 threads against one web server.



  • 42.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Broadcom Employee
    Posted Aug 02, 2012 11:31 PM

    Can you send an email to vmmark-info@vmware.com



  • 43.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 03, 2012 01:01 AM

    Could I know what's that email for? Is it that somebody getting the email will help me to resolve it? Or it's only for reporting the issue? Thanks.



  • 44.  RE: ds2webdriver.exe stops working if run from STAX, but works fine if run separately in a cmd window

    Posted Aug 14, 2012 12:17 AM

    Big thanks to all the help jamesz08 gave. This issue is finally resolved.

    From jamesz08:

    ============

    comment out the nameserver line in resolv.conf on your ds2db vm.  as far  as I can tell Mysql is trying to do a reverse lookup of the webvm, this  causes connection timeouts at higher loads.