How to use dynamically generated file names

[Legacy User]

In my script I query a DB for tests/data and save a temporary .csv file locally and use a "Read rows from a Delimited Data File" step to pass the data to other script steps. In the File Location field of "Read rows from Delimited Data File" I want to use a {{dynamically_generated_file_name}}  that includes a timestamp so other tests don't collide (these tests are being run on the Lisa Server as CVS monitor suites). My script complains that the file doesn't exist and fails to find the file after it is generated, resulting in a Java File IO "not found" exception. I see the file generated prior to when it's needed to be loaded.

 

The below link indicates what I'm trying should be possible. Any guidance appreciated. Thanks.

 

Dynamic excel sheet in "Read rows from Excel"

[Legacy User]

Hi Jason, what you are attempting to accomplish sounds reasonable.  Is it possible for you to share a picture or code snippet of the step that saves the CSV file and the step that reads the rows?  

[Legacy User]

Hi Jason, I tried a scenario similar to yours and it seems to work.  Following is an example using DevTest 9.1.  My sample model is as follows:

- The Set a FileName step does nothing more than store a file name into a filter called "fl_fileName".

 

- Create Sample CSV Data does multiple things. 

The Parse Text as Response step places a comma delimited file (header1 & header2, colVal1 & colVal2) into a property called "fl_csvFileData". 

Then a Timestamp filter creates an "fl_time" property with a date/time. 

Followed by a Save Property Value to File filter using

   - the fl_csvFileData as the Filter In value and

   - the Location of: {{LISA_RELATIVE_PROJ_ROOT}}/Data/CSV_DATA/{{fl_fileName}}_{{fl_time}}.csv

 

- The Read CSV file, opens the dataset using a File Location of:  {{LISA_RELATIVE_PROJ_ROOT}}/Data/CSV_DATA/{{fl_fileName}}_{{fl_time}}.csv

 

When I run this model in ITR mode, I can see that DevTest creates the data, saves the file, and the dataset consumes the first row.

I attached the sample test case in case it helps.  I would start by checking the reference to your file location or file name.  Perhaps an upper/lower case character or location may be creating an issue for you.

Attachment(s)

csv_ExampleTC.zip   1 KB 1 version

[Legacy User]

Thanks for the detailed reply and sample! I'm looking through your sample and seeing how I can apply that approach to my script.

 

In response to your questions, this is how I'm doing it currently:

1) read DB data, write results to local csv using script step. snipped for setting file path/name (timestamp is set earlier and exists)

     String TestFileName = "test_{{timestamp}}.csv";

     String TCFileCSV = "{{LISA_RELATIVE_PROJ_ROOT}}/Data/OutputFiles/" + TestFileName;

     testExec.setStateValue("TCFileCSV", TCFileCSV);

Java code in this script create the file, I see the property created with the correct path/file and I'm able to copy path and open the file in ITR as I step through.

2) In the 2nd step, I try to read the file I created from the property I set in the first step

 

I see that you're parsing the property in your example. Is that what I'm missing, something that loads the property before a step tries to use it?

[Legacy User]

Hum.  Can you try the following to see if it makes any difference...

 

String TestFileName = "test_" + testExec.parseInstate(testExec.getStateValue("timestamp") + ".csv";

String TCFileCSV = testExec.parseInState(testExec.getStateValue("LISA_RELATIVE_PROJ_ROOT") + "/Data/OutputFiles/" + TestFileName;

 

testExec.setStateValue("TCFileCSV", TCFileCSV);

[Legacy User]

Sorry, syntax error - missed last ")" in the logic above.

[Legacy User]

Or, to simplify - there is no need to parse in state on these values. .  I was thinking of parseInState on the "TCFileCSV" value in the last statement.

 

testExec.getStatevalue("timestamp")   and testExec.getStateValue("LISA_RELATIVE_PROJ_ROOT")  

[Legacy User]

So, trouble isn't when file is written. Java code works fine and is able to use the {{properties}} and properly writes the file in the correct location with correct timestamp. The CSV reader doesn't like that the file doesn't exist already (complains at run time) and then fails to loads the file that exists. That's what's so confusing, the property is set and path is correct.

 

This is with DevTest 8.2.0.244, which I believe contains the Dynamic File capability.

[Legacy User]

Not sure about the behavior in 8.2.x...  I think the issue may be with the direction of the slashes in the path...  As a test, I dumped using the relative project root and created a Config property having a key/value pair of:

FILE_PATH     C:\\_Data\\LISA_Projects\\WIP

 

Then, I used this code in a script to build the file location before saving and reading the CSV file.

String myFile = "{{fl_fileName}}_{{fl_time}}.csv";

testExec.setStateValue("fl_path", "{{FILE_PATH}}\\Data\\CSV_DATA\\" + myFile);

 

Concatenating the path with a Config property was unnecessary.  I was playing a bit with how DevTest deals with properties inside strings to make sure conversion was not an issue. 

I believe switching the "/" to a "\\" as a directory separator was the import part.

Attached is a version that uses the fl_path property to save the CSV file and read it in.  I am hopeful this construct will work for you in 8.2.

Attachment(s)

csvExample.zip   2 KB 1 version

[Legacy User]

Hi Joel, thanks for this new info. I really appreciate all the help and effort you've given to try and solve this. I tried with some different path formats, per your suggestion, but no luck.

The path string that's set for the file and shows in the logs is below, and if i copy this into the read csv step and the file exists before executing, it works fine.

 

 

I tried formatting the slashes as \\, but the {{LISA_RELATIVE_PROJ_ROOT}} part of the path is logged with / slash, so the formatting end up being \ and / and none of the slash formatting seemed to make a difference as i was prompted that the file didn't exist, but may be able to resolve at runtime, then same failure IO exception/unable to find file.

 

seems this could be:

1) 8.2 doesn't have dynamic file functionality, or, but maybe not the way i'm implementing

2) i need to parse the {{property}} before the read csv step - i will try this next

[Legacy User]

Thanks, I tried with ) and method capitalization fixes and still get this at run time (pops up during ITR), then below error in logs (as mentioned earlier, i can copy that .csv path and load the file after it is created/before the Read Local CSV step)

 

 

Message:     Error initializing parser

----------------------------------------------------------------------------

| Trapped Exception: Error initializing CSV reader

| Trapped Message:  

 

 

============================================================================

| Exception:

============================================================================

| Message:     Error initializing CSV reader

----------------------------------------------------------------------------

| Trapped Exception: Unable to load file: name=C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv path:[C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] absolutePath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] canonicalPath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv]

| Trapped Message:   java.io.FileNotFoundException: Unable to load file: name=C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv path:[C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] absolutePath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] canonicalPath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv]

----------------------------------------------------------------------------

STACK TRACE

java.io.FileNotFoundException: Unable to load file: name=C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv path:[C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] absolutePath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv] canonicalPath:[C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv]

  at com.itko.util.StreamHelp.fileOpenerWithInfo(StreamHelp.java:775)

  at com.itko.util.StreamHelp.fileOpener(StreamHelp.java:670)

  at com.itko.util.StreamHelp.toReader(StreamHelp.java:912)

  at com.itko.lisa.test.DataFile.loadFromExternal(DataFile.java:193)

  at com.itko.lisa.test.DataFile.initParser(DataFile.java:654)

  at com.itko.lisa.test.DataFile.getNextRecord(DataFile.java:222)

  at com.itko.lisa.test.DataFile.getRecord(DataFile.java:216)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at com.itko.lisa.net.RemoteMethodExec.execute(RemoteMethodExec.java:57)

  at com.itko.lisa.net.ClientBase.invokeColocated(ClientBase.java:284)

  at com.itko.lisa.net.ClientBase.invokeMethod(ClientBase.java:268)

  at com.itko.lisa.net.DataSetClient.getRecord(DataSetClient.java:54)

  at com.itko.lisa.test.DataSetManager.getDataSetResponse(DataSetManager.java:523)

  at com.itko.lisa.test.DataSetManager.processDataSet(DataSetManager.java:546)

  at com.itko.lisa.test.DataSetManager.processUnsynchedDataSets(DataSetManager.java:404)

  at com.itko.lisa.test.DataSetManager.processDataSetList(DataSetManager.java:167)

  at com.itko.lisa.test.TestNode.modifyVariables(TestNode.java:1535)

  at com.itko.lisa.test.TestNode.executeNode(TestNode.java:939)

  at com.itko.lisa.test.TestCase.execute(TestCase.java:1263)

  at com.itko.lisa.test.TestCase.execute(TestCase.java:1178)

  at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1163)

  at com.itko.lisa.utils.ExecSubProcessNode.doFullExec(ExecSubProcessNode.java:353)

  at com.itko.lisa.utils.ExecSubProcessNode.execute(ExecSubProcessNode.java:267)

  at com.itko.lisa.test.TestNode.executeNode(TestNode.java:981)

  at com.itko.lisa.test.TestCase.execute(TestCase.java:1263)

  at com.itko.lisa.test.TestCase.execute(TestCase.java:1178)

  at com.itko.lisa.test.TestCase.executeNextNode(TestCase.java:1163)

  at com.itko.lisa.editor.WalkThruPanel.prepAndExecNode(WalkThruPanel.java:1048)

  at com.itko.lisa.editor.WalkThruPanel.access$900(WalkThruPanel.java:68)

  at com.itko.lisa.editor.WalkThruPanel$10.doCallback(WalkThruPanel.java:961)

  at com.itko.util.swing.panels.ProcessingDialog$2.run(ProcessingDialog.java:194)

  at java.lang.Thread.run(Unknown Source)

Caused by: java.io.FileNotFoundException: C:\Lisa8\Lisa8\C:/Users/jjobe2/UDT_10_DEVS_NEW/ESGDevTest/Data/OutputFiles/test_06-23_115924.csv (The system cannot find the file specified)

  at java.io.FileInputStream.open(Native Method)

  at java.io.FileInputStream.<init>(Unknown Source)

  at java.io.FileInputStream.<init>(Unknown Source)

  at com.itko.util.StreamHelp.fileOpenerWithInfo(StreamHelp.java:761)

  ... 34 more

============================================================================