Service Virtualization

Expand all | Collapse all

Scripted DataSet to read a file

Jump to Best Answer
  • 1.  Scripted DataSet to read a file

    Posted 02-20-2018 04:09 PM

    I have a need to use a data set and read a csv file.  I can't use the built in CSV data set reader because it assumes the first row of the data set contains column headers.  In my case the first row of the csv contains actual data.  I was considering using a scripted dataset to open read through the file, but I just can't seem to get the results I am looking for.  I seem to be able to open and read the file in a scripted dataset, but it doesn't seem to increment through the file.  It just sits looping on row 1.

     

    Does anyone have any suggestions or examples for doing this?  I have some ideas I can implement with some modest code, but it just seems like there should be an easier way.



  • 2.  Re: Scripted DataSet to read a file

    Posted 02-20-2018 04:52 PM

    Hi Tom,

     

    One hacky way to do it would be to have a separate test case that prepends the header and saves the doc as a separate file to be read by a subsequent test.  Unfortunately, you can't do it in the same test case because dataset files need to be present when the test is staged.

     

    A more elegant solution, if you're comfortable with your Java skills, is to use the SDK to create a custom dataset:

     

    https://docops.ca.com/devtest-solutions/10-2/en/using/using-the-sdk/custom-data-sets

     

    --Mike



  • 3.  Re: Scripted DataSet to read a file
    Best Answer

    Posted 02-20-2018 04:59 PM

    Hi Tom,

     

    Actually, I suspect I know why your scripted dataset is always returning the first row.

     

    Your script will get invoked every time DevTest needs another row of data.  On the first invocation, open the input file and, before returning the data, store the stream on testExec.  On subsequent calls, don't reopen the file.  You just need to pull the next record from the already open stream.  When you're out of data, close the stream.

     

    --Mike



  • 4.  Re: Scripted DataSet to read a file

    Posted 02-21-2018 09:15 AM

    @Mike. Thanks.  I had tried taking the route of opening the buffered reader only the first time into the scripted dataset, but when I put the open into a conditional test, the open seems to work, but the subsequent readLine() fails!

     

    Is there something odd with the scripted datasets and file reads?

     

    Maybe I'm missing something, but I may have to open a case if someone can't illustrate a working example.



  • 5.  Re: Scripted DataSet to read a file

    Posted 02-21-2018 09:19 AM

    Hi Tom,

     

    Please post your script.

     

    Thanks.

     

    --Mike



  • 6.  Re: Scripted DataSet to read a file

    Posted 02-21-2018 09:29 AM

    Here it is in the simplest form I can demonstrate:

    _logger.info("trackingReader: " + trackingReader);

    if (trackingReader == void)
        {
        //Read the records in the csv file and output to the tracking file
        //For each ParcelNo output one static record and timestamp, one movement with timestamp change, and one del/nondel record
        FileReader fr = new FileReader(HERMES_WATCH_FOLDER + "/" + UploadFileName);
        BufferedReader trackingReader = new BufferedReader(fr);
        _logger.info("trackingReader: " + trackingReader);
        }

     

    String line = trackingReader.readLine();
    _logger.info("line: " + line);
    if (line == null)
    {
        _logger.info("line is null: " + line);
        trackingReader.close();
        fr.close();
        break;
    }

     

    In this form it fails during String line = trackingReader.readLine();

     

    If I simply remove the if surrounding the creation of the FileReader/BufferedReader it does correctly return a line.



  • 7.  Re: Scripted DataSet to read a file

    Posted 02-21-2018 10:01 AM

    Hi Tom,

     

    I was able to independently recreate the behavior you're describing.  Diving deeper into it, I wonder how it could possibly *ever* work.  I'll open a ticket, and I'll add your email address.

     

    Please email to mike.gavaghan@ca.com:

     

    1) The version of DevTest you're using

    2) You're site ID (if you know it - if not, that's okay)

    3) The company you work for (the license owner).

     

    Thanks.

     

    --Mike



  • 8.  Re: Scripted DataSet to read a file

    Posted 02-23-2018 02:13 PM

    After consulting with our support tem, it seems Scripted DataSet mostly works.  The reader needs to be stored in the Shared Model Map (I was mistakenly putting in testExec).  If you do that, the dataset will correctly step through each line in the file.

     

    However, here's the catch: it only resets itself after completely stepping through the entire file.  If you only get part way through the datasets (say, you've read lines 1, 2, and 3) and then start the test case over, it picks up where it left off (line 4).  The solution to this is to create a step at the beginning of your test case to ensure the reader has been removed from the Shared Model Map.

     

    Here's the script I used:

     

    import java.io.*;
    import java.util.*;
    import com.itko.lisa.vse.*;

     

    Map data = null;

     

    // check to see if we've already opened a reader
    BufferedReader reader = (BufferedReader) SharedModelMap.getObject("MyNamespace", "ds_reader");

     

    // if not found, open it
    if (reader == null)
    {
        FileInputStream fis = new FileInputStream(LISA_PROJ_ROOT+ "/Data/MyData.csv");
        reader = new BufferedReader(new InputStreamReader(fis));
        SharedModelMap.putObject("MyNamespace", "ds_reader", reader);
    }

    // see if there's more data
    String row = reader.readLine();

    if (row == null)
    {
        // nope.  close stream and remove from state
        reader.close();
        SharedModelMap.remove("MyNamespace", "ds_reader");
    }
    else
    {
        // put this row of data into a map
        String[] cols = row.split(",");

        data = new HashMap();
        data.put( "ds_color", cols[0] );
        data.put( "ds_letter", cols[1] );
        data.put( "ds_number", cols[2] );
    }

    return data;



  • 9.  Re: Scripted DataSet to read a file

    Posted 04-06-2018 02:03 AM

    Hi Mike_Gavaghan,

     

    Can you please assist with the scripted dataset for .txt file, where the first line contains the actual data followed by multiple lines down the line.

     

    And how to fetch each row and row length after trimming spaces at the end of the data in each line or record.

     

    Any kind of suggestion or approach is highly appreciated.

     

    Thanks in advance!!!

     

    Regards,

    Bibhu