Read_from_file Operator. How to read more than 1024 lines?

[Mario.M.Gomes]

Hi Community,

 

I am facing a problem with reading txt files. The "read_from_file operator" only reads 1024 rows and I have files bigger than that. My idea is to check the number of rows in a file and run the loopback operator a few times by adding the result to a process variable. Is there an operator that tells the number of rows in a file?

Any other way to read a file with more than 1024 lines?

 

Thank's in advance. 

[Andrew Thompson]

The read from file operator allows you to specify the the line number to start and stop reading.  You would need to read the additional lines from a new operator and write to a new variable in Process Automation.

 

I don't see a built in way to get the row number out of a file using the out-of-the-box PAM operators.  The "Get File Attributes" operator returns things like size, but I don't see a row count.  You might have to build a java function to do that.

[Mario.M.Gomes]

Hi Andrew,

 

I realized that I would have to build something using some programming language. But before that i wanted to know if anyone in the community had this and the ideas used. I've created a solution that I found easy to implement and maintain. I will show my solution in this same post.

 

Thank you for your help.

[deama10]

Hi Mario, I just run a few tests with this operator and the latest pam version (4.3.02) and I was able to get an datase of 1867 lines (it's a txt file).

 

Did you test just unchecking the "Return file contents in a string array" and split the output by "\n"?

[Mario.M.Gomes]

Hi Deama10,

 

This solution is very good, but I think it would be limited by the size of the file, but this solution should work for simpler cases. In my case it was important to bring information in array form for easier use ... I've created a solution and I'll show it here.

 

Thank you

[Mario.M.Gomes]

Hello guys,

 

My solution to this case was to use an MS-DOS function to count the rows of the file. I used the FIND function to count the rows, since my file has the csv format, so I searched the lines that had "," and used that value to run the operator as many times as needed. The result of the load I stored in a process variable. Here are the operators and code I used.

 

 

MS-DOS function:

 

find /C /I "," file_name.txt

 

Code to set start and end limits:

if (Process[OpName].CurrentLoopIteration == 0) {
Process.Inicio01 = 2;
Process.Fim01 = 1023;
}
else {
Process.Inicio01 = Process[OpName].CurrentLoopIteration * 1024;
Process.Fim01 = Process.Inicio01 + 1023;
}

 

Please, feel free to comment on the solution.

 

Thank you all.

[deama10]

This on post execution would also return an array.

Process.linhas = [];
Process.linhas  = Process[OpName].DatasetVariable.split('\n');

 

I wasn't able to reproduce the cap of 1024 rows on the latest process automation version. 

[Mario.M.Gomes]

Marcel,

 

I'm using CA PAM 4.3 SP1. 

 

 

Did CA change this limit in the newer version? Is there any documentation of this?

 

Thanks