View Only

TECH TIP: Unicode File Scanner Utility

  • 1.  TECH TIP: Unicode File Scanner Utility

    Posted Mar 02, 2016 12:44 PM
      |   view attached

    For users of Visual Studio that uses the Harvest SCM plugin, we have found that many users

    inadvertently create project files in Unicode UTF16 format which is incompatible for checking in

    these files into Harvest SCM as text files. If the check in attempt is made then no error message

    popup is displayed, however in the Harvest SCM output pane at the bottom of the IDE, you may

    find an entry that informs that the file in question contains a non-ascii text character.


    Harvest currently only supports up to Unicode UTF8 and treats them the same as ascii text files.


    There are a few workarounds to mitigate this problem and one is to set all files as binary files in

    the administrator GUI. The downside to this is the your company would have to procure a third

    party program such as BeyondCompare etc. that can perform comparisons and merges of binary

    files. Another workaround is to open the file in windows notepad and save the file in "ANSI" format

    then reattempt to add or check in the file to Harvest SCM repository.


    If setting all files as binary in the administrator GUI is not a viable option then the best course of

    action would be to first identify any files within your project for Unicode or non-ascii signatures and

    use the notepad correction prior to check in attempts.


    For this effort, I have provided a windows utility called "FileScanner.exe" that you can use to scan

    your project directory recursively and it will provide a list of files that has the extension you wish to

    scan and it discovers a Unicode signature. While the list is populating in the utility, you can double

    click or r-click on a file list item to view the file in HexViewer (provided by the FileScanner install) and

    it will go to the offending Unicode signature character.


    Here is a screenshot of the FileScanner utility in action, I have set the directory to scan and also

    entered the file extensions to scan only and to check for the Unicode signatures by checking the

    proper checkboxes:



    As you can see, you can select a file item then r-click on the item to open a popup menu and

    select the HexViewer to see where the FF FE code was found:


    Attached is the utility zip file that contain the self extracting install executable. If you wish to uninstall

    the utility, you can uninstall it using the standard windows uninstall applet. The install requires the

    VS2008 Microsoft x86 redistributals, so it will also install these components as part of the complete

    installation. You must run the install as the system administrator privileges.