Data Loss Prevention

Get number of tokens generated per document scanned

  • 1.  Get number of tokens generated per document scanned

    Posted Mar 20, 2025 06:04 PM

    Hello,

    We need to increase the Lexer.MaximumNumberOfTokens parameter (Network prevent detection servers for Email/Web).
    But before proceeding with tests, I would like to know if it is possible to obtain the number of tokens generated per scanned document.

    I've found in the detection server logs the following information (detection_server\logs\debug\EdmMatcherX.log):

    Do these values 'single token count', 'two token count', and 'more than three token count' represent the number of tokens extracted per document?
    Thank you for your help.