Log File Connector with delimited data

The Log File Connector sends a delimited message to a log file for each event bus hit. The message is formatted with a delimiter as specified by the configuration.

  • If the text in the message contains the delimiter, then the character is URL encoded in the string. For example, the comma character (,) would be encoded as %2C.
  • The line length limit is 10,000 characters, including the delimiters.
    Note: This output connector should be implemented only on a dedicated file server with a Discover pipeline. This connector allows several input sources to write to the same log file at one time and frees up disk I/O from the main Discover system.

Below the FILE_Directory, a new directory is created for each day of data with a directory name in YYYYMMDD format. The log file names have the following format:


TL_EB_0_YYYYMMDD_mmmm.log

Where:

  • YYYY - year
  • MM - month
  • DD - day
  • mmmm - Number of minutes past midnight for current day: 0000 - 1439

These files are structured in the following manner:


OutputType=FILE
File Delim: any character or TAB (note: not CSV compliant)
FILE_Delim=|
FILE_Directory=c:\Discover\logfiles
Row Time in minutes - log will be rolled every N minutes
FILE_RollTime=5

Log File schema

The Log connector uses the default schema.