Configuration file

The name of the configuration file that Data Flow service uses is called Dataflowsconfig.xml and is located in the default installation path. The file contains three sections: Data Sources, Data Flows, and Settings. All tags and attribute names in the file must be in lower case. There is also an XSD file that you can use to validate the configuration file on startup.

<DataSources>

The <DataSources> tag of the Configuration File represents a collection of the different data sources that the solution is configured to interact with. For a configuration to be valid, two datasources are required at the minimum. The <datasourcename> attribute should be unique.

<DataSource>

The <datasource> tag is a child node of the <datasources> tag in the configuration document and represents the configuration information for a single datasource.


Attribute name	Default value	Required	Description
datasourcename		Yes	This attribute is used to uniquely identify the datasource. With this attribute, datasources can be mapped to specific adapters within each data flow
connectionstring		Yes	This is the URL of the respective data sources. For example: https://<ipaddress/hostname>:{Port}/api (BigFix) https://<InstanceID>.Service-Now.com/api/now/table (ServiceNow)
username		System generated	This attribute is managed through the ProvideCredentials command. The data is encrypted prior to being persisted in the configuration file
password		System generated	This attribute is managed through the ProvideCredentials command. The data is encrypted prior to being persisted in the configuration file
verifycert	True	No	This attribute enables or disables SSL certificate validation with this data source
proxy_host	N/A	Yes	This attribute provides the proxy server host along with a port number (format: HTTP:// or HTTPS://proxy_host:proxy_port).
proxy_username	N/A	Optional	This attribute is managed through the configureproxy command. The data is encrypted prior to being persisted in the configuration file.
proxy_password	N/A	Optional	This attribute is managed through the configureproxy command. The data is encrypted prior to being persisted in the configuration file.

Note: If the verify cert is set to True in case of proxy, ensure the proxy machine certificate is added to the client of the machine.

<DataFlows>

The <DataFlows> tag of the configuration file represents a collection of the different data flows that the solution is configured to execute.

<DataFlow>

Each <DataFlow> tag represents an instance of the flow of data from one system to another and consists of a Source Adapter tag and a Target Adapter tag.


Attribute name	Required	Description
displayname	Yes	This attribute is used to describe the individual data flow
datatype	Yes	By default it is set to asset
schedule*	Yes	Find more information about schedule here

Schedule*

Cron Time String Format: The Cron time string format consists of five fields that Cron converts into a time interval. Cron then uses this interval to determine how often to schedule the DataFlow. 5 place values specifies Minute, Hour, Day of the Month, Month, Day of the week respectively.

Table 1.
Character	Descriptor	Acceptable values
1	Minute	0 to 59, or * (no specific value)
2	Hour	0 to 23, or * for any value. All times UTC
3	Day of month	1 to 31, or * (no specific value)
4	Month	1 to 12, or * (no specific value)
5	Day of the week	0 to 7 (0 and 7 both represent Sunday), or * (no specific value)

Example usecases:

Execute data flow after specific minutes past the hour, every hour.
Execute data flow at specific time of UTC every Monday.
Execute data flow every five minutes.
Execute data flow every second hour, on the hour.

Example1: Cron time string of 0 10 15 * * executes a command on the 15th of each month at 10:00 A.M. UTC.

Example2: Cron time string of 10/30 10 * * * executes a command on the 10th minute of 10th hour of each day and for every 30 minutes there after.

Note: The scheduler value should be configured one minute later to the current time if the dataflow needs to be executed immediately once the start of DataFlows service. For example, if the current time is 11:35:30 the scheduler can be configured to 36 11 * * *.

Note: When initially testing the integration, “now” can be used in the scheduler and Dataflow execution can be tested manually. Once the configuration works, then configure the schedule as per convenience. Ex: schedule = “now”.

<FilterQuery>

The <FilterQuery> tag is used to filter the records that are monitored by a particular adapter tag. This tag consists of a native query for the selected adapter type. For BigFixRest, this would be a session relevance, while in ServiceNowRest it would be a SysParam_Query value.

For details on session relevance, see The Relevance Language and on ServiceNow filters and queries, see Operators available for filters and queries.

Note: The <FilterQuery> does not work when it is specified in the following format:

<![CDATA[*********]]>

</filterquery>

The <FilterQuery> should be specified in one line. Ex:

<SourceAdapter>

The <SourceAdapter> tag identifies the source system from which the data is extracted. It must include a Properties collection, with a minimum of two properties being valid. Optionally, it may also consist of a <FilterQuery> that leverages the native querying syntax of the AdapterClass that is leveraged.


Attribute name	Required	Description
displayname	Yes	This attribute is used to describe this adapter configuration
adapterclass	Yes	Options: bigfix or servicenow This attribute determines which adapter is used to extract data from the source data source
datasourcename	Yes	This attribute value must match the name of a data source defined in the data sources collection. It is used to provide connection information to the adapter

<TargetAdapter>

The <TargetAdapter> tag identifies the target system in to which the data is loaded. It must include a Properties collection, with a minimum of two properties being valid. Optionally, it may also consist of a <FilterQuery> that leverages the native querying syntax of the AdapterClass being leveraged.


Attribute name	Required	Description
displayname	Yes	This attribute is used to describe this adapter configuration
adapterclass	Yes	Options: bigfix or servicenow This attribute determines which adapter is used to extract data from the source data source
datasourcename	Yes	This attribute value must match the name of a data source defined in the data sources collection. It is used to provide connection information to the adapter

<device_properties>

The <device_properties> tag represents a collection of properties in a specific adapter. Each property in this collection is mapped by position to the collection in the corresponding target or source adapter.

The <sourcekey> tag represents defines primary key of the adapter. Only one property in each collection must be designated as a Primary Key.

The <targetkey> tag represents defines foreign key of the adapter.

Note: The source and the target column positions should not be changed as defined in the default configuration file.

<Property>

The <Property> tag represents a single column of data that is either extracted from or loaded in to a system. It may include simple transformation logic to facilitate the transformation of the data received.


Attribute name	Required	Description
displayname	Yes	This attribute is used to describe the property being configured
propertyname	Yes	This attribute is used to identify the corresponding column using a notation specific to each adapter
datatype	Yes	The datatype which could be STRING, DATETIME, DATE, INT, DECIMAL, or BOOLEAN

Note: BigFix Columns can be identified by using two patterns: Retrieved Property ID (int) or [SiteName]-[Analysis Name]-[PropertyNumber]

<IdentityProperty>

The <IdentityProperty> tag represents a single column of data that is either extracted from or loaded into a system. It may include simple transformation logic to facilitate the transformation of the data received.


Attribute name	Required	Description
displayname	Yes	This attribute is used to describe the property being configured
propertyname	Yes	This attribute is used to identify the corresponding column using a notation specific to each adapter
datatype	Yes	The datatype which could be STRING, DATETIME, DATE, INT, DECIMAL, or BOOLEAN
weight	Yes	This attribute assigns a weight to the property, which is used for the weighted confidence matching of records. Type: Int.
delimiter	No	This attribute transforms entry into multiple identity properties. To disable transformation, provide empty delimiter: "".

Note: Device Correlation is the mapping of machine records between two systems. The customer is able to configure "identity properties" for an adapter within a datasource, and facilitate the correlation of the records based upon a Weighted Confidence Algorithm.

<TransformationLogic>

The <TransformationLogic> tag can be used with a property tag to manipulate the value returned from a system. These transformations are simple python experiences, and typical use cases would be parsing maintenance windows or classification based upon a certain value.

Syntax for transformation logic: <transformationlogic><![CDATA[str(this_value).lower()]]></transformationlogic></identityproperty>

<Settings>

The <Settings> tag represents a collection of settings for the solution. For a detailed list of settings, see Configuration settings.


Attribute name	Required	Description
key	Yes	This attribute is the name of the setting that is being configured
value		This attribute is the value of the setting that is being configured