> For the complete documentation index, see [llms.txt](https://docs.dynalogic.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/jobs.md).

# Jobs

Jobs are specific tasks that are executed according to a scheduled time.\
Raptor has several types of jobs:

## Job configuration

Jobs can be found in the Raptor configuration panel in the tab 'Jobs'.

<figure><img src="https://1567270722-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FbMdBZV3cNP56k0aYYCI6%2Fuploads%2Fgit-blob-b6c803b23282d8a07e23348cb9c3aba2c10282c1%2Fimage%20(183).png?alt=media" alt=""><figcaption></figcaption></figure>

Every job has a status indicated by a color. The colors can be:

* Gray: job has not run yet
* Purple: a job execution is running
* Green: last job execution was finished successfully
* Orange: last job execution ran into an error and will retry later
* Red: last job execution failed after multiple retries

A job also has a type. Available job types in Raptor are:

<table data-full-width="false"><thead><tr><th width="261">Job Type</th><th width="441">Function</th></tr></thead><tbody><tr><td>D365FO Taxonomy</td><td>Retrieves an export from Dynamics 365 F&#x26;O and generates tags based on it's contents.</td></tr><tr><td>D365FO Users</td><td>Retrieves an export from Dynamics 365 F&#x26;O and creates Raptor users based on it's contents.</td></tr><tr><td>D365FO Kofax Masterdata</td><td>Retrieves an export from Dynamics 365 F&#x26;O and pushes it as masterdata to Tungsten AP Essentials (formerly Kofax).</td></tr><tr><td>D365CE Taxonomy</td><td>Retrieves an export from Dynamics 365 CE and generates tags based on it's contents.</td></tr><tr><td>D365BC Taxonomy</td><td>Retrieves an export from Dynamics 365 BC and generates tags based on it's contents.</td></tr><tr><td>Azure AD - Users</td><td>Retrieves users from Azure Entra ID (formerly Active Directory) and creates them as Raptor user.</td></tr><tr><td>SQL Taxonomy</td><td>Retrieves an export from a SQL database and generated tags based on it's contents.</td></tr><tr><td>Dataverse Taxonomy</td><td>Retrieves an export from Dataverse and generated tags based on it's contents.</td></tr><tr><td>Dataverse Kofax Masterdata</td><td>Retrieves an export from Dataverse and pushes it as masterdata to Tungsten AP Essentials (formerly Kofax).</td></tr><tr><td>SQL Kofax Masterdata</td><td>Retrieves an export from a SQL database and pushes it as masterdata to Tungsten AP Essentials (formerly Kofax).</td></tr></tbody></table>

Next to a type and status, every job has:

* A unique ID
* A cron schedule: this is the schedule for the job execution in the cron-format. Cron schedules can be easily created using the website crontab: <https://crontab.guru/>

{% hint style="danger" %}
Cron schedules should be set up in a reasonable format. Scheduling jobs in a hyper frequent manner (e.g. every minute 24/7) will result in the job being deactivated.\
\
Jobs that proactively generate tags are usually not critical for the working of a production environment, as tags can always created ad-hoc. For these type of jobs we advise a schedule limited of max. 3 times a day, preferably outside business hours or in the weekend.
{% endhint %}

* A service connection (Source): the connection used to retrieve data. See [#service-connections](#service-connections "mention")
* A service connection (Destination): the connection used to push data. See [#service-connections](#service-connections "mention")
* A service account (On behalf of): this is the Raptor user that will be used to make any API calls to Raptor during the job, if applicable.
* Delta packager: the packager service that will be used to retrieve data for the job.

## Service connections

Service connections are used by jobs to connect to external services, such as Dynamics 365, Tungsten AP Essentials or a SQL database. Service connections allow you to store credentials in a safe, central place. Updating a service connection will have an effect on all jobs making use of that connection.

Service connections can be found in the 'Jobs' configuration, tab 'Service Connections'.

### Creating a service connection

To create a service connection, click the '+' icon.\
Provide a meaningful name for your connection and select the service type that you will be connecting to. Based on the selected service type, you can choose the purpose of this connection under 'Data'.<br>

<figure><img src="https://1567270722-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FbMdBZV3cNP56k0aYYCI6%2Fuploads%2Fgit-blob-460b27aeed4976dd229a38177d1d477f4391b690%2Fimage%20(185).png?alt=media" alt=""><figcaption></figcaption></figure>

Click 'Save' to create the service connection and fill in the required fields.

### Dynamics 365 service connections

Connections to Dynamics 365 are made with the OAuth2.0 protocol using an [App Registration](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) in Azure. Raptor uses the ID of the client application as well as the secret value and Directory ID.\
Depending on which Dynamics environment you are connecting to, the app registration may need different API permissions.

#### Dynamics 365 F\&O <a href="#dynamics-365-f-and-o" id="dynamics-365-f-and-o"></a>

For F\&O, the app registration needs following API permissions:

* Dynamics ERP
  * AX.FullAccess (Delegated)
  * CustomService.FullAccess (Delegated)
  * Odata.FullAccess (Delegated)
* Microsoft Graph
  * User.Read (Delegated)

{% hint style="info" %}
Make sure that the client app ID is registered in your F\&O environment under System administration > Setup > Microsoft Entra ID applications.
{% endhint %}

#### Dynamics 365 CE

For CE, the app registration needs following API permissions:

* Dynamics CRM
  * user\_impersonation (Delegated)
* Microsoft Graph
  * User.Read (Delegated)

{% hint style="info" %}
Make sure that the client app is registered as an [application user](https://learn.microsoft.com/en-us/power-platform/admin/manage-application-users) with sufficient permissions in the Power Platform portal.
{% endhint %}

#### Dynamics 365 BC

For BC, the app registration needs following API permissions:<br>

* Dynamics 365 Business Central
  * API.ReadWrite.All (Application)
* Microsoft Graph
  * User.Read (Delegated)

{% hint style="info" %}
Make sure that the client app is registered in the Microsoft Entra Application form in Business Central with sufficient permissions.
{% endhint %}

### Tungsten service connection

The Tungsten service connection allows you to connect Raptor to Tungsten AP Essentials to push masterdata (Master Data type) or export processed invoices (Document Export type):<br>

<figure><img src="https://1567270722-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FbMdBZV3cNP56k0aYYCI6%2Fuploads%2Fgit-blob-9609ba622188e907e48e5d6f68716e74cc4bd8e6%2Fimage%20(718).png?alt=media" alt=""><figcaption></figcaption></figure>

Creating a Tungsten service connection requires a service account in Tungsten with sufficient permissions on the masterdata and/or invoices. You will need:

* The user name of the service account
* The password of the service account
* The Tungsten API key (filled in by default)

## D365FO Taxonomy <a href="#d365fo-taxonomy" id="d365fo-taxonomy"></a>

The D365FO Taxonomy job retrieves masterdata from Dynamics 365 F\&O using an export project in it's [Data Management functionality](https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/data-entities-data-packages). Raptor will retrieve the export created by F\&O and generate tags from it using the [Entity Configuration](https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/entity-configuration.md).\
Taxonomy jobs are useful to proactively create tags that might be useful for Raptor users.

{% hint style="info" %}
In most cases, Raptor integrations are capable of making tags ad-hoc, whenever they are needed. Taxonomy jobs are therefore not essential for a working production environment. Their primary purpose is to make tags available for manual search in Raptor.
{% endhint %}

D365FO Taxonomy jobs use a Source service connection to a Dynamics 365 F\&O environment (see [#service-connections](#service-connections "mention")) and use the Default delta packager.

### Delta Fetch settings <a href="#d365fo-taxonomy-delta-fetch-settings" id="d365fo-taxonomy-delta-fetch-settings"></a>

To create a D365F\&O Taxonomy job, an Export project must be available in your Dynamics 365 F\&O environment. The export project can contains the list of entities that you wish to create tags for in Raptor. Be sure to choose 'XML-element' as source data format for each entity.

<figure><img src="https://1567270722-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FbMdBZV3cNP56k0aYYCI6%2Fuploads%2Fgit-blob-31e614952fba19cf5a471042968a7679fa6c62d1%2Fimage%20(188).png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
Set the default refresh type for each entity to 'Incremental push only' to prevent unnecessary load on your F\&O environment and Raptor tenant. Incremental push requires activating [change tracking](https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/entity-change-track) on the entity.
{% endhint %}

{% hint style="info" %}
Note that tags will not be created for an entity if there is no [Entity Configuration](https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/entity-configuration.md) set up for it.
{% endhint %}

**Uri**: the URL of your F\&O environment (e.g. <https://finopsaos.cloudax.dynamics.com>).

**Definition Group Id**: Group name of the export project

**Package name**: the name given to the created exports

**Legal Entity**: the legal entity in F\&O which the export should be made for

**Retention Days**: amount of days that the export should remain available

## SQL Taxonomy Job <a href="#sql-taxonomy-job" id="sql-taxonomy-job"></a>

The SQL Taxonomy job retrieves masterdata from a SQL database. Raptor will use the exported data to generate tags from it using the [Entity Configuration](https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/entity-configuration.md).\
Taxonomy jobs are useful to proactively create tags that might be useful for Raptor users.

{% hint style="info" %}
In most cases, Raptor integrations are capable of making tags ad-hoc, whenever they are needed. Taxonomy jobs are therefore not essential for a working production environment. Their primary purpose is to make tags available for manual search in Raptor.
{% endhint %}

SQL taxonomy Taxonomy jobs use a Source service connection to a SQL database (see [#service-connections](#service-connections "mention")) and can use the Default delta packager (for publicly accessible databases) or a custom delta packager for on-premise databases. Contact a 9altitudes consultant if a custom delta packager needs to be set up.

To configure a SQL sync job, the following components must be defined:

1. **Target Database Connection**\
   Specify the connection string for the source SQL database.
2. **SQL Queries**\
   Provide the SQL queries to retrieve the required data. You can:
   * Use views or direct `SELECT` statements
   * Include SQL parameters (e.g., timestamp filtering)
   * Rename columns using `AS`
   * Apply any SQL logic, as long as the final output is a single result set (table)
3. **Delta Sync (Optional)**\
   To enable delta (incremental) synchronization:
   * Define a storage connection string to track job progress
   * Specify a schema where progress metadata will be stored
   * Use a timestamp parameter in your queries to fetch only changed/new records

All delta-related settings are defined in the **`Delta Fetch`** section of the configuration.

**Example Configuration**

Below is a sample JSON configuration for a SQL sync job:

```json
{
  "source": "SQL",
  "raptorTenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "TaxonomyConfig": {
    "UseEntitySource": "EXAMPLE"
  },
  "packageInfo": {
    "package": "TEST-PACKAGE",
    "source": {
      "connectionString": "Server=SRV\\DB;Database=XX;User Id=USR;Password=PASS;"
    },
    "storage": {
      "connectionString": "inherit",
      "schema": "raptor-integration"
    },
    "entities": [
      {
        "name": "INVOICES",
        "select": "
        DECLARE @date AS NVARCHAR(50);
        IF (@timestamp IS NULL) 
          SET @date = ''; 
        ELSE 
          SET @date = CONVERT(varchar, @timestamp, 23); 
        
        SELECT 
          [inv].[id] AS [ID], 
          [inv].[net amount] AS [AMOUNT],
          [inv].[ref] AS [CUSTOMER]
        FROM [dbo].[invoice-table] AS [inv]
        WHERE [inv].[change] >= @date OR [inv].createdAt >= @date",
        "timestampParameter": "timestamp"
      },
      {
        "name": "ORDERS",
        "select": "SELECT * FROM [dbo].[ORDERS]"
      },
      {
        "name": "USERS",
        "select": "SELECT * FROM [dbo].[user table]",
        "timestampParameter": "none"
      }
    ]
  }
}
```

**Explanation of the Configuration**

* **Entities Exported:**\
  The configuration defines three entities: `INVOICES`, `ORDERS`, and `USERS`. The job output will be a ZIP file containing three XML files:
  * `INVOICES.xml`
  * `ORDERS.xml`
  * `USERS.xml`
* **Delta Sync Behavior:**
  * The `INVOICES` query uses the `@timestamp` parameter, so only changed or newly created records are fetched.
  * The `ORDERS` and `USERS` queries do not filter on timestamp, so they will always return the full table contents.
* **Column Mapping and Custom Logic:**\
  The `INVOICES` query demonstrates key features:
  * Selecting specific columns
  * Renaming columns using `AS`
  * Incorporating conditional logic while ensuring a single `SELECT` statement is returned
* **Storage Configuration:**
  * `connectionString: "inherit"` allows the storage connection to be inherited when running in a hosted environment (e.g., 9Altitudes server park).
  * For on-premise execution, you may need to explicitly define this connection string.
  * The `schema` is used to store delta progress and must exist in the database. For cloud deployments, it will be created by 9Altitudes. For on-premise use, ensure the schema exists and the configured login has full access (read/write/create).

***

**Important Notes**

* Column names must be valid XML tag names.
* Column names must match those defined in the [Entity Configuration](https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/entity-configuration.md).
* You can override the default entity configuration source by setting `TaxonomyConfig.UseEntitySource`. In the example above, it uses the configuration from source `"EXAMPLE"`.
* To perform a full sync instead of a delta sync, you can change the `package` name. This resets the stored timestamp.

## D365FO Tungsten Masterdata <a href="#d365fo-tungsten-masterdata" id="d365fo-tungsten-masterdata"></a>

The 'D365FO Tungsten Masterdata' job in Raptor allows you to take an export from D365 Finance & Operations using the Data Management feature, and upload it as masterdata to Tungsten AP Essentials.

### Connecting D365 and Tungsten <a href="#d365fo-tungsten-masterdata-connecting-d365-and-tungsten" id="d365fo-tungsten-masterdata-connecting-d365-and-tungsten"></a>

To set up a Tungsten Masterdata job, you need to create a service connection to both systems:

Dynamics 365 F\&O: [#dynamics-365-f-and-o](#dynamics-365-f-and-o "mention")

Tungsten AP Essentials (Master data): [#tungsten-service-connection](#tungsten-service-connection "mention")

{% hint style="info" %}
Jobs created before April 2025 might not use Service connections yet and have their credentials stored inline in the 'Authentication' or 'Entity configuration' section of the job. It is advised to switch to using Service credentials for more secure and easier maintainable credential storage.
{% endhint %}

### Setting up the Data Management project <a href="#d365fo-tungsten-masterdata-setting-up-the-data-management-project" id="d365fo-tungsten-masterdata-setting-up-the-data-management-project"></a>

In D365 F\&O, an export project should be created which will export the masterdata needed in Tungsten. During customer onboarding, this project is usually created beforehand with the name 'RAPTOR\_KOFAX'. By default it contains the entities Vendors, Bank accounts and Purchase orders.\
All entities should be set to 'Incremental push only' to avoid continuous large exports.

### Delta Fetch Settings <a href="#d365fo-tungsten-masterdata-delta-fetch-settings" id="d365fo-tungsten-masterdata-delta-fetch-settings"></a>

Once both systems are connected through service connection, a new Raptor job can be created.\
Here the Service connection (Source) is the D365 service connection and the Service connection (Destination) is the Tungsten service connection.

In the 'Delta Fetch Settings', the Data Management export project in D365 should be specified.\
Enter the required parameters:

* Uri: this should be the same as in the F\&O service connection
* Definition Group Id: the name of the export project in Data Management (typically called 'RAPTOR-KOFAX)
* Package Name: name of the package as your internal reference, free to choose
* Legal Entity: the legal entity for which the data should be exported (eg. DAT)
* Retention days: determines for how many days the data export should be stored in Raptor - for recovery

### Entity configuration <a href="#d365fo-tungsten-masterdata-entity-configuration" id="d365fo-tungsten-masterdata-entity-configuration"></a>

In the 'Entity configuration' panel, the mapping from D365 to Tungsten is specified in JSON format.

The 'organizationId' parameter should contain the ID of the buyer in Tungsten which you wish to sync the masterdata to.

The 'mappings' parameter contains the mapping from the F\&O data entity to the Tungsten tables.\
The configuration below contains the default mappings.\
For a full overview of the available fields in Tungsten, visit <https://services.readsoftonline.com/documentation/rest#masterdata>

The 'incrementalPush' parameter determines if the data export is added in add-or-update mode (true) or replace mode (false). Replace mode will delete all existing masterdata for that table and replace it with the exported data from F\&O.

{% hint style="info" %}
If you wish to push a boolean field to Tungsten (such as the 'Blocked' field), make sure that the field in D365 is exported as "1" or "0".
{% endhint %}

```
{
     "kofaxConnection": {
          "organizationId": "15203a050cae4e232323c149224b48103"
     },
     "mappings": [
          {
               "EntityNameExternal": "VENDVENDORV2ENTITY",
               "masterdataObjectName": "suppliers",
               "fieldMapping": {
                    "VENDORACCOUNTNUMBER": "SupplierNumber",
                    "VENDORORGANIZATIONNAME": "Name",
                    "TAXEXEMPTNUMBER": "TaxRegistrationNumber",
                    "ADDRESSCITY": "City",
                    "ADDRESSSTREET": "Street",
                    "ADDRESSZIPCODE": "PostalCode",
                    "ORGANIZATIONNUMBER": "OrganizationNunmber",
                    "PRIMARYPHONENUMBER": "TelephoneNumber"
               },
               "incrementalPush": true
          },
          {
               "EntityNameExternal": "PURCHPURCHASEORDERHEADERV2ENTITY",
               "masterdataObjectName": "purchaseorders",
               "fieldMapping": {
                    "PURCHASEORDERNUMBER": "OrderNumber",
                    "ORDERVENDORACCOUNTNUMBER": "SupplierNumber",
                    "CURRENCYCODE": "CurrencyCode",
                    "ACCOUNTINGDATE": "DateCreated"
               },
               "incrementalPush": true
          },
          {
               "EntityNameExternal": "VENDVENDORBANKACCOUNTENTITY",
               "masterdataObjectName": "supplierbankaccounts",
               "fieldMapping": {
                    "BANKNAME": "BankName",
                    "VENDORACCOUNTNUMBER": "SupplierNumber",
                    "ROUTINGNUMBER": "BankNumber",
                    "IBAN": "AccountNumber"
               },
               "incrementalPush": true
          }
     ]
}
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.dynalogic.com/birds-document-warehouse-product-overview/setup-and-configuration/jobs.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
