2. Process

The process for importing these files involves placing the JSON file into an S3 bucket provided by Ideagen. The system will automatically consume the file and process it into your tenant.

The following must be considered:

• All imports must be in JSON format* and meet the JSON standard in order to be uploaded.

• During the processing phase the respective user interfaces will be disabled in the administration area until the job has completed.

  • Organisational Unit management

  • Groups – Organisational Unit / People management

  • Managing Users

  • User Profiles

  This helps to avoid interference whilst the data is being processed in Calypso. Once complete, these modules will be re-enabled.

• If you wish to import Organisational Units, do it before importing any other data types and wait for it to finish before starting other supported imports.

• It is your responsibility to make backups of files you wish to keep. For example, if you upload a file with the same filename as an existing file, it will overwrite it so you must make a backup of the existing file first if you wish to keep it.

• If you import a file before the previous upload is completed, the process will cancel for the previous upload.

• You must not do more than one import per hour. 

• The filename must not contain any spaces or special characters, but dashes and underscores are allowed. The filename should be kept as simple as possible. It is recommended you add a date to the filename to avoid overwriting older files. Overwriting the same file will still work, but you will lose the previous version and will be unable to diagnose any failures.

• If browsing using an S3 client such as Cyberduck or CloudBerry, you will not have access to list folders at bucket level and an error will be thrown suggesting you do not have permission. You should set the connection to go directly to your tenant folder. This may require multiple connections using the same Access Key to the different tenant folders such as Live, Training, Test etc.

*More information about JSON Format: The format is designed to be readable and easy to follow structurally. Each file will contain one list with 1 to 20,000 list items, with child list items following the hierarchical layout of JSON (a child list item will be nested within its parent).

3. Restrictions

Ideagen reserves the right to clear the data stored in the folder without warning so that it complies with the following restrictions.

• No files other that those required for use in the import process may be stored.

• The total size of all files in your designated S3 bucket folder must not exceed 10GB.

• No file should be stored longer than 1 year.

4. Getting GUIDs from platform API

The API can be accessed through a web interface: https://platform.gaelenlighten.com/api/swagger-ui/

You will need to fill in your details at the top.

• Tenant Url: This should be your URL.

• Username: This should be the username of your admin.

• Password: This should be the admin user’s password.

Find the section related to what data type you want to import and expand the section to view the operations available. For each operation there are different parameters you can filter on. For example if you wanted to find an organisational unit based on its code MYORGUNIT, you can expand the orgunits : Ideagen Aviation Safety (Cora) Organisational Units section, expand the /orgunits operation, under Parameters for Code put “MYORGUNIT” as the value, then click the Try it out! button. The Response Body will have information about MYORGUNIT.

5. Import failures

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

5.1. File validation

The system checks the JSON is well-formed and all fields that are mandatory are present. Should the validation fail at any point, the entire sync file will be rejected.

5.2. Pre-processing validation checks

Before processing a file for import or sync the following is checked and on failure a sync / import will not be processed:

• The JSON is well-formed.
• Mandatory fields are present and have values.
• Data that requires to be in a specific format, e.g. e-mail is in a valid format.
• There are no duplications, e.g. username, email and code values must be unique.
• If a relationship to a security group or organisational unit that does not exist is being attempted.
• If the sync attempts to make an item a child of a parent that does not exist.

If the file is rejected, a failed import email will not be sent out at this point. 

5.3. User sync only

If a user sync file passes the file validation checks , the system will start to sync each user. The sync can still fail at this point, but it will be on a per-user basis. Most common failure is that username or email already exists. This individual user will fail, but the process for the remaining users will continue.

If individual user(s) fail to sync, a 'Failed Import' notification containing the error logs will be sent to our Ideagen Support team and the nominated client contact.

2. Importing custom lists

The information in this section is for importing Custom Lists into the applications Custom List Management module. When Creating or Archiving a Custom List, most fields are mandatory as the List Sync will update the List item to match the data in sync file.

You MUST include ALL existing and new list items in the import file to keep them active. 

You must also make sure the data is up-to-date, including any changes that were made manually in the UI.

2.1. Creating and updating

This feature allows you to create new and update existing Custom List items in bulk.

Please consider:

• When creating new Custom List items, any existing List items must be included in the file to remain active.

• For existing List items, remember to include all existing information. Fields that are omitted will be cleared or set to a default value.

• When creating a or changing the hierarchy structure, you must create the Top-Level Custom List item first before creating or moving any Sub-Level Custom List item. This cannot do both in the same import file.

• You cannot update the Code values using the sync tool, this can only be done manually.

2.2. Archiving

This feature allows you to archive existing Custom List items in bulk.

Please consider:

• Any Custom List items that are excluded from the sync file will be archived.

• If you archive Custom List item(s) that contain Sub-Level Custom List item(s), all Sub-Level Custom List items will also be archived.

2.4. List sync values

*Mandatory

3. Creating a JSON file

You can do multiple kinds of different operations in one .json file. For example you can create, update, and archive Custom List items all in one file, however, the .json for Custom Lists must have only one custom list in the file, and is identified by its ListId and Name values. If you want to do multiple lists, you can upload multiple .json files, one for each custom list.

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

Below are examples of how Custom List entries should be formatted in the JSON file, these examples cover the commands to created, updated and archive Custom Lists in the Custom List Management module. 

3.1 Create a simple list (No parent)

{
  "Name": "Aircraft Types",
  "ListItems": [
    {
      "Name": "Boeing 737"
    },
    {
      "Name": "Airbus A320",
      "Code": "A320"
    }
  ]
}

3.2 Create a list with hierarchical items

{
  "Name": "Aircraft Types",
  "ListItems": [
    {
      "Name": "Boeing 737"
    },
    {
      "Name": "Airbus A320",
      "Code": "A320",
      "Children": [
        {
          "Name": "Airbus A320-200",
          "Code": "A320-200",
          "Children": [
            {
              "Name": "Old A320-200 Model"
            }
          ]
        },
        {
          "Name": "Airbus A320-999"
        }
      ]
    }
  ]
}

3.3 Update an existing custom list item

{
  "ListId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Name": "Aircraft Types",
  "ListItems": [
    {
      "Name": "Boeing 787",
      "Code": "B787"
    }
  ]
}

3.4 Archive a custom list item

As stated in the guidelines, when an Custom List item is omitted it will be archived. As an example say we have the following existing Custom List.

{
  "ListId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Name": "Aircraft Types",
  "ListItems": [
    {
      "Name": "Boeing 737"
    },
    {
      "Name": "Airbus A320",
      "Code": "A320",
      "Children": [
        {
          "Name": "Airbus A320-200",
          "Code": "A320-200",
          "Children": [
            {
              "Name": "Old A320-200 Model"
            }
          ]
        },
        {
          "Name": "Airbus A320-999"
        }
      ]
    }
  ]
}

We will now archive the Airbus A320 customer list item by omitting it from the import file. As stated in the guidelines, all of its sub-level OUs will be archived as well so we will also omit them.

{
  "ListId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "Name": "Aircraft Types",
  "ListItems": [
    {
      "Name": "Boeing 787",
      "Code": "B787"
    }
  ]
}

4. Uploading a list sync file 

Once you have finished creating your JSON file, it can then be uploaded to the S3 location that has been provided to you.

• Bucket: customer-integration-feed

• Folder: <tenant name>/listsync

Please ensure you have read and fully understood the process before uploading your file.

It is important to remember that it is your responsibility to manage and maintain your user data and the upload of the JSON file.

5. Import failures

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

5.1. File validation

The system checks the JSON is well-formed and all fields that are mandatory are present. Should the validation fail at any point, the entire sync file will be rejected.

5.2. Pre-processing validation checks

Before processing a file for import or sync the following is checked and on failure a sync / import will not be processed:

• The JSON is well-formed.
• Mandatory fields are present and have values.
• There are no duplications, e.g. code values must be unique per list item.

If the file is rejected, a failed import email will not be sent out at this point. 

2. Importing organisational unit lists

The information in this section is for importing OUs into the applications Organisational Unit Management module. When Creating or Archiving an OU, most fields are mandatory as the Org Sync will update the OU item to match the data in sync file.

You MUST include ALL existing and new organisational units in the import file to keep them active. 

You must also make sure the data is up-to-date, including any changes that were made manually in the UI.

2.1. Creating and updating

This feature allows you to create new and update existing Organisational Unit list items in bulk.

Please consider:

• When creating new OU items, any existing OU items must be included in the file to remain active.

• For existing OU items, remember to include all existing information. Fields that are omitted will be cleared or set to a default value.

• When creating a or changing the hierarchy structure, you must create the Top-Level OU item first before creating or moving any Sub-Level OU item. This cannot do both in the same import file.

• You cannot update the Code values using the sync tool, this can only be done manually.

2.2. Archiving

This feature allows you to archive existing Organisational Unit list items in bulk.

Please consider:

• Any OU items that are excluded from the sync file will be archived.

• If you archive OU item(s) that contain Sub-Level OU item(s), all Sub-Level OU items will also be archived.

• The default OU item cannot be archived.

2.3. Org sync values

*Mandatory

3. Creating a JSON file

You should provide an array of organisational units in the file. This is denoted with the [ and ] characters at the start and end of the file. You can do multiple kinds of different operations in one .json file. For example you can create, update, and archive organisational units all in one file. 

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

Below are examples of how OU entries should be formatted in the JSON file, these examples cover the commands to created, updated and archive OUs in the organisational units module. 

3.1 Create a top-level organisational unit (No parent)

[
  {
    "Name": "Engineering",
    "Code": "ENG",
    "Description": "Engineering Department Description"
  }
]

3.2 Create a sub-level organisational unit

[
  {
    "OrganisationalUnitId": "79148888-e405-4701-99ce-7b6ee8c3d336",
    "Name": "Engineering",
    "Code": "ENG",
    "Description": "Engineering Department Description"
  },
  {
    "Name": "Software Development",
    "ParentId": "79148888-e405-4701-99ce-7b6ee8c3d336"
  }
]

3.3 Update an existing organisational unit

[
  {
    "OrganisationalUnitId": "79148888-e405-4701-99ce-7b6ee8c3d336",
    "Name": "Engineering Updated Name",
    "Code": "ENG",
    "Description": "Updated Engineering Department Description",
    "ParentId": "a223c41b-0b7e-438b-9f58-6f8c82c3012b"
  }
]

3.4 Archive an organisational unit

As stated in the guidelines, when an organisational unit is omitted it will be archived. As an example say we have the following existing organisational units.

[
  {
    "OrganisationalUnitId": "9e61819d-aaf5-4199-9a0b-b8d482a081f3",
    "Name": "Europe Region",
    "Code": "EUR",
    "Description": "All employees situated in any branches in Europe."
  },
  {
    "OrganisationalUnitId": "3cbe4a54-8d59-4d63-8cf8-b75078152320",
    "Name": "Security",
    "Code": "EUR-SEC",
    "ParentId": "9e61819d-aaf5-4199-9a0b-b8d482a081f3"
  },
  {
    "OrganisationalUnitId": "e22d2de5-18d6-4069-871f-75f229346372",
    "Name": "Malaysia Region",
    "Code": "MAL"
  },
  {
    "OrganisationalUnitId": "08bdf406-f23c-4561-9e79-eab2e04dadc6",
    "Name": "Security",
    "Code": "MAL-SEC",
    "ParentId: "e22d2de5-18d6-4069-871f-75f229346372"
  },
  {
    "OrganisationalUnitId": "8e291fac-fae5-4fdc-bf47-d4a4a53a2600",
    "Name": "Identity & Access Management",
    "Code": "MAL-SEC-IAM",
    "ParentId": "08bdf406-f23c-4561-9e79-eab2e04dadc6"
  }
]

We will now archive the Malaysia Region unit by omitting it from the import file. As stated in the guidelines, all of its sub-level OUs will be archived as well so we will also omit them.

[
  {
    "OrganisationalUnitId": "9e61819d-aaf5-4199-9a0b-b8d482a081f3",
    "Name": "Europe Region",
    "Code": "EUR",
    "Description": "All employees situated in any branches in Europe."
  },
  {
    "OrganisationalUnitId": "3cbe4a54-8d59-4d63-8cf8-b75078152320",
    "Name": "Security",
    "Code": "EUR-SEC",
    "ParentId": "EUR"
  }
]

3.5 Multiple organisation units and operations

In this example we create a Testing organisational unit and an Engineering organisational unit. We also create a Software Development organisational unit with a description that will be a child of the Engineering unit. The Testing organisational unit will also be updated with a description. To follow the data formatting guidelines, the first import file will be as follows.

[
  {
    "Name": "Testing",
    "Code": "TEST"
  },
  {
    "Name": "Engineering",
    "Code": "ENG"
  }
]

Now in the second import file we will do the other operations.

[
  {
    "Name": "Software Development",
    "ParentId": "ENG",
    "Description": "Technical personnel."
  },
  {
    "Name": "Testing",
    "Code": "TEST",
    "Description": "All testers, excluding those on probation."
  }
]

4. Uploading a org sync file 

Once you have finished creating your JSON file, it can then be uploaded to the S3 location that has been provided to you.

• Bucket: customer-integration-feed

• Folder: <tenant name>/orgsync

Please ensure you have read and fully understood the process before uploading your file.

It is important to remember that it is your responsibility to manage and maintain your user data and the upload of the JSON file.

5. Import failures

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

5.1. File validation

The system checks the JSON is well-formed and all fields that are mandatory are present. Should the validation fail at any point, the entire sync file will be rejected.

5.2. Pre-processing validation checks

Before processing a file for import or sync the following is checked and on failure a sync / import will not be processed:

• The JSON is well-formed.
• Mandatory fields are present and have values.
• There are no duplications, e.g. code values must be unique.
• If the sync attempts to make an item a child of a parent that does not exist.

If the file is rejected, a failed import email will not be sent out at this point. 

2. Importing users

The information in this section is for importing users into the applications People module. When Creating or Updating a user, most fields are mandatory as the User Sync will update the User profile to match the data in sync file.

2.1. Creating

This feature allows you to create new users in bulk.

Please consider: 

• A user cannot be created if their username, email address and/or ExternalId values are already in use.

• If a user is to be created with the same ExternalId as a previously archived user, that user will instead be reinstated rather than a new user being created.

• When adding a user to a Group or Organisational Unit, these items must already exist in the application before the user is created.

2.2. Updating

This feature allows you to update existing users in bulk.

Please consider:

• The username value is case sensitive and must match exactly how it appears in the application. If there is a slight discrepancy the application will attempt to create a new user instead.

• When updating a user, any existing information that is not intended to change must be provided, otherwise it will be cleared or set to a default value. Some fields can be omitted, this is detailed in the User Sync Values table.

2.3. Archiving

This feature allows you to archive existing users in bulk.

Please consider:

• The ReassignedUserId field and value must be included when archiving users.

• When archiving a user, the target user for reassigned tasks must already exists or is to be created in the same file. If the reassigned user has been created in a previous sync or current sync, then the ExternalId of that user can be used.

2.3. Reinstating

This feature allows you to reinstate archived users in bulk.

2.4. User sync values

*Mandatory **Mandatory for account to be SSO enabled.

3. Creating a JSON file

You should provide an array of users in the file. This is denoted with the [ and ] characters at the start and end of the file. You can do multiple kinds of different operations in one .json file. For example you can create, update, archive and reinstate users all in one file.

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

Below are examples of how user entries should be formatted in the JSON file, these examples cover the commands to created, updated, archive or reinstate a user in the people module. 

3.1 Create a basic user

[
  {
    "Status": 0,
    "Forename": "John",
    "Surname": "Smith",
    "OrganisationalUnit": "Engineering",
    "EnableLogin": true,
	"UserName": "john.smith",
    "Email": "john.smith@company.com"
  }
]

3.2 Create an external/SSO user

[
  {
    "Status": 0,
    "Forename": "Jane",
    "Surname": "Doe",
    "OrganisationalUnit": "Engineering",
    "IsExternallyManaged": true,
    "ProviderId": "Okta-SSO-Provider",
    "ExternalId": "JD001",
    "UserName": "JD001",
    "Email": "jane.doe@company.com"
  }
]

3.3 Update an existing user

[
  {
    "Status": 1,
    "Forename": "Jane",
    "Surname": "Doe-Updated",
    "OrganisationalUnit": "Engineering",
    "ExternalId": "JD001",
    "UserName": "jane.doe",
    "Email": "jane.newemail@company.com",
    "JobTitle": "Senior Engineer"
  }
]

3.4 Archive a user

[
  {
    "Status": 2,
    "ExternalId": "JD001",
    "ReassignedUserId": "AG523"
  }
]

3.5 Reinstate an archived user

[
  {
    "Status": 3,
    "Forename": "Jane",
    "Surname": "Doe-Updated",
    "OrganisationalUnit": "Engineering",
    "ExternalId": "JD001",
    "UserName": "jane.doe"
  }
]

3.6 All available properties

Here is an example that has all the required and optional properties available in the system. You can use this as a reference for when you want to create or update a user with additional details.

[
  {
    "Status": 0,
    "Forename": "John",
    "Surname": "Smith",
	"OrganisationalUnit": "Engineering",
	"EnableLogin": true,
    "IsExternallyManaged": true,
	"ProviderId": "Okta-SSO-Provider",
	"ExternalId": "JS219",
    "UserName": "JS219",
    "Email": "john.smith@company.com",
    "JobTitle": "Senior Engineer",
    "TelephoneNumber": "+1-555-0100",
    "MobileNumber": "01 555 0101",
    "Culture": "en-US",
    "Timezone": "Central America Standard Time",
    "UserGroups": ["Engineer", "Project Manager"]
  }
]

3.7 Multiple users and operations

In this example we create an external user Jane Doe with some optional properties. Then we update Jane Doe and archive an existing user AB402 and reassign anything tied to them to Jane Doe. To follow the data formatting guidelines, the first import file will be as follows.

[
  {
    "Status": 0,
    "Forename": "Jane",
    "Surname": "Doe",
    "OrganisationalUnit": "Engineering",
    "EnableLogin": true,
    "IsExternallyManaged": true,
    "ProviderId": "Okta-SSO-Provider",
    "ExternalId": "JD001",
    "UserName": "JD001",
    "Email": "jane.doe@company.com",
    "UserGroups": ["Engineer", "Project Manager"]
  }
]

Now in the second import file we will do the other operations. As mentioned in the guidelines, since Jane’s UserGroups property is omitted, it will remain the same and Jane will remain assigned to the user groups Engineer and Project Manager.

[
  {
    "Status": 1,
    "Forename": "Jane",
    "Surname": "Doe",
    "OrganisationalUnit": "Engineering",
    "EnableLogin": true,
    "IsExternallyManaged": true,
    "ProviderId": "Okta-SSO-Provider",
    "ExternalId": "JD001",
    "UserName": "JD001",
    "Email": "jane.doe@company.com",
    "JobTitle": "Senior Engineer",
    "MobileNumber": "01 555 2300"
  },
  {
    "Status": 2,
    "ExternalId": "AB402",
    "ReassignedUserId": "JD001"
  }
]

4. Uploading a user sync file 

Once you have finished creating your JSON file, it can then be uploaded to the S3 location that has been provided to you.

• Bucket: customer-integration-feed

• Folder: <tenant name>/usersync

Please ensure you have read and fully understood the process before uploading your file.

It is important to remember that it is your responsibility to manage and maintain your user data and the upload of the JSON file.

5. Import failures

It is important to consider that if the JSON file is not formatted correctly, the file will be rejected and the sync will fail.

5.1. File validation

The system checks the JSON is well-formed and all fields that are mandatory are present. Should the validation fail at any point, the entire sync file will be rejected.

5.2. Pre-processing validation checks

Before processing a file for import or sync the following is checked and on failure a sync / import will not be processed:

• The JSON is well-formed.
• Mandatory fields are present and have values.
• Data that requires to be in a specific format, e.g. username and e-mail is in a valid format.
• There are no duplications, e.g. username, email and ExternalId values must be unique.

If the file is rejected, a failed import email will not be sent out at this point. 

5.3. Post file validation (user sync only)

If a user sync file passes the file validation checks , the system will start to sync each user. The sync can still fail at this point, but it will be on a per-user basis. 

5.4. Processing checks

Most common failure is that username or email already exists. If an individual user should fail to sync this will be logged and the process for the remaining users will continue. When processing each user entry in the file the following is checked:

• If a relationship to a security group or organisational unit that does not exist is being attempted.
• A username, email and/or ExternalId values are not already in use.

If individual user(s) fail to sync, a 'Failed Import' notification containing the error logs will be sent to our Ideagen Support team and the nominated client contact.