Okta log collection with SumoJanus scripts shows duplicate event ingestion missing the latest events

Problem:

Starting April 18, 2020, we received customer reports indicating an issue with their Okta log collection. The Okta log collection had been working fine for months, if not years.

The collector status was operational. However, new logs were not being collected. There was a difference between the published timestamp and the "collected" timestamp which was showing the time of ingestion. Essentially, the collector was ingesting the same older log events over and over repeatedly.

Cause:

There was an update to some Okta tenants used by our customers relating to the Link HTTP header sent with lower case "l" as opposed to an uppercase “L” as sent previously. 

https://developer.okta.com/docs/reference/api-overview/#link-header

How do I know if I know my installation is affected?

Use the “curl” utility, and an existing Okta API token and your Okta domain, run the following command after updating the highlighted sections: 

curl -v -X GET \

-H "Accept: application/json" \

-H "Content-Type: application/json" \

-H "Authorization: SSWS ${api_token}" \

"https://${yourOktaDomain}/api/v1/logs?since=2017-10-01T00:00:00.000Z"

In the results, if you see HTTP headers with a lowercase “link” as opposed to “Link”, that will indicate your SumoJanus Okta scripts are affected by this problem. For example: 

link: <https://hostedfor.okta.com/api/v1/logs?after=1587580307366_1>; rel="self"

link: <https://hostedfor.okta.com/api/v1/logs?after=1587582126074_1>; rel="next"

Resolution:

Please follow the instructions to configure log collection by downloading and installing the SumoJanus package from this page for your operating system. If you would like to update your existing SumoJanus installation instead, follow the instructions below

The easiest way is to install the new tarballs is to treat this is as a new deployment downloading the new tarball via Step 2 and copying over some files from the existing installation, such as: 

1) conf/sumologic.properties 

2) data/okta_checkpoint.dat

Specifically, please make sure to check if any customizations for “stream_pos_path” or “path” in conf/sumologic.properties and JAVA_HOME in bin/SumoJanus_Okta.bash or bin/SumoJanus_Okta.bat need to be updated for the new directory.

We recommend testing for success with the CLI command from the SumoJanus directory to make sure the updated Okta collector scripts work

bin/SumoJanus_Okta.bash 

bin/SumoJanus_Okta.bat  

If the new tarball was deployed into a new directory, you will need to edit the script source in the UI under the Manage Data -> Collection tab to update the path to the script and the working directory.

If you maintain the same directory naming and transfer the previous installation into a sumojanus-okta.old directory, this step can be avoided.