I have a hosted collector with AWS S3 or AWS CloudTrail sources and data ingestion is delayed by few hours, how should I avoid ingestion delays?

Answer: The most likely reason for ingestion delays for AWS S3 or CloudTrail sources is the time taken by the collector to scan the objects. If you have thousands of objects in the path expression, the collector could take more than a few minutes to a few hours to scan and list these objects and this could cause ingestion delays.

The best option to fix this issue is to enable "S3 Event Notifications". Sumo Logic S3 integration combines scan based discovery and event-based discovery into a unified integration that gives you the ability to maintain a low-latency integration for new content and provide assurances that no data was missed or dropped. When you enable event based notifications S3 will automatically publish new files to Amazon Simple Notification Service (SNS) topics that Sumo Logic can be subscribed to. This notifies Sumo Logic immediately when new files are added to your S3 bucket so we can collect it.

Enabling event-based notifications is an S3 bucket-level operation that subscribes to an SNS topic. An SNS topic is an access point that Sumo Logic can dynamically subscribe to in order to receive event notifications. When creating a Source that collects from an S3 bucket Sumo assigns an endpoint URL to the Source. The URL is for you to use in the AWS subscription to the SNS topic so AWS notifies Sumo when there are new files.

when creating an AWS S3 or CloudTrail source, you have to configure Log File Discovery.

You have the option to set up Amazon Simple Notification Service (SNS) to notify the Sumo Logic of new items in your S3 bucket. A scan interval is required and automatically applied to detect log files.

Scan Interval. Sumo Logic will periodically scan your S3 bucket for new items in addition to SNS notifications. Automatic is recommended to not incur additional AWS charges. This sets the scan interval based on if subscribed to an SNS topic endpoint and how often new files are detected over time. You may enter a set frequency to scan your S3 bucket for new data. To learn more about Scan Interval considerations, see About setting the S3 Scan Interval.
SNS Subscription Endpoint (Optional). New files will be collected by Sumo Logic as soon as the notification is received. This will provide faster collection versus having to wait for the next scan to detect the new file.
A. To set up the subscription you need to get an endpoint URL from Sumo to provide to AWS. This process will save your Source and begin scanning your S3 bucket when the endpoint URL is generated. Click on Create URL and use the provided endpoint URL when creating your subscription in step C.

Make sure that the topic and the bucket are in the same region. C. Again go to Services > Simple Notification Service and click Create Subscription. Paste the Topic ARN from step B above. Select HTTPS as the protocol and enter the Endpoint URL provided while creating the S3 source in Sumo Logic. Click Create subscription and a confirmation request will be sent to Sumo Logic. The request will be automatically confirmed by Sumo Logic. D. Select the Topic created in step B and navigate to Actions > Edit Topic Policy. Use the following policy template, replace the SNS-topic-ARN and bucket-name placeholders in the Resource section of the JSON policy with your actual SNS topic ARN and S3 bucket name: {
    "Version": "2008-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": [
            "SNS:Publish"
        ],
        "Resource": "SNS-topic-ARN",
        "Condition": {
            "ArnLike": {
                "aws:SourceArn": "arn:aws:s3:*:*:bucket-name"
            }
        }
    }]
} E. Go to Services > S3 and select the bucket to which you want to attach the notifications. Navigate to Properties > Events > Add Notification. Enter a Name for the event notification. In the Events section select, All object create events. In the Send to section (notification destination) select SNS Topic. An SNS section becomes available, select the name of the topic you created in step B from the dropdown. Click Save.