I have created an S3 Source in Sumo Logic and trying to collect logs from an S3 bucket into Sumo Logic but I am not seeing any data show up in my Sumo Logic search.
- Can you please check the Collector Status page(Sumo Logic-> Manage Data -> Collection -> Status), verify are you seeing any bar chart showing collection for that specific collector, if yes:
- Please, run the search by Receipt Time and verify now you are able to see log messages or not.
If you still not getting data after completion of the above 2 steps then there are at least three possibilities which may be causing this issue.
1.) AWS S3 JSON access policy:
Please check that you have assigned a valid S3 policy to your IAM user. We recommend users to assign below JSON policy to grant access to S3 bucket content:
Make sure to replace the
your_bucketname placeholders in the
Resource section of the JSON policy with your actual S3 bucket name.
- All of the Action parameters shown above are required. Make sure to include both of the Amazon Resource Name (ARN) statements in the "Resource" section of the policy. Both statements are required to allow full access to the bucket contents and the bucket itself.
2.) Wrong Path Expression:
If your AWS S3 JSON access policy looks correct then please check the path expression supplied within your source to ensure it matches the folder structure of your file objects. Some common errors with the path expression are as follows:
- The S3 bucket name is included as part of the path. The bucket name should not be included as part of the "path expression" unless it exists as a folder and is part of the folder structure.
- The path expressions do include a leading forward slash. The path expression supplied should not include a leading forward slash character and
- The path expression does not match the path for the file objects to collect. To collect all logs at a hierarchical level, you can use a portion of the source path along with a single asterisk as a wildcard to denote the rest of the path and file object names. Alternately, to match all file objects in a bucket, you can supply a "*" as the path expression
- The path expression has multiple wild card asterisk "*". You can use only one wild card in the path expression.
If you want to collect logs from a specific folder then define that folder name and then a * to denote the file objects within that path. For example, if your file objects are contained within a folder named test you can define the path expression as test/* to get all the objects within the test folder. Note this will also include any objects located in subfolders of this directory.
3.) There are no file objects within your bucket with a last modified date earlier than the collection begin time setting.
If your AWS S3 JSON policy and path expressions are correct next check the configured data collection beginning time in Sumo Logic for that specific S3 source and compare this with the last modified dates of the objects you expect to be collected from the S3 bucket.
Where you can find this: In Sumo Logic Collection page -> Select your S3 source -> Edit -> Collection Should begin on the configuration page, to get more idea please see below video
If the last modified date of your S3 bucket objects is older than the collection beginning time in Sumo Logic then you have to recreate your S3 Source into Sumo Logic and change your data collection beginning time to the time which includes the last modified time of the objects within your S3 bucket.
In case if you want to ingest data of all time from your S3 bucket to Sumo Logic then you can select All Time as collection beginning time from the drop-down option in Sumo Logic.
If you still facing any issues please reach out to Sumo Logic Technical Support