Cost of Processing Rules
Hello,
I am setting up a S3 cloudtrail collector and I would like to not ingest events that are from s3.amazonaws.com using the following regex:
^.*"eventSource":"s3.amazonaws.com".*$
The reason I am not configuring cloudtrail to not publish s3 events is because I still want those events to appear in cloudwatch. I also want to avoid creating a separate cloudtrail.
One solution I am looking into is using processing roles with the regex above via the exclude rule, however I can confused on how this affects pricing. Does excluded data still contribute to my Ingestion budget as ingested data? Is there any cost to excluding data?
Any clarification or if you have any other ideas for how I resolve this would be greatly appreciated!
Thank you.
-
Hi Jared,
Data filtered by a Collector using Processing Rules does not count towards your daily data volume quota. Filtered data is fed to the Collector, but does not upload to the Sumo Logic cloud. There is no additional cost as far as Sumo Logic is concerned.
Using processing rule on a hosted collector to filter out 50% or more of your data on a single Source may be subject to temporary throttling. Excessive filtration can lead to temporary increases in ingest latency. Throttling reduces the effect excessive filtration has on ingestion latency. We encourage you to limit the proportion of data you are filtering out.
Another point to note is your Regular expressions must be RE2 compliant.You can read more about processing rules here: https://help.sumologic.com/Manage/Collection/Processing-Rule
Hope this helps and answers your questions.
Thanks.
Please sign in to leave a comment.
Comments
1 comment