Is it true that when using partitions or scheduled view to create filtered/refined data sets, those extra data sets cost towards ingest?
Normally there is no charge but under some use cases, there could be a charge as explained below.
If a data is duplicated in partitions then there will be a charge for the additional copy of the duplicated messages. If the partitions are created without an overlapping data set, then there is no charge for the ingestion.
For example, in the following example, the data in sourceCategory prod/Apache is targeted by two partitions one using a wild card and other using a specific sourceCategory
Refer the Best Practices for creating partitions
There are 2 use cases in schedule view where the customer will be charged:
- If you are running an non-aggregate query(without any group by operator) to create a scheduled view then for those views that include raw data there will be a charge
- If you are running an aggregate query but if you count the log messages by the _raw field then it will consider that column result as raw data and in this scenario there is a charge for this ingest.
To avoid a charge while creating scheduled views, please ensure that the queries are aggregated and avoid aggregating using the _raw field. Please refer the following Doc link for more information.