How to include tomcat log files that end in .txt for only the past month, or any recent time frame

Comments

3 comments

  • Avatar
    Kevin

    The dates files are rotations of the original file. Since the Collector is reading that original file and will continue to send data from that file after any rotation occurs you do not need to specifically collect those old dated files, unless you are trying to get the content from those when first setting up your Collector/Source. 

    If you are trying to get these old files when first configuring a Source on your Collector you can specify a "Collection Should Begin" time within the Source configuration, which will tell the Collector to only read from files that have been last modified since the relative date selected. This should limit how much old data is sent to the service from these files when the Source is first created. 




    Once the Collector is reading from the most recent file you simply need to specify that file name when you query for your logs, using the timerange selection to display only the messages for the timerange you need.

    0
    Comment actions Permalink
  • Avatar
    Raye Raskin

    Kevin,

    Thanks for the quick response.

    Question 1: If I keep the *.txt files blacklisted you're saying the data will still accumulate from the SITE file?

    Question 2: Maybe there should be a way to reset the "Collection should begin" date after the collection has already been created. That would be much easier than deleting and recreating our dozens of collectors.

    0
    Comment actions Permalink
  • Avatar
    Kevin

    1.) Correct the Collector will read the contents of the SITE file and when a rotation occurs it will then start reading any new contents in the new SITE file created as part of the rotation.  

    And actually, you shouldn't really need to blacklist the .txt files. The Collector will know if it has already read the content of those files, even after a rotation occurs and the names change. The Collector keeps a series of fingerprints based on the first 2kb of content within the files. It then compares these fingerprints to the first 2kb of content it finds in any files matching the supplied path expression, regardless of the name of the file. So if the SITE file is written and then this rotates and the name changes to SITE-yyyy-mm-dd.txt  the Collector will see the contents of that rotated file match a fingerprint it has on record and it will not read those contents again. The new SITE file created when the rotation occurs will then be picked up and that will be read, since the contents will be different than previous versions, usually because the timestamps of those messages will be new.  

    When you first create a Source the Collector may read those dated files if they are included in the path expression, but only the ones that have a last modified date later than the "Collection Should Begin" time supplied when the Source is first created. This is what allows you to ingest some of the older files at initial creation, but prevents these from being re-read as time goes on. 

    2.) I believe the "Collection Should Begin" date can be edited on an existing Source. You should be able to just open the Source for edit and update this value and save. 

    Hope that all makes sense ;) 

    0
    Comment actions Permalink

Please sign in to leave a comment.