Multiple sources.json for syncSources feature

Comments

13 comments

  • Avatar
    Jamie Snell

    When multiple json files, you must have only 1 source per file.  The JSON changes slightly. You must use a "source" element instead of "sources" and it can't be an array.

  • Avatar
    Michael Hill

    Jamie, 

    Can you give some examples of this?  One of the things that I've found is lacking with sumologic is we are left to guess about these kinds of things instead of having clear examples and use cases.

  • Avatar
    Jamie Snell

    Here's an example. If you want to ingest multiple logfiles you'd just have an example.json file for each log file.

  • Avatar
    Robert White

    Thanks Jamie.  Works as described.  Hopefully Sumo will get the doco up for this feature soon enough.  It'd be good if you could define multiple sources in the same file because I'd like to have...

    common.json along with webapp.json and httpd.json rather than 12 different source files but we''ll wait and see - maybe it'll come in a future release.  The fact that this feature exists at all is a massive :) for me!

  • Avatar
    Dean Thomas

    Hey guys,

    I'm glad you've found the support for multiple source files on the syncSource feature valuable.  I also want to apologize that our documentation of that feature hasn't kept pace with the development/deployment cycle.  I've alerted our Doc team to prioritize this, as it's obviously something many customers are going to want to leverage.

    Robert - if you want to expound on your specific requirements that would make a single file more useful, please do.  Our Collector Product Manager is out for the week, but he'll see that information here when he returns.

    Jamie, thanks for supporting the community!  ;-)

    Cheers,

    Dean

    VP Customer Success - Sumo Logic

  • Avatar
    Jamie Snell

    I'd also like to have many sources configured in a single syncSources JSON file.  That way we could have one file for the common Linux System logs, one file that contains all the mongo logs, one for all the nginx logs.. then just depending on what's installed on the server, we could just add the json files we need.  

    For example, some of our test servers might have nginx and mongo running on linux, so we'd just have all 3, mongo.json, nginx.json and linux-common.json.  Where in prod, it would just be nginx and linux-common.

  • Avatar
    Robert White

    +1 for what Jamie said.

    Currently I define /etc/sumo.d as my directory to watch and then i have to drop in a json file for every single source.  E.g. /var/logmessages, /var/log/maillog, /var/log/yum.log etc. These are common across all my Linux servers so I'd like to just have one common.json file with all those sources in there.  Then i can just drop in a specific file for the particular app server i'm running.

    E.g.

    /etc/sumo.d/common.json and /etc/sumo.d/mywebapp.json on server1 and then /etc/sumo.d/common.json and /etc/sumo.d/mymobileapp.json on server2

  • Avatar
    Jason Floyd

    I'd love to see this ability as well! It's already a vast improvement using local .json files for each source. But being able to group multiple sources into multiple files would be great.

  • Avatar
    Jose Muniz

    Robert and Jamie,

    Thanks a lot for your feedback! I am glad you were able to get the Local File based configuration feature to work.

    We are in the process of rolling out the directory-based synchronization, which is why the documentation is not up yet. We are definitely designing support for multiple sources on a single file in this mode, but there are some design considerations we are working out (e.g. how to deal with conflicting source / collector configurations coming from different files).

    I am excited to continue hearing your feedback as you work with this feature, so please reach out and let us know how it's working for you.

    Jose Alberto
    Engineering Manager, Data Collection

  • Avatar
    Luke Rohde

    After struggling to make this work for several hours today, I figured out that it's not enough to point syncSources to a directory - the config files for each source need to have a .json extension to be consumed. They are silently ignored otherwise.

  • Avatar
    dbrown

    Happy I stumbled upon this. Either my doc searching is terrible or the documentation still hasn't been updated.

  • Avatar
    Christian Essner

    What happened to the example on the March 31, 2015 07:25 post?

     

    We desperately need this feature as well. Is it still a priority with the Doc team?

  • Avatar
    Jamie Snell

    @christian  Not sure what happened to it.. but here's another.

    {
    "api.version": "v1",
      "source": {
        "sourceType" : "LocalFile",
        "name": "Error Log",
        "hostName": "server1",
        "pathExpression": "/var/log/errors.log",
        "category": "errors",
        "useAutolineMatching": true,
        "multilineProcessingEnabled": true,
        "timeZone": "UTC",
       "automaticDateParsing": true,
        "forceTimeZone": false,
        "defaultDateFormat": "dd/MMM/yyyy HH:mm:ss"
      }
    }

    that'd be the whole file.. then you need to have one of those for each file or sourceCategory that you want to be injested


    your sumo.conf would then look like 
    name=[collector name]
    accessid=[access id]
    accesskey=[access key]
    syncSources=/path/to/sumo/json/configs/
    ephemeral=true

Please sign in to leave a comment.