Multiple sources.json for syncSources feature
The syncSources feature is relativly new but i had been told by Sumo that it was going to support multiple sources.json files. This does not seem to be in the case.
If i point syncSources to a big long sources.json file it works well but when i point to /etc/sumo.d/ -- a directory that contains a few different json sources files then it fails :(
Should i raise this as a feature request or is this functionality just not working because of a bug (as it's a new, undocumented feature)?
Come on Sumo, it's sooo close to what i need! Imagine just dropping in a new json sources file for a new app installed or if i just installed apache.
-
Thanks Jamie. Works as described. Hopefully Sumo will get the doco up for this feature soon enough. It'd be good if you could define multiple sources in the same file because I'd like to have...
common.json along with webapp.json and httpd.json rather than 12 different source files but we''ll wait and see - maybe it'll come in a future release. The fact that this feature exists at all is a massive :) for me!
-
Hey guys,
I'm glad you've found the support for multiple source files on the syncSource feature valuable. I also want to apologize that our documentation of that feature hasn't kept pace with the development/deployment cycle. I've alerted our Doc team to prioritize this, as it's obviously something many customers are going to want to leverage.
Robert - if you want to expound on your specific requirements that would make a single file more useful, please do. Our Collector Product Manager is out for the week, but he'll see that information here when he returns.
Jamie, thanks for supporting the community! ;-)
Cheers,
Dean
VP Customer Success - Sumo Logic
-
I'd also like to have many sources configured in a single syncSources JSON file. That way we could have one file for the common Linux System logs, one file that contains all the mongo logs, one for all the nginx logs.. then just depending on what's installed on the server, we could just add the json files we need.
For example, some of our test servers might have nginx and mongo running on linux, so we'd just have all 3, mongo.json, nginx.json and linux-common.json. Where in prod, it would just be nginx and linux-common.
-
+1 for what Jamie said.
Currently I define /etc/sumo.d as my directory to watch and then i have to drop in a json file for every single source. E.g. /var/logmessages, /var/log/maillog, /var/log/yum.log etc. These are common across all my Linux servers so I'd like to just have one common.json file with all those sources in there. Then i can just drop in a specific file for the particular app server i'm running.
E.g.
/etc/sumo.d/common.json and /etc/sumo.d/mywebapp.json on server1 and then /etc/sumo.d/common.json and /etc/sumo.d/mymobileapp.json on server2
-
Robert and Jamie,
Thanks a lot for your feedback! I am glad you were able to get the Local File based configuration feature to work.
We are in the process of rolling out the directory-based synchronization, which is why the documentation is not up yet. We are definitely designing support for multiple sources on a single file in this mode, but there are some design considerations we are working out (e.g. how to deal with conflicting source / collector configurations coming from different files).
I am excited to continue hearing your feedback as you work with this feature, so please reach out and let us know how it's working for you.
Jose Alberto
Engineering Manager, Data Collection -
@christian Not sure what happened to it.. but here's another.
{
"api.version": "v1",
"source": {
"sourceType" : "LocalFile",
"name": "Error Log",
"hostName": "server1",
"pathExpression": "/var/log/errors.log",
"category": "errors",
"useAutolineMatching": true,
"multilineProcessingEnabled": true,
"timeZone": "UTC",
"automaticDateParsing": true,
"forceTimeZone": false,
"defaultDateFormat": "dd/MMM/yyyy HH:mm:ss"
}
}
that'd be the whole file.. then you need to have one of those for each file or sourceCategory that you want to be injested
your sumo.conf would then look like
name=[collector name]
accessid=[access id]
accesskey=[access key]
syncSources=/path/to/sumo/json/configs/
ephemeral=true
Please sign in to leave a comment.
Comments
13 comments