Collect Logs from Azure Blob Storage
-
We are using this to pull in "IIS" logs from our web apps. I've noticed that sometimes the lines come in as one message (example below). It seems to almost always happen when a new log file is created on the hour. The collector has infer line breaks enabled just like our other collectors on our physical machines importing IIS logs. I'm not sure if I have something misconfigured on the sumo side or this is just how it's going to be processing blob files.
#Fields: date time s-sitename cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2018-09-06 13:01:34 sitename GET /api/resources/health/8/0 X-ARR-LOG-ID=4e10ddfd-7f3e-4651-a39c-96703899adca 80 - 24.107.161.140 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/69.0.3497.81+Safari/537.36 - http://w2.www.edu/results?search=health host.net 200 0 0 1594 1015 814
2018-09-06 13:01:34 sitename GET /api/documents/health/3/8/0 X-ARR-LOG-ID=7975b6ce-136e-4281-ad92-30b32476788b 80 - 24.107.161.140 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/69.0.3497.81+Safari/537.36 - http://w2.www.edu/results?search=health host.net 200 0 0 3910 1021 937
2018-09-06 13:01:36 sitename GET /api/documents/health/2/8/0 X-ARR-LOG-ID=67f33aed-04e1-4049-ade2-e96ead3311dc 80 - 24.107.161.140 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/69.0.3497.81+Safari/537.36 - http://w2.www.edu/results?search=health host.net 200 0 0 2225 1021 2314
2018-09-06 13:01:36 sitename GET /api/documents/health/4/8/0 X-ARR-LOG-ID=a3d7702d-def4-4258-9409-d191f1e37f8b 80 - 24.107.161.140 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/69.0.3497.81+Safari/537.36 - http://w2.www.edu/results?search=health host.net 200 0 0 2657 1021 2314 -
Can you define Boundary Regex at Hosted collector setting and see if it works ?
.*\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}.*
http://sumologic.link/a729f5dbd9eb
-
The data being sent from Azure to Sumo (excluding IIS logs for some reason) are showing logs from 7 days ago. Running the "task-producer - BlobTaskProducer" function, which shows that your code is grabbing 7 day old logs... the timestamp of when the function is run shows today's date and time. Is there an issue with the function?
2019-07-30T20:48:19.579 [Info] Tasks Created: [{"startByte":200244,"endByte":205375,"url":blah...APP-2019-07-23-20-fe940d-12780.applicationLog.csv":[205376]}
Please sign in to leave a comment.
Comments
6 comments