Lookup against CSV that's stored within SumoLogic itself?

Follow

Kenneth Ord

• August 22, 2019 15:34

My search results for log data return IP addresses of hosts which I need to be resolved to hostnames. I have a CSV hosted on a web server and I can get SumoLogic to look up each IP in the CSV from the external web server and return the hostname for inclusion in the results. I'd like to stop using the web server and instead have the data held directly in SumoLogic. Is this possible? None of the IP address and hostname data is held anywhere in my logs and I don't want to create a collector simply to look up the contents of a file to get the data into SumoLogic. Is there any way I can upload data like a CSV and use it?

 

Thanks

0

• Official comment

  

  Ryan Johnson

  • August 23, 2019 01:30
  • Edited

  Hi Kenneth!

  Historically I would recommend one of the following:

  1. Externally HTTPS hosted CSV (what you've been doing)
  2. Dedicated collector to monitor and upload the file (what you'd rather not do), or;
  3. Combining the data from either step 1 or 2, and apply a unnecessarily complex search query to save this to a lookup stored in the platform (what no-one should have to do)

  Now the (soon to be) good news is that we'll be announcing a major uplift of our lookup capabilities at our user conference 'Illuminate' next month which will address your requirements (and then some!). So while I don't have a nice solution for you today, I can assure you than help is on the way :-)

  Cheers!

  - RJ

  Comment actions Permalink
• 

  Kenneth Ord

  • August 23, 2019 09:22

  Thanks, Ryan, the new enhancements sound very intriguing :-)

  Ken

  0

  Comment actions Permalink
• 

  Jason Gianfrancesco

  • September 18, 2019 15:58

  Hi guys.  Was the announcement on new lookup capabilities made? 

  0

  Comment actions Permalink
• 

  Bowei Chi

  • October 09, 2019 19:55

  I'm looking for updates on the lookup operator as well.  Please!

  0

  Comment actions Permalink
• 

  Angad Singh

  • October 16, 2019 23:58

  Hi Guys, 

  I wanted to give a quick update about the improvements to a lookup table. Here are a couple of things we are going to address.

  1.) Users will be able to manage lookup tables using UI. They will be able to perform CRUD operations from UI, including uploading data into Lookup file from a CSV file directly. 

   2.) Fine-grain RBAC control on lookups. Users will be to grant View/Edit & manage access to various users/groups within their organization. This will help in managing lookup tables.
  
  3.)  Lookup size limit will be increased from 8MB to 100MB
  
  4.) Lookups will perform 10x faster, so you should be able to use them in a non-aggregate query without any problems.
  
  Currently, our plan is to have these ready for beta by the end of this year. if you are interested in trying out the functionality in beta, reach out to the rep associated with the account so they can get you into the beta. 
  
  Angad
  PM, Sumo

  1

  Comment actions Permalink
• 

  Bowei Chi

  • December 13, 2019 22:27

  Is the combination of collector + "save" query the only way to generate/update the local lookup file?

  It sounds like the beta UI allows you to upload CSV directly.  What about via API?

  We need to update the lookup table on a daily basis in an automated manner.  So direct access and update to the look up table would be ideal (no collectors / scheduled save query).

  0

  Comment actions Permalink
• 

  Kenneth Barry

  • March 26, 2020 18:49

  Bowei Chi
  In the past, when I have needed to do this:

  1. Push data to a hosted HTTPS Source.
  2. (a few minutes later, to be safe): Use the SearchJob API to programmatically FORCE a "save" of the lookup data just ingested (step 1)

  You can even use the "append" to keep it "updated: throughout the day, and do a full "reset" once daily. 

  Just one more reason the SearchJob API is my one of my favorite parts of Sumo

  0

  Comment actions Permalink

Please sign in to leave a comment.