Dealing with Base64 encoded log data

Comments

7 comments

  • Avatar
    Kevin

    Hi Mark,

    If you are using the instructions provided in the GCP Collection documentation and have updated your HTTP Source with the thirdPartyRef details listed in this doc, this should tell that Source to decode the "data" value from your GCP logs on ingest. 

    The values in the JSON that determine the decoding of the "data" string are the "contentType" and "serviceType" values, which both need to be set to "GoogleCloudLogs"

    Here is a sample portion of the JSON, with the important portions highlighted. 

    "contentType": "GoogleCloudLogs",
    "thirdPartyRef":{
         "resources":[
           {
             "serviceType": "GoogleCloudLogs",
             "path":{
               "type":"NoPathExpression"
             },
             "authentication": {
               "type": "GoogleCloudAuthentication",
               "validations": [
                 {
                   "type":"GoogleCloudValidationDoc",
                   "name":"<google-validation-html-filename>",
                   "content": "<google-validation-html-file-content>"
                 }
               ]
             }
           }
         ]
       }

     

    Within the Sumo Logic Collector Management page you can view the current JSON configuration for a Source by selecting the "i" icon to the far right of the Source name. Within this JSON check if the above values are set. If not you may need to post a new update via the Collection API to set these values.  Note: When you first get the JSON via the Collector API there are a number of keys returned, one of these keys is named "alive." Make sure this key is removed prior to posting back your updates as I have seen this may cause the update to silently fail.

    If you continue to have any problems please let me know and I can pull this into a support case and work with you to try and get this corrected. 

    -Kevin

    1
    Comment actions Permalink
  • Avatar
    Mark Drummond

    Thanks Kevin. Looks like that is probably the issue. The contentType and serviceType are both currently set to "other".

    0
    Comment actions Permalink
  • Avatar
    Mark Drummond

    Kevin,

    We're still having some trouble with this. Would you be able to create that ticket?

    Thanks,

    Mark

    0
    Comment actions Permalink
  • Avatar
    Mark Drummond

    Looking at the current config for appengine/onboarding/dev, I can see that 'contentType' is now set to 'GoogleCloudLogs', but 'serviceType' is still set to 'other'. I will try updating the config.

    0
    Comment actions Permalink
  • Avatar
    Mark Drummond

    I started following the process more or less from scratch. I regenerated the HTTP source URL in Sumo, added the new source URL to API creds in GCP, downloaded the verification HTML file, created a new API key pair, downloaded the 'source.json' and edited it with the updated information, including 'serviceType: GoogleCloudLogs', and pushed the updated JSON to the source URL.

    In the response, serviceType has switched back to 'other'.

    $ grep serviceType source.json 
            "serviceType":"GoogleCloudLogs",

    Response after pushing the edited source.json:

        "thirdPartyRef":{
          "resources":[{
            "serviceType":"Other",
    0
    Comment actions Permalink
  • Avatar
    Tom TANG

    I have the same issue as Mark described.

    I try to change the source.json, and use cURL with eTag to do Http PUT update, and the response tell me that the service type become Other immediately.

    The log I am getting is base 64 encoded.

    "serviceType":"Other"
    0
    Comment actions Permalink
  • Avatar
    Kevin

    Here is a proper JSON for updating the HTTP Source for GCP.  We need to strip any non-required values from the JSON in order to address the issue of the content types not being set correctly.  Just update the "id", "name", "category", "validations.name" and "validations.content" to match with your HTTP Source. 

    {
      "source":{
        "id":xxxxxxxxx,
        "name":"Google Cloud Logs",
        "category":"google_cloud_logs",
        "contentType":"GoogleCloudLogs",
        "thirdPartyRef":{
          "resources":[{
            "serviceType":"GoogleCloudLogs",
            "path":{
              "type":"NoPathExpression"
            },
            "authentication":{
              "type":"GoogleCloudAuthentication",
              "validations":[{
                "type":"GoogleCloudValidationDoc",
                "name":"googlexxxxxxxxxxx.html",
                "content":"google-site-verification: googlexxxxxxxxxx.html"
              }]
            }
          }]
        },
        "messagePerRequest":false,
        "sourceType":"HTTP"
      }
    }
    0
    Comment actions Permalink

Please sign in to leave a comment.