How to send Hyper-Q logs into Azure Log Analytics

Pre-requisites:

  1. VM with successful Hyper-Q installation
  2. Log Analytics workspace in Azure Portal

Step 1: Disable SELinux on Hyper-Q VM

  1. Open file /etc/selinux/config using below command and set SELINUX=permissive

                                 vi /etc/selinux/config

  1. To avoid rebooting the VM after above change run below command:

                               sudo setenforce 0

 

Step 2: Install and connect the Azure Log Analytics agent

  1. Go to Log Analytics Workspace
  2. Create New workspace (Click "Create" Button on top left) [Optional]
    • Provide Resource Group Name in which you want to set Log Analytics
    • Provide name of the workspace
    • Pick appropriate region
    • Click "Next: Pricing tier" and pick appropriate tier
    • Click "Next: Tags" and add appropriate tags
    • Click "Next: Review + Create" (wait for “Validation successful” message)
    • Click "Create" (wait for deployment to finish)
    • Go to resource once deployment finishes
  3. Installing agent on Hyper-Q VM
    • Click on Agents management (under "Settings section" ) and choose "Linux Servers" tab
    • Copy the complete wget command from "Download and onboard agent for Linux" section
    • Run above command on Hyper-Q VM and wait for it to get successfully installed (might take few minutes to download and configure). [Message on Terminal: Shell bundle exiting with code 0 implies successful installation]

Once above steps are successfully done you will see a message "1 Linux computers connected" under "Linux Servers" on Log Analytics Workspace

 

Step 3: Configuring Custom Logging

  1. Go to Log Analytics Workspace → Advanced Settings (under "Settings section")→ Data → Custom Logs
  2. Enable the flag: "Apply below configuration to my linux machines" and click "Save" button (top left corner)
  3. Click on "Add+" button
  4. Upload sample error file generated by Hyper-Q (reach out to Datometry team in case of questions) and click Next – you should see some records from file displayed on screen
  5. Select "Timestamp" field under "Select record delimiter" option and from drop-down choose format YYYY-MM-DD HH:MM:SS and click "Next"
  6. Select "Linux" and add the path /opt/datometry/logs/error_log*.csv , click "+" and then click "Next"
  7. Give custom logs name: hyper_q_error_log (Note: _CL shall be automatically appended) the click "Done"
  8. Repeat Steps 3 to 7 again for default_tracer and system_info logs as follows: 
    • Note1: While repeating on Step 6 please use path /opt/datometry/logs/default_tracer*.csv and /opt/datometry/logs/system_info*.csv for default_tracer and system_info logs respectively
    • Note2: While repeating on Step 7 name custom logs as hyper_q_default_tracer and hyper_q_session_info for default_tracer and system_info logs respectively
  9. Once all 3 logs are set, click Save on top left corner

Step 4: Final validation on Hyper-Q VM

  1. Get the workspaceid from portal [Find it under Agent management section]
  2. Ensure file: sudo vi /etc/opt/microsoft/omsagent/<WORKSPACE_ID>/conf/omsagent.d/customlog.conf exists and has references to all the three log files [*Note: Please replace workspaceid from step 1]

Example of entries in above file:

tag oms.blob.CustomLog.CUSTOM_LOG_BLOB.hyper_q_error_log_CL_
tag oms.blob.CustomLog.CUSTOM_LOG_BLOB.hyper_q_session_info_CL_
tag oms.blob.CustomLog.CUSTOM_LOG_BLOB.hyper_q_default_tracer_CL_

 

Step 5: Viewing Logs to ensure setup is correct:

  1. Go to logs section in workspace – Click "Get Started" for the first time
  2. Paste sample queries below in the editor and click "Run" to verify error and default tracer logs are collected (It may take few mins to half an hour during first run for logs to be pushed to Log Analytics and then it will push logs almost immediately)
hyper_q_error_log_CL
| extend CSVFields = split(trim('"', RawData), '","')
| extend Timestamp =  tostring(CSVFields[0])
| extend Pid =  tostring(CSVFields[1])
| extend SessionId =  toint(CSVFields[2])
| extend CRID = tostring(CSVFields[3])
| extend LSN = tostring(CSVFields[4])
| extend Module =  tostring(CSVFields[5])
| extend Line =  toint(CSVFields[6])
| extend MessageCode =  tostring(CSVFields[7])
| extend Severity =  tostring(CSVFields[8])
| extend Message =  tostring(CSVFields[9])
| extend QueryString =  tostring(CSVFields[10])
hyper_q_default_tracer_CL
| extend CSVFields = split(trim('"', RawData), '","')
| extend Timestamp =  tostring(CSVFields[0])
| extend SessionId =  toint(CSVFields[1])
| extend CRID = tostring(CSVFields[2])
| extend BRID = tostring(CSVFields[3])
| extend LSN =  toint(CSVFields[4])
| extend TagCategory =  tostring(CSVFields[5])
| extend TagName =  tostring(CSVFields[6])
| extend TagValue =  tostring(CSVFields[7])
  1. Result screen like below confirms successful setup

mceclip0.png

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.