Issues automating Least Privelege at scale
I'm working through how to implement the least privelege "Windows_NonAdmin_Config" script in 100+ environments. In at least two, the LM service account we have is the only one with enough admin credentials to change the account to non-admin. I'm testing in our own internal systems to make sure I can get it to work. In my first go of it as both the LM Service account and using my own Admin creds in our environment, I'm getting errors: Has anyone else seen this? I'm going to keep chipping away at it as I'd like to come up with a purely LM solution to the shift due to the scale of the effort in our MSP environment. We do have ConnectWise Automate to utilize if I can't get this working, but right now, I can't even get it going using the instructions provided directly on the VM (in a console window using 'enter-pssession 127.0.0.1 -credential (get-credential)' to get a session with admin priveleges.Powershell in five easy steps
A few years ago, I wrote a powershell tutorial. Thought some may find it useful. On to the Future with Powershell – PowerShell.org It's less programming tutorial than other programming tutorials. I wrote it for the "you can tear this mouse from my cold hands" type of admin who are being forced to learn powershell to do their jobs. It's peppered with some nuggets of geeky humor to keep it rolling forward and is just enough to get someone started using powershell (or any other programming language) by focusing on the larger concepts of Storage, Input, Output, Decisions, Loops... the building blocks of every language from assembly to applescript.
Bulk Removal of System.Categories Value
We accidentally added the value 'ZertoAppliance' to the system.categories field for 800+ devices due to a poorly written PropertySource. And because we have DataSources and EventSources that apply based on that PropertySource we had about 800 non-Zerto devices trying to grab Zerto data. LogicMonitor does not automatically remove a value from system.categories field if the PropertySource is no longer valid. The only option to bulk remove this value in this situation is via the API. Below is a compilation of calls to the API using Powershell that corrected this issue for us. Hopefully it helps the community, even if simply giving examples of calling the API with Powershell. Disclaimer: Run at your own risk - this does not have any 'pause-continue' logic in it. Adding links to the KBs that were helpful to me in writing this solution. You will notice a lot of the code below is simply copied and pasted from the KBs. REST API v1 Examples | LogicMonitor REST API Basic Filters | LogicMonitor REST API Advanced Filters | LogicMonitor Get Devices | LogicMonitor Update a Device | LogicMonitor <# Use TLS 1.2 #> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 <# Account Info #> $accessId = 'myAccessId' $accessKey = 'myAccessKey' $company = 'myCompany' <# Request Details #> $httpVerb = 'GET' $resourcePath = '/device/devices' $queryParams = '?fields=customProperties,name,id&filter=customProperties.name:system.categories,customProperties.value~ZertoAppliance&size=1000' <# Construct URL #> $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams <# Get current time in milliseconds #> $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) <# Concatenate Request Details #> $requestVars = $httpVerb + $epoch + $resourcePath <# Construct Signature #> $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) <# Construct Headers #> $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') <# Make Request #> $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers <# Print status and body of response #> $status = $response.status $data = $response.data.items <# Set value in system.categories to be removed #> $badString = 'ZertoAppliance' <# Initialize an array of devices to be changed #> $results = @() <# Iterate through the response data #> ForEach ($item in $data) { <# Get the current value of system.categories #> $currentValue = ($item.customProperties | ? {$_.name -eq 'system.categories'}).value <# Only add to results array if matches #> if ( $currentValue -match $badString ) { <# Rebuild value of system.categories #> $newValue = ($currentValue.split(',') | ? {$_ -ne $badString}) -join "," <# Create PSObject of device id, name, and values #> $result = New-Object PSObject -Property @{ id = $item.id name = $item.name currentData = $currentValue newData = $newValue } <# Add PSObject to array of devices to be changed #> $results += $result } else { # Nothing to do because it's not mislabeled # Add verbose logging if you want output } } <# Dump the results array to validate the value is being removed #> $results <# Iterate through the array of devices to be changed #> ForEach ($item in $results) { <# Request Info #> $httpVerb = 'PATCH' $resourcePath = '/device/devices/' + $item.id $queryParams = '?patchFields=customProperties&opType=replace' $data = '{"customProperties":[{"name":"system.categories","value":"' + $item.newData + '"}]}' <# Construct URL #> $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams <# Get current time in milliseconds #> $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) <# Concatenate Request Details #> $requestVars = $httpVerb + $epoch + $data + $resourcePath <# Construct Signature #> $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) <# Construct Headers #> $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') <# Make Request #> $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers -Body $data <# Dump the result of each PATCH call #> $response }
Issues with Set-LMWebsiteGroup
I'm working with the lm-powershell-module and I'm able to get all my website information and the groups id's but when I try to update the group information so I can bulk move websites nothing seems to happen. I have tried many different ways such as: Set-LMWebsiteGroup -Name 'CNVPABACUSS252/Servicing/Loan' -ParentGroupName 'Production/Internal/DevOps/Individual Nodes/Abacus Sync' Set-LMWebsiteGroup -Name $website.name -ParentGroupName 'Production/Internal/DevOps/Individual Nodes/Abacus Sync' -ParentGroupId 77 Set-LMWebsiteGroup -Id 3526 -ParentGroupId 77 I'm sure my syntax is not correct somewhere, but I'm drawing a blank on what it is. Thanks for your help.NOC Rollup Status Dashboards for MSPs
LM doesn't come with it out of the box, so I built the NOC Dashboard I've wanted. It provides high level, at-a-glance health indicators for each of our client environments we manage. This makes a great "big board" for a NOC room or a second screen status board for work from home NOC/Support folks. I do have three examples in this code for ways to filter for specific teams/purposes. This all collapses for ease of reference correctly in Powershell ISE on windows. Line 282 references a dataSource I wrote that counts frequency of specific eventlog events to illustrate potential brute force attempts (CTM are my initials, we tag our scripts to make finding the best source of answers faster in the future - old habit from pen & paper change logs from a previous job). As any screenshots would contain client names, I'm unable to post any screen shots of the results of this, but my current settings for my Main dashboard are (This is the first Dashboard I've made that looks better in UIv4 than 3): ... #!!! These two need to be changed. First is a string, second an integer #!!! See the comment block below for instructions # The first chunk of your company's logicmonitor URL $company = "yourCompanyNameHere" # ID of the group to be used as a source for the NOC widget items $parentGroupID = <parentGroupID> <# Netgain Technology, llc ( https://netgaincloud.com ) 2/26/2024 - Developed by Cole McDonald Disclaimer: Neither Netgain nor Cole McDonald are not responsible for any unexpected results this script may cause in your environment. To deploy this: - COLLECTOR: you will need a collector for scripting, this will be the single applies to target. You may need to increase the script timeout depending on the size of your device deployment. - DASHBOARD: you will need a Dashboard with a NOC widget on it. The name can be whatever you'd like, there will be a name change in the "name" property for the initial array. In the case of the first example here, "NOC - Master" - PARENT GROUP: you will need to identify the ID# of the group you wish to use as the source for the subgroup list and set the $parentGroupID to the appropriate ID# Purpose: Create an auto-updating high level NOC dashboard that can show - Rollup state for a list of client subgroups from our \Clients group - Group Indicators for a specific dataSource - Group indicators for a subset of devices within each group After the API region, there are three separate dashboards referenced to illustrate the 3 methods for using this dataSource. NOTE: my code uses backticks for line continuation. Where possible in my code, each line indicates a single piece of information about the script's algorithm and the first character in each line from a block indicates the line's relationship to the one above it. #> #region Rest API Initialization and Functions # Init variables used in the RESTApi functions $URLBase = "https://$company.logicmonitor.com/santaba/rest" $accessID = "##ApiAccessID.key##" $accessKey = "##ApiAccessKey.key##" #-------- The Functions ---------- function Send-Request { param ( $cred , $URL , $accessid = $null, $accesskey = $null, $data = $null, $version = '3' , $httpVerb = "GET" ) if ( $accessId -eq $null) { exit 1 } <# Use TLS 1.2 #> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 <# Get current time in milliseconds #> $epoch = [Math]::Round( ( New-TimeSpan ` -start (Get-Date -Date "1/1/1970") ` -end (Get-Date).ToUniversalTime()).TotalMilliseconds ) <# Concatenate Request Details #> $requestVars = $httpVerb + $epoch + $data + $resourcePath <# Construct Signature #> $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes( $accessKey ) $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) ) $signatureHex = [System.BitConverter]::ToString( $signatureBytes ) -replace '-' $signature = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) ) <# Construct Headers #> $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add( "Authorization", $auth ) $headers.Add( "Content-Type" , 'application/json' ) # uses version 2 of the API $headers.Add( "X-version" , $version ) <# Make Request #> $response = Invoke-RestMethod ` -Uri $URL ` -Method $httpVerb ` -Body $data ` -Header $headers ` -erroraction SilentlyContinue ` -warningaction SilentlyContinue Return $response } function Get-LMRestAPIObjectListing { param ( $URLBase , $resourcePathRoot , # "/device/devices" $size = 1000 , $accessKey , $accessId , $version = '2' ) $output = @() $looping = $true $counter = 0 while ($looping) { #re-calc offset based on iteration $offset = $counter * $size $resourcePath = $resourcePathRoot $queryParam = "?size=$size&offset=$offset" $url = $URLBase + $resourcePath + $queryParam # Make Request $response = Send-Request ` -accesskey $accessKey ` -accessid $accessId ` -URL $url ` -version $version if ( $response.items.count -eq $size ) { # Return set is full, more items to retrieve $output += $response.items $counter++ } elseif ( $response.items.count -gt 0 ) { # Return set is not full, store date, end loop $output += $response.items $looping = $false } else { # Return set is empty, no data to store, end loop $looping = $false } } write-output $output } # Get Dashboards $resourcePath = "/dashboard/dashboards" $dashboards = Get-LMRestAPIObjectListing ` -resourcePathRoot $resourcePath ` -accessKey $accessKey ` -accessId $accessID ` -URLBase $URLBase # Get Widgets $resourcePath = "/dashboard/widgets" $widgets = Get-LMRestAPIObjectListing ` -resourcePathRoot $resourcePath ` -accessKey $accessKey ` -accessId $accessID ` -URLBase $URLBase # Get Groups $resourcePath = "/device/groups" $Groups = Get-LMRestAPIObjectListing ` -resourcePathRoot $resourcePath ` -accessKey $accessKey ` -accessId $accessID ` -URLBase $URLBase #endregion function generateJSON { param( $dashInfo, $clientnames, $deviceDisplayName = "*", $DSDisplayName = "*" ) $itemArray = @() foreach ($name in $clientnames) { $itemArray += @{ "type" = "device" "deviceGroupFullPath" = "Clients/$name" "deviceDisplayName" = $deviceDisplayName "dataSourceDisplayName" = $DSDisplayName "instanceName" = "*" "dataPointName" = "*" "groupBy" = "deviceGroup" "name" = "`#`#RESOURCEGROUP`#`#" } } # Write JSON back to the API for that widget $outputJSON = "`n`t{`n`t`t`"items`" : [`n" foreach ($item in $itemArray) { $elementJSON = @" { `"type`" : `"$($item.type)`", `"dataPointName`" : `"$($item.dataPointName)`", `"instanceName`" : `"$($item.instanceName)`", `"name`" : `"$($item.name)`", `"dataSourceDisplayName`" : `"$($item.dataSourceDisplayName)`", `"groupBy`" : `"$($item.groupBy)`", `"deviceGroupFullPath`" : `"$($item.deviceGroupFullPath)`", `"deviceDisplayName`" : `"$($item.deviceDisplayName)`" } "@ if ($item -ne $itemArray[-1]) { $outputJSON += "$elementJSON,`n" } else { # Last Item $outputJSON += "$elementJSON`n`t`t]`n`t}" } } write-output $outputJSON } # Get Client Names from groups $clientnames = ( $groups ` | where parentid -eq $parentGroupID ` | where name -notmatch "^\." ).name | sort #ID Master Dashboard # declare dashboard name and set default id and widgetid to use in the loop later $masterDash = @{ id=0; widgetid=0; name="NOC - Master" } $master = $dashboards | ? name -eq $masterDash.name if (($master.name).count -eq 1) { $masterDash.id = $master.id $masterDash.widgetid = $master.widgetsConfig[0].psobject.Properties.name $outputJSON = generateJSON ` -dashInfo $masterDash ` -clientnames $clientnames $resourcePath = "/dashboard/widgets/$($masterDash.widgetid)" $url = $URLBase + $resourcePath $widget = Send-Request ` -accessKey $accessKey ` -accessId $accessID ` -data $outputJSON ` -URL $URL ` -httpVerb "PATCH" } #ID Network Dashboard # declare dashboard name and set default id and widgetid to use in the loop later $networkDash = @{ id=0; widgetid=0; name="NOC - Network" } # preset filters for specific dashboard targeting by device $networkDeviceDisplayNameString = "*(meraki|kemp)*" $network = $dashboards | ? name -eq $networkDash.name if (($network.name).count -eq 1) { $networkDash.id = $network.id $networkDash.widgetid = $network.widgetsConfig[0].psobject.Properties.name $outputJSON = generateJSON ` -dashInfo $networkDash ` -clientnames $clientnames ` -deviceDisplayName $networkDeviceDisplayNameString $resourcePath = "/dashboard/widgets/$($networkDash.widgetid)" $url = $URLBase + $resourcePath $widget = Send-Request ` -accessKey $accessKey ` -accessId $accessID ` -data $outputJSON ` -URL $URL ` -httpVerb "PATCH" } #ID Security Dashboard # declare dashboard name and set default id and widgetid to use in the loop later $securityDash = @{ id=0; widgetid=0; name="NOC - Security" } # preset filters for specific dashboard targeting by datasource $securityDataSourceDisplayNameString = "Event Frequency Sec:4625 CTM" $security = $dashboards | ? name -eq $securityDash.name if (($security.name).count -eq 1) { $securityDash.id = $security.id $securityDash.widgetid = $security.widgetsConfig[0].psobject.Properties.name $outputJSON = generateJSON ` -dashInfo $securityDash ` -clientnames $clientnames ` -DSDisplayName $securityDataSourceDisplayNameString $resourcePath = "/dashboard/widgets/$($securityDash.widgetid)" $url = $URLBase + $resourcePath $widget = Send-Request ` -accessKey $accessKey ` -accessId $accessID ` -data $outputJSON ` -URL $URL ` -httpVerb "PATCH" }
Example script for automated alert actions via External Alerting
Below is a PowerShell script that's a handy starting point if you want to trigger actions based on specific alert types. In a nutshell, it takes a number of parameters from each alertand has a section of if/elsestatements where you can specify what to do based on the alert.It leverages LogicMonitor'sExternal Alertingfeature so the script runs local to whatever Collector(s)you configure it on. I included a couple of example actions forpinging a device and forrestarting a service.It also includes some handy (optional) functions for logging as well as attaching a noteto thealert in LogicMonitor. NOTE: this script is provided as-is and you will need to customize it to suit your needs. Automated actions are something that must be approached with careful planning and caution!! LogicMonitor cannot be responsible for inadvertent consequences of using thisscript. If you want try it out, here's how to get started: Update the variables in the appropriate section near the top of the script with optional API credentialsand/or log settings. Also change any of the if/elseif statements (starting around line #95) to suit your needs. Save the script onto your Collector server.I named the file"alert_central.ps1" but feel free to call it something else. Make note of it’s full path (ex: “C:\scripts\alert_central.ps1”). NOTE: it’s notrecommended to place it under the Collector's agent/lib directory (typically "C:\Program Files (x86)\LogicMonitor\Agent\lib") since that location can be overwritten by collector upgrades. In your LogicMonitor portal go to Settings, then External Alerting. Click the Add button. Set the 'Groups' field as needed to limit the actions to alerts from any appropriategroup of resources. (Be sure the group's devices would be reachable from the Collector running the script) Choose the appropriate Collector in the Collectorfield. Set Delivery Mechanismto "Script" Enter the name you saved the scriptas (in step #2)in theScriptfield (ex. "alert_central.ps1"). Paste the following into the Script Command Linefield (NOTE: if you add other parameters here then be sure to also add them to the 'Param' line at the top of the script): "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##""##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" Example of the completed Add External Alerting dialog Click Save. This uses LogicMonitor's External Alerting featureso there are some things to be aware of: Since the script is called foreveryalert, the section of if/then statements at the bottom of the script is important for filtering what specific alerts you want to take action on. The Collector(s) oversee the running of thescript, so be conscience to any additional overhead the script actions may cause. It could take up to 60 seconds for the script to trigger from the time the alert comes in. This example is a PowerShell script so best suited for Windows-based collectors, but could certainly be re-written as a shell script for Linux-based collectors. Here's a screenshot of acleared alert where the script auto-restarted a Windows service and attached a note based on its actions. Example note the script added to the alert reflecting the automated action that was taken Below is the PowerShell script: # ---- # This PowerShell script can be used as a starting template for enabling # automated remediation for alerts coming from LogicMonitor. # In LogicMonitor, you can use the External Alerting feature to pass all alerts # (or for a specific group of resources) to this script. # ---- # To use this script: # 1. Update the variables in the appropriate section below with optional API and log settings. # 2. Drop this script onto your Collector server under the Collector's agent/lib directory. # 3. In your LogicMonitor portal go to Settings, then click External Alerting. # 4. Click the Add button. # 5. Set the 'Groups' field as needed to limit the actions to a specific group of resources. # 6. Choose the appropriate Collector in the 'Collector' field. # 7. Set 'Delivery Mechanism' to "Script" # 8. Enter "alert_central.ps1" in the 'Script' field. # 9. Paste the following into the 'Script Command Line' field: # "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##" "##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" # 10. Click Save. # The following line captures alert information passed from LogicMonitor (defined in step #9 above)... Param ($alertID = "", $alertStatus = "", $severity = "", $hostName = "", $sysName = "", $dsName = "", $instance = "", $datapoint = "", $metricValue = "", $alertURL = "", $dpDescription = "") ###--- SET THE FOLLOWING VARIABLES AS APPROPRIATE ---### # OPTIONAL: LogicMonitor API info for updating alert notes (the API user will need "Acknowledge" permissions)... $accessId = '' $accessKey = '' $company = '' # OPTIONAL: Set a filename in the following variable if you want specific alerts logged. (example: "C:\lm_alert_central.log")... $logFile = '' # OPTIONAL: Destination for syslog alerts... $syslogServer = '' ############################################################### ## HELPER FUNCTIONS (you likely won't need to change these) ## # Function for logging the alert to a local text file if one was specified in the $logFile variable above... Function LogWrite ($logstring = "") { if ($logFile -ne "") { $tmpDate = Get-Date -Format "dddd MM/dd/yyyy HH:mm:ss" # Using a mutex to handle file locking if multiple instances of this script trigger at once... $LogMutex = New-Object System.Threading.Mutex($false, "LogMutex") $LogMutex.WaitOne()|out-null "$tmpDate, $logstring" | out-file -FilePath $logFile -Append $LogMutex.ReleaseMutex()|out-null } } # Function for attaching a note to the alert... function AddNoteToAlert ($alertID = "", $note = "") { # Only execute this if the appropriate API information has been set above... if ($accessId -ne '' -and $accessKey -ne '' -and $company -ne '') { # Encode the note... $encodedNote = $note | ConvertTo-Json # API and URL request details... $httpVerb = 'POST' $resourcePath = '/alert/alerts/' + $alertID + '/note' $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath $data = '{"ackComment":' + $encodedNote + '}' # Get current time in milliseconds... $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) # Concatenate general request details... $requestVars_00 = $httpVerb + $epoch + $data + $resourcePath # Construct signature... $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars_00)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) # Construct headers... $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') # Make request to add note.. $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers # Change the following if you want to capture API errors somewhere... # LogWrite "API call response: $response" } } function SendTo-SysLog ($IP = "", $Facility = "local7", $Severity = "notice", $Content = "Your payload...", $SourceHostname = $env:computername, $Tag = "LogicMonitor", $Port = 514) { switch -regex ($Facility) { 'kern' {$Facility = 0 * 8 ; break } 'user' {$Facility = 1 * 8 ; break } 'mail' {$Facility = 2 * 8 ; break } 'system' {$Facility = 3 * 8 ; break } 'auth' {$Facility = 4 * 8 ; break } 'syslog' {$Facility = 5 * 8 ; break } 'lpr' {$Facility = 6 * 8 ; break } 'news' {$Facility = 7 * 8 ; break } 'uucp' {$Facility = 8 * 8 ; break } 'cron' {$Facility = 9 * 8 ; break } 'authpriv' {$Facility = 10 * 8 ; break } 'ftp' {$Facility = 11 * 8 ; break } 'ntp' {$Facility = 12 * 8 ; break } 'logaudit' {$Facility = 13 * 8 ; break } 'logalert' {$Facility = 14 * 8 ; break } 'clock' {$Facility = 15 * 8 ; break } 'local0' {$Facility = 16 * 8 ; break } 'local1' {$Facility = 17 * 8 ; break } 'local2' {$Facility = 18 * 8 ; break } 'local3' {$Facility = 19 * 8 ; break } 'local4' {$Facility = 20 * 8 ; break } 'local5' {$Facility = 21 * 8 ; break } 'local6' {$Facility = 22 * 8 ; break } 'local7' {$Facility = 23 * 8 ; break } default {$Facility = 23 * 8 } #Default is local7 } switch -regex ($Severity) { '^(ac|up)' {$Severity = 1 ; break } # LogicMonitor "active", "ack" or "update" '^em' {$Severity = 0 ; break } #Emergency '^a' {$Severity = 1 ; break } #Alert '^c' {$Severity = 2 ; break } #Critical '^er' {$Severity = 3 ; break } #Error '^w' {$Severity = 4 ; break } #Warning '^n' {$Severity = 5 ; break } #Notice '^i' {$Severity = 6 ; break } #Informational '^d' {$Severity = 7 ; break } #Debug default {$Severity = 5 } #Default is Notice } $pri = "<" + ($Facility + $Severity) + ">" # Note that the timestamp is local time on the originating computer, not UTC. if ($(get-date).day -lt 10) { $timestamp = $(get-date).tostring("MMM d HH:mm:ss") } else { $timestamp = $(get-date).tostring("MMM dd HH:mm:ss") } # Hostname does not have to be in lowercase, and it shouldn't have spaces anyway, but lowercase is more traditional. # The name should be the simple hostname, not a fully-qualified domain name, but the script doesn't enforce this. $header = $timestamp + " " + $sourcehostname.tolower().replace(" ","").trim() + " " #Cannot have non-alphanumerics in the TAG field or have it be longer than 32 characters. if ($tag -match '[^a-z0-9]') { $tag = $tag -replace '[^a-z0-9]','' } #Simply delete the non-alphanumerics if ($tag.length -gt 32) { $tag = $tag.substring(0,31) } #and truncate at 32 characters. $msg = $pri + $header + $tag + ": " + $content # Convert message to array of ASCII bytes. $bytearray = $([System.Text.Encoding]::ASCII).getbytes($msg) # RFC3164 Section 4.1: "The total length of the packet MUST be 1024 bytes or less." # "Packet" is not "PRI + HEADER + MSG", and IP header = 20, UDP header = 8, hence: if ($bytearray.count -gt 996) { $bytearray = $bytearray[0..995] } # Send the message... $UdpClient = New-Object System.Net.Sockets.UdpClient $UdpClient.Connect($IP,$Port) $UdpClient.Send($ByteArray, $ByteArray.length) | out-null } # Empty placeholder for capturing any note we might want to attach back to the alert... $alertNote = "" # Placeholder for whether we want to capture an alert in our log. Set to true if you want to log everything. $logThis = $false ############################################################### ## CUSTOMIZE THE FOLLOWING AS NEEDED TO HANDLE SPECIFIC ALERTS FROM LOGICMONITOR... # Actions to take if the alert is new or re-opened (note: status will be "active" or "clear")... if ($alertStatus -eq 'active') { # Perform actions based on the type of alert... # Ping alerts... if ($dsName -eq 'Ping' -and $datapoint -eq 'PingLossPercent') { # Insert action to take if a device becomes unpingable. In this example we'll do a verification ping & capture the output... $job = ping -n 4 $sysName # Restore line feeds to the output... $job = [string]::join("`n", $job) # Add ping results as a note on the alert... $alertNote = "Automation script output: $job" # Log the alert... $logThis = $true # Restart specific Windows services... } elseif ($dsName -eq 'WinService-' -and $datapoint -eq 'State') { # List of Windows Services to match against. Only if one of the following are alerting will we try to restart it... $serviceList = @("Print Spooler","Service 2") # Note: The PowerShell "-Contains" operator is exact in it's matching. Replace it with "-Match" for a loser match. if ($serviceList -Contains $instance) { # Get an object reference to the Windows service... $tmpService = Get-Service -DisplayName "$instance" -ComputerName $sysName # Only trigger if the service is still stopped... if ($tmpService.Status -eq "Stopped") { # Start the service... $tmpService | Set-Service -Status Running # Capture the current state of the service as a note on the alert... $alertNote = "Attempted to auto-restart the service. Its new status is " + $tmpService.Status + "." } # Log the alert... $logThis = $true } # Actions to take if a website stops responding... } elseif ($dsName -eq 'HTTPS-' -and $datapoint -eq 'CantConnect') { # Insert action here to take if there's a website error... # Example of sending a syslog message to an external server... $syslogMessage = "AlertID:$alertID,Host:$sysName,AlertStatus:$alertStatus,LogicModule:$dsName,Instance:$instance,Datapoint:$datapoint,Value:$metricValue,AlertDescription:$dpDescription" SendTo-SysLog $syslogServer "" $severity $syslogMessage $hostName "" "" # Attach a note to the LogicMonitor alert... $alertNote = "Sent syslog message to " + $syslogServer # Log the alert... $logThis = $true } } ############################################################### ## Final functions for backfilling notes and/or logging as needed ## (you likely won't need to change these) # Section that updates the LogicMonitor alert if 'alertNote' is not empty... if ($alertNote -ne "") { AddNoteToAlert $alertID $alertNote } if ($logThis) { # Log the alert (only triggers if a filename is given in the $logFile variable near the top of this script)... LogWrite "$alertID,$alertStatus,$severity,$hostName,$sysName,$dsName,$instance,$datapoint,$metricValue,$alertURL,$dpDescription" }
Finding the culprit for TCP_StatsCollector ConnectionsEstablished alert for Windows collectors
From the collector’s device page in the LM Portal or the collectors page, get to a debug console, then here’s your !POSH one-liner to get info about the destination device that is holding your ports captive. netstat -an| sls establish | foreach { ($_ -split "\s+")[3] } | group | sort count | select count, name -last 10 In the Netstat, a shows all, n shows IP addresses rather than solving the DNS for it. TheSelect-String (aliased as sls)passes only the “Established” connection entries from the netstat down the pipeline. The foreach{} splits each line ($_ is the current object being iterated by the foreach loop) on contiguous whitespace (I use this a lot!) and takes the third element (remote address:port) to passdown the pipeline It then passes Group-Object (aliased as group) which bundles identical strings and Sort-Object (aliased as sort)by the count property of the group object. The select displays grabs the calculated match count and the name properties to limit display and just shows the -last 10 of them (which are the biggest number of matched lines due to the sort previously applied. This should give you the target/s for troubleshooting further.
