Forum Discussion

Pooja's avatar
5 years ago

How to monitor opensource elasticsearch in Logic Monitor

Hello All,

I have installed opensource(Not AWS) elasticsearch on my servers. Now I want to monitor the elasticsearch parameters in Logic Monitor. How can i do that? Are there any plugins available to monitor elasticsearch in LM. I found an article for monitoring AWS Elasticsearch through Cloudwatch , but nothing other than that. Any help would be greatly appreciated.

Thanks in advance!!

  •  

    We have 3 datasources that we use for our internal Elk clusters.  Maybe they can help you or at least point you in the right direction for writing your own.  This is the one for shard health.

    <?xml version="1.0" encoding="UTF-8" ?>
    <feed  version="1.0" hasPendingRequests="false" >
      <company></company>
      <status>200</status>
      <errmsg>OK</errmsg>
      <interval>0</interval>
        <entry type="predatasource">
            <version>1573572263</version>
            <name>Elasticsearch_Cluster_Shards_veri</name>
            <displayedas>Cluster Shards_veri</displayedas>
            <description>Monitors Elasticsearch Cluster shard health.</description>
            <collector>webpage</collector>
            <hasMultiInstances>false</hasMultiInstances>
            <schedule>60</schedule>
            <appliesTo>hasCategory(&#34;elasticsearch&#34;) or system.deviceId == &#34;6986&#34;</appliesTo>
            <wildcardauto>false</wildcardauto>
            <wildcardpersist>false</wildcardpersist>
            <wildcardlinuxscript></wildcardlinuxscript>
            <wildcardlinuxcmdline></wildcardlinuxcmdline>
            <wildcardwinscript></wildcardwinscript>
            <wildcardwincmdline></wildcardwincmdline>
            <wildcardgroovyscript></wildcardgroovyscript>
            <wildcardschedule>1440</wildcardschedule>
            <wildcarddisable>false</wildcarddisable>
            <wildcarddeleteinactive>false</wildcarddeleteinactive>
            <agdmethod>none</agdmethod>
            <agdparams></agdparams>
            <group></group>
            <tags>cluster,shards,elasticsearch,health</tags>
            <technology></technology>
            <adlist><![CDATA[{"agdmethod":"none","agdparams":"","id":0,"filters":[],"params":{}}]]></adlist>
            <schemaVersion>2</schemaVersion>
            <dataSourceType>1</dataSourceType>
            <attributes>
            <attribute>
                <name>ip</name>
                <value></value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>port</name>
                <value>9200</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>followRedirect</name>
                <value>false</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>connectTimeout</name>
                <value>1000</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>readTimeout</name>
                <value>1000</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>request</name>
                <value>GET /_cluster/health?pretty&#38;human HTTP/1.0</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>ssl</name>
                <value>N</value>
                <comment></comment>
            </attribute>
            </attributes>
            <datapoints>
            <datapoint>
                <name>active_primary_shards</name>
                <dataType>7</dataType>
                <type>2</type>
                <postprocessormethod>json</postprocessormethod>
                <postprocessorparam>active_primary_shards</postprocessorparam>
                <usevalue>body</usevalue>
                <alertexpr></alertexpr>
                <alertmissing>1</alertmissing>
                <alertsubject></alertsubject>
                <alertbody></alertbody>
                <enableanomalyalertsuppression></enableanomalyalertsuppression>
                <adadvsettingenabled>false</adadvsettingenabled>
                <warnadadvsetting></warnadadvsetting>
                <erroradadvsetting></erroradadvsetting>
                <criticaladadvsetting></criticaladadvsetting>
                <description>Number of active primary shards.</description>
                <maxvalue></maxvalue>
                <minvalue>0</minvalue>
                <userparam1></userparam1>
                <userparam2></userparam2>
                <userparam3></userparam3>
                <iscomposite>false</iscomposite>
                <rpn></rpn>
                <alertTransitionIval>0</alertTransitionIval>
                <alertClearTransitionIval>0</alertClearTransitionIval>
            </datapoint>
            <datapoint>
                <name>active_shards</name>
                <dataType>7</dataType>
                <type>2</type>
                <postprocessormethod>json</postprocessormethod>
                <postprocessorparam>active_shards</postprocessorparam>
                <usevalue>body</usevalue>
                <alertexpr></alertexpr>
                <alertmissing>1</alertmissing>
                <alertsubject></alertsubject>
                <alertbody></alertbody>
                <enableanomalyalertsuppression></enableanomalyalertsuppression>
                <adadvsettingenabled>false</adadvsettingenabled>
                <warnadadvsetting></warnadadvsetting>
                <erroradadvsetting></erroradadvsetting>
                <criticaladadvsetting></criticaladadvsetting>
                <description>Number of active shards.</description>
                <maxvalue></maxvalue>
                <minvalue>0</minvalue>
                <userparam1></userparam1>
                <userparam2></userparam2>
                <userparam3></userparam3>
                <iscomposite>false</iscomposite>
                <rpn></rpn>
                <alertTransitionIval>0</alertTransitionIval>
                <alertClearTransitionIval>0</alertClearTransitionIval>
            </datapoint>
            <datapoint>
                <name>initializing_shards</name>
                <dataType>7</dataType>
                <type>2</type>
                <postprocessormethod>json</postprocessormethod>
                <postprocessorparam>initializing_shards</postprocessorparam>
                <usevalue>body</usevalue>
                <alertexpr></alertexpr>
                <alertmissing>1</alertmissing>
                <alertsubject></alertsubject>
                <alertbody></alertbody>
                <enableanomalyalertsuppression></enableanomalyalertsuppression>
                <adadvsettingenabled>false</adadvsettingenabled>
                <warnadadvsetting></warnadadvsetting>
                <erroradadvsetting></erroradadvsetting>
                <criticaladadvsetting></criticaladadvsetting>
                <description>Number of initializing shards.</description>
                <maxvalue></maxvalue>
                <minvalue>0</minvalue>
                <userparam1></userparam1>
                <userparam2></userparam2>
                <userparam3></userparam3>
                <iscomposite>false</iscomposite>
                <rpn></rpn>
                <alertTransitionIval>0</alertTransitionIval>
                <alertClearTransitionIval>0</alertClearTransitionIval>
            </datapoint>
            <datapoint>
                <name>relocating_shards</name>
                <dataType>7</dataType>
                <type>2</type>
                <postprocessormethod>json</postprocessormethod>
                <postprocessorparam>relocating_shards</postprocessorparam>
                <usevalue>body</usevalue>
                <alertexpr></alertexpr>
                <alertmissing>1</alertmissing>
                <alertsubject></alertsubject>
                <alertbody></alertbody>
                <enableanomalyalertsuppression></enableanomalyalertsuppression>
                <adadvsettingenabled>false</adadvsettingenabled>
                <warnadadvsetting></warnadadvsetting>
                <erroradadvsetting></erroradadvsetting>
                <criticaladadvsetting></criticaladadvsetting>
                <description>Number of relocating shards.</description>
                <maxvalue></maxvalue>
                <minvalue>0</minvalue>
                <userparam1></userparam1>
                <userparam2></userparam2>
                <userparam3></userparam3>
                <iscomposite>false</iscomposite>
                <rpn></rpn>
                <alertTransitionIval>0</alertTransitionIval>
                <alertClearTransitionIval>0</alertClearTransitionIval>
            </datapoint>
            <datapoint>
                <name>unassigned_shards</name>
                <dataType>7</dataType>
                <type>2</type>
                <postprocessormethod>json</postprocessormethod>
                <postprocessorparam>unassigned_shards</postprocessorparam>
                <usevalue>body</usevalue>
                <alertexpr></alertexpr>
                <alertmissing>1</alertmissing>
                <alertsubject></alertsubject>
                <alertbody></alertbody>
                <enableanomalyalertsuppression></enableanomalyalertsuppression>
                <adadvsettingenabled>false</adadvsettingenabled>
                <warnadadvsetting></warnadadvsetting>
                <erroradadvsetting></erroradadvsetting>
                <criticaladadvsetting></criticaladadvsetting>
                <description>Number of unassigned shards.</description>
                <maxvalue></maxvalue>
                <minvalue></minvalue>
                <userparam1></userparam1>
                <userparam2></userparam2>
                <userparam3></userparam3>
                <iscomposite>false</iscomposite>
                <rpn></rpn>
                <alertTransitionIval>0</alertTransitionIval>
                <alertClearTransitionIval>0</alertClearTransitionIval>
            </datapoint>
            </datapoints>
            <graphs>
            <graph>
                <name>Shards</name>
                <title>Shards</title>
                <verticallabel>shards</verticallabel>
                <rigid>false</rigid>
                <maxvalue>NaN</maxvalue>
                <minvalue>0.0</minvalue>
                <displayprio>1</displayprio>
                <timescale>1day</timescale>
                <base1024>false</base1024>
                <graphdatapoints>
            <graphdatapoint>
                <name>active_primary_shards</name>
                <datapointname>active_primary_shards</datapointname>
                <cf>2</cf>
            </graphdatapoint>
            <graphdatapoint>
                <name>active_shards</name>
                <datapointname>active_shards</datapointname>
                <cf>2</cf>
            </graphdatapoint>
            <graphdatapoint>
                <name>initializing_shards</name>
                <datapointname>initializing_shards</datapointname>
                <cf>2</cf>
            </graphdatapoint>
            <graphdatapoint>
                <name>relocating_shards</name>
                <datapointname>relocating_shards</datapointname>
                <cf>2</cf>
            </graphdatapoint>
            <graphdatapoint>
                <name>unassigned_shards</name>
                <datapointname>unassigned_shards</datapointname>
                <cf>2</cf>
            </graphdatapoint>
                </graphdatapoints>
                <graphvirtualdatapoints>
                </graphvirtualdatapoints>
                <graphdatas>
                <graphdata>
                    <type>1</type>
                    <legend>Active Primary</legend>
                    <color>red</color>
                    <datapointname>active_primary_shards</datapointname>
                    <isvirtualdatapoint>false</isvirtualdatapoint>
                </graphdata>
                <graphdata>
                    <type>1</type>
                    <legend>Active</legend>
                    <color>orange</color>
                    <datapointname>active_shards</datapointname>
                    <isvirtualdatapoint>false</isvirtualdatapoint>
                </graphdata>
                <graphdata>
                    <type>1</type>
                    <legend>Initializing</legend>
                    <color>yellow</color>
                    <datapointname>initializing_shards</datapointname>
                    <isvirtualdatapoint>false</isvirtualdatapoint>
                </graphdata>
                <graphdata>
                    <type>1</type>
                    <legend>Relocating</legend>
                    <color>green</color>
                    <datapointname>relocating_shards</datapointname>
                    <isvirtualdatapoint>false</isvirtualdatapoint>
                </graphdata>
                <graphdata>
                    <type>1</type>
                    <legend>Unassigned</legend>
                    <color>blue</color>
                    <datapointname>unassigned_shards</datapointname>
                    <isvirtualdatapoint>false</isvirtualdatapoint>
                </graphdata>
                </graphdatas>
            </graph>
            </graphs>
            <overviewgraphs>
            </overviewgraphs>
            <scripts>
            </scripts>
        </entry>
    </feed>

     

  • Last one for cluster health status.

    <?xml version="1.0" encoding="UTF-8" ?>
    <feed  version="1.0" hasPendingRequests="false" >
      <company></company>
      <status>200</status>
      <errmsg>OK</errmsg>
      <interval>0</interval>
        <entry type="predatasource">
            <version>1573572264</version>
            <name>Elasticsearch_Cluster_Health_Status_veri</name>
            <displayedas>Cluster Health Status_veri</displayedas>
            <description>Elasticsearch Cluster status: 2 == green, 1 == yellow, 0 == red.</description>
            <collector>script</collector>
            <hasMultiInstances>false</hasMultiInstances>
            <schedule>60</schedule>
            <appliesTo>hasCategory(&#34;elasticsearch&#34;) or system.deviceId == &#34;6986&#34;</appliesTo>
            <wildcardauto>false</wildcardauto>
            <wildcardpersist>false</wildcardpersist>
            <wildcardlinuxscript></wildcardlinuxscript>
            <wildcardlinuxcmdline></wildcardlinuxcmdline>
            <wildcardwinscript></wildcardwinscript>
            <wildcardwincmdline></wildcardwincmdline>
            <wildcardgroovyscript></wildcardgroovyscript>
            <wildcardschedule>1440</wildcardschedule>
            <wildcarddisable>false</wildcarddisable>
            <wildcarddeleteinactive>false</wildcarddeleteinactive>
            <agdmethod>none</agdmethod>
            <agdparams></agdparams>
            <group></group>
            <tags>cluster,elasticsearch,health,status</tags>
            <technology></technology>
            <adlist><![CDATA[{"agdmethod":"none","agdparams":"","id":0,"filters":[],"params":{}}]]></adlist>
            <schemaVersion>2</schemaVersion>
            <dataSourceType>1</dataSourceType>
            <attributes>
            <attribute>
                <name>scripttype</name>
                <value>embed</value>
                <comment></comment>
            </attribute>
            <attribute>
                <name>scriptgroovy</name>
                <value>import groovy.json.JsonSlurper
    import org.apache.http.client.methods.HttpGet
    import org.apache.http.conn.ssl.SSLConnectionSocketFactory
    import org.apache.http.conn.ssl.SSLContextBuilder
    import org.apache.http.conn.ssl.TrustSelfSignedStrategy
    import org.apache.http.impl.client.HttpClients
    import org.apache.http.util.EntityUtils
    
    def hostname = hostProps.get(&#34;system.hostname&#34;)
    def slurper = new JsonSlurper()
    
    def cluster_status_map = [&#34;green&#34; : 2,
                              &#34;yellow&#34;: 1,
                              &#34;red&#34;   : 0]
    
    def request_endpoint = &#34;http://${hostname}:9200/_cluster/health&#34;
    def request_headers = [&#34;User-Agent&#34;  : &#34;LogicMonitor Polling Agent&#34;,
                           &#34;Content-Type&#34;: &#34;application/json&#34;];
    
    /*
     * http_get - do an HTTP GET with apache http client
     *  @param string request_endpoint
     *  @param map request_headers
     *  @return map [code,status,body]
     */
    
    def http_get(request_endpoint, request_headers)
    {
        def http_get = new HttpGet(request_endpoint);
        // what type of request?
        if (request_endpoint.startsWith(&#39;https:&#39;))
        {
            // https -- create an ssl-safe client
            ssl_builder = new SSLContextBuilder();
            ssl_builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
            ssl_sf = new SSLConnectionSocketFactory(ssl_builder.build(), SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
            client = HttpClients.custom().setSSLSocketFactory(ssl_sf).build();
        }
        else
        {
            // vanilla http -- use the default client
            client = HttpClients.createDefault();
        }
        // iterate over request headers
        request_headers.each()
        { key, value -&#62;
            http_get.setHeader(key, value);
        }
        def http_response = client.execute(http_get);
        def response_message = http_response.getEntity();
        def response_status = http_response.getStatusLine();
        def response_code = response_status.getStatusCode();
        def response_body = EntityUtils.toString(response_message);
        http_response.close();
        return 
    ;
    }
    
    // Make HTTP GET call
    def get_response = http_get(request_endpoint, request_headers);
    
    // Parse result to JSON
    def result = slurper.parseText(get_response[&#39;body&#39;])
    
    // Print the results
    println cluster_status_map[result.status]
    
    return 0;
    </value> <comment></comment> </attribute> <attribute> <name>windowsscript</name> <value></value> <comment></comment> </attribute> <attribute> <name>linuxscript</name> <value></value> <comment></comment> </attribute> <attribute> <name>windowscmdline</name> <value></value> <comment></comment> </attribute> <attribute> <name>linuxcmdline</name> <value></value> <comment></comment> </attribute> </attributes> <datapoints> <datapoint> <name>cluster_status</name> <dataType>7</dataType> <type>2</type> <postprocessormethod>none</postprocessormethod> <postprocessorparam></postprocessorparam> <usevalue>output</usevalue> <alertexpr>= 1 0</alertexpr> <alertmissing>1</alertmissing> <alertsubject></alertsubject> <alertbody>The ElasticSearch Cluster at ##HOST## is reporting a state of ##VALUE## putting it in a ##LEVEL## state. The cluster has been in this state since ##START## or for ##DURATION##.</alertbody> <enableanomalyalertsuppression></enableanomalyalertsuppression> <adadvsettingenabled>false</adadvsettingenabled> <warnadadvsetting></warnadadvsetting> <erroradadvsetting></erroradadvsetting> <criticaladadvsetting></criticaladadvsetting> <description>ElasticSearch Cluster status: 2 = green, 1 = yellow, 0 = red.</description> <maxvalue>2</maxvalue> <minvalue>0</minvalue> <userparam1></userparam1> <userparam2></userparam2> <userparam3></userparam3> <iscomposite>false</iscomposite> <rpn></rpn> <alertTransitionIval>1</alertTransitionIval> <alertClearTransitionIval>0</alertClearTransitionIval> </datapoint> </datapoints> <graphs> <graph> <name>Elasticsearch Cluster Health Status</name> <title>Elasticsearch Cluster Health Status</title> <verticallabel>status</verticallabel> <rigid>false</rigid> <maxvalue>2.0</maxvalue> <minvalue>0.0</minvalue> <displayprio>1</displayprio> <timescale>1day</timescale> <base1024>false</base1024> <graphdatapoints> <graphdatapoint> <name>cluster_status</name> <datapointname>cluster_status</datapointname> <cf>1</cf> </graphdatapoint> </graphdatapoints> <graphvirtualdatapoints> </graphvirtualdatapoints> <graphdatas> <graphdata> <type>1</type> <legend>Status</legend> <color>blue</color> <datapointname>cluster_status</datapointname> <isvirtualdatapoint>false</isvirtualdatapoint> </graphdata> </graphdatas> </graph> </graphs> <overviewgraphs> </overviewgraphs> <scripts> </scripts> </entry> </feed>

     

  • Anonymous's avatar
    Anonymous

    @Jeff8682, I suggest you submit these to the LM Exchange for easier distribution. This has the added benefit of getting a security review of the code.