Threathunting with Elasticsearch

All kinds of interesting information can be found using the Elasticsearch API. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

I find it easier than using Kibana, but Kibana was necessary to figure out the query language. Using the same queries, you can save out results and sort through them.

The simplest example would be something like this.

#!/bin/bash
curl -XPOST "http://elasticsearch:9200/_search" -d'
{
 "query": {
  "query_string": {
   "query": "*"
  }
 }
}'

That will pull 1024 results from all indices. You can format the results with jq

apt-get install jq

And then pipe this on the end

| jq '.hits.hits[]._source | del(.["type"]) | del(.["@version"])'

Depending on how your indices are setup, you can grab all kinds of useful information.

#!/bin/bash
rm output
rm hits
rm noqualys

window=now-30d

curl -s -XPOST "http://elasticsearch:9200/logstash-*/_search?size=10000" -d'
 {
 "query": {
   "filtered": {
     "query": {
       "query_string": {
         "query": "\"bro_http\" AND \"POST\"",
         "analyze_wildcard": true
         }
       },
       "filter": {
         "bool": {
           "must": [
            {
           "range": {
             "@timestamp": {
               "gte" : "'$window'" 
                }
              }
            }
          ],
          "must_not": []
        }
      }
    }
  }
 }' | jq '.hits.hits[]._source | del(.["type"]) | del(.["@version"])' >> hits

cat hits | egrep -v "64\\.39\\.(11[01]|10[0-9]|9[6-9])\\.(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])" > noqualys
cat noqualys | grep -oP "[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3},514,rslog|signature_id.*?," | grep -B1 signature| grep -v -- "--" | paste -s -d' \n'| sort |uniq -c | sort -nr | sed 's/,514,rslog//g' | sed 's/\"//g' | sed 's/,$//g' | sed 's/\\//g' > output

sed -i -e 's/10\.1\.1\.20/NAME/g' output

while read p; do
 sid=$(echo $p | cut -d ":" -f2);
 sigtotal=$(echo $p; echo -e '\t'; eval "grep sid:\\$sid allsurirules" | grep -oE "msg:.{0,100}"|sed "s/;.*//"); 
 echo $sigtotal; 
done < output

That will pull 10,000 results on the query “bro_http” AND “POST”.  From there, you can filter down to find what you’re looking for and sort through the logs, create counts and metrics, and have the data available to manipulate further. More examples here.