Wednesday, March 19, 2014

toggle Splunk panels

With Splunk version 6 you can easily adjust and extend dashboards with javascript code (i.e. jQuery). You can either convert Simple XML dashboards into plain HTML and include javascript directly in the dashboard source code, or you can keep the  Simple XML format and tell Splunk to inject the javascript into a Splunk panel of type "html".

While the first alternative (plain HTML dashobard) is easy to understand and to implement, it rapidly produces a lot of messy and non-reusable code. Furthermore you cannot convert HTML back to Simple XML. So for example if you like to add a new Splunk panel to your already converted HTML dashboard, you have to implement this panel in HTML as well.... this is quite an error prone and complicated process.

The second alternative (injecting javascript into Simple XML) is a lot cleaner and you can easily reuse the code in other dashboards. However, the initial effort may be a bit higher to implement the javascript code this way.
You have to have a little bit of knowledge about the Splunk programming interfaces and unfortunately that part of Splunk is not really good documented..

The following example is a small jQuery slideToggle implementation to give you a starting point for implementing your own Splunk6  extensions.
The goal is to have a toggle functionality for your Splunk panels. That means you will be able to toggle panels on and off as you like. This helps keeping dashboards clean and tidy.

See the working example on youtube: https://www.youtube.com/watch?v=VbNN9Q41kaw

The Simple XML

let's implement the Simple XML panel first. We start with one row and one chart panel. 
For this example we use a Splunk search that counts events by sourcetype within the last 5 minutes

<dashboard>
  <label>slidepanel</label>
  <row>
    <chart>
      <title>Internal sourcetypes over the last 5 minutes</title>
      <searchString>index=_internal | stats count by sourcetype</searchString>
      <earliestTime>-5m</earliestTime>
      <option name="charting.chart.stackMode">stacked</option>
    </chart>
  </row>
</dashboard>

To inject the javascript code into Simple XML, we will add a custom HTML panel above the existing chart panel. The HTML panel contains calls javascript code and adds separate panel-title and a link to toggle the chart panel on and off

The new Simple XML code looks as follows

<dashboard script="autodiscover.js">
  <label>slidepanel</label>
  <row grouping="2">
    <html>
      <div id="slider1" class="splunk-view" 
        data-require="app/your-app/components/slidepanel/slidepanelgroup" 
        data-options='{
          "items": ["panel1"]}'/>
          "title": "this is a toggle example for Splunk6 dashboards",

          "hide":  "yes"
    </html>
    <chart id="panel1">
      <title>Internal sourcetypes over the last 5 minutes</title>
      <searchString>index=_internal | stats count by sourcetype</searchString>
      <earliestTime>-5m</earliestTime>
      <option name="charting.chart.stackMode">stacked</option>
    </chart>
  </row>
</dashboard>


  • the first line loads autodiscover.js. This script will automatically load all javascript code in the app directory
  • we add a row-grouping of two because we add a new HTML panel to our row
  • the HTML panel references our main javascript file that will later be placed in a file calledyour-app/appserver/static/components/slidepanel/slidepanelgroup.js
  • the HTML panel also sets following parameters
    • items: an array of panel ids in this row-group
    • title: a title for the new html panel
    • hide: this indicates that the panel will initially be hidden at page 
  • finally we set an id for every panel element in the row-group. Here we set id="panel1" for our chart
That's it. Next step is to implement the two javascript files

loading javascript from Simple XML

The autodiscover.js script is taken form Splunk bubblechart example (See Splunk Dashboard example app). Place it into your app's appserever/static folder

the content of autodiscover.js looks as follows:

require.config({

    paths: {
        "app": "../app"
    }
});
require(['splunkjs/mvc/simplexml/ready!'], function(){
    require(['splunkjs/ready!'], function(){
        // The splunkjs/ready loader script will 
        // automatically instantiate all elements
        // declared in the dashboard's HTML.
    });

});

custom javascript code

The actual javascript code for our example goes into the file your-app/appserver/static/components/slidepanel/slidepanelgroup.js

First, we load some standard javascript and splunk libraries and include a custom css file.

define(function(require, exports, module) {

    var _ = require('underscore');
    var mvc = require('splunkjs/mvc');

    require("css!./slidepanelgroup.css");

[...]


the css defines two buttons to toggle the panel either up or down (collapse or expand)

.collaps {
  background-image: url(collapse.png); 
  background-repeat: no-repeat;
  float: right;
  padding-right: 20px;
  cursor: pointer;
  display: inline;
  background-size: 90% 100% ;
}

.expand {
  background-image: url(expand.png);
  background-repeat: no-repeat;
  float: right;
  padding-right: 20px;
  cursor: pointer;
  margin: 0px;
  display: inline;
  background-size: 90% 100% ;
}


Then we create our own custom SlidePanelView that extends the Splunk BaseSplunkView


var SlidePanelView = require('splunkjs/mvc/basesplunkview').extend({

        events: {
            [...]
        },
        render: function() {
            [...]
            return this;
        }
    });

    return SlidePanelView;
});

  • The object literal "events" will be called as soon as the user generates an event. In our case we will define a button (hyperlink) to toggle our panel. So we will put the code for button.click() events in here
  • The object literal "render" will be called on page load. We will put all our code that have to be initially loaded in here. That will be the panel title, a toggle link, etc.

render:

The code for the render part looks as follows

        render: function() {
            this.$('.btn-pill').remove();
            if (this.settings.has('items')) {
                var hide = this.settings.get('hide') || "no"
                var items = this.settings.get('items'), $el = this.$el;
                var first_panel = mvc.Components.get(items[0]);
                var h = $('<h2></h2>');
                var title = this.settings.get("title") || "";
                var img = $('<div> &nbsp; </div>');
                img.attr('class', "collaps");
                img.attr('alt', '#' + items[0]).data('item', items);
                img.appendTo($el);
                h.text(title);
                h.appendTo($el);
                if (hide == "yes") {
                  // initially toggle elements with option hide=yes
                  img.attr('class', "expand");
                  _(items).each(function(id) {
                    var component = mvc.Components.get(id);
                    if (component) {
                      component.$el.hide();
                    }
                  });
                }
            }
            return this;

        }

  • we get the data-options from the Simple XML. title, hide and the array of items. 
  • create a new HTML header tag an set its title accordingly
  • create a new HTML div tag as a placeholder for the collapse and expand buttons and set the class initially to "collaps"
  • in the "alt" property of the div tag, we add a list of all our items. this list will later be used to know which panels we have to toggle when a click event is fired
  • we add the newly created div and and header tag to our HTML DOM.
    the "$el" element is the HTML panel we've defined in the Simple XML. So calling h.appendTo($el) will append the header to our HTML panel
  • Now we check if "hide=yes" is set. If so, we have to initially hide (toggle-off) all panels in our items list
  • We loop over all items and call mvc.Components.get(..) to get all panel elements from the HTML DOM.
  • For every panel component we call the hide() function. This will initially toggle all panels in our row-group

events:

The code for the events part looks as follows

        events: {
            'click .expand': function(e) {
                var img = $(e.currentTarget);
                var items = img.data('item');
                _(items).each(function(id) {
                  var component = mvc.Components.get(id);
                  if (component) {
                    component.$el.slideToggle(1000);
                    // SVG panels need a resize event after toggling
                    component.$el.resize();
                }
                });
                img.attr("class", img.attr("class") == "expand" ? "collaps": "expand");
            },

            'click .collaps': function(e) {
                var img = $(e.currentTarget);
                var items = img.data('item');
                _(items).each(function(id) {
                  var component = mvc.Components.get(id);
                  if (component) {
                    component.$el.slideToggle(1000);
                    // SVG panels need a resize event after toggling
                    component.$el.resize();
                }
                });
                img.attr("class", img.attr("class") == "expand" ? "collaps": "expand");
            }

        }

  • we identify all click events on classes of type "expand" or "collaps". (that's our HTML div we defined in the render part earlier)
  • get the div and the array of items. The array contains all panels we have in our row-group in the Simple XML
  • for each item we get the corresponding component using Splunk's mvc.Component.get(..) function and call jQuery slideToggle() function. This will either toggle-on or toggle-off the panel
  • To make sure that SVG panels (i.e. bubblecharts) will be repainted after toggling them on, we call resize()
.. so that's basically all. 


get the full source code on gitub as fully functional splunk example app.

Or check out Splunk base to download an the example App custom_simplexml_extensions

Troubleshoot 

  1. Reload your Browser Cache
  2. If the Button images will not show up, check the path of "expand" and "collapse" in slidepanelgroup.js. It should inculde your app's name
  3. Use the browser developer tools to check if autodiscover.js and slidepanelgroup.js are loaded
  4. Use the browser developer tools to debug the slidepanelgroup.js

Friday, March 7, 2014

clickstream analysis with Splunk

Analyzing logfiles from webservers is a common use case in Splunk. Usually people are counting page clicks, analyzing status codes and so on. 

A while ago, I was asked if it would be possible to extract and analyze a user path with Splunk by means of our raw webserver log files. My short answer was "yes of course". But then it took me some time to get my final search working correctly.

The result looks quite simple.. here it is


base search

define your initial search and extract the two fields "page" and "client_id". This should look something like this

> index=your_weblogs sourcetype=your_st | 
  fields page client_id


hint: I've put my base search in a macro called "base_search" so that I can reuse it in the following searches. 

current page and next_page

With our base search at hand, let's extract the current page and next_page attributes per event and per client

> `base_search`| 
   streamstats current=f last(page) as next_page by client_id 

Finally, let's count the occurrence of equal paths

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id |
  stats count(next_page) as count by page next_page

This should give you a table with all paths from page to next_page for all users. If you'd like to analyze one specific page, just append a search query

To see where people are mostly going to after the "login_page":

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id  |
  stats count(next_page) as count by page next_page |
  search page="login_page"

To see where people are mostly coming from, before they reach the "payment" page

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id  |
  stats count(next_page) as count by page next_page |
  search next_page="payment"


visualize with sankey

A possible way to visualize the results would be a d3.js sankey diagram (http://bost.ocks.org/mike/sankey).
You can include it in your Splunk dashboard and get a pretty nice but still simple path analytics page with Splunk..





A blog entry and code samples to show how to include sankey in a Splunk dashboard will follow.. post a comment if you'd like to see it soon.

include search results as HTML content in Splunk6 simple xml

There is currently no native implementation in Splunk6 simple xml for including search results as HTML content in dashboard panels.

But Splunk6 allows it to easily inject custom javascript code into its simle xml structure. So here is a simple example that transforms a search result into HTML code and displays it in a simple xml dasboard panel

The basic search in this example counts the number of events over the last 7 days in the _internal index

> index=_internal |stats count


The simple XML

The simple XML panel we want to display in our panel should look as follows

<row>
<html>
  <h2>HTML content example</h2>
  <hr />
  <font size="3" color="red">
    <div id="htmlcontent-search"
      class="splunk-manager splunk-searchmanager"
      data-require="splunkjs/mvc/searchmanager"
      data-options='{
        "app": "htmlcontent-example",
        "search":  "index=_internal |stats count(sourcetype)",
        "earliest_time": "-7d",
        "preview": true
      }'/>
    <div id="htmlcontent"
      class="splunk-view"
      data-require="app/htmlcontent-example/components/htmlcontent/htmlcontent"
      data-options='{
      "managerid": "htmlcontent-search"
    }'/>
  </font>
  <hr />
</html>
</row>


As you can see we place the code in the App "htmlcontent-example" and refer to a directory called "app/htmlcontent-example/components/htmlcontent/htmlcontent". In this directory we will later place our javascript code. 


autoload javascript code

But first we have to make sure that the code in the htmlcontent directory will be loaded by Splunk when needed. Therefore we place a small javascript discover file called "autodiscover.js" in the app's static directory and call it in our dashboard

$app_home/appserver/static/autodiscover.js:

require.config({
    paths: {
        "app": "../app"
    }
});

require(['splunkjs/mvc/simplexml/ready!'], function(){

    require(['splunkjs/ready!'], function(){
    });
});

Include autodiscover.js in the <dashboard> tag of your simple xml

<dashboard script="autodiscover.js">

[...]
</dashboard>


the custom javascript 

Transformation into HTML code is done in a custom javascript file called "htmlcontent.js" 

$app_home/appserver/static/components/htmlcontent/htmlcontent.js

The script extends the SimpleSplunkView class from Splunk. It takes the first element from the search result and writes it back to the panel element.

define(function(require, exports, module) {
  [...]
  var HTMLContent = SimpleSplunkView.extend({

    [...]
    updateView: function(viz, data) {
      var myResults = data[0];
      this.$el.html(myResults);
    }
  });
  return HTMLContent;
});

Finally, the dashboard panel should look something like this:





.. That's it, get the full source code on >> github