clickstream analysis with Splunk

Analyzing logfiles from webservers is a common use case in Splunk. Usually people are counting page clicks, analyzing status codes and so on. 

A while ago, I was asked if it would be possible to extract and analyze a user path with Splunk by means of our raw webserver log files. My short answer was "yes of course". But then it took me some time to get my final search working correctly.

The result looks quite simple.. here it is

base search

define your initial search and extract the two fields "page" and "client_id". This should look something like this

> index=your_weblogs sourcetype=your_st | 
  fields page client_id


hint: I've put my base search in a macro called "base_search" so that I can reuse it in the following searches. 

current page and next_page

With our base search at hand, let's extract the current page and next_page attributes per event and per client

> `base_search`| 
   streamstats current=f last(page) as next_page by client_id 

Finally, let's count the occurrence of equal paths

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id |
  stats count(next_page) as count by page next_page

This should give you a table with all paths from page to next_page for all users. If you'd like to analyze one specific page, just append a search query

To see where people are mostly going to after the "login_page":

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id  |
  stats count(next_page) as count by page next_page |
  search page="login_page"

To see where people are mostly coming from, before they reach the "payment" page

> `base_search`| 
  streamstats current=f last(page) as next_page by client_id  |
  stats count(next_page) as count by page next_page |
  search next_page="payment"

visualize with sankey

A possible way to visualize the results would be a d3.js sankey diagram (http://bost.ocks.org/mike/sankey).
You can include it in your Splunk dashboard and get a pretty nice but still simple path analytics page with Splunk..

A blog entry and code samples to show how to include sankey in a Splunk dashboard will follow.. post a comment if you'd like to see it soon.