Parsing Log Files

Last updated 7 months ago

Apache Log File

Let's first list the important information that we may need from the Apache logs

  • IP address

  • Time stamp

  • HTTP method

  • URI path

  • Response code

  • User agent

To read a log file, I prefer to read it as lines

apache_logs = File.readlines "/var/log/apache2/access.log"

I was looking for a simple regular expression for Apache logs. I found one here with small tweak.

apache_regex = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (.{0})- \[([^\]]+?)\] "(GET|POST|PUT|DELETE) ([^\s]+?) (HTTP\/1\.1)" (\d+) (\d+) "-" "(.*)"/

So I came up with this small method which parses and converts Apache "access.log" file to an array contains a list of hashes with our needed information.

#!/usr/bin/env ruby
# KING SABRI | @KINGSABRI
apache_logs = File.readlines "/var/log/apache2/access.log"
def parse(logs)
apache_regex = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - (.{0})- \[([^\]]+?)\] "(GET|POST|PUT|DELETE) ([^\s]+?) (HTTP\/1\.1)" (\d+) (\d+) ([^\s]+?) "(.*)"/
result_parse = []
logs.each do |log|
parser = log.scan(apache_regex)[0]
# If can't parse the log line for any reason.
if log.scan(apache_regex)[0].nil?
puts "Can't parse: #{log}\n\n"
next
end
parse =
{
:ip => parser[0],
:user => parser[1],
:time => parser[2],
:method => parser[3],
:uri_path => parser[4],
:protocol => parser[5],
:code => parser[6],
:res_size => parser[7],
:referer => parser[8],
:user_agent => parser[9]
}
result_parse << parse
end
return result_parse
end
require 'pp'
pp parse(apache_logs)

Returns

[{:ip=>"127.0.0.1",
:user=>"",
:time=>"12/Dec/2015:20:09:05 +0300",
:method=>"GET",
:uri_path=>"/",
:protocol=>"HTTP/1.1",
:code=>"200",
:res_size=>"3525",
:referer=>"\"-\"",
:user_agent=>
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"},
{:ip=>"127.0.0.1",
:user=>"",
:time=>"12/Dec/2015:20:09:05 +0300",
:method=>"GET",
:uri_path=>"/icons/ubuntu-logo.png",
:protocol=>"HTTP/1.1",
:code=>"200",
:res_size=>"3689",
:referer=>"\"http://localhost/\"",
:user_agent=>
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"},
{:ip=>"127.0.0.1",
:user=>"",
:time=>"12/Dec/2015:20:09:05 +0300",
:method=>"GET",
:uri_path=>"/favicon.ico",
:protocol=>"HTTP/1.1",
:code=>"404",
:res_size=>"500",
:referer=>"\"http://localhost/\"",
:user_agent=>
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"}]

Note: The Apache LogFormat is configured as LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined which is the default configurations.

  • %h is the remote host (i.e. the client IP address)

  • %l is the identity of the user determined by identd (not usually used since not reliable)

  • %u is the user name determined by HTTP authentication

  • %t is the time the request was received.

  • %r is the request line from the client. ("GET / HTTP/1.0")

  • %>s is the status code sent from the server to the client (200, 404 etc.)

  • %b is the size of the response to the client (in bytes)

  • Referer is the page that linked to this URL.

  • User-agent is the browser identification string.

IIS Log File

Here is a basic IIS log regular expression

iis_regex = /(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}) (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) ([^\s]++?) (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) (\d{2}) (GET|POST|PUT|DELETE) ([^\s]++?) - (\d+) (\d+) (\d+) (\d+) ([^\s]++?) (.*)/