Class: RubyLogParser

Inherits:
Object
  • Object
show all
Defined in:
lib/rubylogparser.rb

Constant Summary collapse

VERSION = 
"0.1.3"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(lp_executable = nil) ⇒ RubyLogParser

Initialize is used to check that the Log Parser executable can be found and to config the various command line defaults.



61
62
63
64
 
# File 'lib/rubylogparser.rb', line 61

def initialize(lp_executable = nil)
  self.valid_executable(lp_executable)
  self.set_defaults
end

Instance Attribute Details

#cmd_options ⇒ Object

Stores the various options that will be used to build the command line that the process is eventually started with



23
24
25
 
# File 'lib/rubylogparser.rb', line 23

def cmd_options
  @cmd_options
end

#cmd_string ⇒ Object

The command line string that IO.popen uses to open the LogParser parser



26
27
28
 
# File 'lib/rubylogparser.rb', line 26

def cmd_string
  @cmd_string
end

#elements_output ⇒ Object (readonly)

Query statistics are turned on by default and controlled with the “ -stats” switch. The output includes Elements Processed, Elements Output, and Execution time in seconds and is available in these class variables after all input has been read.



54
55
56
 
# File 'lib/rubylogparser.rb', line 54

def elements_output
  @elements_output
end

#elements_processed ⇒ Object (readonly)

Query statistics are turned on by default and controlled with the “ -stats” switch. The output includes Elements Processed, Elements Output, and Execution time in seconds and is available in these class variables after all input has been read.



54
55
56
 
# File 'lib/rubylogparser.rb', line 54

def elements_processed
  @elements_processed
end

#error_flag ⇒ Object

A flag which is nil until remaining log parser output is statistics, errors or other non-data related output



30
31
32
 
# File 'lib/rubylogparser.rb', line 30

def error_flag
  @error_flag
end

#execution_time ⇒ Object (readonly)

Query statistics are turned on by default and controlled with the “ -stats” switch. The output includes Elements Processed, Elements Output, and Execution time in seconds and is available in these class variables after all input has been read.



54
55
56
 
# File 'lib/rubylogparser.rb', line 54

def execution_time
  @execution_time
end

#field_names ⇒ Object (readonly)

Column headings on the output data are controlled with the “ -q” quiet mode switch. When enabled (the default), the various column names are stored in this array. This is populated immediately after the process is created and before any data is read in using the self.read_hash or self.read_array methods.



40
41
42
 
# File 'lib/rubylogparser.rb', line 40

def field_names
  @field_names
end

#logparser_exe=(value) ⇒ Object (writeonly)

Stores the location of the LogParser.exe file.



57
58
59
 
# File 'lib/rubylogparser.rb', line 57

def logparser_exe=(value)
  @logparser_exe = value
end

#logparser_pipe ⇒ Object

The pipe to the IO.popen process to read LogParser output data



33
34
35
 
# File 'lib/rubylogparser.rb', line 33

def logparser_pipe
  @logparser_pipe
end

#parse_errors ⇒ Object (readonly)

LogParser reports parse errors when it can’t do something it expects to. One example is when access is denied to registry keys. All parse errors are stuffed into this array so they can be examined at the end of processing.



45
46
47
 
# File 'lib/rubylogparser.rb', line 45

def parse_errors
  @parse_errors
end

#queryInfo ⇒ Object (readonly)

Output when using the “ -queryInfo” switch will be stored here



48
49
50
 
# File 'lib/rubylogparser.rb', line 48

def queryInfo
  @queryInfo
end

Instance Method Details

#build_command_string(options = nil) ⇒ Object

The LogParser process is opened using a string formatted for a command line. All of the command line options can be viewed in more detail by running “logparser -h” to see online help. The command line string is built using the currently configured options including any values in the input hash used to override or add specific functionality.



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
 
# File 'lib/rubylogparser.rb', line 144

def build_command_string(options = nil)
  @cmd_string = nil
  if !options.nil? then
    options.each {|key, value| @cmd_options[key] = value}
  end
  if (valid_input_format(@cmd_options['i']) &&
     valid_output_format(@cmd_options['o']) &&
     @cmd_options['sql'] != nil) then
    @cmd_string = "\"#{@logparser_exe}\"" 
    @cmd_string += " -i:#{@cmd_options['i']}" unless @cmd_options['i'].nil?
    @cmd_options['sql'].gsub!(/[\r\n]/, ' ') # remove any linefeeds from formatted SQL statement
    @cmd_string += " \"#{@cmd_options['sql']}\""
    @cmd_string += " -o:#{@cmd_options['o']}" unless @cmd_options['o'].nil?
    @cmd_string += " -q:ON" if cmd_options['q']
    @cmd_string += " -e:#{@cmd_options['e']}" if @cmd_options['e'] >= 0
    @cmd_string += " -iw:ON" if @cmd_options['iw']
    @cmd_string += " -stats:OFF" if @cmd_options['stats'] == false
    @cmd_string += " -c" if @cmd_options['c']
    @cmd_string += " -multiSite:ON" if @cmd_options['multiSite']
    @cmd_string += " -saveDefaults" if @cmd_options['saveDefaults']
    @cmd_string += " -restoreDefaults" if @cmd_options['restoreDefaults']
    @cmd_string += " -rtp:#{@cmd_options['rtp']}" if @cmd_options['rtp'] != 0 # turns off "Press a key"
    @cmd_string += " -queryInfo" if @cmd_options['queryInfo']
  end
  return @cmd_string
end

#create_process ⇒ Object

The LogParser process is created using IO.popen and then subsequent output from LogParser is read via a pipe.



173
174
175
176
177
178
179
180
181
182
183
 
# File 'lib/rubylogparser.rb', line 173

def create_process
  @logparser_pipe = nil
  msg = nil
  begin
    @logparser_pipe = IO.popen(@cmd_string, "rb")
  rescue Exception => e
    msg = e.to_s
    # puts msg # when debugging
  end
  return @logparser_pipe
end

#data_hash(data_array) ⇒ Object

To return the data in a hash, the data line is first read into an array. Then the field keys (initially loaded by the field_definitions method when the LogParser process was created), are matched up with the respective data values.



262
263
264
265
266
267
268
269
270
271
 
# File 'lib/rubylogparser.rb', line 262

def data_hash(data_array)
  data_hash = nil
  if !data_array.nil? then
    data_hash = Hash.new
    @field_names.each_index {|x|
      data_hash[@field_names[x]] = data_array[x]
    }
  end
  return data_hash
end

#field_definitions(names_line) ⇒ Object

The first line of output contains the field names (by default). This information is used to name the various hash keys when data is returned in a hash.



210
211
212
213
214
215
216
 
# File 'lib/rubylogparser.rb', line 210

def field_definitions(names_line)
  @field_names = Array.new
  if !names_line.nil? then
    @field_names = names_line.split(/,/)
  end
  return @field_names
end

#open_query(input, query, output, options) ⇒ Object

LogParser needs a data input source, a SQL selection query, and an output location. A hash can also be passed to override default values or provide additional command line switches. An interesting exception that needs to be handled is LogParser’s “-queryInfo” switch which makes LogParser evaluate the input parameters and return info on the various column datatypes.



278
279
280
281
282
283
284
285
286
287
288
289
290
291
 
# File 'lib/rubylogparser.rb', line 278

def open_query(input, query, output, options)
  @cmd_options['i'] = input
  @cmd_options['sql'] = query
  @cmd_options['o'] = output
  @cmd_string = self.build_command_string(options)

  self.create_process
  
  if @cmd_options['queryInfo'] == true then
    @queryInfo = self.read
  elsif @cmd_options['q'] == false then
    self.field_definitions(self.readline)
  end
end

#parse_errors_and_stats(text_input) ⇒ Object

Extract statistics and any parse errors from output



243
244
245
246
247
248
249
250
251
252
253
254
255
256
 
# File 'lib/rubylogparser.rb', line 243

def parse_errors_and_stats(text_input)
  if text_input =~ /Statistics:\r\n(.*)/ then
    @elements_processed = $1 if text_input =~ /^Elements processed:\s*([\d,]+)\r\n/i
    @elements_output = $1 if text_input =~ /^Elements output:\s*([\d,]+)\r\n/i
    @execution_time = $1 if text_input =~ /^Execution time:\s*([\d\.]+) seconds\r\n/i
  end
    
  if text_input =~ /Task completed with parse errors.*?\r\n(.*)/ims then
    @parse_errors = $1
    if @parse_errors =~ /(.*?)Statistics:\r\n/ims then
      @parse_errors = $1
    end
  end  
end

#process_array_line(next_line) ⇒ Object

The output from LogParser is return in comma separated value (CSV) format. Using split to determine the fields is much faster than CSV. Occasionally however, a field will have a comma in it which will cause split to break a field in two. In cases where the actual number of fields returned by split does not match the expected number, the CSV library is called to try the split. This usually fixes the problem.



224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
 
# File 'lib/rubylogparser.rb', line 224

def process_array_line(next_line)
  if !next_line.nil? then 
    array_line = next_line.split(/,/)
    if array_line.length == @field_names.length then
      return array_line
    else
      CSV.parse(next_line) {|row|
        if row.length == @field_names.length then
          return row
        else
          return nil
        end
      }
    end
  end
  return nil
end

#process_line(line) ⇒ Object

When Log Parser outputs the first blank line (“rn”), it indicates that the data output is complete and anything that will follow are the errors or query statistics.



198
199
200
201
202
203
204
205
 
# File 'lib/rubylogparser.rb', line 198

def process_line(line)
  if (line == "\r\n" || line =~ /Task completed with parse errors/i) then
    self.error_flag = true # remaining output should now be error
    return nil
  else
    return line.chomp!
  end
end

#read ⇒ Object

Return everything that remains in the Log Parser pipe



186
187
188
 
# File 'lib/rubylogparser.rb', line 186

def read
  @logparser_pipe.read
end

#read_array(blnHeader_row) ⇒ Object

When data is best processed all at once, this method will return an array of arrays. The array will have a header row with the field names by default unless LogParser is started in “quiet” mode (i.e. -q:ON).



296
297
298
299
300
301
302
303
304
305
306
307
308
 
# File 'lib/rubylogparser.rb', line 296

def read_array(blnHeader_row)
  data_array = Array.new
  i=0
  if blnHeader_row
    data_array[i] = @field_names
    i += 1
  end
  while array = self.process_array_line(self.readline) do
    data_array[i] = array
    i += 1
  end
  return data_array
end

#read_array_line ⇒ Object

Similar to read_hash, this method will parse a single return the LogParser output in an array. If no output remains, this will return nil.



319
320
321
 
# File 'lib/rubylogparser.rb', line 319

def read_array_line
  self.process_array_line(self.readline)
end

#read_hash ⇒ Object

Often times, the easiest way to use the LogParser output is to parse a single line into a hash that is keyed by the field names. Once all LogParser output has been read, this will return nil.



313
314
315
 
# File 'lib/rubylogparser.rb', line 313

def read_hash
  self.data_hash(self.process_array_line(self.readline))  
end

#readline ⇒ Object

Reads a single line of output from the LogParser pipe.



191
192
193
 
# File 'lib/rubylogparser.rb', line 191

def readline
  self.process_line(@logparser_pipe.readline)
end

#set_defaults ⇒ Object

LogParser has a variety of switches to control the input, output, SQL and process options. Most can be viewed by opening up LogParser at a command prompt and typing “c:Logparser> logparser -h”. There are additional switches that are not documented in the “logparser -h” output used to for controlling things like chart appearance. See the self.open_query method for more information.



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
 
# File 'lib/rubylogparser.rb', line 97

def set_defaults
  @cmd_options = {
    'sql' => nil,  # SQL query empty to start with
    'i' => nil,    # input mode
    'o' => 'CSV',  # output mode
    'q' => false,  # quiet mode - when true field headings supressed
    'e' => -1,     # max number of errors 
    'iw' => false, # ignore warnings
    'stats' => true, # display statistics after executing query
    'c' => false,     # use built-in conversion query
    'multiSite' => false, # send BIN conversion output to multiple files - see help
    'saveDefaults' => false, # save specificed options as default values
    'restoreDefaults' => false, # restore factory defaults
    'rtp' => 0,   # -1 turns off "Press a key..." prompts normally found in DOS window when NAT output selected
    'queryInfo' => false # display query processing info
  }
  return @cmd_options
end

#valid_executable(lp_executable = nil) ⇒ Object

Simple test to make sure the LogParser executable can be found. By default, it checks the IIS Resource Kit install path, the stand-alone LogParser install path, and then if the LogParser.exe file is found on the path. If an executable path is provided, this is tried instead allowing the user to pick the desired path if one or more LogParser versions are installed.



71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/rubylogparser.rb', line 71

def valid_executable(lp_executable = nil)
  @logparser_exe = nil
  if lp_executable.nil? then
    file_locations = ['c:\Program Files\IIS Resources\Log Parser 2.2\LogParser.exe', 'c:\Program Files\Log Parser 2.2\LogParser.exe', 'c:\Program Files (x86)\Log Parser 2.2\LogParser.exe', 'LogParser.exe']
  else
    file_locations = [ lp_executable ]
  end
  
  file_locations.each {|location|
    if File::executable? location then
      @logparser_exe = location
    end
  }
  if @logparser_exe.nil?
    raise RuntimeError, "The Log Parser executable could not be found."
  end
  
  return @logparser_exe
end

#valid_input_format(input) ⇒ Object

Input formats are checked against valid formats to catch simple errors



117
118
119
120
121
122
123
124
125
126
 
# File 'lib/rubylogparser.rb', line 117

def valid_input_format(input)
  if input.nil? then
    return true
  end
  input_formats = %w(IISW3C NCSA IIS IISODBC BIN IISMSID HTTPERR URLSCAN
                     CSV TSV W3C XML EVT ETW NETMON REG ADS TEXTLINE
                     TEXTWORD FS COM)
  input_formats.each {|format| return true if format==input.upcase}
  return false
end

#valid_output_format(output) ⇒ Object

Output formats are checked against valid formats to catch simple errors



129
130
131
132
133
134
135
136
137
 
# File 'lib/rubylogparser.rb', line 129

def valid_output_format(output)
  if output.nil? then
    return true
  end
  output_formats = %w(CSV TSV XML DATAGRID CHART SYSLOG NEUROVIEW NAT
                      W3C IIS SQL TPL NULL)
  output_formats.each{|format| return true if format==output.upcase}
  return false
end