Class: Mu::Xtractr
- Inherits:
-
Object
- Object
- Mu::Xtractr
- Defined in:
- lib/mu/xtractr.rb,
lib/mu/xtractr/flow.rb,
lib/mu/xtractr/term.rb,
lib/mu/xtractr/about.rb,
lib/mu/xtractr/field.rb,
lib/mu/xtractr/flows.rb,
lib/mu/xtractr/views.rb,
lib/mu/xtractr/packet.rb,
lib/mu/xtractr/stream.rb,
lib/mu/xtractr/content.rb,
lib/mu/xtractr/packets.rb,
lib/mu/xtractr/service.rb,
lib/mu/xtractr/stream/http.rb,
lib/mu/xtractr/test/tc_flow.rb,
lib/mu/xtractr/test/tc_host.rb,
lib/mu/xtractr/test/tc_term.rb,
lib/mu/xtractr/test/tc_field.rb,
lib/mu/xtractr/test/tc_flows.rb,
lib/mu/xtractr/test/tc_views.rb,
lib/mu/xtractr/test/tc_packet.rb,
lib/mu/xtractr/test/tc_stream.rb,
lib/mu/xtractr/test/tc_packets.rb,
lib/mu/xtractr/test/tc_service.rb,
lib/mu/xtractr/test/tc_xtractr.rb,
lib/mu/xtractr/test/stream/tc_http.rb
Overview
Mu::Xtractr
:main: Mu::Xtractr
This gem is Ruby front-end to the RESTful API that xtractr[http://www.pcapr.net/xtractr] provides. We primarily use this for unit testing xtractr’s API, but on its own this gem provides for a powerful programmable interface into xtractr and is a super fast way to extract information out of large pcaps.
Getting Started
First download xtractr from www.pcapr.net/xtractr. Follow the instructions to index your pcap. Finally run xtractr in browse mode and then you can hang out in IRB poking around flows and packets.
Examples
You can run the xtractr-gem from within IRB which makes it a fun interactive network forensics tool. Make sure you are running the xtractr binary in browse mode. Turning on auto-completion in IRB also makes it easier to try out different things and interactively experiment with the API:
$ irb -rirb/completion -rmu/xtractr
All of the examples below work off the test/test.pcap bundled with the gem. We’ll also assume that you’ve done this at the start of the IRB session:
irb> xtractr = Mu::Xtractr.new
Top DNS query names
We first pull out all DNS flows and then map/reduce the unique values of the dns.qry.name.
irb> xtractr.flows('flow.service:DNS').values('dns.qry.name')
Services used by the top talker (based on bytes sent/received)
We first sum the total number of bytes using the src address as the key. The sum function returns the matches sorted by the #bytes. We then use the first object (the top talker) to in turn map/reduce the unique list of services supported by it.
irb> xtractr.flows.sum('flow.src', 'flow.bytes').first.count('flow.service')
Generating #new pcaps based on search criteria
We first get a list of the unique HTTP methods in the index and then for each of methods, query for all the packets and then save them into a new pcap.
irb> xtractr.packets.count('http.request.method').each { |c| c.packets.save("#{c.value}.pcap") }
– rdoc –exclude test –force-update –inline-source –op mu/xtractr/doc –main Mu::Xtractr
Direct Known Subclasses
Defined Under Namespace
Classes: About, Content, Field, Flow, Flows, Host, Packet, Packets, Service, Stream, Term, Test, Views
Instance Attribute Summary collapse
-
#address ⇒ Object
readonly
Return the IP address of the xtractr instance.
-
#port ⇒ Object
readonly
Return the listening port of the xtractr instance.
-
#relurl ⇒ Object
readonly
Relative URL.
Class Method Summary collapse
-
.create(url) ⇒ Object
Create a new instance to connect to the xtractr binary using a url.
Instance Method Summary collapse
-
#about ⇒ Object
Fetch the meta data about the index.
-
#field(name) ⇒ Object
Fetch a field of the given name.
-
#fields(regex = nil) ⇒ Object
Fetch the list of fields in the index.
-
#flow(id) ⇒ Object
Return the id’th flow from the index.
-
#flows(q = '*') ⇒ Object
Return an iterator that can yield each flow that matched the query.
-
#get(url, opts = {}) ⇒ Object
Fetch the URL and return the response body, as is.
-
#host(address) ⇒ Object
Fetch a host of the given address.
-
#hosts(regex = nil) ⇒ Object
Fetch the list of hosts in the index.
-
#initialize(address = 'localhost', port = 8080, relurl = nil) ⇒ Xtractr
constructor
Create a new instance to connect to the xtractr binary running in browse mode.
-
#inspect ⇒ Object
:nodoc:.
-
#json(url, opts = {}) ⇒ Object
Fetch the URL with the GET parameters and interpret the response body as a JSON object.
-
#packet(id) ⇒ Object
Return the id’th packet from the index.
-
#packets(q = '*') ⇒ Object
Return an iterator that can yield each packet that matched the query.
-
#service(name) ⇒ Object
Fetch a service of the given name.
-
#services(regex = nil) ⇒ Object
Fetch the list of services in the index.
Constructor Details
#initialize(address = 'localhost', port = 8080, relurl = nil) ⇒ Xtractr
99 100 101 102 103 104 105 106 107 108 109 110 |
# File 'lib/mu/xtractr.rb', line 99 def initialize address='localhost', port=8080, relurl=nil @address = address @port = port @relurl = relurl || '/' @relurl << '/' if @relurl[-1,1] != '/' #unless about.version =~ /^4\.5\.(svn|41604)$/ # puts "boh" # puts "xtractr version #{about.version} out of date!" # puts "please download a new one from http://www.pcapr.net/xtractr" # raise #end end |
Instance Attribute Details
#address ⇒ Object (readonly)
Return the IP address of the xtractr instance
79 80 81 |
# File 'lib/mu/xtractr.rb', line 79 def address @address end |
#port ⇒ Object (readonly)
Return the listening port of the xtractr instance
82 83 84 |
# File 'lib/mu/xtractr.rb', line 82 def port @port end |
#relurl ⇒ Object (readonly)
Relative URL
85 86 87 |
# File 'lib/mu/xtractr.rb', line 85 def relurl @relurl end |
Class Method Details
.create(url) ⇒ Object
Create a new instance to connect to the xtractr binary using a url.
Xtractr.create 'http://some.host:8080/'
90 91 92 93 |
# File 'lib/mu/xtractr.rb', line 90 def self.create url uri = URI.parse url self.new uri.host, uri.port, uri.path end |
Instance Method Details
#about ⇒ Object
Fetch the meta data about the index. This includes information about the total number of packets, flows as well as the duration of the entire set of pcaps in the index.
xtractr.about
116 117 118 |
# File 'lib/mu/xtractr.rb', line 116 def about @about ||= About.new json('api/about') end |
#field(name) ⇒ Object
Fetch a field of the given name.
xtractr.field 'http.server'
134 135 136 137 138 |
# File 'lib/mu/xtractr.rb', line 134 def field name obj = fields.find { |f| f.name == name } raise ArgumentError, "Unknown field #{name}" if not obj return obj end |
#fields(regex = nil) ⇒ Object
Fetch the list of fields in the index. The fields are only available if the –mode forensics was used during the indexing process.
xtractr.fields
xtractr.fields /^http/
xtractr.fields 'http.server'
125 126 127 128 129 130 |
# File 'lib/mu/xtractr.rb', line 125 def fields regex=nil regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String result = (@fields ||= json 'api/fields') result = result.select { |name| name =~ regex } if regex return result.map { |name| Field.new self, name } end |
#flow(id) ⇒ Object
Return the id’th flow from the index.
xtractr.flow 1
200 201 202 203 204 205 |
# File 'lib/mu/xtractr.rb', line 200 def flow id result = json "api/flows", :start => id, :limit => id rows = result['rows'] raise ArgumentError, "Unknown flow #{id}" if rows.empty? return Flow.new(self, rows[0]) end |
#flows(q = '*') ⇒ Object
Return an iterator that can yield each flow that matched the query. If q is a Range, then it’s used to extract the set of flows that match all of those ids.
xtractr.flows.each { |flow| ... }
xtractr.flows(1..10).each { |flow| ... }
xtractr.flows("flow.src:192.168.1.1").each { |flow| ... }
188 189 190 191 192 193 194 195 196 |
# File 'lib/mu/xtractr.rb', line 188 def flows(q='*') # :yields: flow if q.is_a? Range first = q.first last = q.last last -= 1 if q.exclude_end? q = "flow.id:[#{first} #{last}]" end return Flows.new(self, :q => q) end |
#get(url, opts = {}) ⇒ Object
Fetch the URL and return the response body, as is
242 243 244 245 246 247 248 249 250 251 |
# File 'lib/mu/xtractr.rb', line 242 def get url, opts={} # :nodoc: _url = relurl + url if opts.size _url << '?' _url << opts.keys.map { |key| key.to_s + '=' + opts[key].to_s }.join('&') end _url = "http://#{address}:#{port}" + URI.escape(_url) return Net::HTTP.get(URI.parse(_url)) end |
#host(address) ⇒ Object
Fetch a host of the given address.
xtractr.host '192.168.1.1'
155 156 157 158 159 |
# File 'lib/mu/xtractr.rb', line 155 def host address obj = hosts.find { |h| h.address == address } raise ArgumentError, "Unknown host #{address}" if not obj return obj end |
#hosts(regex = nil) ⇒ Object
Fetch the list of hosts in the index. The optional regex (or String) can be used to filter the hosts list.
xtractr.hosts
xtractr.hosts /192.168/
xtractr.hosts '10.10'
145 146 147 148 149 150 151 |
# File 'lib/mu/xtractr.rb', line 145 def hosts regex=nil regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String result = (@hosts ||= json 'api/hosts') rows = result['rows'] rows = rows.select { |row| row['name'] =~ regex } if regex return rows.map { |row| Host.new self, row['name'] } end |
#inspect ⇒ Object
:nodoc:
253 254 255 |
# File 'lib/mu/xtractr.rb', line 253 def inspect # :nodoc: "#<xtractr #{address}:#{port}>" end |
#json(url, opts = {}) ⇒ Object
Fetch the URL with the GET parameters and interpret the response body as a JSON object
234 235 236 237 238 239 |
# File 'lib/mu/xtractr.rb', line 234 def json url, opts={} # :nodoc: res = get url, opts js = JSON.parse(res) raise ArgumentError, js['reason'] if js.is_a?(Hash) and js['error'] return js end |
#packet(id) ⇒ Object
Return the id’th packet from the index.
xtractr.packet 1
225 226 227 228 229 230 |
# File 'lib/mu/xtractr.rb', line 225 def packet id result = json "api/packets", :start => id, :limit => id rows = result['rows'] raise ArgumentError, "Unknown packet #{id}" if rows.empty? return Packet.new(self, rows[0]) end |
#packets(q = '*') ⇒ Object
Return an iterator that can yield each packet that matched the query. If q is a Range, then it’s used to extract the set of packets that match all of those ids.
xtractr.packets.each { |pkt| ... }
xtractr.packets(5..32).each { |pkt| ... }
xtractr.packets("http.user.agent:mozilla").each { |pkt| ... }
213 214 215 216 217 218 219 220 221 |
# File 'lib/mu/xtractr.rb', line 213 def packets(q='*') # :yields: packet if q.is_a? Range first = q.first last = q.last last -= 1 if q.exclude_end? q = "pkt.id:[#{first} #{last}]" end return Packets.new(self, :q => q) end |
#service(name) ⇒ Object
Fetch a service of the given name.
xtractr.service 'dns'
176 177 178 179 180 |
# File 'lib/mu/xtractr.rb', line 176 def service name obj = services.find { |s| s.name.downcase == name.downcase } raise ArgumentError, "Unknown service #{name}" if not obj return obj end |
#services(regex = nil) ⇒ Object
Fetch the list of services in the index. The optional regex (or String) can be used to filter the services lists.
xtractr.services
xtractr.services /http/
xtractr.services 'sip'
166 167 168 169 170 171 172 |
# File 'lib/mu/xtractr.rb', line 166 def services regex=nil regex = Regexp.new(regex, Regexp::IGNORECASE) if regex.is_a? String result = (@services ||= json 'api/services') rows = result['rows'] rows = rows.select { |row| row['name'] =~ regex } if regex return rows.map { |row| Service.new self, row['name'] } end |