Class: RestScrapyd

Inherits:
Object
  • Object
show all
Includes:
RestCore
Defined in:
lib/rest-scrapyd.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#project ⇒ Object

Default project name (can also be passed as hash argument to constructor).



31
32
33
 
# File 'lib/rest-scrapyd.rb', line 31

def project
  @project
end

Instance Method Details

#addversion(version, egg, project = self.project) ⇒ Hash<String, Object>

Add a version to a project, creating the project if it doesn’t exist.

Parameters:

  • version (String) —

    Project version

  • egg (String, IO) —

    Filename or file to upload, must be a Scrapy egg

Returns:

  • (Hash<String, Object>) —

    Response, right now just {“spiders”: <n>}

See Also:



38
39
40
41
 
# File 'lib/rest-scrapyd.rb', line 38

def addversion(version, egg, project=self.project)
  egg = File.open(egg, 'rb') if egg.is_a? String
  post('addversion.json', project: project, version: version, egg: egg)
end

#cancel(job, project = self.project) ⇒ String

Cancel a scheduled or running job

Parameters:

  • job (String) —

    Jobid to cancel

  • project (String) (defaults to: self.project) —

    Project name

Returns:

  • (String) —

    Previous state of job

See Also:



68
69
70
 
# File 'lib/rest-scrapyd.rb', line 68

def cancel(job, project=self.project)
  post('cancel.json', project: project, job: job)['prevstate']
end

#delproject(project) ⇒ Object

Delete a project and all its uploaded versions.

Parameters:

  • project (String) —

    Project name

See Also:



114
115
116
 
# File 'lib/rest-scrapyd.rb', line 114

def delproject(project)
  post('delproject.json', project: project)
end

#delversion(version, project = self.project) ⇒ Object

Delete a project version.

Parameters:

  • version (String) —

    Project version

  • project (String) (defaults to: self.project) —

    Project name

See Also:



107
108
109
 
# File 'lib/rest-scrapyd.rb', line 107

def delversion(version, project=self.project)
  post('delversion.json', project: project, version: version)
end

#listjobs(project = self.project) ⇒ Hash<String, Array<Hash>>

Get the list of pending, running and finished jobs of some project.

Parameters:

  • project (String) (defaults to: self.project) —

    Project name

Returns:

  • (Hash<String, Array<Hash>>) —

    Jobs for project

See Also:



99
100
101
 
# File 'lib/rest-scrapyd.rb', line 99

def listjobs(project=self.project)
  get('listjobs.json', project: project).reject{|k,v| k=='status'}
end

#listprojects ⇒ Array<String>

Get the list of projects uploaded to this Scrapy server.

Returns:

  • (Array<String>) —

    Available projects

See Also:



75
76
77
 
# File 'lib/rest-scrapyd.rb', line 75

def listprojects
  get('listprojects.json')['projects']
end

#listspiders(project = self.project) ⇒ Array<String>

Get the list of spiders available in the last version of some project.

Parameters:

  • project (String) (defaults to: self.project) —

    Project name

Returns:

  • (Array<String>) —

    Available spiders for project

See Also:



91
92
93
 
# File 'lib/rest-scrapyd.rb', line 91

def listspiders(project=self.project)
  get('listspiders.json', project: project)['spiders']
end

#listversions(project = self.project) ⇒ Array<String>

Get the list of versions available for some project.

Parameters:

  • project (String) (defaults to: self.project) —

    Project name

Returns:

  • (Array<String>) —

    Available versions for project

See Also:



83
84
85
 
# File 'lib/rest-scrapyd.rb', line 83

def listversions(project=self.project)
  get('listversions.json', project: project)['versions']
end

#schedule(spider, version, settings = {}, arguments = {}) ⇒ String

Schedule a spider run (also known as a job).

Parameters:

  • spider (String) —

    Spider name

  • version (String) —

    Project version

  • settings (Hash<String, String>) (defaults to: {}) —

    Additional Scrapy settings

  • arguments (Hash<String, String>) (defaults to: {}) —

    Additional spider arguments

Options Hash (arguments):

  • :project (String) — default: {self.project} —

    Project name

Returns:

  • (String) —

    Job id

See Also:



51
52
53
54
55
56
57
58
59
60
61
 
# File 'lib/rest-scrapyd.rb', line 51

def schedule(spider, version, settings={}, arguments={})
  # +setting+ may appear multiple times, so can't use hash
  params = {project: self.project, spider: spider}.to_a
  params += settings.to_a.map{|k,v| ["setting", "#{k}=#{v}"]}
  params += arguments.to_a
  # rest-core doesn't (yet) support array payload, encode ourselves
  params.map{|p| p[0] = p[0].to_s} # like stringify_keys
  params = RestCore::Middleware.percent_encode(params)
  s_headers = headers.merge({'Content-Type' => 'application/x-www-form-urlencoded'})
  post('schedule.json', params, {}, headers: s_headers)['jobid']
end