Class: RestScrapyd

Inherits:
Object
  • Object
show all
Includes:
RestCore
Defined in:
lib/rest-scrapyd.rb

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#projectObject

Default project name (can also be passed as hash argument to constructor).



31
32
33
# File 'lib/rest-scrapyd.rb', line 31

def project
  @project
end

Instance Method Details

#addversion(version, egg, project = self.project) ⇒ Hash<String, Object>

Add a version to a project, creating the project if it doesn’t exist.

Parameters:

  • version (String)

    Project version

  • egg (String, IO)

    Filename or file to upload, must be a Scrapy egg

Returns:

  • (Hash<String, Object>)

    Response, right now just {“spiders”: <n>}

See Also:



38
39
40
41
# File 'lib/rest-scrapyd.rb', line 38

def addversion(version, egg, project=self.project)
  egg = File.open(egg, 'rb') if egg.is_a? String
  post('addversion.json', project: project, version: version, egg: egg)
end

#cancel(job, project = self.project) ⇒ String

Cancel a scheduled or running job

Parameters:

  • job (String)

    Jobid to cancel

  • project (String) (defaults to: self.project)

    Project name

Returns:

  • (String)

    Previous state of job

See Also:



68
69
70
# File 'lib/rest-scrapyd.rb', line 68

def cancel(job, project=self.project)
  post('cancel.json', project: project, job: job)['prevstate']
end

#delproject(project) ⇒ Object

Delete a project and all its uploaded versions.

Parameters:

  • project (String)

    Project name

See Also:



114
115
116
# File 'lib/rest-scrapyd.rb', line 114

def delproject(project)
  post('delproject.json', project: project)
end

#delversion(version, project = self.project) ⇒ Object

Delete a project version.

Parameters:

  • version (String)

    Project version

  • project (String) (defaults to: self.project)

    Project name

See Also:



107
108
109
# File 'lib/rest-scrapyd.rb', line 107

def delversion(version, project=self.project)
  post('delversion.json', project: project, version: version)
end

#listjobs(project = self.project) ⇒ Hash<String, Array<Hash>>

Get the list of pending, running and finished jobs of some project.

Parameters:

  • project (String) (defaults to: self.project)

    Project name

Returns:

  • (Hash<String, Array<Hash>>)

    Jobs for project

See Also:



99
100
101
# File 'lib/rest-scrapyd.rb', line 99

def listjobs(project=self.project)
  get('listjobs.json', project: project).reject{|k,v| k=='status'}
end

#listprojectsArray<String>

Get the list of projects uploaded to this Scrapy server.

Returns:

  • (Array<String>)

    Available projects

See Also:



75
76
77
# File 'lib/rest-scrapyd.rb', line 75

def listprojects
  get('listprojects.json')['projects']
end

#listspiders(project = self.project) ⇒ Array<String>

Get the list of spiders available in the last version of some project.

Parameters:

  • project (String) (defaults to: self.project)

    Project name

Returns:

  • (Array<String>)

    Available spiders for project

See Also:



91
92
93
# File 'lib/rest-scrapyd.rb', line 91

def listspiders(project=self.project)
  get('listspiders.json', project: project)['spiders']
end

#listversions(project = self.project) ⇒ Array<String>

Get the list of versions available for some project.

Parameters:

  • project (String) (defaults to: self.project)

    Project name

Returns:

  • (Array<String>)

    Available versions for project

See Also:



83
84
85
# File 'lib/rest-scrapyd.rb', line 83

def listversions(project=self.project)
  get('listversions.json', project: project)['versions']
end

#schedule(spider, version, settings = {}, arguments = {}) ⇒ String

Schedule a spider run (also known as a job).

Parameters:

  • spider (String)

    Spider name

  • version (String)

    Project version

  • settings (Hash<String, String>) (defaults to: {})

    Additional Scrapy settings

  • arguments (Hash<String, String>) (defaults to: {})

    Additional spider arguments

Options Hash (arguments):

  • :project (String) — default: {self.project}

    Project name

Returns:

  • (String)

    Job id

See Also:



51
52
53
54
55
56
57
58
59
60
61
# File 'lib/rest-scrapyd.rb', line 51

def schedule(spider, version, settings={}, arguments={})
  # +setting+ may appear multiple times, so can't use hash
  params = {project: self.project, spider: spider}.to_a
  params += settings.to_a.map{|k,v| ["setting", "#{k}=#{v}"]}
  params += arguments.to_a
  # rest-core doesn't (yet) support array payload, encode ourselves
  params.map{|p| p[0] = p[0].to_s} # like stringify_keys
  params = RestCore::Middleware.percent_encode(params)
  s_headers = headers.merge({'Content-Type' => 'application/x-www-form-urlencoded'})
  post('schedule.json', params, {}, headers: s_headers)['jobid']
end