Class: Databricks::Resources::Dbfs

Inherits:
Databricks::Resource show all
Defined in:
lib/databricks/resources/dbfs.rb

Overview

Provide the DBFS API cf. docs.databricks.com/dev-tools/api/latest/dbfs.html

Instance Attribute Summary

Attributes inherited from Databricks::Resource

#properties

Instance Method Summary collapse

Methods inherited from Databricks::Resource

#add_properties, #initialize, #inspect, #new_resource, #sub_resource, sub_resources

Constructor Details

This class inherits a constructor from Databricks::Resource

Instance Method Details

#delete(path, recursive: false) ⇒ Object

Delete a path

Parameters

  • path (String): Path to delete

  • recursive (Boolean): Do we delete recursively? [default: false]



42
43
44
45
46
47
48
49
50
 
# File 'lib/databricks/resources/dbfs.rb', line 42

def delete(path, recursive: false)
  post_json(
    'dbfs/delete',
    {
      path: path,
      recursive: recursive
    }
  )
end

#list(path) ⇒ Object

List a path

Parameters

  • path (String): Path to be listed

Result

  • Array<String>: List of DBFS paths



17
18
19
 
# File 'lib/databricks/resources/dbfs.rb', line 17

def list(path)
  (get_json('dbfs/list', { path: path })['files'] || []).map { |properties| new_resource(:file, properties) }
end

#put(path, local_file) ⇒ Object

Put a new file

Parameters

  • path (String): Path to the file to create

  • local_file (String): Path to the local file to put



26
27
28
29
30
31
32
33
34
35
 
# File 'lib/databricks/resources/dbfs.rb', line 26

def put(path, local_file)
  post(
    'dbfs/put',
    {
      path: path,
      contents: ::File.new(local_file, 'rb'),
      overwrite: true
    }
  )
end

#read(path, offset: 0, length: 524_288) ⇒ Object

Read a file. Decodes the content in the json response (that is originally Base64-encoded).

Parameters

  • path (String): Path to the file to read

  • offset (Integer): Offset to read from [default: 0]

  • length (Integer): Number of nytes to read from (max 1MB) [default: 524_288]



59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/databricks/resources/dbfs.rb', line 59

def read(path, offset: 0, length: 524_288)
  raw_json = get_json(
    'dbfs/read',
    {
      path: path,
      offset: offset,
      length: length
    }
  )
  {
    'bytes_read' => raw_json['bytes_read'],
    'data' => Base64.decode64(raw_json['data'])
  }
end