Method: Polars::IO#read_ndjson

Defined in:
lib/polars/io/ndjson.rb

#read_ndjson(source, schema: nil, schema_overrides: nil, infer_schema_length: N_INFER_DEFAULT, batch_size: 1024, n_rows: nil, low_memory: false, rechunk: false, row_index_name: nil, row_index_offset: 0, ignore_errors: false, storage_options: nil, credential_provider: "auto", retries: nil, file_cache_ttl: nil, include_file_paths: nil) ⇒ DataFrame

Read into a DataFrame from a newline delimited JSON file.

Parameters:

  • source (String) —

    Path to a file.

  • schema (Object) (defaults to: nil) —

    The DataFrame schema may be declared in several ways:

    • As a dict of \{name:type} pairs; if type is nil, it will be auto-inferred.
    • As a list of column names; in this case types are automatically inferred.
    • As a list of (name,type) pairs; this is equivalent to the hash form.

    If you supply a list of column names that does not match the names in the underlying data, the names given here will overwrite them. The number of names given in the schema should match the underlying data dimensions.

  • schema_overrides (Hash) (defaults to: nil) —

    Support type specification or override of one or more columns; note that any dtypes inferred from the schema param will be overridden.

  • infer_schema_length (Integer) (defaults to: N_INFER_DEFAULT) —

    Infer the schema length from the first infer_schema_length rows.

  • batch_size (Integer) (defaults to: 1024) —

    Number of rows to read in each batch.

  • n_rows (Integer) (defaults to: nil) —

    Stop reading from JSON file after reading n_rows.

  • low_memory (Boolean) (defaults to: false) —

    Reduce memory pressure at the expense of performance.

  • rechunk (Boolean) (defaults to: false) —

    Reallocate to contiguous memory when all chunks/ files are parsed.

  • row_index_name (String) (defaults to: nil) —

    If not nil, this will insert a row count column with give name into the DataFrame.

  • row_index_offset (Integer) (defaults to: 0) —

    Offset to start the row_count column (only use if the name is set).

  • ignore_errors (Boolean) (defaults to: false) —

    Return Null if parsing fails because of schema mismatches.

  • storage_options (Hash) (defaults to: nil) —

    Options that indicate how to connect to a cloud provider.

    The cloud providers currently supported are AWS, GCP, and Azure. See supported keys here:

    • aws
    • gcp
    • azure
    • Hugging Face (hf://): Accepts an API key under the token parameter: \ {'token': '...'}, or by setting the HF_TOKEN environment variable.

    If storage_options is not provided, Polars will try to infer the information from environment variables.

  • credential_provider (Object) (defaults to: "auto") —

    Provide a function that can be called to provide cloud storage credentials. The function is expected to return a hash of credential keys along with an optional credential expiry time.

  • retries (Integer) (defaults to: nil) —

    Number of retries if accessing a cloud instance fails.

  • file_cache_ttl (Integer) (defaults to: nil) —

    Amount of time to keep downloaded cloud files since their last access time, in seconds. Uses the POLARS_FILE_CACHE_TTL environment variable (which defaults to 1 hour) if not given.

  • include_file_paths (String) (defaults to: nil) —

    Include the path of the source file(s) as a column with this name.

Returns:



65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'lib/polars/io/ndjson.rb', line 65

def read_ndjson(
  source,
  schema: nil,
  schema_overrides: nil,
  infer_schema_length: N_INFER_DEFAULT,
  batch_size: 1024,
  n_rows: nil,
  low_memory: false,
  rechunk: false,
  row_index_name: nil,
  row_index_offset: 0,
  ignore_errors: false,
  storage_options: nil,
  credential_provider: "auto",
  retries: nil,
  file_cache_ttl: nil,
  include_file_paths: nil
)
  credential_provider_builder = _init_credential_provider_builder(
    credential_provider, source, storage_options, "read_ndjson"
  )

  scan_ndjson(
    source,
    schema: schema,
    schema_overrides: schema_overrides,
    infer_schema_length: infer_schema_length,
    batch_size: batch_size,
    n_rows: n_rows,
    low_memory: low_memory,
    rechunk: rechunk,
    row_index_name: row_index_name,
    row_index_offset: row_index_offset,
    ignore_errors: ignore_errors,
    include_file_paths: include_file_paths,
    retries: retries,
    storage_options: storage_options,
    credential_provider: credential_provider_builder,
    file_cache_ttl: file_cache_ttl,
  ).collect
end