Class: Polars::DataFrame
- Inherits:
-
Object
- Object
- Polars::DataFrame
- Includes:
- Plot
- Defined in:
- lib/polars/data_frame.rb
Overview
Two-dimensional data structure representing data as a table with rows and columns.
Instance Method Summary collapse
-
#!=(other) ⇒ DataFrame
Not equal.
-
#%(other) ⇒ DataFrame
Returns the modulo.
-
#*(other) ⇒ DataFrame
Performs multiplication.
-
#+(other) ⇒ DataFrame
Performs addition.
-
#-(other) ⇒ DataFrame
Performs subtraction.
-
#/(other) ⇒ DataFrame
Performs division.
-
#<(other) ⇒ DataFrame
Less than.
-
#<=(other) ⇒ DataFrame
Less than or equal.
-
#==(other) ⇒ DataFrame
Equal.
-
#>(other) ⇒ DataFrame
Greater than.
-
#>=(other) ⇒ DataFrame
Greater than or equal.
-
#[](*args) ⇒ Object
Returns subset of the DataFrame.
-
#[]=(*key, value) ⇒ Object
Set item.
-
#clear(n = 0) ⇒ DataFrame
(also: #cleared)
Create an empty copy of the current DataFrame.
-
#columns ⇒ Array
Get column names.
-
#columns=(columns) ⇒ Object
Change the column names of the DataFrame.
-
#delete(name) ⇒ Series
Drop in place if exists.
-
#describe ⇒ DataFrame
Summary statistics for a DataFrame.
-
#drop(*columns) ⇒ DataFrame
Remove column from DataFrame and return as new.
-
#drop_in_place(name) ⇒ Series
Drop in place.
-
#drop_nulls(subset: nil) ⇒ DataFrame
Return a new DataFrame where the null values are dropped.
-
#dtypes ⇒ Array
Get dtypes of columns in DataFrame.
-
#each(&block) ⇒ Object
Returns an enumerator.
-
#each_row(named: true, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
-
#equals(other, null_equal: true) ⇒ Boolean
(also: #frame_equal)
Check if DataFrame is equal to other.
-
#estimated_size(unit = "b") ⇒ Numeric
Return an estimation of the total (heap) allocated size of the DataFrame.
-
#explode(columns) ⇒ DataFrame
Explode
DataFrame
to long format by exploding a column with Lists. -
#extend(other) ⇒ DataFrame
Extend the memory backed by this
DataFrame
with the values fromother
. -
#fill_nan(fill_value) ⇒ DataFrame
Fill floating point NaN values by an Expression evaluation.
-
#fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) ⇒ DataFrame
Fill null values using the specified value or strategy.
-
#filter(predicate) ⇒ DataFrame
Filter the rows in the DataFrame based on a predicate expression.
-
#flags ⇒ Hash
Get flags that are set on the columns of this DataFrame.
-
#fold(&operation) ⇒ Series
Apply a horizontal reduction on a DataFrame.
-
#gather_every(n, offset = 0) ⇒ DataFrame
(also: #take_every)
Take every nth row in the DataFrame and return as a new DataFrame.
-
#get_column(name) ⇒ Series
Get a single column as Series by name.
-
#get_column_index(name) ⇒ Series
(also: #find_idx_by_name)
Find the index of a column by name.
-
#get_columns ⇒ Array
Get the DataFrame as a Array of Series.
-
#group_by(by, maintain_order: false) ⇒ GroupBy
(also: #groupby, #group)
Start a group by operation.
-
#group_by_dynamic(index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window") ⇒ DataFrame
(also: #groupby_dynamic)
Group based on a time value (or index value of type
:i32
,:i64
). -
#hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) ⇒ Series
Hash and combine the rows in this DataFrame.
-
#head(n = 5) ⇒ DataFrame
Get the first
n
rows. -
#height ⇒ Integer
(also: #count, #length, #size)
Get the height of the DataFrame.
-
#hstack(columns, in_place: false) ⇒ DataFrame
Return a new DataFrame grown horizontally by stacking multiple Series to it.
-
#include?(name) ⇒ Boolean
Check if DataFrame includes column.
-
#initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) ⇒ DataFrame
constructor
Create a new DataFrame.
-
#insert_column(index, series) ⇒ DataFrame
(also: #insert_at_idx)
Insert a Series at a certain column index.
-
#interpolate ⇒ DataFrame
Interpolate intermediate values.
-
#is_duplicated ⇒ Series
Get a mask of all duplicated rows in this DataFrame.
-
#is_empty ⇒ Boolean
(also: #empty?)
Check if the dataframe is empty.
-
#is_unique ⇒ Series
Get a mask of all unique rows in this DataFrame.
-
#item ⇒ Object
Return the dataframe as a scalar.
-
#iter_rows(named: false, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
-
#join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) ⇒ DataFrame
Join in SQL-like fashion.
-
#join_asof(other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false) ⇒ DataFrame
Perform an asof join.
-
#lazy ⇒ LazyFrame
Start a lazy query from this point.
-
#limit(n = 5) ⇒ DataFrame
Get the first
n
rows. -
#map_rows(return_dtype: nil, inference_size: 256, &f) ⇒ Object
(also: #apply)
Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
-
#max(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their maximum value.
-
#mean(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their mean value.
-
#median ⇒ DataFrame
Aggregate the columns of this DataFrame to their median value.
-
#merge_sorted(other, key) ⇒ DataFrame
Take two sorted DataFrames and merge them by the sorted key.
-
#min(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their minimum value.
-
#n_chunks(strategy: "first") ⇒ Object
Get number of chunks used by the ChunkedArrays of this DataFrame.
-
#n_unique(subset: nil) ⇒ DataFrame
Return the number of unique rows, or the number of unique row-subsets.
-
#null_count ⇒ DataFrame
Create a new DataFrame that shows the null counts per column.
-
#partition_by(groups, maintain_order: true, include_key: true, as_dict: false) ⇒ Object
Split into multiple DataFrames partitioned by groups.
-
#pipe(func, *args, **kwargs, &block) ⇒ Object
Offers a structured way to apply a sequence of user-defined functions (UDFs).
-
#pivot(on, index: nil, values: nil, aggregate_function: nil, maintain_order: true, sort_columns: false, separator: "_") ⇒ DataFrame
Create a spreadsheet-style pivot table as a DataFrame.
-
#plot(x = nil, y = nil, type: nil, group: nil, stacked: nil) ⇒ Vega::LiteChart
included
from Plot
Plot data.
-
#product ⇒ DataFrame
Aggregate the columns of this DataFrame to their product values.
-
#quantile(quantile, interpolation: "nearest") ⇒ DataFrame
Aggregate the columns of this DataFrame to their quantile value.
-
#rechunk ⇒ DataFrame
This will make sure all subsequent operations have optimal and predictable performance.
-
#rename(mapping) ⇒ DataFrame
Rename column names.
-
#replace(column, new_col) ⇒ DataFrame
Replace a column by a new Series.
-
#replace_column(index, series) ⇒ DataFrame
(also: #replace_at_idx)
Replace a column at an index location.
-
#reverse ⇒ DataFrame
Reverse the DataFrame.
-
#rolling(index_column:, period:, offset: nil, closed: "right", by: nil) ⇒ RollingGroupBy
(also: #groupby_rolling, #group_by_rolling)
Create rolling groups based on a time column.
-
#row(index = nil, by_predicate: nil, named: false) ⇒ Object
Get a row as tuple, either by index or by predicate.
-
#rows(named: false) ⇒ Array
Convert columnar data to rows as Ruby arrays.
-
#sample(n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil) ⇒ DataFrame
Sample from this DataFrame.
-
#schema ⇒ Hash
Get the schema.
-
#select(*exprs, **named_exprs) ⇒ DataFrame
Select columns from this DataFrame.
-
#set_sorted(column, descending: false) ⇒ DataFrame
Indicate that one or multiple columns are sorted.
-
#shape ⇒ Array
Get the shape of the DataFrame.
-
#shift(n, fill_value: nil) ⇒ DataFrame
Shift values by the given period.
-
#shift_and_fill(periods, fill_value) ⇒ DataFrame
Shift the values by a given period and fill the resulting null values.
-
#shrink_to_fit(in_place: false) ⇒ DataFrame
Shrink DataFrame memory usage.
-
#slice(offset, length = nil) ⇒ DataFrame
Get a slice of this DataFrame.
-
#sort(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column.
-
#sort!(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column in-place.
-
#std(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their standard deviation value.
-
#sum(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their sum value.
-
#tail(n = 5) ⇒ DataFrame
Get the last
n
rows. -
#to_a ⇒ Array
Returns an array representing the DataFrame.
-
#to_csv(**options) ⇒ String
Write to comma-separated values (CSV) string.
-
#to_dummies(columns: nil, separator: "_", drop_first: false) ⇒ DataFrame
Get one hot encoded dummy variables.
-
#to_h(as_series: true) ⇒ Hash
Convert DataFrame to a hash mapping column name to values.
-
#to_hashes ⇒ Array
Convert every row to a dictionary.
-
#to_numo ⇒ Numo::NArray
Convert DataFrame to a 2D Numo array.
-
#to_s ⇒ String
(also: #inspect)
Returns a string representing the DataFrame.
-
#to_series(index = 0) ⇒ Series
Select column as Series at index location.
-
#to_struct(name) ⇒ Series
Convert a
DataFrame
to aSeries
of typeStruct
. -
#transpose(include_header: false, header_name: "column", column_names: nil) ⇒ DataFrame
Transpose a DataFrame over the diagonal.
-
#unique(maintain_order: true, subset: nil, keep: "first") ⇒ DataFrame
Drop duplicate rows from this DataFrame.
-
#unnest(names) ⇒ DataFrame
Decompose a struct into its fields.
-
#unpivot(on, index: nil, variable_name: nil, value_name: nil) ⇒ DataFrame
(also: #melt)
Unpivot a DataFrame from wide to long format.
-
#unstack(step:, how: "vertical", columns: nil, fill_values: nil) ⇒ DataFrame
Unstack a long table to a wide form without doing an aggregation.
-
#upsample(time_column:, every:, by: nil, maintain_order: false) ⇒ DataFrame
Upsample a DataFrame at a regular frequency.
-
#var(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their variance value.
-
#vstack(df, in_place: false) ⇒ DataFrame
Grow this DataFrame vertically by stacking a DataFrame to it.
-
#width ⇒ Integer
Get the width of the DataFrame.
-
#with_column(column) ⇒ DataFrame
Return a new DataFrame with the column added or replaced.
-
#with_columns(*exprs, **named_exprs) ⇒ DataFrame
Add columns to this DataFrame.
-
#with_row_index(name: "index", offset: 0) ⇒ DataFrame
(also: #with_row_count)
Add a column at index 0 that counts the rows.
-
#write_avro(file, compression = "uncompressed") ⇒ nil
Write to Apache Avro file.
-
#write_csv(file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil) ⇒ String?
Write to comma-separated values (CSV) file.
-
#write_ipc(file, compression: "uncompressed") ⇒ nil
Write to Arrow IPC binary stream or Feather file.
-
#write_ipc_stream(file, compression: "uncompressed") ⇒ Object
Write to Arrow IPC record batch stream.
-
#write_json(file = nil, pretty: false, row_oriented: false) ⇒ nil
Serialize to JSON representation.
-
#write_ndjson(file = nil) ⇒ nil
Serialize to newline delimited JSON representation.
-
#write_parquet(file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil, data_page_size: nil) ⇒ nil
Write to Apache Parquet file.
Constructor Details
#initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) ⇒ DataFrame
Create a new DataFrame.
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
# File 'lib/polars/data_frame.rb', line 21 def initialize(data = nil, schema: nil, columns: nil, schema_overrides: nil, orient: nil, infer_schema_length: 100, nan_to_null: false) schema ||= columns if defined?(ActiveRecord) && (data.is_a?(ActiveRecord::Relation) || data.is_a?(ActiveRecord::Result)) raise ArgumentError, "Use read_database instead" end if data.nil? self._df = self.class.hash_to_rbdf({}, schema: schema, schema_overrides: schema_overrides) elsif data.is_a?(Hash) data = data.transform_keys { |v| v.is_a?(Symbol) ? v.to_s : v } self._df = self.class.hash_to_rbdf(data, schema: schema, schema_overrides: schema_overrides, nan_to_null: nan_to_null) elsif data.is_a?(::Array) self._df = self.class.sequence_to_rbdf(data, schema: schema, schema_overrides: schema_overrides, orient: orient, infer_schema_length: infer_schema_length) elsif data.is_a?(Series) self._df = self.class.series_to_rbdf(data, schema: schema, schema_overrides: schema_overrides) else raise ArgumentError, "DataFrame constructor called with unsupported type; got #{data.class.name}" end end |
Instance Method Details
#!=(other) ⇒ DataFrame
Not equal.
192 193 194 |
# File 'lib/polars/data_frame.rb', line 192 def !=(other) _comp(other, "neq") end |
#%(other) ⇒ DataFrame
Returns the modulo.
275 276 277 278 279 280 281 282 |
# File 'lib/polars/data_frame.rb', line 275 def %(other) if other.is_a?(DataFrame) return _from_rbdf(_df.rem_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.rem(other._s)) end |
#*(other) ⇒ DataFrame
Performs multiplication.
227 228 229 230 231 232 233 234 |
# File 'lib/polars/data_frame.rb', line 227 def *(other) if other.is_a?(DataFrame) return _from_rbdf(_df.mul_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.mul(other._s)) end |
#+(other) ⇒ DataFrame
Performs addition.
251 252 253 254 255 256 257 258 |
# File 'lib/polars/data_frame.rb', line 251 def +(other) if other.is_a?(DataFrame) return _from_rbdf(_df.add_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.add(other._s)) end |
#-(other) ⇒ DataFrame
Performs subtraction.
263 264 265 266 267 268 269 270 |
# File 'lib/polars/data_frame.rb', line 263 def -(other) if other.is_a?(DataFrame) return _from_rbdf(_df.sub_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.sub(other._s)) end |
#/(other) ⇒ DataFrame
Performs division.
239 240 241 242 243 244 245 246 |
# File 'lib/polars/data_frame.rb', line 239 def /(other) if other.is_a?(DataFrame) return _from_rbdf(_df.div_df(other._df)) end other = _prepare_other_arg(other) _from_rbdf(_df.div(other._s)) end |
#<(other) ⇒ DataFrame
Less than.
206 207 208 |
# File 'lib/polars/data_frame.rb', line 206 def <(other) _comp(other, "lt") end |
#<=(other) ⇒ DataFrame
Less than or equal.
220 221 222 |
# File 'lib/polars/data_frame.rb', line 220 def <=(other) _comp(other, "lt_eq") end |
#==(other) ⇒ DataFrame
Equal.
185 186 187 |
# File 'lib/polars/data_frame.rb', line 185 def ==(other) _comp(other, "eq") end |
#>(other) ⇒ DataFrame
Greater than.
199 200 201 |
# File 'lib/polars/data_frame.rb', line 199 def >(other) _comp(other, "gt") end |
#>=(other) ⇒ DataFrame
Greater than or equal.
213 214 215 |
# File 'lib/polars/data_frame.rb', line 213 def >=(other) _comp(other, "gt_eq") end |
#[](*args) ⇒ Object
Returns subset of the DataFrame.
316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 |
# File 'lib/polars/data_frame.rb', line 316 def [](*args) if args.size == 2 row_selection, col_selection = args # df[.., unknown] if row_selection.is_a?(Range) # multiple slices # df[.., ..] if col_selection.is_a?(Range) raise Todo end end # df[2, ..] (select row as df) if row_selection.is_a?(Integer) if col_selection.is_a?(::Array) df = self[0.., col_selection] return df.slice(row_selection, 1) end # df[2, "a"] if col_selection.is_a?(::String) || col_selection.is_a?(Symbol) return self[col_selection][row_selection] end end # column selection can be "a" and ["a", "b"] if col_selection.is_a?(::String) || col_selection.is_a?(Symbol) col_selection = [col_selection] end # df[.., 1] if col_selection.is_a?(Integer) series = to_series(col_selection) return series[row_selection] end if col_selection.is_a?(::Array) # df[.., [1, 2]] if Utils.is_int_sequence(col_selection) series_list = col_selection.map { |i| to_series(i) } df = self.class.new(series_list) return df[row_selection] end end df = self[col_selection] return df[row_selection] elsif args.size == 1 item = args[0] # select single column # df["foo"] if item.is_a?(::String) || item.is_a?(Symbol) return Utils.wrap_s(_df.get_column(item.to_s)) end # df[idx] if item.is_a?(Integer) return slice(_pos_idx(item, 0), 1) end # df[..] if item.is_a?(Range) return Slice.new(self).apply(item) end if item.is_a?(::Array) && item.all? { |v| Utils.strlike?(v) } # select multiple columns # df[["foo", "bar"]] return _from_rbdf(_df.select(item.map(&:to_s))) end if Utils.is_int_sequence(item) item = Series.new("", item) end if item.is_a?(Series) dtype = item.dtype if dtype == String return _from_rbdf(_df.select(item)) elsif dtype == UInt32 return _from_rbdf(_df.take_with_series(item._s)) elsif [UInt8, UInt16, UInt64, Int8, Int16, Int32, Int64].include?(dtype) return _from_rbdf( _df.take_with_series(_pos_idxs(item, 0)._s) ) end end end # Ruby-specific if item.is_a?(Expr) || item.is_a?(Series) return filter(item) end raise ArgumentError, "Cannot get item of type: #{item.class.name}" end |
#[]=(*key, value) ⇒ Object
Set item.
418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 |
# File 'lib/polars/data_frame.rb', line 418 def []=(*key, value) if key.length == 1 key = key.first elsif key.length != 2 raise ArgumentError, "wrong number of arguments (given #{key.length + 1}, expected 2..3)" end if Utils.strlike?(key) if value.is_a?(::Array) || (defined?(Numo::NArray) && value.is_a?(Numo::NArray)) value = Series.new(value) elsif !value.is_a?(Series) value = Polars.lit(value) end self._df = with_column(value.alias(key.to_s))._df elsif key.is_a?(::Array) row_selection, col_selection = key if Utils.strlike?(col_selection) s = self[col_selection] elsif col_selection.is_a?(Integer) raise Todo else raise ArgumentError, "column selection not understood: #{col_selection}" end s[row_selection] = value if col_selection.is_a?(Integer) replace_column(col_selection, s) elsif Utils.strlike?(col_selection) replace(col_selection, s) end else raise Todo end end |
#clear(n = 0) ⇒ DataFrame Also known as: cleared
Create an empty copy of the current DataFrame.
Returns a DataFrame with identical schema but no data.
2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 |
# File 'lib/polars/data_frame.rb', line 2760 def clear(n = 0) if n == 0 _from_rbdf(_df.clear) elsif n > 0 || len > 0 self.class.new( schema.to_h { |nm, tp| [nm, Series.new(nm, [], dtype: tp).extend_constant(nil, n)] } ) else clone end end |
#columns ⇒ Array
Get column names.
102 103 104 |
# File 'lib/polars/data_frame.rb', line 102 def columns _df.columns end |
#columns=(columns) ⇒ Object
Change the column names of the DataFrame.
135 136 137 |
# File 'lib/polars/data_frame.rb', line 135 def columns=(columns) _df.set_column_names(columns) end |
#delete(name) ⇒ Series
Drop in place if exists.
2720 2721 2722 |
# File 'lib/polars/data_frame.rb', line 2720 def delete(name) drop_in_place(name) if include?(name) end |
#describe ⇒ DataFrame
Summary statistics for a DataFrame.
1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 |
# File 'lib/polars/data_frame.rb', line 1195 def describe describe_cast = lambda do |stat| columns = [] self.columns.each_with_index do |s, i| if self[s].is_numeric || self[s].is_boolean columns << stat[0.., i].cast(:f64) else # for dates, strings, etc, we cast to string so that all # statistics can be shown columns << stat[0.., i].cast(:str) end end self.class.new(columns) end summary = _from_rbdf( Polars.concat( [ describe_cast.( self.class.new(columns.to_h { |c| [c, [height]] }) ), describe_cast.(null_count), describe_cast.(mean), describe_cast.(std), describe_cast.(min), describe_cast.(max), describe_cast.(median) ] )._df ) summary.insert_column( 0, Polars::Series.new( "describe", ["count", "null_count", "mean", "std", "min", "max", "median"], ) ) summary end |
#drop(*columns) ⇒ DataFrame
Remove column from DataFrame and return as new.
2682 2683 2684 |
# File 'lib/polars/data_frame.rb', line 2682 def drop(*columns) lazy.drop(*columns).collect(_eager: true) end |
#drop_in_place(name) ⇒ Series
Drop in place.
2710 2711 2712 |
# File 'lib/polars/data_frame.rb', line 2710 def drop_in_place(name) Utils.wrap_s(_df.drop_in_place(name)) end |
#drop_nulls(subset: nil) ⇒ DataFrame
Return a new DataFrame where the null values are dropped.
1576 1577 1578 |
# File 'lib/polars/data_frame.rb', line 1576 def drop_nulls(subset: nil) lazy.drop_nulls(subset: subset).collect(_eager: true) end |
#dtypes ⇒ Array
Get dtypes of columns in DataFrame. Dtypes can also be found in column headers when printing the DataFrame.
153 154 155 |
# File 'lib/polars/data_frame.rb', line 153 def dtypes _df.dtypes end |
#each(&block) ⇒ Object
Returns an enumerator.
309 310 311 |
# File 'lib/polars/data_frame.rb', line 309 def each(&block) get_columns.each(&block) end |
#each_row(named: true, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
4521 4522 4523 |
# File 'lib/polars/data_frame.rb', line 4521 def each_row(named: true, buffer_size: 500, &block) iter_rows(named: named, buffer_size: buffer_size, &block) end |
#equals(other, null_equal: true) ⇒ Boolean Also known as: frame_equal
Check if DataFrame is equal to other.
1388 1389 1390 |
# File 'lib/polars/data_frame.rb', line 1388 def equals(other, null_equal: true) _df.equals(other._df, null_equal) end |
#estimated_size(unit = "b") ⇒ Numeric
Return an estimation of the total (heap) allocated size of the DataFrame.
Estimated size is given in the specified unit (bytes by default).
This estimation is the sum of the size of its buffers, validity, including nested arrays. Multiple arrays may share buffers and bitmaps. Therefore, the size of 2 arrays is not the sum of the sizes computed from this function. In particular, StructArray's size is an upper bound.
When an array is sliced, its allocated size remains constant because the buffer unchanged. However, this function will yield a smaller number. This is because this function returns the visible size of the buffer, not its total capacity.
FFI buffers are included in this estimation.
942 943 944 945 |
# File 'lib/polars/data_frame.rb', line 942 def estimated_size(unit = "b") sz = _df.estimated_size Utils.scale_bytes(sz, to: unit) end |
#explode(columns) ⇒ DataFrame
Explode DataFrame
to long format by exploding a column with Lists.
2958 2959 2960 |
# File 'lib/polars/data_frame.rb', line 2958 def explode(columns) lazy.explode(columns).collect(no_optimization: true) end |
#extend(other) ⇒ DataFrame
Extend the memory backed by this DataFrame
with the values from other
.
Different from vstack
which adds the chunks from other
to the chunks of this
DataFrame
extend
appends the data from other
to the underlying memory
locations and thus may cause a reallocation.
If this does not cause a reallocation, the resulting data structure will not have any extra chunks and thus will yield faster queries.
Prefer extend
over vstack
when you want to do a query after a single append.
For instance during online operations where you add n
rows and rerun a query.
Prefer vstack
over extend
when you want to append many times before doing a
query. For instance when you read in multiple files and when to store them in a
single DataFrame
. In the latter case, finish the sequence of vstack
operations with a rechunk
.
2622 2623 2624 2625 |
# File 'lib/polars/data_frame.rb', line 2622 def extend(other) _df.extend(other._df) self end |
#fill_nan(fill_value) ⇒ DataFrame
Note that floating point NaNs (Not a Number) are not missing values!
To replace missing values, use fill_null
.
Fill floating point NaN values by an Expression evaluation.
2923 2924 2925 |
# File 'lib/polars/data_frame.rb', line 2923 def fill_nan(fill_value) lazy.fill_nan(fill_value).collect(no_optimization: true) end |
#fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) ⇒ DataFrame
Fill null values using the specified value or strategy.
2883 2884 2885 2886 2887 2888 2889 2890 |
# File 'lib/polars/data_frame.rb', line 2883 def fill_null(value = nil, strategy: nil, limit: nil, matches_supertype: true) _from_rbdf( lazy .fill_null(value, strategy: strategy, limit: limit, matches_supertype: matches_supertype) .collect(no_optimization: true) ._df ) end |
#filter(predicate) ⇒ DataFrame
Filter the rows in the DataFrame based on a predicate expression.
1161 1162 1163 |
# File 'lib/polars/data_frame.rb', line 1161 def filter(predicate) lazy.filter(predicate).collect end |
#flags ⇒ Hash
Get flags that are set on the columns of this DataFrame.
160 161 162 |
# File 'lib/polars/data_frame.rb', line 160 def flags columns.to_h { |name| [name, self[name].flags] } end |
#fold(&operation) ⇒ Series
Apply a horizontal reduction on a DataFrame.
This can be used to effectively determine aggregations on a row level, and can be applied to any DataType that can be supercasted (casted to a similar parent type).
An example of the supercast rules when applying an arithmetic operation on two DataTypes are for instance:
i8 + str = str f32 + i64 = f32 f32 + f64 = f64
4330 4331 4332 4333 4334 4335 4336 4337 |
# File 'lib/polars/data_frame.rb', line 4330 def fold(&operation) acc = to_series(0) 1.upto(width - 1) do |i| acc = operation.call(acc, to_series(i)) end acc end |
#gather_every(n, offset = 0) ⇒ DataFrame Also known as: take_every
Take every nth row in the DataFrame and return as a new DataFrame.
4558 4559 4560 |
# File 'lib/polars/data_frame.rb', line 4558 def gather_every(n, offset = 0) select(F.col("*").gather_every(n, offset)) end |
#get_column(name) ⇒ Series
Get a single column as Series by name.
2800 2801 2802 |
# File 'lib/polars/data_frame.rb', line 2800 def get_column(name) self[name] end |
#get_column_index(name) ⇒ Series Also known as: find_idx_by_name
Find the index of a column by name.
1248 1249 1250 |
# File 'lib/polars/data_frame.rb', line 1248 def get_column_index(name) _df.get_column_index(name) end |
#get_columns ⇒ Array
Get the DataFrame as a Array of Series.
2778 2779 2780 |
# File 'lib/polars/data_frame.rb', line 2778 def get_columns _df.get_columns.map { |s| Utils.wrap_s(s) } end |
#group_by(by, maintain_order: false) ⇒ GroupBy Also known as: groupby, group
Start a group by operation.
1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 |
# File 'lib/polars/data_frame.rb', line 1684 def group_by(by, maintain_order: false) if !Utils.bool?(maintain_order) raise TypeError, "invalid input for group_by arg `maintain_order`: #{maintain_order}." end GroupBy.new( self, by, maintain_order: maintain_order ) end |
#group_by_dynamic(index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window") ⇒ DataFrame Also known as: groupby_dynamic
Group based on a time value (or index value of type :i32
, :i64
).
Time windows are calculated and rows are assigned to windows. Different from a normal group by is that a row can be member of multiple groups. The time/index window could be seen as a rolling window, with a window size determined by dates/times/values instead of slots in the DataFrame.
A window is defined by:
- every: interval of the window
- period: length of the window
- offset: offset of the window
The every
, period
and offset
arguments are created with
the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
In case of a group_by_dynamic on an integer column, the windows are defined by:
- "1i" # length 1
- "10i" # length 10
2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 |
# File 'lib/polars/data_frame.rb', line 2024 def group_by_dynamic( index_column, every:, period: nil, offset: nil, truncate: true, include_boundaries: false, closed: "left", by: nil, start_by: "window" ) DynamicGroupBy.new( self, index_column, every, period, offset, truncate, include_boundaries, closed, by, start_by ) end |
#hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) ⇒ Series
Hash and combine the rows in this DataFrame.
The hash value is of type :u64
.
4595 4596 4597 4598 4599 4600 4601 |
# File 'lib/polars/data_frame.rb', line 4595 def hash_rows(seed: 0, seed_1: nil, seed_2: nil, seed_3: nil) k0 = seed k1 = seed_1.nil? ? seed : seed_1 k2 = seed_2.nil? ? seed : seed_2 k3 = seed_3.nil? ? seed : seed_3 Utils.wrap_s(_df.hash_rows(k0, k1, k2, k3)) end |
#head(n = 5) ⇒ DataFrame
Get the first n
rows.
1515 1516 1517 |
# File 'lib/polars/data_frame.rb', line 1515 def head(n = 5) _from_rbdf(_df.head(n)) end |
#height ⇒ Integer Also known as: count, length, size
Get the height of the DataFrame.
69 70 71 |
# File 'lib/polars/data_frame.rb', line 69 def height _df.height end |
#hstack(columns, in_place: false) ⇒ DataFrame
Return a new DataFrame grown horizontally by stacking multiple Series to it.
2524 2525 2526 2527 2528 2529 2530 2531 2532 2533 2534 |
# File 'lib/polars/data_frame.rb', line 2524 def hstack(columns, in_place: false) if !columns.is_a?(::Array) columns = columns.get_columns end if in_place _df.hstack_mut(columns.map(&:_s)) self else _from_rbdf(_df.hstack(columns.map(&:_s))) end end |
#include?(name) ⇒ Boolean
Check if DataFrame includes column.
302 303 304 |
# File 'lib/polars/data_frame.rb', line 302 def include?(name) columns.include?(name) end |
#insert_column(index, series) ⇒ DataFrame Also known as: insert_at_idx
Insert a Series at a certain column index. This operation is in place.
1114 1115 1116 1117 1118 1119 1120 |
# File 'lib/polars/data_frame.rb', line 1114 def insert_column(index, series) if index < 0 index = columns.length + index end _df.insert_column(index, series._s) self end |
#interpolate ⇒ DataFrame
Interpolate intermediate values. The interpolation method is linear.
4628 4629 4630 |
# File 'lib/polars/data_frame.rb', line 4628 def interpolate select(F.col("*").interpolate) end |
#is_duplicated ⇒ Series
Get a mask of all duplicated rows in this DataFrame.
3435 3436 3437 |
# File 'lib/polars/data_frame.rb', line 3435 def is_duplicated Utils.wrap_s(_df.is_duplicated) end |
#is_empty ⇒ Boolean Also known as: empty?
Check if the dataframe is empty.
4642 4643 4644 |
# File 'lib/polars/data_frame.rb', line 4642 def is_empty height == 0 end |
#is_unique ⇒ Series
Get a mask of all unique rows in this DataFrame.
3460 3461 3462 |
# File 'lib/polars/data_frame.rb', line 3460 def is_unique Utils.wrap_s(_df.is_unique) end |
#item ⇒ Object
Return the dataframe as a scalar.
Equivalent to df[0,0]
, with a check that the shape is (1,1).
466 467 468 469 470 471 |
# File 'lib/polars/data_frame.rb', line 466 def item if shape != [1, 1] raise ArgumentError, "Can only call .item if the dataframe is of shape (1,1), dataframe is of shape #{shape}" end self[0, 0] end |
#iter_rows(named: false, buffer_size: 500, &block) ⇒ Object
Returns an iterator over the DataFrame of rows of Ruby-native values.
4474 4475 4476 4477 4478 4479 4480 4481 4482 4483 4484 4485 4486 4487 4488 4489 4490 4491 4492 4493 4494 4495 4496 4497 4498 4499 4500 4501 4502 4503 4504 4505 |
# File 'lib/polars/data_frame.rb', line 4474 def iter_rows(named: false, buffer_size: 500, &block) return to_enum(:iter_rows, named: named, buffer_size: buffer_size) unless block_given? # load into the local namespace for a modest performance boost in the hot loops columns = self.columns # note: buffering rows results in a 2-4x speedup over individual calls # to ".row(i)", so it should only be disabled in extremely specific cases. if buffer_size offset = 0 while offset < height zerocopy_slice = slice(offset, buffer_size) rows_chunk = zerocopy_slice.rows(named: false) if named rows_chunk.each do |row| yield columns.zip(row).to_h end else rows_chunk.each(&block) end offset += buffer_size end elsif named height.times do |i| yield columns.zip(row(i)).to_h end else height.times do |i| yield row(i) end end end |
#join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) ⇒ DataFrame
Join in SQL-like fashion.
2364 2365 2366 2367 2368 2369 2370 2371 2372 2373 2374 2375 2376 |
# File 'lib/polars/data_frame.rb', line 2364 def join(other, left_on: nil, right_on: nil, on: nil, how: "inner", suffix: "_right", join_nulls: false) lazy .join( other.lazy, left_on: left_on, right_on: right_on, on: on, how: how, suffix: suffix, join_nulls: join_nulls ) .collect(no_optimization: true) end |
#join_asof(other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false) ⇒ DataFrame
Perform an asof join.
This is similar to a left-join except that we match on nearest key rather than equal keys.
Both DataFrames must be sorted by the asof_join key.
For each row in the left DataFrame:
- A "backward" search selects the last row in the right DataFrame whose 'on' key is less than or equal to the left's key.
- A "forward" search selects the first row in the right DataFrame whose 'on' key is greater than or equal to the left's key.
The default is "backward".
2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 |
# File 'lib/polars/data_frame.rb', line 2234 def join_asof( other, left_on: nil, right_on: nil, on: nil, by_left: nil, by_right: nil, by: nil, strategy: "backward", suffix: "_right", tolerance: nil, allow_parallel: true, force_parallel: false ) lazy .join_asof( other.lazy, left_on: left_on, right_on: right_on, on: on, by_left: by_left, by_right: by_right, by: by, strategy: strategy, suffix: suffix, tolerance: tolerance, allow_parallel: allow_parallel, force_parallel: force_parallel ) .collect(no_optimization: true) end |
#lazy ⇒ LazyFrame
Start a lazy query from this point.
3467 3468 3469 |
# File 'lib/polars/data_frame.rb', line 3467 def lazy wrap_ldf(_df.lazy) end |
#limit(n = 5) ⇒ DataFrame
Get the first n
rows.
Alias for #head.
1484 1485 1486 |
# File 'lib/polars/data_frame.rb', line 1484 def limit(n = 5) head(n) end |
#map_rows(return_dtype: nil, inference_size: 256, &f) ⇒ Object Also known as: apply
The frame-level apply
cannot track column names (as the UDF is a black-box
that may arbitrarily drop, rearrange, transform, or add new columns); if you
want to apply a UDF such that column names are preserved, you should use the
expression-level apply
syntax instead.
Apply a custom/user-defined function (UDF) over the rows of the DataFrame.
The UDF will receive each row as a tuple of values: udf(row)
.
Implementing logic using a Ruby function is almost always significantly slower and more memory intensive than implementing the same logic using the native expression API because:
- The native expression engine runs in Rust; UDFs run in Ruby.
- Use of Ruby UDFs forces the DataFrame to be materialized in memory.
- Polars-native expressions can be parallelised (UDFs cannot).
- Polars-native expressions can be logically optimised (UDFs cannot).
Wherever possible you should strongly prefer the native expression API to achieve the best performance.
2438 2439 2440 2441 2442 2443 2444 2445 |
# File 'lib/polars/data_frame.rb', line 2438 def map_rows(return_dtype: nil, inference_size: 256, &f) out, is_df = _df.map_rows(f, return_dtype, inference_size) if is_df _from_rbdf(out) else _from_rbdf(Utils.wrap_s(out).to_frame._df) end end |
#max(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their maximum value.
3727 3728 3729 3730 3731 3732 3733 3734 3735 |
# File 'lib/polars/data_frame.rb', line 3727 def max(axis: 0) if axis == 0 lazy.max.collect(_eager: true) elsif axis == 1 Utils.wrap_s(_df.max_horizontal) else raise ArgumentError, "Axis should be 0 or 1." end end |
#mean(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their mean value.
3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 |
# File 'lib/polars/data_frame.rb', line 3845 def mean(axis: 0, null_strategy: "ignore") case axis when 0 lazy.mean.collect(_eager: true) when 1 Utils.wrap_s(_df.mean_horizontal(null_strategy)) else raise ArgumentError, "Axis should be 0 or 1." end end |
#median ⇒ DataFrame
Aggregate the columns of this DataFrame to their median value.
3960 3961 3962 |
# File 'lib/polars/data_frame.rb', line 3960 def median lazy.median.collect(_eager: true) end |
#merge_sorted(other, key) ⇒ DataFrame
Take two sorted DataFrames and merge them by the sorted key.
The output of this operation will also be sorted. It is the callers responsibility that the frames are sorted by that key otherwise the output will not make sense.
The schemas of both DataFrames must be equal.
4757 4758 4759 |
# File 'lib/polars/data_frame.rb', line 4757 def merge_sorted(other, key) lazy.merge_sorted(other.lazy, key).collect(_eager: true) end |
#min(axis: 0) ⇒ DataFrame
Aggregate the columns of this DataFrame to their minimum value.
3759 3760 3761 3762 3763 3764 3765 3766 3767 |
# File 'lib/polars/data_frame.rb', line 3759 def min(axis: 0) if axis == 0 lazy.min.collect(_eager: true) elsif axis == 1 Utils.wrap_s(_df.min_horizontal) else raise ArgumentError, "Axis should be 0 or 1." end end |
#n_chunks(strategy: "first") ⇒ Object
Get number of chunks used by the ChunkedArrays of this DataFrame.
3695 3696 3697 3698 3699 3700 3701 3702 3703 |
# File 'lib/polars/data_frame.rb', line 3695 def n_chunks(strategy: "first") if strategy == "first" _df.n_chunks elsif strategy == "all" get_columns.map(&:n_chunks) else raise ArgumentError, "Strategy: '{strategy}' not understood. Choose one of {{'first', 'all'}}" end end |
#n_unique(subset: nil) ⇒ DataFrame
Return the number of unique rows, or the number of unique row-subsets.
4133 4134 4135 4136 4137 4138 4139 4140 4141 4142 4143 4144 4145 4146 4147 4148 4149 |
# File 'lib/polars/data_frame.rb', line 4133 def n_unique(subset: nil) if subset.is_a?(StringIO) subset = [Polars.col(subset)] elsif subset.is_a?(Expr) subset = [subset] end if subset.is_a?(::Array) && subset.length == 1 expr = Utils.wrap_expr(Utils.parse_into_expression(subset[0], str_as_lit: false)) else struct_fields = subset.nil? ? Polars.all : subset expr = Polars.struct(struct_fields) end df = lazy.select(expr.n_unique).collect df.is_empty ? 0 : df.row(0)[0] end |
#null_count ⇒ DataFrame
Create a new DataFrame that shows the null counts per column.
4183 4184 4185 |
# File 'lib/polars/data_frame.rb', line 4183 def null_count _from_rbdf(_df.null_count) end |
#partition_by(groups, maintain_order: true, include_key: true, as_dict: false) ⇒ Object
Split into multiple DataFrames partitioned by groups.
3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 |
# File 'lib/polars/data_frame.rb', line 3308 def partition_by(groups, maintain_order: true, include_key: true, as_dict: false) if groups.is_a?(::String) groups = [groups] elsif !groups.is_a?(::Array) groups = Array(groups) end if as_dict out = {} if groups.length == 1 _df.partition_by(groups, maintain_order, include_key).each do |df| df = _from_rbdf(df) out[df[groups][0, 0]] = df end else _df.partition_by(groups, maintain_order, include_key).each do |df| df = _from_rbdf(df) out[df[groups].row(0)] = df end end out else _df.partition_by(groups, maintain_order, include_key).map { |df| _from_rbdf(df) } end end |
#pipe(func, *args, **kwargs, &block) ⇒ Object
It is recommended to use LazyFrame when piping operations, in order to fully take advantage of query optimization and parallelization. See #lazy.
Offers a structured way to apply a sequence of user-defined functions (UDFs).
1616 1617 1618 |
# File 'lib/polars/data_frame.rb', line 1616 def pipe(func, *args, **kwargs, &block) func.call(self, *args, **kwargs, &block) end |
#pivot(on, index: nil, values: nil, aggregate_function: nil, maintain_order: true, sort_columns: false, separator: "_") ⇒ DataFrame
Create a spreadsheet-style pivot table as a DataFrame.
2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 |
# File 'lib/polars/data_frame.rb', line 2999 def pivot( on, index: nil, values: nil, aggregate_function: nil, maintain_order: true, sort_columns: false, separator: "_" ) index = Utils.(self, index) on = Utils.(self, on) if !values.nil? values = Utils.(self, values) end if aggregate_function.is_a?(::String) case aggregate_function when "first" aggregate_expr = F.element.first._rbexpr when "sum" aggregate_expr = F.element.sum._rbexpr when "max" aggregate_expr = F.element.max._rbexpr when "min" aggregate_expr = F.element.min._rbexpr when "mean" aggregate_expr = F.element.mean._rbexpr when "median" aggregate_expr = F.element.median._rbexpr when "last" aggregate_expr = F.element.last._rbexpr when "len" aggregate_expr = F.len._rbexpr when "count" warn "`aggregate_function: \"count\"` input for `pivot` is deprecated. Use `aggregate_function: \"len\"` instead." aggregate_expr = F.len._rbexpr else raise ArgumentError, "Argument aggregate fn: '#{aggregate_fn}' was not expected." end elsif aggregate_function.nil? aggregate_expr = nil else aggregate_expr = aggregate_function._rbexpr end _from_rbdf( _df.pivot_expr( on, index, values, maintain_order, sort_columns, aggregate_expr, separator ) ) end |
#plot(x = nil, y = nil, type: nil, group: nil, stacked: nil) ⇒ Vega::LiteChart Originally defined in module Plot
Plot data.
#product ⇒ DataFrame
Aggregate the columns of this DataFrame to their product values.
3986 3987 3988 |
# File 'lib/polars/data_frame.rb', line 3986 def product select(Polars.all.product) end |
#quantile(quantile, interpolation: "nearest") ⇒ DataFrame
Aggregate the columns of this DataFrame to their quantile value.
4017 4018 4019 |
# File 'lib/polars/data_frame.rb', line 4017 def quantile(quantile, interpolation: "nearest") lazy.quantile(quantile, interpolation: interpolation).collect(_eager: true) end |
#rechunk ⇒ DataFrame
This will make sure all subsequent operations have optimal and predictable performance.
4157 4158 4159 |
# File 'lib/polars/data_frame.rb', line 4157 def rechunk _from_rbdf(_df.rechunk) end |
#rename(mapping) ⇒ DataFrame
Rename column names.
1063 1064 1065 |
# File 'lib/polars/data_frame.rb', line 1063 def rename(mapping) lazy.rename(mapping).collect(no_optimization: true) end |
#replace(column, new_col) ⇒ DataFrame
Replace a column by a new Series.
1417 1418 1419 1420 |
# File 'lib/polars/data_frame.rb', line 1417 def replace(column, new_col) _df.replace(column.to_s, new_col._s) self end |
#replace_column(index, series) ⇒ DataFrame Also known as: replace_at_idx
Replace a column at an index location.
1283 1284 1285 1286 1287 1288 1289 |
# File 'lib/polars/data_frame.rb', line 1283 def replace_column(index, series) if index < 0 index = columns.length + index end _df.replace_column(index, series._s) self end |
#reverse ⇒ DataFrame
Reverse the DataFrame.
1032 1033 1034 |
# File 'lib/polars/data_frame.rb', line 1032 def reverse select(Polars.col("*").reverse) end |
#rolling(index_column:, period:, offset: nil, closed: "right", by: nil) ⇒ RollingGroupBy Also known as: groupby_rolling, group_by_rolling
Create rolling groups based on a time column.
Also works for index values of type :i32
or :i64
.
Different from a dynamic_group_by
the windows are now determined by the
individual values and are not of constant intervals. For constant intervals use
group_by_dynamic
The period
and offset
arguments are created either from a timedelta, or
by using the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
In case of a group_by_rolling on an integer column, the windows are defined by:
- "1i" # length 1
- "10i" # length 10
1781 1782 1783 1784 1785 1786 1787 1788 1789 |
# File 'lib/polars/data_frame.rb', line 1781 def rolling( index_column:, period:, offset: nil, closed: "right", by: nil ) RollingGroupBy.new(self, index_column, period, offset, closed, by) end |
#row(index = nil, by_predicate: nil, named: false) ⇒ Object
The index
and by_predicate
params are mutually exclusive. Additionally,
to ensure clarity, the by_predicate
parameter must be supplied by keyword.
When using by_predicate
it is an error condition if anything other than
one row is returned; more than one row raises TooManyRowsReturned
, and
zero rows will raise NoRowsReturned
(both inherit from RowsException
).
Get a row as tuple, either by index or by predicate.
4378 4379 4380 4381 4382 4383 4384 4385 4386 4387 4388 4389 4390 4391 4392 4393 4394 4395 4396 4397 4398 4399 4400 4401 4402 4403 4404 4405 4406 4407 4408 4409 4410 4411 4412 |
# File 'lib/polars/data_frame.rb', line 4378 def row(index = nil, by_predicate: nil, named: false) if !index.nil? && !by_predicate.nil? raise ArgumentError, "Cannot set both 'index' and 'by_predicate'; mutually exclusive" elsif index.is_a?(Expr) raise TypeError, "Expressions should be passed to the 'by_predicate' param" end if !index.nil? row = _df.row_tuple(index) if named columns.zip(row).to_h else row end elsif !by_predicate.nil? if !by_predicate.is_a?(Expr) raise TypeError, "Expected by_predicate to be an expression; found #{by_predicate.class.name}" end rows = filter(by_predicate).rows n_rows = rows.length if n_rows > 1 raise TooManyRowsReturned, "Predicate #{by_predicate} returned #{n_rows} rows" elsif n_rows == 0 raise NoRowsReturned, "Predicate #{by_predicate} returned no rows" end row = rows[0] if named columns.zip(row).to_h else row end else raise ArgumentError, "One of 'index' or 'by_predicate' must be set" end end |
#rows(named: false) ⇒ Array
Convert columnar data to rows as Ruby arrays.
4435 4436 4437 4438 4439 4440 4441 4442 4443 4444 |
# File 'lib/polars/data_frame.rb', line 4435 def rows(named: false) if named columns = self.columns _df.row_tuples.map do |v| columns.zip(v).to_h end else _df.row_tuples end end |
#sample(n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil) ⇒ DataFrame
Sample from this DataFrame.
4223 4224 4225 4226 4227 4228 4229 4230 4231 4232 4233 4234 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 |
# File 'lib/polars/data_frame.rb', line 4223 def sample( n: nil, frac: nil, with_replacement: false, shuffle: false, seed: nil ) if !n.nil? && !frac.nil? raise ArgumentError, "cannot specify both `n` and `frac`" end if n.nil? && !frac.nil? frac = Series.new("frac", [frac]) unless frac.is_a?(Series) return _from_rbdf( _df.sample_frac(frac._s, with_replacement, shuffle, seed) ) end if n.nil? n = 1 end n = Series.new("", [n]) unless n.is_a?(Series) _from_rbdf(_df.sample_n(n._s, with_replacement, shuffle, seed)) end |
#schema ⇒ Hash
Get the schema.
178 179 180 |
# File 'lib/polars/data_frame.rb', line 178 def schema columns.zip(dtypes).to_h end |
#select(*exprs, **named_exprs) ⇒ DataFrame
Select columns from this DataFrame.
3559 3560 3561 |
# File 'lib/polars/data_frame.rb', line 3559 def select(*exprs, **named_exprs) lazy.select(*exprs, **named_exprs).collect(_eager: true) end |
#set_sorted(column, descending: false) ⇒ DataFrame
Indicate that one or multiple columns are sorted.
4769 4770 4771 4772 4773 4774 4775 4776 |
# File 'lib/polars/data_frame.rb', line 4769 def set_sorted( column, descending: false ) lazy .set_sorted(column, descending: descending) .collect(no_optimization: true) end |
#shape ⇒ Array
Get the shape of the DataFrame.
57 58 59 |
# File 'lib/polars/data_frame.rb', line 57 def shape _df.shape end |
#shift(n, fill_value: nil) ⇒ DataFrame
Shift values by the given period.
3377 3378 3379 |
# File 'lib/polars/data_frame.rb', line 3377 def shift(n, fill_value: nil) lazy.shift(n, fill_value: fill_value).collect(_eager: true) end |
#shift_and_fill(periods, fill_value) ⇒ DataFrame
Shift the values by a given period and fill the resulting null values.
3410 3411 3412 |
# File 'lib/polars/data_frame.rb', line 3410 def shift_and_fill(periods, fill_value) shift(periods, fill_value: fill_value) end |
#shrink_to_fit(in_place: false) ⇒ DataFrame
Shrink DataFrame memory usage.
Shrinks to fit the exact capacity needed to hold the data.
4530 4531 4532 4533 4534 4535 4536 4537 4538 4539 |
# File 'lib/polars/data_frame.rb', line 4530 def shrink_to_fit(in_place: false) if in_place _df.shrink_to_fit self else df = clone df._df.shrink_to_fit df end end |
#slice(offset, length = nil) ⇒ DataFrame
Get a slice of this DataFrame.
1451 1452 1453 1454 1455 1456 |
# File 'lib/polars/data_frame.rb', line 1451 def slice(offset, length = nil) if !length.nil? && length < 0 length = height - offset + length end _from_rbdf(_df.slice(offset, length)) end |
#sort(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column.
1340 1341 1342 1343 1344 |
# File 'lib/polars/data_frame.rb', line 1340 def sort(by, reverse: false, nulls_last: false) lazy .sort(by, reverse: reverse, nulls_last: nulls_last) .collect(no_optimization: true) end |
#sort!(by, reverse: false, nulls_last: false) ⇒ DataFrame
Sort the DataFrame by column in-place.
1356 1357 1358 |
# File 'lib/polars/data_frame.rb', line 1356 def sort!(by, reverse: false, nulls_last: false) self._df = sort(by, reverse: reverse, nulls_last: nulls_last)._df end |
#std(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their standard deviation value.
3893 3894 3895 |
# File 'lib/polars/data_frame.rb', line 3893 def std(ddof: 1) lazy.std(ddof: ddof).collect(_eager: true) end |
#sum(axis: 0, null_strategy: "ignore") ⇒ DataFrame
Aggregate the columns of this DataFrame to their sum value.
3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 |
# File 'lib/polars/data_frame.rb', line 3807 def sum(axis: 0, null_strategy: "ignore") case axis when 0 lazy.sum.collect(_eager: true) when 1 Utils.wrap_s(_df.sum_horizontal(null_strategy)) else raise ArgumentError, "Axis should be 0 or 1." end end |
#tail(n = 5) ⇒ DataFrame
Get the last n
rows.
1546 1547 1548 |
# File 'lib/polars/data_frame.rb', line 1546 def tail(n = 5) _from_rbdf(_df.tail(n)) end |
#to_a ⇒ Array
Returns an array representing the DataFrame
295 296 297 |
# File 'lib/polars/data_frame.rb', line 295 def to_a rows(named: true) end |
#to_csv(**options) ⇒ String
Write to comma-separated values (CSV) string.
757 758 759 |
# File 'lib/polars/data_frame.rb', line 757 def to_csv(**) write_csv(**) end |
#to_dummies(columns: nil, separator: "_", drop_first: false) ⇒ DataFrame
Get one hot encoded dummy variables.
4048 4049 4050 4051 4052 4053 |
# File 'lib/polars/data_frame.rb', line 4048 def to_dummies(columns: nil, separator: "_", drop_first: false) if columns.is_a?(::String) columns = [columns] end _from_rbdf(_df.to_dummies(columns, separator, drop_first)) end |
#to_h(as_series: true) ⇒ Hash
Convert DataFrame to a hash mapping column name to values.
478 479 480 481 482 483 484 |
# File 'lib/polars/data_frame.rb', line 478 def to_h(as_series: true) if as_series get_columns.to_h { |s| [s.name, s] } else get_columns.to_h { |s| [s.name, s.to_a] } end end |
#to_hashes ⇒ Array
Convert every row to a dictionary.
Note that this is slow.
497 498 499 500 501 502 503 504 |
# File 'lib/polars/data_frame.rb', line 497 def to_hashes rbdf = _df names = columns height.times.map do |i| names.zip(rbdf.row_tuple(i)).to_h end end |
#to_numo ⇒ Numo::NArray
Convert DataFrame to a 2D Numo array.
This operation clones data.
518 519 520 521 522 523 524 525 |
# File 'lib/polars/data_frame.rb', line 518 def to_numo out = _df.to_numo if out.nil? Numo::NArray.vstack(width.times.map { |i| to_series(i).to_numo }).transpose else out end end |
#to_s ⇒ String Also known as: inspect
Returns a string representing the DataFrame.
287 288 289 |
# File 'lib/polars/data_frame.rb', line 287 def to_s _df.to_s end |
#to_series(index = 0) ⇒ Series
Select column as Series at index location.
553 554 555 556 557 558 |
# File 'lib/polars/data_frame.rb', line 553 def to_series(index = 0) if index < 0 index = columns.length + index end Utils.wrap_s(_df.select_at_idx(index)) end |
#to_struct(name) ⇒ Series
Convert a DataFrame
to a Series
of type Struct
.
4672 4673 4674 |
# File 'lib/polars/data_frame.rb', line 4672 def to_struct(name) Utils.wrap_s(_df.to_struct(name)) end |
#transpose(include_header: false, header_name: "column", column_names: nil) ⇒ DataFrame
This is a very expensive operation. Perhaps you can do it differently.
Transpose a DataFrame over the diagonal.
1004 1005 1006 1007 |
# File 'lib/polars/data_frame.rb', line 1004 def transpose(include_header: false, header_name: "column", column_names: nil) keep_names_as = include_header ? header_name : nil _from_rbdf(_df.transpose(keep_names_as, column_names)) end |
#unique(maintain_order: true, subset: nil, keep: "first") ⇒ DataFrame
Note that this fails if there is a column of type List
in the DataFrame or
subset.
Drop duplicate rows from this DataFrame.
4093 4094 4095 4096 4097 4098 4099 4100 |
# File 'lib/polars/data_frame.rb', line 4093 def unique(maintain_order: true, subset: nil, keep: "first") self._from_rbdf( lazy .unique(maintain_order: maintain_order, subset: subset, keep: keep) .collect(no_optimization: true) ._df ) end |
#unnest(names) ⇒ DataFrame
Decompose a struct into its fields.
The fields will be inserted into the DataFrame
on the location of the
struct
type.
4708 4709 4710 4711 4712 4713 |
# File 'lib/polars/data_frame.rb', line 4708 def unnest(names) if names.is_a?(::String) names = [names] end _from_rbdf(_df.unnest(names)) end |
#unpivot(on, index: nil, variable_name: nil, value_name: nil) ⇒ DataFrame Also known as: melt
Unpivot a DataFrame from wide to long format.
Optionally leaves identifiers set.
This function is useful to massage a DataFrame into a format where one or more columns are identifier variables (index) while all other columns, considered measured variables (on), are "unpivoted" to the row axis leaving just two non-identifier columns, 'variable' and 'value'.
3101 3102 3103 3104 3105 3106 |
# File 'lib/polars/data_frame.rb', line 3101 def unpivot(on, index: nil, variable_name: nil, value_name: nil) on = on.nil? ? [] : Utils.(self, on) index = index.nil? ? [] : Utils.(self, index) _from_rbdf(_df.unpivot(on, index, value_name, variable_name)) end |
#unstack(step:, how: "vertical", columns: nil, fill_values: nil) ⇒ DataFrame
This functionality is experimental and may be subject to changes without it being considered a breaking change.
Unstack a long table to a wide form without doing an aggregation.
This can be much faster than a pivot, because it can skip the grouping phase.
3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 |
# File 'lib/polars/data_frame.rb', line 3180 def unstack(step:, how: "vertical", columns: nil, fill_values: nil) if !columns.nil? df = select(columns) else df = self end height = df.height if how == "vertical" n_rows = step n_cols = (height / n_rows.to_f).ceil else n_cols = step n_rows = (height / n_cols.to_f).ceil end n_fill = n_cols * n_rows - height if n_fill > 0 if !fill_values.is_a?(::Array) fill_values = [fill_values] * df.width end df = df.select( df.get_columns.zip(fill_values).map do |s, next_fill| s.extend_constant(next_fill, n_fill) end ) end if how == "horizontal" df = ( df.with_column( (Polars.arange(0, n_cols * n_rows, eager: true) % n_cols).alias( "__sort_order" ) ) .sort("__sort_order") .drop("__sort_order") ) end zfill_val = Math.log10(n_cols).floor + 1 slices = df.get_columns.flat_map do |s| n_cols.times.map do |slice_nbr| s.slice(slice_nbr * n_rows, n_rows).alias("%s_%0#{zfill_val}d" % [s.name, slice_nbr]) end end _from_rbdf(DataFrame.new(slices)._df) end |
#upsample(time_column:, every:, by: nil, maintain_order: false) ⇒ DataFrame
Upsample a DataFrame at a regular frequency.
The every
and offset
arguments are created with
the following string language:
- 1ns (1 nanosecond)
- 1us (1 microsecond)
- 1ms (1 millisecond)
- 1s (1 second)
- 1m (1 minute)
- 1h (1 hour)
- 1d (1 day)
- 1w (1 week)
- 1mo (1 calendar month)
- 1y (1 calendar year)
- 1i (1 index count)
Or combine them: "3d12h4m25s" # 3 days, 12 hours, 4 minutes, and 25 seconds
2113 2114 2115 2116 2117 2118 2119 2120 2121 2122 2123 2124 2125 2126 2127 2128 2129 2130 2131 |
# File 'lib/polars/data_frame.rb', line 2113 def upsample( time_column:, every:, by: nil, maintain_order: false ) if by.nil? by = [] end if by.is_a?(::String) by = [by] end every = Utils.parse_as_duration_string(every) _from_rbdf( _df.upsample(by, time_column, every, maintain_order) ) end |
#var(ddof: 1) ⇒ DataFrame
Aggregate the columns of this DataFrame to their variance value.
3934 3935 3936 |
# File 'lib/polars/data_frame.rb', line 3934 def var(ddof: 1) lazy.var(ddof: ddof).collect(_eager: true) end |
#vstack(df, in_place: false) ⇒ DataFrame
Grow this DataFrame vertically by stacking a DataFrame to it.
2573 2574 2575 2576 2577 2578 2579 2580 |
# File 'lib/polars/data_frame.rb', line 2573 def vstack(df, in_place: false) if in_place _df.vstack_mut(df._df) self else _from_rbdf(_df.vstack(df._df)) end end |
#width ⇒ Integer
Get the width of the DataFrame.
84 85 86 |
# File 'lib/polars/data_frame.rb', line 84 def width _df.width end |
#with_column(column) ⇒ DataFrame
Return a new DataFrame with the column added or replaced.
2488 2489 2490 2491 2492 |
# File 'lib/polars/data_frame.rb', line 2488 def with_column(column) lazy .with_column(column) .collect(no_optimization: true, string_cache: false) end |
#with_columns(*exprs, **named_exprs) ⇒ DataFrame
Add columns to this DataFrame.
Added columns will replace existing columns with the same name.
3671 3672 3673 |
# File 'lib/polars/data_frame.rb', line 3671 def with_columns(*exprs, **named_exprs) lazy.with_columns(*exprs, **named_exprs).collect(_eager: true) end |
#with_row_index(name: "index", offset: 0) ⇒ DataFrame Also known as: with_row_count
Add a column at index 0 that counts the rows.
1648 1649 1650 |
# File 'lib/polars/data_frame.rb', line 1648 def with_row_index(name: "index", offset: 0) _from_rbdf(_df.with_row_index(name, offset)) end |
#write_avro(file, compression = "uncompressed") ⇒ nil
Write to Apache Avro file.
769 770 771 772 773 774 775 776 777 778 |
# File 'lib/polars/data_frame.rb', line 769 def write_avro(file, compression = "uncompressed") if compression.nil? compression = "uncompressed" end if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end _df.write_avro(file, compression) end |
#write_csv(file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil) ⇒ String?
Write to comma-separated values (CSV) file.
694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 |
# File 'lib/polars/data_frame.rb', line 694 def write_csv( file = nil, has_header: true, include_header: nil, sep: ",", quote: '"', batch_size: 1024, datetime_format: nil, date_format: nil, time_format: nil, float_precision: nil, null_value: nil ) include_header = has_header if include_header.nil? if sep.length > 1 raise ArgumentError, "only single byte separator is allowed" elsif quote.length > 1 raise ArgumentError, "only single byte quote char is allowed" elsif null_value == "" null_value = nil end if file.nil? buffer = StringIO.new buffer.set_encoding(Encoding::BINARY) _df.write_csv( buffer, include_header, sep.ord, quote.ord, batch_size, datetime_format, date_format, time_format, float_precision, null_value ) return buffer.string.force_encoding(Encoding::UTF_8) end if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end _df.write_csv( file, include_header, sep.ord, quote.ord, batch_size, datetime_format, date_format, time_format, float_precision, null_value, ) nil end |
#write_ipc(file, compression: "uncompressed") ⇒ nil
Write to Arrow IPC binary stream or Feather file.
788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 |
# File 'lib/polars/data_frame.rb', line 788 def write_ipc(file, compression: "uncompressed") return_bytes = file.nil? if return_bytes file = StringIO.new file.set_encoding(Encoding::BINARY) end if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end if compression.nil? compression = "uncompressed" end _df.write_ipc(file, compression) return_bytes ? file.string : nil end |
#write_ipc_stream(file, compression: "uncompressed") ⇒ Object
Write to Arrow IPC record batch stream.
See "Streaming format" in https://arrow.apache.org/docs/python/ipc.html.
827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 |
# File 'lib/polars/data_frame.rb', line 827 def write_ipc_stream( file, compression: "uncompressed" ) return_bytes = file.nil? if return_bytes file = StringIO.new file.set_encoding(Encoding::BINARY) elsif Utils.pathlike?(file) file = Utils.normalize_filepath(file) end if compression.nil? compression = "uncompressed" end _df.write_ipc_stream(file, compression) return_bytes ? file.string : nil end |
#write_json(file = nil, pretty: false, row_oriented: false) ⇒ nil
Serialize to JSON representation.
584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 |
# File 'lib/polars/data_frame.rb', line 584 def write_json( file = nil, pretty: false, row_oriented: false ) if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end to_string_io = !file.nil? && file.is_a?(StringIO) if file.nil? || to_string_io buf = StringIO.new buf.set_encoding(Encoding::BINARY) _df.write_json(buf, pretty, row_oriented) json_bytes = buf.string json_str = json_bytes.force_encoding(Encoding::UTF_8) if to_string_io file.write(json_str) else return json_str end else _df.write_json(file, pretty, row_oriented) end nil end |
#write_ndjson(file = nil) ⇒ nil
Serialize to newline delimited JSON representation.
627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 |
# File 'lib/polars/data_frame.rb', line 627 def write_ndjson(file = nil) if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end to_string_io = !file.nil? && file.is_a?(StringIO) if file.nil? || to_string_io buf = StringIO.new buf.set_encoding(Encoding::BINARY) _df.write_ndjson(buf) json_bytes = buf.string json_str = json_bytes.force_encoding(Encoding::UTF_8) if to_string_io file.write(json_str) else return json_str end else _df.write_ndjson(file) end nil end |
#write_parquet(file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil, data_page_size: nil) ⇒ nil
Write to Apache Parquet file.
871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 |
# File 'lib/polars/data_frame.rb', line 871 def write_parquet( file, compression: "zstd", compression_level: nil, statistics: false, row_group_size: nil, data_page_size: nil ) if compression.nil? compression = "uncompressed" end if Utils.pathlike?(file) file = Utils.normalize_filepath(file) end if statistics == true statistics = { min: true, max: true, distinct_count: false, null_count: true } elsif statistics == false statistics = {} elsif statistics == "full" statistics = { min: true, max: true, distinct_count: true, null_count: true } end _df.write_parquet( file, compression, compression_level, statistics, row_group_size, data_page_size ) end |