Class: Braintrust::Resources::Experiment

Inherits:
Object
  • Object
show all
Defined in:
lib/braintrust/resources/experiment.rb

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Experiment

Returns a new instance of Experiment.



6
7
8
 
# File 'lib/braintrust/resources/experiment.rb', line 6

def initialize(client:)
  @client = client
end

Instance Method Details

#create(params = {}, opts = {}) ⇒ Braintrust::Models::Experiment

Create a new experiment. If there is an existing experiment in the project with the same name as the one specified in the request, will return the existing experiment unmodified

Parameters:

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :project_id (String) —

    Unique identifier for the project that the experiment belongs under

  • :base_exp_id (String) —

    Id of default base experiment to compare against when viewing this experiment

  • :dataset_id (String) —

    Identifier of the linked dataset, or null if the experiment is not linked to a dataset

  • :dataset_version (String) —

    Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.

  • :description (String) —

    Textual description of the experiment

  • :ensure_new (Boolean) —

    Normally, creating an experiment with the same name as an existing experiment will return the existing one un-modified. But if ensure_new is true, registration will generate a new experiment with a unique name in case of a conflict.

  • :metadata (Hash) —

    User-controlled metadata about the experiment

  • :name (String) —

    Name of the experiment. Within a project, experiment names are unique

  • :public (Boolean) —

    Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization

  • :repo_info (RepoInfo) —

    Metadata about the state of the repo when the experiment was created

Returns:



35
36
37
38
39
40
41
42
 
# File 'lib/braintrust/resources/experiment.rb', line 35

def create(params = {}, opts = {})
  req = {}
  req[:method] = :post
  req[:path] = "/v1/experiment"
  req[:body] = params
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end

#delete(experiment_id, opts = {}) ⇒ Braintrust::Models::Experiment

Delete an experiment object by its id

Parameters:

  • experiment_id (String) —

    Experiment id

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Returns:



129
130
131
132
133
134
135
 
# File 'lib/braintrust/resources/experiment.rb', line 129

def delete(experiment_id, opts = {})
  req = {}
  req[:method] = :delete
  req[:path] = "/v1/experiment/#{experiment_id}"
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end

#feedback(experiment_id, params = {}, opts = {}) ⇒ nil

Log feedback for a set of experiment events

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :feedback (Array<Feedback>) —

    A list of experiment feedback items

Returns:

  • (nil) 


147
148
149
150
151
152
153
154
 
# File 'lib/braintrust/resources/experiment.rb', line 147

def feedback(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :post
  req[:path] = "/v1/experiment/#{experiment_id}/feedback"
  req[:body] = params
  req[:model] = NilClass
  @client.request(req, opts)
end

#fetch(experiment_id, params = {}, opts = {}) ⇒ Braintrust::Models::ExperimentFetchResponse

Fetch the events in an experiment. Equivalent to the POST form of the same path, but with the parameters in the URL query rather than in the request body

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :limit (Integer) —

    limit the number of traces fetched

    Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest _xact_id. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier _xact_id. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by id) from your combined result set.

    The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

  • :max_root_span_id (String) —

    DEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.

    Together, max_xact_id and max_root_span_id form a pagination cursor

    Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id). See the documentation of limit for an overview of paginating fetch queries.

  • :max_xact_id (String) —

    DEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.

    Together, max_xact_id and max_root_span_id form a pagination cursor

    Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id). See the documentation of limit for an overview of paginating fetch queries.

  • :version (String) —

    Retrieve a snapshot of events from a past time

    The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Returns:



205
206
207
208
209
210
211
212
 
# File 'lib/braintrust/resources/experiment.rb', line 205

def fetch(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :get
  req[:path] = "/v1/experiment/#{experiment_id}/fetch"
  req[:query] = params
  req[:model] = Braintrust::Models::ExperimentFetchResponse
  @client.request(req, opts)
end

#fetch_post(experiment_id, params = {}, opts = {}) ⇒ Braintrust::Models::ExperimentFetchPostResponse

Fetch the events in an experiment. Equivalent to the GET form of the same path, but with the parameters in the request body rather than in the URL query

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :cursor (String) —

    An opaque string to be used as a cursor for the next page of results, in order from latest to earliest.

    The string can be obtained directly from the cursor property of the previous fetch query

  • :filters (Array<Filter>) —

    A list of filters on the events to fetch. Currently, only path-lookup type filters are supported, but we may add more in the future

  • :limit (Integer) —

    limit the number of traces fetched

    Fetch queries may be paginated if the total result size is expected to be large (e.g. project_logs which accumulate over a long time). Note that fetch queries only support pagination in descending time order (from latest to earliest _xact_id. Furthermore, later pages may return rows which showed up in earlier pages, except with an earlier _xact_id. This happens because pagination occurs over the whole version history of the event log. You will most likely want to exclude any such duplicate, outdated rows (by id) from your combined result set.

    The limit parameter controls the number of full traces to return. So you may end up with more individual rows than the specified limit if you are fetching events containing traces.

  • :max_root_span_id (String) —

    DEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.

    Together, max_xact_id and max_root_span_id form a pagination cursor

    Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id). See the documentation of limit for an overview of paginating fetch queries.

  • :max_xact_id (String) —

    DEPRECATION NOTICE: The manually-constructed pagination cursor is deprecated in favor of the explicit 'cursor' returned by object fetch requests. Please prefer the 'cursor' argument going forwards.

    Together, max_xact_id and max_root_span_id form a pagination cursor

    Since a paginated fetch query returns results in order from latest to earliest, the cursor for the next page can be found as the row with the minimum (earliest) value of the tuple (_xact_id, root_span_id). See the documentation of limit for an overview of paginating fetch queries.

  • :version (String) —

    Retrieve a snapshot of events from a past time

    The version id is essentially a filter on the latest event transaction id. You can use the max_xact_id returned by a past fetch as the version to reproduce that exact fetch.

Returns:



270
271
272
273
274
275
276
277
 
# File 'lib/braintrust/resources/experiment.rb', line 270

def fetch_post(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :post
  req[:path] = "/v1/experiment/#{experiment_id}/fetch"
  req[:body] = params
  req[:model] = Braintrust::Models::ExperimentFetchPostResponse
  @client.request(req, opts)
end

#insert(experiment_id, params = {}, opts = {}) ⇒ Braintrust::Models::ExperimentInsertResponse

Insert a set of events into the experiment

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :events (Array<Event::UnnamedTypeWithunionParent10|Event::UnnamedTypeWithunionParent9>) —

    A list of experiment events to insert

Returns:



289
290
291
292
293
294
295
296
 
# File 'lib/braintrust/resources/experiment.rb', line 289

def insert(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :post
  req[:path] = "/v1/experiment/#{experiment_id}/insert"
  req[:body] = params
  req[:model] = Braintrust::Models::ExperimentInsertResponse
  @client.request(req, opts)
end

#list(params = {}, opts = {}) ⇒ Braintrust::ListObjects<Braintrust::Models::Experiment>

List out all experiments. The experiments are sorted by creation date, with the most recently-created experiments coming first

Parameters:

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :ending_before (String) —

    Pagination cursor id.

    For example, if the initial item in the last page you fetched had an id of foo, pass ending_before=foo to fetch the previous page. Note: you may only pass one of starting_after and ending_before

  • :experiment_name (String) —

    Name of the experiment to search for

  • :ids (Array<String>|String) —

    Filter search results to a particular set of object IDs. To specify a list of IDs, include the query param multiple times

  • :limit (Integer) —

    Limit the number of objects to return

  • :org_name (String) —

    Filter search results to within a particular organization

  • :project_name (String) —

    Name of the project to search for

  • :starting_after (String) —

    Pagination cursor id.

    For example, if the final item in the last page you fetched had an id of foo, pass starting_after=foo to fetch the next page. Note: you may only pass one of starting_after and ending_before

Returns:



113
114
115
116
117
118
119
120
121
 
# File 'lib/braintrust/resources/experiment.rb', line 113

def list(params = {}, opts = {})
  req = {}
  req[:method] = :get
  req[:path] = "/v1/experiment"
  req[:query] = params
  req[:page] = Braintrust::ListObjects
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end

#replace(params = {}, opts = {}) ⇒ Braintrust::Models::Experiment

NOTE: This operation is deprecated and will be removed in a future revision of the API. Create or replace a new experiment. If there is an existing experiment in the project with the same name as the one specified in the request, will return the existing experiment unmodified, will replace the existing experiment with the provided fields

Parameters:

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :project_id (String) —

    Unique identifier for the project that the experiment belongs under

  • :base_exp_id (String) —

    Id of default base experiment to compare against when viewing this experiment

  • :dataset_id (String) —

    Identifier of the linked dataset, or null if the experiment is not linked to a dataset

  • :dataset_version (String) —

    Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.

  • :description (String) —

    Textual description of the experiment

  • :ensure_new (Boolean) —

    Normally, creating an experiment with the same name as an existing experiment will return the existing one un-modified. But if ensure_new is true, registration will generate a new experiment with a unique name in case of a conflict.

  • :metadata (Hash) —

    User-controlled metadata about the experiment

  • :name (String) —

    Name of the experiment. Within a project, experiment names are unique

  • :public (Boolean) —

    Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization

  • :repo_info (RepoInfo) —

    Metadata about the state of the repo when the experiment was created

Returns:



325
326
327
328
329
330
331
332
 
# File 'lib/braintrust/resources/experiment.rb', line 325

def replace(params = {}, opts = {})
  req = {}
  req[:method] = :put
  req[:path] = "/v1/experiment"
  req[:body] = params
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end

#retrieve(experiment_id, opts = {}) ⇒ Braintrust::Models::Experiment

Get an experiment object by its id

Parameters:

  • experiment_id (String) —

    Experiment id

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Returns:



50
51
52
53
54
55
56
 
# File 'lib/braintrust/resources/experiment.rb', line 50

def retrieve(experiment_id, opts = {})
  req = {}
  req[:method] = :get
  req[:path] = "/v1/experiment/#{experiment_id}"
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end

#summarize(experiment_id, params = {}, opts = {}) ⇒ Braintrust::Models::ExperimentSummarizeResponse

Summarize experiment

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :comparison_experiment_id (String) —

    The experiment to compare against, if summarizing scores and metrics. If omitted, will fall back to the base_exp_id stored in the experiment metadata, and then to the most recent experiment run in the same project. Must pass summarize_scores=true for this id to be used

  • :summarize_scores (Boolean) —

    Whether to summarize the scores and metrics. If false (or omitted), only the metadata will be returned.

Returns:



349
350
351
352
353
354
355
356
 
# File 'lib/braintrust/resources/experiment.rb', line 349

def summarize(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :get
  req[:path] = "/v1/experiment/#{experiment_id}/summarize"
  req[:query] = params
  req[:model] = Braintrust::Models::ExperimentSummarizeResponse
  @client.request(req, opts)
end

#update(experiment_id, params = {}, opts = {}) ⇒ Braintrust::Models::Experiment

Partially update an experiment object. Specify the fields to update in the payload. Any object-type fields will be deep-merged with existing content. Currently we do not support removing fields or setting them to null.

Parameters:

  • experiment_id (String) —

    Experiment id

  • params (Hash) (defaults to: {}) —

    Attributes to send in this request.

  • opts (Hash|RequestOptions) (defaults to: {}) —

    Options to specify HTTP behaviour for this request.

Options Hash (params):

  • :base_exp_id (String) —

    Id of default base experiment to compare against when viewing this experiment

  • :dataset_id (String) —

    Identifier of the linked dataset, or null if the experiment is not linked to a dataset

  • :dataset_version (String) —

    Version number of the linked dataset the experiment was run against. This can be used to reproduce the experiment after the dataset has been modified.

  • :description (String) —

    Textual description of the experiment

  • :metadata (Hash) —

    User-controlled metadata about the experiment

  • :name (String) —

    Name of the experiment. Within a project, experiment names are unique

  • :public (Boolean) —

    Whether or not the experiment is public. Public experiments can be viewed by anybody inside or outside the organization

  • :repo_info (RepoInfo) —

    Metadata about the state of the repo when the experiment was created

Returns:



80
81
82
83
84
85
86
87
 
# File 'lib/braintrust/resources/experiment.rb', line 80

def update(experiment_id, params = {}, opts = {})
  req = {}
  req[:method] = :patch
  req[:path] = "/v1/experiment/#{experiment_id}"
  req[:body] = params
  req[:model] = Braintrust::Models::Experiment
  @client.request(req, opts)
end