Module: Ai4r::NeuralNetwork::WeightInitializations

Defined in:
lib/ai4r/neural_network/weight_initializations.rb

Overview

Collection of common weight initialization strategies.

Class Method Summary collapse

Class Method Details

.he(structure) ⇒ Object

He initialization suitable for ReLU activations



30
31
32
33
34
35
 
# File 'lib/ai4r/neural_network/weight_initializations.rb', line 30

def he(structure)
  lambda do |layer, _i, _j|
    limit = Math.sqrt(6.0 / structure[layer])
    (rand * 2 * limit) - limit
  end
end

.uniform ⇒ Object

Uniform distribution in [-1, 1)



17
18
19
 
# File 'lib/ai4r/neural_network/weight_initializations.rb', line 17

def uniform
  ->(_n, _i, _j) { (rand * 2) - 1 }
end

.xavier(structure) ⇒ Object

Xavier/Glorot initialization based on layer dimensions



22
23
24
25
26
27
 
# File 'lib/ai4r/neural_network/weight_initializations.rb', line 22

def xavier(structure)
  lambda do |layer, _i, _j|
    limit = Math.sqrt(6.0 / (structure[layer] + structure[layer + 1]))
    (rand * 2 * limit) - limit
  end
end