renom.layers.function

class renom.layers.function.parameterized. Model

Abstract class of neural network model.

forward ( )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

train ( )

Context manager to control whether a computational graph will be created or not.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z
...
>>> x = rm.Variable(np.random.rand(3, 3))
>>> y = np.random.rand(3, 2)
>>> model = Model()
>>> z = model(x)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(z, y)
...
>>> dx1 = loss.grad().get(x)
>>> print("Gradient1 of x is \n{}".format(dx1))
Gradient1 of x is
array([[ 0.85432934,  0.51205811],
       [ 0.20379112,  0.62481132],
       [ 0.49004569,  0.35310219]])
>>>
>>> loss = rm.mean_squared_error(z, y)
>>> dx2 = loss.grad().get(x)
>>> print("Gradient2 of x is \n{}".format(dx2))
Gradient2 of x is
None

prevent_update ( )

This context manager can controls that whether model’s weight parameter be updated.

Example

>>> import numpy as np
>>> import renom as rm
>>> model = rm.Sequential([
...     rm.Dense(1)
... ])
>>> x = np.random.rand(2, 2)
>>> y = np.random.rand(2, 1)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(model(x), y)
>>>
>>> print("Before", model[0].params.w)
Before
 [[ 0.03417877]
 [-0.29819158]]
>>> loss.grad().update()
>>> print("Updated", model[0].params.w)
Updated
 [[ 0.526793  ]
 [ 0.00882804]]
>>>
>>>
>>> print("Before", model[0].params.w)
Before
 [[ 0.526793  ]
 [ 0.00882804]]
>>>
>>> # Performs update inside the context manager.
>>> with model.prevent_update():
...     loss.grad().update()
>>> print("Not updated", model[0].params.w)
Not updated
 [[ 0.526793  ]
 [ 0.00882804]]

values ( )

Generates nested tuple of underlying models and params of models.

Each model generates tuple of two dictionary. The first dictionary contains child models, keyed by attribute name. The second dictionary contains parameters of the model, keyed by attribute name.

Example

(
    # child models of self
    {
        'layer1': (
            {},     # child model of self.layer1
            {       # params of layer1
                'w': [1,2],   # self.layer1.params.w
                'b': [3,4],   # self.layer1.params.b
            }
        ),
        'layer2': (
            {},     # child model of self.layer2
            {       # params of layer2
                'w': [1,2],   # self.layer2.params.w
                'b': [3,4],   # self.layer2.params.b
            }
    },
    # params of self
    {}
)

join_grads ( grads , others )

Merge gradients of other models. Others is a list of tuple of (model, grads) to be merged. Models listed in the others should have same structure with self.

save ( filename )

Save model attributes. For save attributes, please register attributes to the dictionary which is named as ‘SERIALIZED’.

Following example shows how to do it.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class MyModel(rm.Model):
...     SERIALIZED = ('_moving_avg', ) # Put any attributes for saving.
...     def __init__(self):
...         super(MyModel, self).__init__()
...         self._l1 = rm.Dense(2)
...         self._l2 = rm.Dense(1)
...         self._moving_avg = 0
...     def forward(self, x):
...         h = self._l1(x)
...         h = rm.relu(h)
...         h = self._l2(h)
...         self._moving_avg = np.float32(self._moving_avg*0.5 + rm.sum(h)*0.5)
...         return h
...
>>> model = MyModel()
>>> model(np.random.rand(12, 4))
>>> print(model._moving_avg)
1.95637
>>> model.save("test.h5") # Save
>>> model = MyModel() # Reset model object.
>>> model.load("test.h5") # Load
>>> print(model._moving_avg)
1.95637

Parameters: filename ( str ) – File name to save model.

load ( filename )

Load saved weights to model.

Parameters: filename ( str ) – File name of saved model.

Example

>>> model = rm.Dense(2)
>>> model.load("model.hd5")

set_initializer ( initializer )

Set initializer Setting all weights to initializer.

Following example shows how to do it.

Parameters: initializer ( Initializer ) – Initializing Object.

Example

>>> import renom as rm
>>> import numpy as np
>>> from renom.utility.initializer import Orthogonal
>>>
>>> class MyModel(rm.Model):
...     def __init__(self):
...         super(MyModel, self).__init__()
...         self._l1 = rm.Dense(2)
...         self._l2 = rm.Dense(1)
...         self._moving_avg = 0
...     def forward(self, x):
...         h = self._l1(x)
...         h = rm.relu(h)
...         h = self._l2(h)
...         return h
>>>
>>> model = MyModel()
>>> model.set_initializer(Orthogonal())
>>>
>>> x = np.random.random((3,4))
>>> y = model(x)
>>>
>>> weight=model._l1.params.w
>>> print("weight\n",weight)
weight
[[-0.43140578  0.39115947]
[-0.53214586 -0.61308974]
[-0.5807101   0.5656842 ]
[-0.43986994 -0.3887372 ]]
>>>
>>> print("dot product\n",np.dot(weight.T,weight))
dot product
[[1. 0.]
[0. 1.]]

class renom.layers.function.parameterized. Sequential ( layers , loss_function=None )

Sequential model.

Parameters: layers ( list ) – A list of layer objects.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> x = np.random.rand(32, 50)
>>> sequential = rm.Sequential([
...         rm.Dense(100),
...         rm.Relu(),
...         rm.Dense(10),
...     ])
...
>>> z = sequential(x)
>>> z.shape
(32, 10)

class renom.layers.function.batch_normalize. BatchNormalize ( input_size=None , momentum=0.99 , mode='activation' , epsilon=1e-05 , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

Batch normalization function [bn] . This layer accelerates learning speed with reducing internal covariate shift and allow us to set high learning rate.

When the forward propagation, if the argument inference is set to False this layer calculates moving average of mean and variance. Other wise the inference is set to True, this layer uses the moving average which calculated in the above mode.

If the argument mode is set to ‘activation’, this layer normalizes prior-layer features per unit. Otherwise the argument mode is set to ‘feature’, this layer normalizes prior-layer features per channel.

The ‘feature’ mode is only effective for 4D tensor input.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• input_size ( int ) – Input unit size.
• momentum ( float ) – Momentum coefficient for the moving average.
• mode ( str ) – ‘activation’ or ‘feature’.
• epsilon ( float ) – Small number added to avoid division by zero.
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.BatchNormalize(momentum=0.99)
>>> layer(x, inference=False)
batch_normalize([[-0.05047419,  0.00471613],
                 [-0.00887055, -0.01459344],
                 [ 0.05934474,  0.00987731]], dtype=float32)

[bn]	Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift(2015)

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.conv2d. Conv2d ( channel=32 , filter=3 , padding=0 , stride=1 , dilation=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

2d convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• channel ( int ) – The dimensionality of the output.
• filter ( tuple , int ) – Filter size of the convolution kernel.
• padding ( tuple , int ) – Size of the zero-padding around the image.
• stride ( tuple , int ) – Stride-size of the convolution.
• dilation ( tupe , int ) – Dilation of the convolution.
• input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Conv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NCHW .

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.convnd. ConvNd ( channel=2 , filter=3 , padding=0 , stride=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.Gaussian object> )

Nd convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class accepts tensors of any dimensionality and produces an output of equal dimensionality as the input

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• channel ( int ) – The dimensionality of the output.
• filter ( int ) – Filter size of the convolution kernel.
• padding ( int ) – Size of the zero - padding around the image.
• stride ( int ) – Stride - size of the convolution.
• input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.ConvNd(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NC(D*) .

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.convnd. Conv3d ( channel=2 , filter=3 , padding=0 , stride=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.Gaussian object> , weight_decay=0 )

Provides an interface for the ConvNd with a more familiar name

Note

Tensor data format is NCHWD .

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.group_conv2d. GroupConv2d ( channel=32 , filter=3 , padding=0 , stride=1 , dilation=1 , groups=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

2d grouped convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• channel ( int ) – The dimensionality of the output.
• filter ( tuple , int ) – Filter size of the convolution kernel.
• padding ( tuple , int ) – Size of the zero-padding around the image.
• stride ( tuple , int ) – Stride-size of the convolution.
• dilation ( tupe , int ) – Dilation of the convolution.
• input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.
• groups ( int ) – Number of groups to split convolution into. Must be divisor of input and output channels.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 16, 32, 32)
>>> layer = rm.GroupConv2d(channel=32, groups=8)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NCHW .

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.deconv2d. Deconv2d ( channel=1 , filter=3 , padding=0 , stride=1 , dilation=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

2d convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class only accepts and outputs 4d tensors.

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• channel ( int ) – The dimensionality of the output.
• filter ( tuple , int ) – Filter size to witch used as convolution kernel.
• padding ( tuple , int ) – Pad around image by 0 according to this size.
• stride ( tuple , int ) – Specifying the strides of the convolution.
• dilation ( tuple , int ) – Dilation of the convolution.
• input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.Deconv2d(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 34, 34)

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.deconvnd. DeconvNd ( channel=2 , filter=3 , padding=0 , stride=1 , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.Gaussian object> )

Nd convolution layer.

This class creates a convolution filter to be convolved with the input tensor. The instance of this class accepts tensors of any dimensionality and produces an output of equal dimensionality as the input

At instantiation, in the case of int input, filter, padding, and stride, the shape will be symmetric.

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• channel ( int ) – The dimensionality of the output.
• filter ( int ) – Filter size of the convolution kernel.
• padding ( int ) – Size of the zero - padding around the image.
• stride ( int ) – Stride - size of the convolution.
• input_size ( tuple ) – Input unit size. This must be a tuple like (Channel, Height, Width).
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, c, h, w = (10, 3, 32, 32)
>>> x = np.random.rand(n, c, h, w)
>>> x.shape
(10, 3, 32, 32)
>>> layer = rm.ConvNd(channel=32)
>>> z = layer(x)
>>> z.shape
(10, 32, 30, 30)

Note

Tensor data format is NC(D*) .

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.dense. Dense ( output_size , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

Fully connected layer as described bellow.

f(x)= w \cdot x + b

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• output_size ( int ) – Output unit size.
• input_size ( int ) – Input unit size.
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 2)
>>> x.shape
(3, 2)
>>> layer = rm.Dense(3)
>>> z = layer(x)
>>> z.shape
(3, 3)

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.dropout. Dropout ( dropout_ratio=0.5 )

Applies Dropout [dropout] to the input.

Dropout function randomly selects a fraction (specified by dropout_ratio) of the data sets them to zero. Remaining data will be rescaled by 1/(1 - dropout_ratio) .

Parameters: dropout_ratio ( float ) – Dropout ratio.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = rm.Variable(np.random.rand(3, 2))
>>> x
Variable([[ 0.49312586,  0.95414829],
          [ 0.7346437 ,  0.9014256 ],
          [ 0.09413767,  0.29910043]], dtype=float32)
>>> layer = rm.Dropout(0.2)
>>> z = layer(x)
>>> z
dropout([[ 0.        ,  1.19268537],
         [ 0.91830462,  1.12678194],
         [ 0.        ,  0.37387553]], dtype=float32)

>>> z = rm.dropout(x, 0.75)
>>> z
dropout([[ 0.65750116,  0.        ],
         [ 0.        ,  1.20190084],
         [ 0.12551689,  0.39880058]], dtype=float32)

[dropout]

Hinton, Geoffrey E.; Srivastava, Nitish; Krizhevsky, Alex; Sutskever, Ilya; Salakhutdinov, Ruslan R. (2012). Improving neural networks by preventing co-adaptation of feature detectors

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.dropout. SpatialDropout ( dropout_ratio=0.5 )

Applies spatial dropout to the input. This function drops feature maps randomly.

Parameters: dropout_ratio ( float ) – Dropout ratio. Raises: AssertionError – An assertion error will be raised if the input tensor dimension is not 4.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = rm.Variable(np.random.rand(3, 3, 1, 1))
Variable([[[[ 0.55784005]],
           [[ 0.99528867]],
           [[ 0.77544725]]],
          [[[ 0.65406305]],
           [[ 0.27961349]],
           [[ 0.43104461]]],
          [[[ 0.66176379]],
           [[ 0.70499772]],
           [[ 0.87102354]]]], dtype=float32)
>>> z = rm.spatial_dropout(x)
>>> z
spatial_dropout([[[[ 1.1156801 ]],
                  [[ 0.        ]],
                  [[ 1.5508945 ]]],
                 [[[ 1.30812609]],
                  [[ 0.55922699]],
                  [[ 0.        ]]],
                 [[[ 0.        ]],
                  [[ 1.40999544]],
                  [[ 1.74204707]]]], dtype=float32)

Note

SpatialDropout only accepts 4d tensor data.

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.embedding. Embedding ( output_size , input_size , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=None )

Embedding layer. This layer is the special case of dense layer. The case is that the input value is onehot encoded. Since the onehot encoded input is very sparse, the matrix product performed in the dense layer is redundant.

The difference between dense layer and embedding layer is bellow.

[Dense layer]

data -> onehot encoding -> onehot data -> dense layer -> embedded data

[Embedding layer]

data -> embedding layer -> embedded data

Parameters:

• output_size ( int ) – Output unit size.
• input_size ( int ) – Input unit size. This is same as number of embedding characters.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> N = 4
>>> a = np.arange(N).reshape(N, 1)
>>> layer = rm.Embedding(output_size=1, input_size=8)
>>> out = layer(a)

Note

1. This layer only accepts matrix which shape is (N, 1) and has integer value. * N is batch size.
2. Both output_size and input_size must be specified.

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.flatten. Flatten

This function flattens an input tensor. It does not affect the batch size.

Example

>>> x = np.random.rand(3, 3, 32, 32)
>>> x = rm.Variable(x)
>>> z = rm.flatten(x)
>>> x.shape
(3, 3, 32, 32)
>>> z.shape
(3, 3072)

>>> # Use as a instance.
>>> layer = rm.Flatten()
>>> z = layer(x)
>>> z.shape
(3, 3072)

class renom.layers.function.gru. Gru ( output_size , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

Gated Recurrent Unit

An LSTM-like RNN unit, which simplifies the LSTM unit by not including a memory core. This simplifies learning of the unit and reduces computational complexity, as the GRU only performs requires 3 input gates, compared to the 4 required by the LSTM.

Parameters:

• output_size ( int ) – Output unit size.
• input_size ( int ) – Input unit size.
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n,d))
>>> layer = rm.Gru(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()
>>> grad.get(x)
Add([[-8.89559174, -0.58861321, -4.67931843],
[-7.27466679, -0.45286781, -3.81758523]], dtype=float32)
>>> layer.truncate()

https://arxiv.org/pdf/1409.1259.pdf

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

truncate ( )

Truncates temporal connection.

class renom.layers.function.lrn. Lrn ( n=5 , k=2 , a=0.0001 , b=0.75 )

Local response normalization function [lrn] .

y_{c_{out},j,i}= \frac{x_{c_{in},i,j}}{(k + a{\sum_{c=max(0, i-n/2)}^{min(N-1, i+n/2)} (x_{c,i,j})})^b}

x_{c,i,j} represents the c th conv filter’s output at the position of (i, j) in the feature map. y_{c_{out},j,i} represents the output of local response normalization. N is the number of the feature maps. n is the adjacent feature map number. The default argument values are recommended.

Parameters:

• n ( int ) – Number of images used to be normalized.
• k ( int ) – Constant.
• a ( float ) – Scale factor.
• b ( float ) – Exponential factor.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.Lrn()
>>> z=layer(x)
>>> z.shape
(3, 3, 32, 32)

[lrn]

Alex Krizhevsky Krizhevsky, , Ilya IlyaSutskever Sutskever, Geoff. ImageNet Classification with Deep Convolutional Neural Networks

class renom.layers.function.layer_normalize. LayerNormalize ( gain=0.1 , bias=1.0 )

Layer Normalization Model [1] Applies a shift to a standard bell curve formation for each input unit. The resultant bell curve can be transformed with the gain/bias parameters, displacing the mean with the bias or the variance with gain.

Parameters:

• gain ( float ) – Initial value of gain.
• bias ( float ) – Initial value of bias.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(2,3)
>>> layer = rm.LayerNormalize(bias=0)
>>> x
array([[0.5833913 , 0.39715111, 0.19503325],
       [0.74007066, 0.34642472, 0.57293373]])
>>> layer(x)
layer_normalize([[ 0.12076415,  0.00333703, -0.12410118],
           [ 0.11587134, -0.12813905,  0.01226771]])

[1]	https://arxiv.org/pdf/1607.06450.pdf

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.weight_normalize. WeightNormalize ( units , gain=0.1 , initializer=<renom.utility.initializer.GlorotNormal object> , input_size=None , weight_decay=0 )

Weight Normalization Model [weight_norm] A modification to the normal dense layer model, where the weight is normalized and multiplied by a trainable gain factor.

The weight in this form is parameterized by the form:

w = v / ||v|| * gain

Note that in this version, gain is taken linear on the input s giving:

gain = s.

The original paper suggests a potential gain parameterization by taking the exponential value of s instead:

gain = exp(s)

There might be support for this later.

Example

>>> import numpy as np
>>> import renom as rm
>>> x = np.random.rand(2,3)
>>> layer = rm.WeightNormalization(4)
>>> layer(x)
weight_normalize([[1.00133252, 1.00713646, 0.98452991, 1.0043143],
            [0.983392 , 1.01545942, 0.99134618, 1.01834679]],
            dtype=float32)

[weight_norm]

https://arxiv.org/abs/1602.07868

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.lstm. Lstm ( output_size , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

Long short time memory [lstm] . Lstm object has 8 weights and 4 biases parameters to learn.

Weights applied to the input of the input gate, forget gate and output gate. W_{ij}, Wgi_{ij}, Wgf_{ij}, Wgo_{ij}

Weights applied to the recuurent input of the input gate, forget gate and output gate. R_{ij}, Rgi_{ij}, Rgf_{ij}, Rgo_{ij}

\begin{split}u^t_{i} &= \sum_{j = 0}^{J-1} W_{ij}x^t_{j} + \sum_{k = 0}^{K-1} R_{ik}y^{t-1}_{k} + b_i \\ gi^t_{i} &= \sum_{j = 0}^{J-1} Wgi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgi_{ik}y^{t-1}_{k} + bi_i \\ gf^t_{i} &= \sum_{j = 0}^{J-1} Wgfi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgf_{ik}y^{t-1}_{k} + bi_f \\ go^t_{i} &= \sum_{j = 0}^{J-1} Wgo_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgo_{ik}y^{t-1}_{k} + bi_o \\ s^t_i &= sigmoid(gi^t_{i})tanh(u^t_{i}) + s^{t-1}_isigmoid(gf^t_{i}) \\ y^t_{i} &= go^t_{i}tanh(s^t_{i})\end{split}

If the argument input_size is passed, this layers’ weight is initialized in the __init__ function. Otherwise, the weight is initialized in its first forward calculation.

Parameters:

• output_size ( int ) – Output unit size.
• input_size ( int ) – Input unit size.
• ignore_bias ( bool ) – If True is given, bias will not be added.
• initializer ( Initializer ) – Initializer object for weight initialization.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n, d))
>>> layer = rm.Lstm(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()    # Backpropagation.
>>> grad.get(x)    # Gradient of x.
Add([[-0.01853334, -0.0585249 ,  0.01290053],
     [-0.0205425 , -0.05837972,  0.00467286]], dtype=float32)
>>> layer.truncate()

[lstm]

Learning Precise Timing with LSTM Recurrent Networks

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

truncate ( )

Truncates temporal connection.

class renom.layers.function.peephole_lstm. PeepholeLstm ( output_size , input_size=None , ignore_bias=False , initializer=<renom.utility.initializer.GlorotNormal object> , weight_decay=0 )

Long short time memory with peephole [plstm] . Lstm object has 11 weights and 4 biases parameters to learn.

Weights applied to the input of the input gate, forget gate and output gate. W_{ij}, Wgi_{ij}, Wgf_{ij}, Wgo_{ij}

Weights applied to the recuurent input of the input gate, forget gate and output gate. R_{ij}, Rgi_{ij}, Rgf_{ij}, Rgo_{ij}

Weights applied to the state input of the input gate, forget gate and output gate. P_{ij}, Pgi_{ij}, Pgf_{ij}, Pgo_{ij}

\begin{split}u^t_{i} &= \sum_{j = 0}^{J-1} W_{ij}x^t_{j} + \sum_{k = 0}^{K-1} R_{ik}y^{t-1}_{k} + P_{i}s^{t-1}_{i} + b_i \\ gi^t_{i} &= \sum_{j = 0}^{J-1} Wgi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgi_{ik}y^{t-1}_{k} + Pgi_{i}s^{t-1}_{i} + bi_i \\ gf^t_{i} &= \sum_{j = 0}^{J-1} Wgfi_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgf_{ik}y^{t-1}_{k} + Pgf_{i}s^{t-1}_{i} + bi_f \\ go^t_{i} &= \sum_{j = 0}^{J-1} Wgo_{ij}x^t_{j} + \sum_{k = 0}^{K-1} Rgo_{ik}y^{t-1}_{k} + Pgo_{i}s^{t}_{i} + bi_o \\ s^t_i &= sigmoid(gi^t_{i})tanh(u^t_{i}) + s^{t-1}_isigmoid(gf^t_{i}) \\ y^t_{i} &= go^t_{i}tanh(s^t_{i})\end{split}

Parameters:

• output_size ( int ) – Output unit size.
• input_size ( int ) – Input unit size.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> n, d, t = (2, 3, 4)
>>> x = rm.Variable(np.random.rand(n, d))
>>> layer = rm.PeepholeLstm(2)
>>> z = 0
>>> for i in range(t):
...     z += rm.sum(layer(x))
...
>>> grad = z.grad()    # Backpropagation.
>>> grad.get(x)    # Gradient of x.
Add([[-0.01853334, -0.0585249 ,  0.01290053],
     [-0.0205425 , -0.05837972,  0.00467286]], dtype=float32)
>>> layer.truncate()

[plstm]

Felix A. Gers, Nicol N. Schraudolph, J ̈urgen Schmidhuber. Learning Precise Timing with LSTM Recurrent Networks

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

truncate ( )

Truncates temporal connection.

class renom.layers.function.parameterized. Model

Abstract class of neural network model.

forward ( )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

train ( )

Context manager to control whether a computational graph will be created or not.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z
...
>>> x = rm.Variable(np.random.rand(3, 3))
>>> y = np.random.rand(3, 2)
>>> model = Model()
>>> z = model(x)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(z, y)
...
>>> dx1 = loss.grad().get(x)
>>> print("Gradient1 of x is \n{}".format(dx1))
Gradient1 of x is
array([[ 0.85432934,  0.51205811],
       [ 0.20379112,  0.62481132],
       [ 0.49004569,  0.35310219]])
>>>
>>> loss = rm.mean_squared_error(z, y)
>>> dx2 = loss.grad().get(x)
>>> print("Gradient2 of x is \n{}".format(dx2))
Gradient2 of x is
None

prevent_update ( )

This context manager can controls that whether model’s weight parameter be updated.

Example

>>> import numpy as np
>>> import renom as rm
>>> model = rm.Sequential([
...     rm.Dense(1)
... ])
>>> x = np.random.rand(2, 2)
>>> y = np.random.rand(2, 1)
>>>
>>> with model.train():
...     loss = rm.mean_squared_error(model(x), y)
>>>
>>> print("Before", model[0].params.w)
Before
 [[ 0.03417877]
 [-0.29819158]]
>>> loss.grad().update()
>>> print("Updated", model[0].params.w)
Updated
 [[ 0.526793  ]
 [ 0.00882804]]
>>>
>>>
>>> print("Before", model[0].params.w)
Before
 [[ 0.526793  ]
 [ 0.00882804]]
>>>
>>> # Performs update inside the context manager.
>>> with model.prevent_update():
...     loss.grad().update()
>>> print("Not updated", model[0].params.w)
Not updated
 [[ 0.526793  ]
 [ 0.00882804]]

values ( )

Generates nested tuple of underlying models and params of models.

Each model generates tuple of two dictionary. The first dictionary contains child models, keyed by attribute name. The second dictionary contains parameters of the model, keyed by attribute name.

Example

(
    # child models of self
    {
        'layer1': (
            {},     # child model of self.layer1
            {       # params of layer1
                'w': [1,2],   # self.layer1.params.w
                'b': [3,4],   # self.layer1.params.b
            }
        ),
        'layer2': (
            {},     # child model of self.layer2
            {       # params of layer2
                'w': [1,2],   # self.layer2.params.w
                'b': [3,4],   # self.layer2.params.b
            }
    },
    # params of self
    {}
)

join_grads ( grads , others )

Merge gradients of other models. Others is a list of tuple of (model, grads) to be merged. Models listed in the others should have same structure with self.

save ( filename )

Save model attributes. For save attributes, please register attributes to the dictionary which is named as ‘SERIALIZED’.

Following example shows how to do it.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class MyModel(rm.Model):
...     SERIALIZED = ('_moving_avg', ) # Put any attributes for saving.
...     def __init__(self):
...         super(MyModel, self).__init__()
...         self._l1 = rm.Dense(2)
...         self._l2 = rm.Dense(1)
...         self._moving_avg = 0
...     def forward(self, x):
...         h = self._l1(x)
...         h = rm.relu(h)
...         h = self._l2(h)
...         self._moving_avg = np.float32(self._moving_avg*0.5 + rm.sum(h)*0.5)
...         return h
...
>>> model = MyModel()
>>> model(np.random.rand(12, 4))
>>> print(model._moving_avg)
1.95637
>>> model.save("test.h5") # Save
>>> model = MyModel() # Reset model object.
>>> model.load("test.h5") # Load
>>> print(model._moving_avg)
1.95637

Parameters: filename ( str ) – File name to save model.

load ( filename )

Load saved weights to model.

Parameters: filename ( str ) – File name of saved model.

Example

>>> model = rm.Dense(2)
>>> model.load("model.hd5")

set_initializer ( initializer )

Set initializer Setting all weights to initializer.

Following example shows how to do it.

Parameters: initializer ( Initializer ) – Initializing Object.

Example

>>> import renom as rm
>>> import numpy as np
>>> from renom.utility.initializer import Orthogonal
>>>
>>> class MyModel(rm.Model):
...     def __init__(self):
...         super(MyModel, self).__init__()
...         self._l1 = rm.Dense(2)
...         self._l2 = rm.Dense(1)
...         self._moving_avg = 0
...     def forward(self, x):
...         h = self._l1(x)
...         h = rm.relu(h)
...         h = self._l2(h)
...         return h
>>>
>>> model = MyModel()
>>> model.set_initializer(Orthogonal())
>>>
>>> x = np.random.random((3,4))
>>> y = model(x)
>>>
>>> weight=model._l1.params.w
>>> print("weight\n",weight)
weight
[[-0.43140578  0.39115947]
[-0.53214586 -0.61308974]
[-0.5807101   0.5656842 ]
[-0.43986994 -0.3887372 ]]
>>>
>>> print("dot product\n",np.dot(weight.T,weight))
dot product
[[1. 0.]
[0. 1.]]

class renom.layers.function.parameterized. Sequential ( layers , loss_function=None )

Sequential model.

Parameters: layers ( list ) – A list of layer objects.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> x = np.random.rand(32, 50)
>>> sequential = rm.Sequential([
...         rm.Dense(100),
...         rm.Relu(),
...         rm.Dense(10),
...     ])
...
>>> z = sequential(x)
>>> z.shape
(32, 10)

forward ( x )

Override this method to allow network to calculate.

Example

>>> import renom as rm
>>> import numpy as np
>>>
>>> class Model(rm.Model):
...     def __init__(self):
...         self._layer1 = rm.Dense(3)
...         self._layer2 = rm.Dense(2)
...     def forward(self, x):  # This part
...         h = rm.relu(self._layer1(x))
...         z = self._layer2(h)
...         return z

class renom.layers.function.pool2d. MaxPool2d ( filter=3 , padding=0 , stride=1 )

Max pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:

• filter ( tuple , int ) – Filter size of the convolution kernel.
• padding ( tuple , int ) – Size of the zero-padding around the image.
• stride ( tuple , int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.MaxPool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.max_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)

class renom.layers.function.pool2d. AveragePool2d ( filter=3 , padding=0 , stride=1 )

Average pooling function. In the case of int input, filter, padding, and stride, the shape will be symmetric.

Parameters:

• filter ( tuple , int ) – Filter size of the convolution kernel.
• padding ( tuple , int ) – Size of the zero-padding around the image.
• stride ( tuple , int ) – Stride-size of the convolution.

Example

>>> import numpy as np
>>> import renom as rm
>>>
>>> x = np.random.rand(3, 3, 32, 32)
>>> layer = rm.AveragePool2d(filter=3)
>>> z = layer(x)
>>> z.shape
(3, 3, 30, 30)
>>> z = rm.average_pool2d(x, filter=(3, 3), stride=(1, 1), padding=(0,0))
>>> z.shape
(3, 3, 30, 30)

class renom.layers.function.unpool2d. MaxUnPool2d

Max unpooling function. Unpools an input in a network where a previous pooling has occured.

Parameters:

• x ( Node , np.ndarray ) – The input to the unpooling method
• prev_pool ( max_pool2d , None ) – The previous pool to be unpooled. In the case of none, the model searches through the history for the previous layer.

Note

The input shape requirement: x.shape == previous_pool.shape

The output shape will be: ret.shape == previous_pool.input.shape

class renom.layers.function.unpool2d. AverageUnPool2d

Average unpooling function. Unpools an input in a network where a previous pooling has occured.

Parameters:

• x ( Node , np.ndarray ) – The input to the unpooling method
• prev_pool ( average_pool2d , None ) – The previous pool to be unpooled. In the case of none, the model searches through the history for the previous layer.

Note

The input shape requirement: x.shape == previous_pool.shape

The output shape will be: ret.shape == previous_pool.input.shape