A toy example - a binary logistic module using torch

In this post, I will be demostrating how to build a simple binary logistic regression module with 2 input units and 1 output using torch. I will be using nn package for this purpose.

Creation for neural network is done in following steps:

• Model creation
• Defining criterion
• Data handling
• Training

Model creation

Let’s being with the model creation. The function to be modelled is as follows:

\[y = \sigma(Wx + b)\]

where sigmoid (logistic function) is defined as:

\[ \sigma(z) = \frac{1}{1 - \exp^{-z}} \]

require 'nn'

module = nn.Sequential()
module:add(nn.Linear(2, 1))
module:add(nn.sigmoid())

simple feed-forward network: It takes the input, feeds it through several layers one after the other, and then finally gives the output.

Such a network container is nn.Sequential which feeds the input through several layers. In this case we have a network with 1 hidden layer.

Defining criterion

Here criterion is a binary cross-entropy Criterion (which expects a 0 or 1 valued targets):

BCE loss function is defined as:

\[ -\sum_i [t_i \log(y_i) + (1 - t_i)\log(1 - y_i)] \]

criterion = nn.BCECriterion()

Data

As this is just a toy example, I am using some random generated data. This dummy dataset contains 10 samples:

inputs = torch.Tensor(10, 2):uniform(-1, 1)
targets = torch.Tensor(10):random(0, 1)

Training

Function for one epoch of stochastic gradient descent (SGD)

require 'dpnn'

function trainEpoc(module, criterion, inputs, targets)
    for i=1, inputs:size(1) do
        local idx = math.random(1, input:size(1))
        local input, target = inputs[idx], targets:narrow(1, idx, 1)
        --forward
        local output = module:forward(input)
        local loss = criterion:forward(output, target)
        --backward
        local gradOutput = criterion:backward(output, target)
        module:zeroGradParameters()
        local gradInput = module:backward(input, gradOutput)
        --update
        module:updateGradParameters(0.9) -- momentum (dpnn)
        module:updateParameters(0.1) -- W = W - 0.1*dL/dW
    end
end

Let’s do 100 epochs to train the module

for i=1, 100 do
    trainEpoc(module, criterion, inputs, targets)
end

Here, I have taken 0.1 as learning rate.