Neural Network for Multiple Output Regression

Question

I have a dataset containing 34 input columns and 8 output columns.

One way to solve the problem is to take the 34 inputs and build individual regression model for each output column.

I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used Multilayer Perceptron but that needs multiple models just like linear regression. Can Sequence to Sequence be a viable option?

I am using TensorFlow. I have code but I think it is more important to understand what I am missing out in terms of the multilayer perceptron theory.

I understand that in MLP if you have one output node it will provide one output. If you have 10 output nodes then it is a multi class problem. You pick the class with the highest probability out of the 10 outputs. But in my case it is certain there will be 8 outputs for same input.

Lets say, for a set of inputs you will get the 3D coordinate of something (X,Y,Z). Like, Inputs = {1,10,5,7} Output = {1,2,1}. So for the same input {1,10,5,7} I need to make models for X value Y value and Z. One solution is to have 3 different models using MLP. But I would like to see if I can have one model. So I thought about using seq2seq. Because the encoder takes a series of input and the decoder provides series of output. But it seems seq2seq in tensorflow cannot handle float values. I can be wrong about this though.

Answer 1

What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the nature of the problem.

Any neural network framework is able to do something like that.

The key to do that is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (34,) and the output (8,).

Update: the usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras; the network is not an MLP but it should be Ok to illustrate the idea.

Answer 2

You can implement this very simply in Python.
Your X will be the collection of training x,y,z coordinates.
Your Y will be the collection of testing x,y,z coordinates.

from sklearn import cross_validation from sklearn.neural_network import MLPRegressor model = MLPRegressor(solver='lbfgs',alpha=0.001,hidden_layer_sizes=(150,)) cross_validation.cross_val_score(model, X, Y,scoring='mean_squared_error') 

Answer 3

This is much easier than you would think - you can simply set your output layer to be a vector instead of a single scalar. Of course there's no magic around here and I advice you to prep your data (perform batch normalization so all outputs will be values between 0 and 1).

If you're using Keras, the way for doing this is by adding a dense layer as the final output layer:

model.add(Dense(8, activation='linear'))