Gradient descent for linear regression using numpy/pandas

Question

I currently follow along Andrew Ng'sMachine Learning Course on Coursera and wanted to implement the gradient descent algorithm in python3 using numpy and pandas.

This is what I came up with:

import os import numpy as np import pandas as pd def get_training_data(path): # path to read data from raw_panda_data = pd.read_csv(path) # append a column of ones to the front of the data set raw_panda_data.insert(0, 'Ones', 1) num_columns = raw_panda_data.shape[1] # (num_rows, num_columns) panda_X = raw_panda_data.iloc[:,0:num_columns-1] # [ slice_of_rows, slice_of_columns ] panda_y = raw_panda_data.iloc[:,num_columns-1:num_columns] # [ slice_of_rows, slice_of_columns ] X = np.matrix(panda_X.values) # pandas.DataFrame -> numpy.ndarray -> numpy.matrix y = np.matrix(panda_y.values) # pandas.DataFrame -> numpy.ndarray -> numpy.matrix return X, y def compute_mean_square_error(X, y, theta): summands = np.power(X * theta.T - y, 2) return np.sum(summands) / (2 * len(X)) def gradient_descent(X, y, learning_rate, num_iterations): num_parameters = X.shape[1] # dim theta theta = np.matrix([0.0 for i in range(num_parameters)]) # init theta cost = [0.0 for i in range(num_iterations)] for it in range(num_iterations): error = np.repeat((X * theta.T) - y, num_parameters, axis=1) error_derivative = np.sum(np.multiply(error, X), axis=0) theta = theta - (learning_rate / len(y)) * error_derivative cost[it] = compute_mean_square_error(X, y, theta) return theta, cost 

This is how one could use the code:

X, y = get_training_data(os.getcwd() + '/data/data_set.csv') theta, cost = gradient_descent(X, y, 0.008, 10000) print('Theta: ', theta) print('Cost: ', cost[-1]) 

Where data/data_set.csv could contain data (model used: 2 + x1 - x2 = y) looking like this:

x1, x2, y 0, 1, 1 1, 1, 2 1, 0, 3 0, 0, 2 2, 4, 0 4, 2, 4 6, 0, 8 

Output:

Theta: [[ 2. 1. -1.]] Cost: 9.13586056551e-26 

I'd especially like to get the following aspects of my code reviewed:

• Overall python style. I'm relatively new to python coming from a C background and not sure if I'm misunderstanding some concepts here.
• numpy/pandas integration. Do I use these packages correctly?
• Correctness of the gradient descent algorithm.
• Efficiency. How can I further improve my code?

Answer 1

Without having the insight (or, honestly, time) to verify your actual algorithm, I can say that your Python is pretty good.

Only minor stuff - this kind of comment - # path to read data from - should be turned into a PEP257-style docstring.

You should add a shebang at the top of your file, probably #!/usr/bin/env python3.

Otherwise, you're off to a good start.

Answer 2

I like your Python style. There is an issue with your algorithm though. numpy.repeat does not work the way you expect it to. Try this code:

import numpy as np theta = np.matrix([1,2,3]) y = 2 X = np.matrix(np.array(range(9)).reshape(3,3)) error = np.repeat((X * theta.T) - y, 3, axis=1) print(error) >>>[[ 6 6 6] [24 24 24] [42 42 42]] print(np.dot(X, theta.T)-y) >>>[[ 6] [24] [42] 

Do you see how numpy.repeat returns a matrix although you want to return a vector?

Answer 3

You can also use numpy.zeros(shape) to initialize the tetha and cost vectors with zeros.