There are 3 different APIs for model evaluation:
1. Estimator score method: Estimator/model object has a ‘score()’ method that provides a default evaluation
2. Scoring parameter: Predefined scoring parameter that can be passed into cross_val_score() method
3. Metric function: Functions defined in the metrics module

Mean Squared Error is an example of Scoring parameter API.
– Similar to Mean Absolute Error where the error is magnified using the square function
– To get the original unit, you can take square root also known as Root Mean Squared Error (or RMSE)

Caveat: cross_val_score() reports scores in ascending order (largest score is best). But MSE is naturally descending scores (the smallest score is best). Thus we need to use ‘neg_mean_squared_error’ to invert the sorting. This also results in the score to be negative even though the value can never be negative.

Note:
– For the regression problem, we will use the Boston house price dataset.
– Estimator/Algorithm: Linear Regression
– Cross-Validation Split: K-Fold (k=10)

---

This recipe includes the following topics:

---

# import modules
import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

# read data file from github
# dataframe: houseDf
gitFileURL = 'https://raw.githubusercontent.com/andrewgurung/data-repository/master/housing.csv'
cols = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']
houseDf = pd.read_csv(gitFileURL, delim_whitespace=True, names = cols)

# convert into numpy array for scikit-learn
houseArr = houseDf.values

# Let's split columns into the usual feature columns(X) and target column(Y)
# Y represents the target 'MEDV' column
X = houseArr[:, 0:13]
Y = houseArr[:, 13]

# set k-fold count
folds = 10

# set seed to reproduce the same random data each time
seed = 7

# split data using KFold
kfold = KFold(n_splits=folds, random_state=seed)

# instantiate a regression model
model = LinearRegression()

# set scoring parameter to 'neg_mean_squared_error'
scoring = 'neg_mean_squared_error'

# call cross_val_score() to run cross validation
resultArr = cross_val_score(model, X, Y, cv=kfold, scoring=scoring)

# calculate mean of scores for all folds
mse = resultArr.mean()

# calculate standard deviation
stdAccuracy = resultArr.std()

# display Mean Squared Error
# descending score(smallest score is best) is denoted by negative even though the value is positive
print("Mean Absolute Error: %.3f, Standard Deviation : %.3f" % (mse, stdAccuracy))

Mean Absolute Error: -34.705, Standard Deviation : 45.574