Matrix Factorization

Compartilhar:

• 
• 
• 
• 

    import pandas as pd
    import numpy as np
    import examples as ex

    matrix_interactions = ex.get_matrix_interaction()
    matrix_interactions
            
            

    ratings = matrix_interactions.values
    k = 5 # numer of latent dimensions

    num_users, num_items = ratings.shape

    # Initialize user and item latent feature matrices
    user_matrix = np.random.normal(scale=1. / k, size=(num_users, k))  # U matrix
    item_matrix = np.random.normal(scale=1. / k, size=(num_items, k))  # V matrix
            

    alpha = 0.01 # learning rate
    lambda_ = 0.02 # regularization term

    """
    - Returns the predicted rating of user i for item j.
    - It does this through the dot product between the user feature vector (users[i, :]) and the item feature vector (items[j, :]).
    - The resulting vector indicates the affinity between the user and the item.
    """
    def predict(users, items, i, j):
        return np.dot(users[i, :], items[j, :].T)

    """
    - This function performs training using Stochastic Gradient Descent (SGD).
    - ratings.nonzero() returns the indices (i, j) of the non-zero positions in the ratings array, i.e. where there is a known rating.
    - ratings[i, j]: actual value of the rating given by user i to item j.
    - predict(users, items, i, j): rating prediction calculated by the model.
    - eij: diferença entre o valor real e o previsto.
    - The term 2 * eij * items[j, :] corrects the direction of the user vector based on the prediction error.
    - The term - lambda_ * users[i, :] is the regularization to avoid overfitting.
    - After the updates, the function returns the adjusted users and items vectors, which contain the final user and item embeddings.
    """
    def sgd(ratings, users, items):
        for i, j in zip(*ratings.nonzero()):
            eij = ratings[i,j] - predict(users, items, i, j)
            users[i, :] += alpha * (2 * eij * items[j, :] - lambda_ * users[i, :])
            items[j, :] += alpha * (2 * eij * users[i, :] - lambda_ * items[j, :])

        return users, items
            

    """
    - This variable will accumulate the squared error of each pair (i, j) in the ratings matrix.
    - ratings.nonzero() returns the indices (i, j) where the ratings are known (nonzero).
    - ratings[i, j]: actual rating given by user i to item j.
    - (ratings[i, j] - y_pred) ** 2: squared error between actual and predicted value.
    """
    def compute_loss(ratings, users, items):
        error = 0
        for i, j in zip(*ratings.nonzero()):
            y_pred = predict(users, items, i, j)
            error += (ratings[i, j] - y_pred) ** 2

        mse = np.mean(error)

        return mse
            

    """
    - This variable will accumulate the squared error of each pair (i, j) in the ratings matrix.
    - ratings.nonzero() returns the indices (i, j) where the ratings are known (nonzero).
    - ratings[i, j]: actual rating given by user i to item j.
    - (ratings[i, j] - y_pred) ** 2: squared error between actual and predicted value.
    """
    def compute_loss(ratings, users, items):
        error = 0
        for i, j in zip(*ratings.nonzero()):
            y_pred = predict(users, items, i, j)
            error += (ratings[i, j] - y_pred) ** 2

        mse = np.mean(error)

        # Regularization term to avoid overfitting
        # Penalizes large magnitudes in user and item vectors
        regularization = lambda_ * (np.linalg.norm(users)**2 + np.linalg.norm(items)**2)

        return mse + regularization
            

    epochs = 1000
    for epoch in range(epochs):
        user_matrix, item_matrix = sgd(ratings, user_matrix, item_matrix)

        if epoch % 100 == 0:
            error = compute_loss(ratings, user_matrix, item_matrix)
            print(f'Epoch {epoch + 1}/{epochs} - Error: {error:.4f}')

    """
    Train output:
    Epoch 1/1000 - Error: 596.5902
    Epoch 101/1000 - Error: 1.6179
    Epoch 201/1000 - Error: 1.6440
    Epoch 301/1000 - Error: 1.6707
    Epoch 401/1000 - Error: 1.6952
    Epoch 501/1000 - Error: 1.7164
    Epoch 601/1000 - Error: 1.7343
    Epoch 701/1000 - Error: 1.7491
    Epoch 801/1000 - Error: 1.7612
    Epoch 901/1000 - Error: 1.7710
    """
            

At the beginning of training, we usually start the matrices randomly. So, the model is practically “guessing” values in the predictions. That’s why the error starts high up (in the example, around 596).

As the SGD algorithm adjusts its parameters, it gradually “discovers” the best way to align these user and item vectors to approximate real interactions. This explains why, after a few iterations, the error drops to around 1.6. The model quickly picks up on the most “obvious” patterns in the data.

After this initial phase of great decline, the error tends to stabilize and even fluctuate a little. It's as if the model has learned most of the users' general preferences, but still needs to refine finer details (or deal with data that doesn't quite follow the general trend). As the number of epochs increases (200, 300, 400...), the model makes small fine adjustments, which means that the error does not fall so drastically, but remains around 1.6 to 1.7.

Why doesn’t the error go to zero? In real applications, there is always some level of noise: users may give inconsistent ratings, certain items may have few ratings, and so on. In addition, we apply “regularization” to prevent the model from simply memorizing every single point in the matrix (so-called overfitting). This regularization “safeguards” the fit a bit and prevents the model from overfitting to noise or incomplete data.

Recommending new music

Now that we have defined all the parts of the process, let's put the code together.

    import numpy as np

    class MatrixFactorization:
        def __init__(self, n_factors, learning_rate, regularization):
            """
            Matrix Factorization using Stochastic Gradient Descent (SGD) to factorize a matrix R into two matrices U and V.
            :param n_factors: Number of latent factors
            :param learning_rate: Learning rate
            :param regularization: Regularization parameter
            """
            self._k = n_factors
            self._alpha = learning_rate
            self._lambda = regularization

            self._original_matrix = None

            self._user_matrix = None
            self._item_matrix = None

            self._num_users = None
            self._num_items = None


        def fit(self, matrix, epochs=100) -> np.ndarray:
            """
            Fit the matrix factorization model to the data
            :param matrix: the matrix to factorize
            :param epochs: the number of epochs to train the model
            :return: the predicted ratings
            """
            self._initialize(matrix)

            for i in range(epochs):
                self._sgd()
                loss = self._compute_loss()
                print(f'Epoch {i + 1}/{epochs} - Training Loss: {loss:.4f}')

            return self.predict()

        def predict(self) -> np.ndarray:
            """
            Predict the ratings for every user and item.
            U matrix * V matrix^T
            :return: the predicted ratings
            """
            return self._user_matrix.dot(self._item_matrix.T)

        def _predict(self, i, j) -> np.ndarray:
            """
            Predict the rating of user i for item j
            U[i, :] * V[j, :].T
            :param i: Index of the user
            :param j: Index of the item
            :return: the predicted rating
            """
            return self._user_matrix[i, :].dot(self._item_matrix[j, :].T)

        def _sgd(self):
            """
            Perform Stochastic Gradient Descent to learn the latent factors U and V by minimizing the loss function using the original matrix R.
            :return: None
            """
            for i, j in zip(*self._original_matrix.nonzero()):
                eij = self._original_matrix[i, j] - self._predict(i, j)

                self._user_matrix[i] += self._alpha * (2 * eij * self._item_matrix[j] - self._lambda * self._user_matrix[i])
                self._item_matrix[j] += self._alpha * (2 * eij * self._user_matrix[i] - self._lambda * self._item_matrix[j])

        def _compute_loss(self) -> float:
            """
            Compute the loss (MSE) of the prediction
            :return: the loss of the prediction
            """
            error = 0
            for i, j in zip(*self._original_matrix.nonzero()):
                error = self._original_matrix[i, j] - self._predict(i, j)
                error += error ** 2

            mse = np.mean(error)
            regularization = self._lambda * (np.linalg.norm(self._user_matrix) + np.linalg.norm(self._item_matrix))
            return mse + regularization


        def _initialize(self, matrix):
            self._original_matrix = matrix.copy()
            self._num_users, self._num_items = self._original_matrix.shape

            self._user_matrix = np.random.normal(scale=1. / self._k, size=(self._num_users, self._k))  # U matrix
            self._item_matrix = np.random.normal(scale=1. / self._k, size=(self._num_items, self._k))  # V matrix
        

Training

Now that we have everything together, we can start training the model. But first, it's important to normalize the ratings values to maintain the range \([0, 5]\). This minimizes the impact of outliers and allows the model to learn patterns and relations more efficiently and consistently.

    from sklearn.preprocessing import MinMaxScaler

    ratings = matrix_interactions.values

    scaler = MinMaxScaler(feature_range=(0, 5))
    normalized_ratings = scaler.fit_transform(ratings)

    model = MatrixFactorization(n_factors=75, learning_rate=0.001, regularization=0.02)
    matrix_factorization_result = model.fit(ratings, epochs=600)
        

Result of matrix_factorization_result

Comparison of actual grade vs model prediction:

Gap predictions

While training the model with Matrix Factorization, some cells in the interactions matrix had values ​​equal to 0. This means that these users did not rate these musics or have any recorded interactions with them. So how did the model manage to predict these missing ratings?

The model fills in these gaps by using the similarity between the latent factors of the users and the musics. For example, even though Darth Vader has never listened to "Brandenburg Concerto No. 3" (a score of 0 in the interaction matrix), the model was able to predict a score of 0.71. This was possible because the factored matrix learned that Darth Vader tends to prefer epic and science fiction music, but has little affinity for classical music.

Let's look at other examples of predictions made by the model:

• Darth Vader and "Brandenburg Concerto No. 3": Prediction: 0.71 This indicates that the model understood that Darth Vader has little interest in this music. The score was low because Darth Vader's latent factors do not align well with the latent factors of classical music.
• Hermione and "The Imperial March": Prediction: 0.97 Hermione didn't show much interest in this music (original score 1), and the model predicted a value close to it. This is because Hermione's latent factors indicate a preference for emotional and classical music, but "The Imperial March" is not among her favorites.
• Ripley and "Clair de Lune": Prediction: 2.02 Even without a prior assessment, the model predicted a moderate score. This is because Ripley has shown some interest in soft, emotional music, but "Clair de Lune" is not entirely in line with her core preferences.
• Spock and "Firework": Prediction: 2.25 Although Spock never interacted with this music, the model predicted limited interest. This is consistent with Spock's latent factors indicating a preference for classical music, while "Firework" is a pop music.

In general, the model makes these predictions by multiplying the latent factors of users and items, adjusted by Stochastic Gradient Descent (SGD). The multiplication between the matrices \(U\) and \(V\) captures the relation between the general preferences of each user and the characteristics of the musics.

As SGD minimizes the error in known predictions, it adjusts the latent factors to correctly reflect individual tastes. This allows the model to generalize and predict gaps in the original matrix, providing personalized music suggestions for each character.

This allows Robbie to suggest new music to each user, personalizing his recommendations.

Similarity between users

    from sklearn.metrics.pairwise import cosine_similarity
    user_similarity = cosine_similarity(matrix_factorization_result)
    df_user_similarity = pd.DataFrame(user_similarity, index=ex.users, columns=ex.users)
    df_user_similarity
        

1. Ripley and Spock (0.996773):
   • High similarity, as both have strong preferences for musics like “Bohemian Rhapsody” and “Back in Black”.
2. Darth Vader and Hermione (0.976032):
   • High similarity, which makes sense, since they both like epic and classical music, like “The Imperial March” and “Hedwig’s Theme”.
3. Darth Vader and Spock (0.856497):
   • Moderate similarity, because although they have some preferences in common, Spock has a greater focus on classical music.

Recommend new musics to Hermione

After training the model, Robbie is ready to make recommendations! He’s looked at the songs Hermione hasn’t listened to yet and analyzed other characters who have similar likes to her.

    def recommend_by_user(user, df_predicted, df_interactions, top_k=3):
        # Calculate user similarity matrix
        user_similarity = cosine_similarity(df_predicted.values)
        user_similarity_df = pd.DataFrame(user_similarity, index=ex.users, columns=ex.users)

        # Get the top similar users
        similarities = user_similarity_df.loc[user].sort_values(ascending=False)[1:top_k+1]
        print('Sser similarity score:')
        print(similarities)

        # Initialize recommendations
        recommendations = pd.Series(0, index=df_predicted.columns, dtype=np.float64)

        # Aggregate recommendations based on similar users
        for user_similar, similarity in similarities.items():
            recommendations = recommendations.add(
                df_predicted.loc[user_similar].reindex(recommendations.index, fill_value=0) * similarity,
                fill_value=0
            )

        # Filter out already known items
        already_known = df_interactions.loc[user] > 0
        recommendations = recommendations[~already_known.reindex(recommendations.index, fill_value=False)]

        return recommendations.sort_values(ascending=False)


    recommend_by_user('Hermione', df_matrix_interactions, matrix_interactions)

    """
    Output:
    Sser similarity score:
    Darth Vader    0.976032
    Spock          0.805946
    Ripley         0.795352
    """
        

Recommendations:

Robbie realized that the characters most similar to Hermione, based on musical likes, are:

• Darth Vader (high similarity: 0.97)
• Spock (moderate similarity: 0.81)
• Ripley (moderate similarity: 0.79)

Darth Vader has the most similar likes to Hermione, so his preferences carried more weight in the recommendations.

Why was "Black in Black" recommended?

• Hermione has never heard "Black in Black" before (rating 0 in the interaction matrix).
• Robbie saw that Darth Vader and Ripley really liked this music:
  • Since Darth Vader has a very high similarity to Hermione, his preferences greatly influenced the recommendation.
• Robbie concluded that Hermione would probably also love this music because of the strong influence of Darth Vader and Ripley.

Why was "Brandenburg Concerto No. 3" recommended?

• Hermione hadn't heard that music either (rating 0 in the interactions matrix).
• Robbie realized that Spock liked this song a lot.
• Additionally, Hermione has shown an interest in classical music (such as "Clair de Lune").
• With Spock's influence and his past preferences, Robbie concluded that this classical music would be a good recommendation.

in summary

The model works like a friend (Robbie) who knows your likes well and compares them with those of other people. He realized that Hermione:

• She shares musical likes with Darth Vader, which led to the recommendation of a rock song like "Black in Black".
• Her has an affinity for classical music, such as Spock, which resulted in the recommendation of "Brandenburg Concerto No. 3.".

Robbie now knows exactly how to surprise Hermione with new musics! 🎶

Conclusion

Matrix Factorization stands out as an essential tool for creating personalized recommendations at scale, helping platforms such as streaming services and e-commerce to better understand their users. By decomposing the matrix of interactions, we can identify hidden patterns of preferences, even when there is missing data.

With Stochastic Gradient Descent (SGD), we iteratively adjusted the matrices, reducing the error between predictions and real data. Regularization ensured a balanced model, without overfitting, while data normalization brought more stability to the learning.

In the end, we saw how the model goes beyond direct interactions, predicting preferences even when data is limited. The story of Robbie and his iconic friends showed us that these techniques can be applied to a variety of scenarios, providing unique and personalized recommendations. 🎶

References

• Koren, Yehuda; Bell, Robert; Volinsky, Chris. "Matrix Factorization Techniques for Recommender Systems". IEEE Computer Society, 2009. Link to article
• Ricci, Francesco; Rokach, Lior; Shapira, Bracha. "Recommender Systems Handbook". Springer, 2022. Link to the book
• Hug, Nicolas. "Surprise: A Python Scikit for Building and Analyzing Recommender Systems". Official documentation
• Said, Alan; Bellogín, Alejandro. "Comparative Recommender System Evaluation". Springer, 2014. Link to article

Post Tags machine-learning