Predicting Love at First Sight - Part 2

SMOT and Model Selection

In Part 1, I went over EDA and a feature selection process. Please visit Part 1 for the review. This part covers synthetic minority oversampling techniques (SMOT) and model selection. Synthetic minority oversampling techniques is a very useful strategy to combat imbalanced classes. Most people usually don’t find matches at speed dating events, thus the data comes with imbalanced classes. From my experience, SMOT is also extremely valuable in increasing model’s accuracy. Almost every time I would use SMOTs, I drastically increased my accuracy scores.

However, going through the interview process for data science positions, I realized that many industry experts use a simple logistic regression to combat imbalanced classes. Here, I want to demonstrate that combating imbalanced classes with more sophisticated techniques can substantially outperform logistic regression. Applying these techniques can save company several thousands or even millions dollars in revenue. In industry, instead of predicting a speed dating match we need to match an advertiser with a publisher or do risk assessment and evaluation for loans. Thus, improving accuracy of an algorithm can substantially increase profitability of a company.

In this project, I implemented several SMOT techniques and chose the one that would give me the best accuracy across seven classifiers: 1) Decision Tree, 2) Logistic Regression, 3) Bagging DT, 4) Random Forest, 5) Extra Trees, 6) Ada Boost, and 7) Gradient Boosting.

Below is an illustration of various synthetic minority oversampling techniques for the male model (in blue) and female model (in purple). As you can see from the graphs, they create additional data for the imbalanced class “match”, but they use different approaches.

MALE MODEL

FEMALE MODEL

I will not go into a detailed description of each technique, but, generally, SMOTs oversample by creating “synthetic” examples rather than by over-sampling with replacement. On the other hand, Random Oversampling works by randomly over-sampling with replacement. Previous research (Chawla, et al., 2002; Ling & Li, 1998; Japkowicz, 2000) noted Random Oversampling doesn’t significantly improve minority class recognition. As you see below, Random Oversampling improved minority class recognition on the balanced training set, but when I tested it on the imbalanced data set, it delivered only slightly better scores compared to logistic regression. The best test accuracy scores for the imbalanced data set were obtained using the regular synthetic minority over-sampling technique (SMOT Regular). Basically, my strategy was to tune and train my model using the SMOT-balanced training set, and then test its performance in comparison with the same algorithm trained on the imbalanced set and a logistic regression trained on the imbalanced data set with adjusted class weights.

In the tables of accuracy scores below, you can see that the SMOTs techniques drastically boost accuracy scores of the balanced training set.

MALE MODEL

FEMALE MODEL

Extra Tree classifier exibited the best perfomace across the SWOTs techniques. Thus, my choice was the Extra Tree classifier trained on the balanced data set using the regular synthetic minority oversampling technique. I tested it on the imbalanced test data set and compared its accuracy to the accuracy of the Extra Tree classifier that was trained and tested on the imbalanced data set and the logistic regression with adjusted class weights that was trained and tested on the imbalanced data set. The table below represents training and test accuracy scores for the tuned algorithms.

As seen in the table above, the SMOT Extra Tree classifier outperformed the Extra Tree Classifier and logistic regression. In addition, as seen below from the confusion matrices, the SMOT Extra Tree classifier substantially improved precision and recall accuracies for the imbalanced class on the test set (14%-19% and 43%-45%, correspondingly).

MALE MODEL

FEMALE MODEL

Moreover, as seen below, the SMOT Extra Tree classifier helped to yield nearly perfect AUC scores for the both models.

Conclusion

In this project, I’ve implemented feature selection techniques and modeling techniques to build two machine learning models. I also applied a synthetic minority oversampling technique to improve accuracy scores of the models as well as precision and recall accuracies for the imbalanced class. This project demonstrated that the SMOT Extra Tree significantly outperformed the logistic regression and boosted prediction accuracy scores by at least 6 % while boosting precision and recall accuracies of the imbalanced class by 14-19 % and 43-45 %, correspondingly. This is a drastic improvement. In industry, most of the time, the imbalanced class is the class that we need to predict accurately, and using simple logistic regression with class weights might not be an ideal solution.

Furthermore, I built two models, for each gender, that predicted finding love at first sight. The findings indicate that there were gender differences in everyday activities as well as in mating and relating. Males were more interested in sports, watching sports on TV, and gaming while females preferred yoga, watching TV, theater, shopping, dining concerts, and art. Males preferred attractive females while females preferred ambitious men. The feature selection process showed that love at first sight does not necessarily guarantee a long-lasting relationship. For example, even though females reported they prefered ambitious men, ambition negatively predicted finding a match for both genders. Sincerity negatively predicated finding a match for males, and was a trivial predictor in finding a match for females. That is, both males and females did not consider sincerity an important factor in finding love at first sight, even though, both genders reported that they wanted sincerity in a relationship (since both genders reported them as important qualities in a potential date). Older females had a much harder time finding a match – for them age negatively predicted love at first sight, and those females who major in Film and Social Work were more likely to find a match compared with those who major in Biology/Chemistry/Physics.

These are just some highlights, and of course, love at first sight is not completely superficial. Both genders would choose a partner who they perceived shared interests with them, and those who valued intelligence in their partners were more likely to find a match. However, these findings cannot be applicable to the general population. Columbia students and students in general do not behave like the general population. To make these findings applicable to the general population, data should be collected from multiple speed dating events from various states, from all age groups, and from people with various economic backgrounds.