Missing Data and the Sound of One Hand Clapping One of my favorite koans is " Two hands clap and there is a sound. What is the sound of one hand clapping?" Im not sure why I like it. Maybe because it is a question that has no answer, yet it seems like it should. The defintion of koan is "A paradoxical anecdote or riddle used in Zen Buddism to demonstrate and provoke enlightenment" Anyway, when I was first introduced to the problem of missing data with respect to building a machine learning model, my response was similar to how I felt about the clapping koan. As it turns out, that was the wrong response. There is a solution to the problem of missing data. Its called imputation. One method of imputation is to replace all missing values in a column with the mean of that column. Another method is to replace all missing values with zero. Both of these methods lead to problems. Mean imputation reduces the variance. Thi...
Posts
What's the Story? I recently completed the Data Science Immersive course from General Assembly. For our first project, we were given data that broke down the scores and participation rates of SAT's and ACT's by state. The SAT and the ACT are the most popular college admission tests. The assignment was to analyze the data and make a presentation to an (imaginary) audience of people from the company that provides the SAT. The SAT and the ACT are published by two different companies that compete for market share. Since the data included participation rates, my first thought was to make a presentation with suggestions as to how to increase market share in the states where where the ACT is outpacing the SAT. I kicked off my analysis by generating a heat map of the correlations between each of the variables. I show it below. Look at the SAT_M (math) on...
STEM, STEAM and Machine Learning Anyone who is somewhat up to date on education buzzwords is surely familiar with STEM=Science, Technology, Engineering, and Mathematics. There is a recent twist on STEM: STEAM. STEAM is STEM with the arts (A) injected into the mix. Its all the rage. To see this, all one needs to do is go to the Google news tab and search on "Steam Education". On any given day, articles from local newspapers from all over the country reporting that the local school district is either discussing how to fund a STEAM program, starting a STEAM program or reporting on the ongoing success of a STEAM program that has been in place for a year or two. The link below will take you to an article from Science Magazine that discusses the need for art in STEM. Note that the article has its own version of the acronym (STEMM). Click here to see the article. It seems to me that a great wa...