Hello and welcome to the first post in my "What I've Learned" series. It's been a while since I've updated my blog and it's been very busy since then: leaving a job, starting another and getting laid off, volunteering, and doing a lot of self-training. On the training front, I just completed a six week Data Analysis with Python: Zero to Pandas course delivered through a collaboration between Jovian.ML and Freecode Camp.

"Data Analysis with Python: Zero to Pandas" is a practical, beginner-friendly and coding-focused introduction to data analysis covering the basics of Python, Numpy, Pandas, data visualization and exploratory data analysis.

I learned data analysis and visualization with Python and NumPy, Pandas, Matplotlib, and Seaborn libraries. During the course, we were tasked with completing three assignments and an overall course project. Since I consider the entire month of October as Halloween season, I thought it would be appropriate to analyze some Halloween related data for my project.

Zero to Pandas Course Project: 2017 Halloween Candy Survey

Dataset

In recent years, a group at The University of British Columbia surveyed people on their Halloween activities. I chose to analyze the data from their 2017 survey, Candy Hierarchy 2017 Survey, and see what I could find. The survey, with over 2000 responses, has information on age, gender, location, and candy preferences.

Data Preparation and Cleaning

As a first step, I uploaded my Jupyter notebook to Jovian.ml, created variables for my project name and the location of my data, and then installed Pandas so that I could work with my data as a dataset.

project_name = "course-project-halloween-candy"

project_file = "https://raw.githubusercontent.com/rdhm113/z-to-p-course-project/main/candyhierarchy2017.csv"

!pip install jovian --upgrade -q

import jovian

jovian.commit(project=project_name)

import pandas as pd

project_file

'https://raw.githubusercontent.com/rdhm113/z-to-p-course-project/main/candyhierarchy2017.csv'

After importing my dataset, I encountered a read error when previewing the dataframe. I imported it again and set an encoding type to eliminate the error.

candy_survey_raw_df = pd.read_csv(project_file, encoding = 'ISO-8859-1')

Previewing and requesting info on the dataframe showed me that there are 115 columns and 2460 rows of data in my dataset.

candy_survey_raw_df

candy_survey_raw_df.info()

RangeIndex: 2460 entries, 0 to 2459
Columns: 120 entries, Internal ID to Click Coordinates (x, y)
dtypes: float64(4), int64(1), object(115)
memory usage: 2.3+ MB

Previewing a list of column headers showed the basic data for the columns. In the absence of a data schema, I used the PDF of the survey to understand the relationship between column headers and possible row values.

candy_survey_raw_df.columns

Index(['Internal ID', 'Q1: GOING OUT?', 'Q2: GENDER', 'Q3: AGE', 'Q4: COUNTRY',
       'Q5: STATE, PROVINCE, COUNTY, ETC', 'Q6 | 100 Grand Bar',
       'Q6 | Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Q6 | Any full-sized candy bar', 'Q6 | Black Jacks',
       ...
       'Q8: DESPAIR OTHER', 'Q9: OTHER COMMENTS', 'Q10: DRESS', 'Unnamed: 113',
       'Q11: DAY', 'Q12: MEDIA [Daily Dish]', 'Q12: MEDIA [Science]',
       'Q12: MEDIA [ESPN]', 'Q12: MEDIA [Yahoo]', 'Click Coordinates (x, y)'],
      dtype='object', length=120)

An important step that I've learned in this course and other data analysis that I've done is to always make a copy of data to work with. This preserves the raw data in case you need to go back and start again or fix errors.

candy_survey_df = candy_survey_raw_df.copy()
candy_survey_df

Next, columns that wouldn't be used for analysis were dropped: Internal ID and survey questions 7-13.

candy_survey_df.drop(['Internal ID', 'Q7: JOY OTHER','Q8: DESPAIR OTHER', 'Q9: OTHER COMMENTS', 'Q10: DRESS', 'Unnamed: 113',
       'Q11: DAY', 'Q12: MEDIA [Daily Dish]', 'Q12: MEDIA [Science]',
       'Q12: MEDIA [ESPN]', 'Q12: MEDIA [Yahoo]', 'Click Coordinates (x, y)'], axis=1, inplace=True)

candy_survey_df.columns

Index(['Q1: GOING OUT?', 'Q2: GENDER', 'Q3: AGE', 'Q4: COUNTRY',
       'Q5: STATE, PROVINCE, COUNTY, ETC', 'Q6 | 100 Grand Bar',
       'Q6 | Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Q6 | Any full-sized candy bar', 'Q6 | Black Jacks',
       'Q6 | Bonkers (the candy)',
       ...
       'Q6 | Three Musketeers', 'Q6 | Tolberone something or other',
       'Q6 | Trail Mix', 'Q6 | Twix',
       'Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein',
       'Q6 | Vicodin', 'Q6 | Whatchamacallit Bars', 'Q6 | White Bread',
       'Q6 | Whole Wheat anything', 'Q6 | York Peppermint Patties'],
      dtype='object', length=108)

Reviewed dateframe info again after the change.

candy_survey_df.info()

RangeIndex: 2460 entries, 0 to 2459
Columns: 108 entries, Q1: GOING OUT? to Q6 | York Peppermint Patties
dtypes: object(108)
memory usage: 2.0+ MB

Calculated the null values to get an idea of how many questions were left unanswered.

candy_survey_df.isnull().sum()

Q1: GOING OUT?                      110
Q2: GENDER                           41
Q3: AGE                              84
Q4: COUNTRY                          64
Q5: STATE, PROVINCE, COUNTY, ETC    100
                                   ... 
Q6 | Vicodin                        789
Q6 | Whatchamacallit Bars           823
Q6 | White Bread                    757
Q6 | Whole Wheat anything           747
Q6 | York Peppermint Patties        705
Length: 108, dtype: int64

The column names needed some clean up for easier use so the following steps were taken. The codes were run multiple times as I kept finding inconsistencies.

Cleaning up column names to remove question numbers: Q1, Q2, etc.

candy_survey_df.columns = candy_survey_df.columns.str[4:] #strip the first 4 leading characters

candy_survey_df.columns

Index(['GOING OUT?', 'GENDER', 'AGE', 'COUNTRY',
       'STATE, PROVINCE, COUNTY, ETC', ' 100 Grand Bar',
       ' Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       ' Any full-sized candy bar', ' Black Jacks', ' Bonkers (the candy)',
       ...
       ' Three Musketeers', ' Tolberone something or other', ' Trail Mix',
       ' Twix',
       ' Vials of pure high fructose corn syrup, for main-lining into your vein',
       ' Vicodin', ' Whatchamacallit Bars', ' White Bread',
       ' Whole Wheat anything', ' York Peppermint Patties'],
      dtype='object', length=108)

Now that the question numbers were removed, I noticed that there were leading white spaces in some of the column names that needed to be removed.

candy_survey_df.columns = candy_survey_df.columns.str.strip() #strip leading and trailing whitespace
candy_survey_df.columns

Index(['GOING OUT?', 'GENDER', 'AGE', 'COUNTRY',
       'STATE, PROVINCE, COUNTY, ETC', '100 Grand Bar',
       'Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)',
       'Any full-sized candy bar', 'Black Jacks', 'Bonkers (the candy)',
       ...
       'Three Musketeers', 'Tolberone something or other', 'Trail Mix', 'Twix',
       'Vials of pure high fructose corn syrup, for main-lining into your vein',
       'Vicodin', 'Whatchamacallit Bars', 'White Bread',
       'Whole Wheat anything', 'York Peppermint Patties'],
      dtype='object', length=108)

Renamed specific columns to make them easier to understand. For example: "Anonymous brown globs that come in black and orange wrappers (a.k.a. Mary Janes)" was not a column name that I wanted to work with.

candy_survey_df.rename(columns={ 'GOING OUT?':'Going Out','Anonymous brown globs that come in black and orange wrappers\t(a.k.a. Mary Janes)':'Mary Janes','Chick-o-Sticks (we don’t know what that is)':'Chick-o-Sticks','Sourpatch Kids (i.e. abominations of nature)':'Sourpatch Kids','Tolberone something or other':'Tolberone','Vials of pure high fructose corn syrup, for main-lining into your vein':'high fructose corn syrup'}, inplace=True)

candy_survey_df.columns

Index(['Going Out', 'GENDER', 'AGE', 'COUNTRY', 'STATE, PROVINCE, COUNTY, ETC',
       '100 Grand Bar', 'Mary Janes', 'Any full-sized candy bar',
       'Black Jacks', 'Bonkers (the candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'high fructose corn syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat anything', 'York Peppermint Patties'],
      dtype='object', length=108)

Converted all column names to title case.

candy_survey_df.columns = candy_survey_df.columns.str.title()

candy_survey_df.columns

Index(['Going Out', 'Gender', 'Age', 'Country', 'State, Province, County, Etc',
       '100 Grand Bar', 'Mary Janes', 'Any Full-Sized Candy Bar',
       'Black Jacks', 'Bonkers (The Candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'High Fructose Corn Syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat Anything', 'York Peppermint Patties'],
      dtype='object', length=108)

After additional work, I noticed that some columns had an unexpected character: 'õ'.

candy_survey_df.columns = candy_survey_df.columns.str.replace('õ', '\'')

candy_survey_df.columns

Index(['Going Out', 'Gender', 'Age', 'Country', 'State, Province, County, Etc',
       '100 Grand Bar', 'Mary Janes', 'Any Full-Sized Candy Bar',
       'Black Jacks', 'Bonkers (The Candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'High Fructose Corn Syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat Anything', 'York Peppermint Patties'],
      dtype='object', length=108)

After column name work it was time to move on to the Age and Country columns.

Clean up the Age and Country columns

The Age and Country survey questions allowed for answers in free-form text fields so a lot of my time was focused on cleaning these.

Country

I started by viewing how many unique country values existed and the country names and counts. There were 128 which could have been true values or not. I needed to do some more digging.

candy_survey_df.Country.nunique()

128

candy_survey_df.Country.value_counts()

USA                             699
United States                   497
usa                             217
Canada                          179
Usa                             139
                               ... 
Korea                             1
U.S.                              1
Indonesia                         1
The United States of America      1
Ahem....Amerca                    1
Name: Country, Length: 128, dtype: int64

In sampling some of the dataset, I immediately noticed inconsistencies in how countries were named, the most visible was for the United States of America.

candy_survey_df.sample(20)

My first bit of cleanup was to remove any leading or trailing whitespace in the Country column.

candy_survey_df.Country = candy_survey_df.Country.str.strip()

candy_survey_df

The Country text-field allowed for numerous spellings and namings of countries and the majority of my project cleaning was spent here. I repeatedly cycled through replace and value counts functions to find and merge as many inconsistencies as possible.

candy_survey_df.Country.value_counts(ascending=True) #Sorting country values from least amount to greatest.

Ahem....Amerca      1
Indonesia           1
Korea               1
New Jersey          1
spain               1
                 ... 
Usa               140
Canada            187
usa               217
United States     534
USA               772
Name: Country, Length: 117, dtype: int64

#Replacing country names to consolidate names.
candy_survey_df.Country = candy_survey_df.Country.replace(['united States','United Statea','united ststes','murrika','USA? Hard to tell anymore..','U S','usas','u s a','The United States of America','unite states','USA! USA! USA!','america','United Sates','Unied States','United State','America','Unites States','United Stated','USSA','USAUSAUSA','The United States','u.s.','Unites States,''america','US of A','united states of america','USA USA USA!!!!','United ststes','\'merica', 'U.S.A.','u.s.a.','U.S.','United staes','United Statss','Us','us', 'USa', 'USAA', 'Usa', 'USA','usa','US', 'united states','United States of America','United states'],'United States')
candy_survey_df.Country = candy_survey_df.Country.replace(['Canada`','CANADA','canada','Can'], 'Canada')
candy_survey_df.Country = candy_survey_df.Country.replace(['U.K.','endland','uk','Uk','UK','United kingdom','England'],'United Kingdom')
candy_survey_df.Country = candy_survey_df.Country.replace(['germany'],'Germany')
candy_survey_df.Country = candy_survey_df.Country.replace(['australia'],'Australia')
candy_survey_df.Country = candy_survey_df.Country.replace(['france'],'France')

#Creating a new series of only the top 15 countries
top_countries = candy_survey_df.Country.value_counts().head(15)

top_countries

United States     2037
Canada             226
United Kingdom      35
Germany             10
Australia            7
Netherlands          6
Scotland             5
Japan                5
Ireland              4
Mexico               4
Switzerland          3
France               3
China                2
Sweden               2
Denmark              2
Name: Country, dtype: int64

There were so many minuscule country counts that became difficult and timeconsuming to catch them all. I decided that my project would focus on the top 10-20 countries.

Age

I expected that there could be white space in the Age data, as I learned with the Country data, so my first step was to strip that.

candy_survey_df.Age = candy_survey_df.Age.str.strip()
candy_survey_df.Age.value_counts()

40                92
34                90
37                89
43                86
36                79
                  ..
old enough         1
46 Halloweens.     1
77                 1
99                 1
5u                 1
Name: Age, Length: 106, dtype: int64

Reviewing the value counts showed that there were non-numerical values. All non-numerical values were converted to null.

candy_survey_df.Age = pd.to_numeric(candy_survey_df.Age, errors='coerce')

candy_survey_df.Age.value_counts()

40.0    92
34.0    90
37.0    89
43.0    86
36.0    79
        ..
77.0     1
88.0     1
4.0      1
99.0     1
1.0      1
Name: Age, Length: 82, dtype: int64

candy_survey_df.Age.describe()

count    2351.000000
mean       42.605444
std        23.844665
min         1.000000
25%        34.000000
50%        41.000000
75%        50.000000
max      1000.000000
Name: Age, dtype: float64

Upon reviewing a summary of the age values, I noticed a problem because there was a max value of 1000. I'd be very concerned if there were people who are 200+ years old answering the survey. I chose to drop the rows with values greater than 110.

candy_survey_df.drop(candy_survey_df[candy_survey_df.Age>110].index, inplace=True)

candy_survey_df.Age.describe()

count    2349.000000
mean       42.083184
std        12.144721
min         1.000000
25%        34.000000
50%        41.000000
75%        50.000000
max       102.000000
Name: Age, dtype: float64

I reviewed the dataframe again for any final touchups and noticed that the very first row contains null values.

candy_survey_df

candy_survey_df = candy_survey_df.drop(candy_survey_df.index[0]) #Dropping first row as it's filled with null values

candy_survey_df

After all of that work, I felt that my dataset was clean enough to go forward with visualization.

Exploratory Analysis and Visualization

My plan going forward was to use Matplotlib and Seaborn for visualization.

I started by importing matplotlib and seaborn and then setting my chart styles.

import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

# chart styling
sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 12
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = 'white'

# Setting a Halloween themed colored palette
colors = ['#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e', '#000000', '#8a29b2', '#ff9936', '#ff6100', '#a9640e']
sns.set_palette(colors)

Going out or staying in?

The first question on the survey asked if the respondents had plans to go out to celebrate Halloween or to just stay home.

Let's start by seeing how many responses are left after clean-up.

candy_survey_df['Going Out']

1        No
2       NaN
3        No
4        No
5        No
       ... 
2455     No
2456     No
2457     No
2458     No
2459     No
Name: Going Out, Length: 2457, dtype: object

Next, let's look at how the 2,457 responses were spread.

candy_survey_df['Going Out'].value_counts() #View results of values

No     2037
Yes     311
Name: Going Out, dtype: int64

I used the seaborn .countplot() function to count the number of values since "Yes" and "No" aren't numerical values.

#Visualize in a bar chart
plt.figure(figsize=(12,6))
plt.xticks(rotation=0)
plt.title('Are you actually going trick or treating yourself?')
sns.countplot(y = 'Going Out', data = candy_survey_df);

The overwhelming majority of respondents said that they would not be going out trick or treating.

Gender

The gender question allowed for four answers: Male, Female, I'd Rather Not Say, and Other.

candy_survey_df.Gender.value_counts() #View results of values

Male                  1467
Female                 839
I'd rather not say      82
Other                   29
Name: Gender, dtype: int64

#Visualize in a bar chart
plt.figure(figsize=(16,6))
plt.xticks(rotation=0)
plt.title('Your gender')
sns.countplot(x = 'Gender', data = candy_survey_df);

I was surprised that the majority of respondents chose "Male" as their gender.

Age

Let's take a look at the ages reported by respondents.

candy_survey_df.Age.value_counts() # View counts of Ages of respondents

40.0     92
34.0     90
37.0     89
43.0     86
36.0     79
         ..
4.0       1
39.4      1
74.0      1
102.0     1
99.0      1
Name: Age, Length: 80, dtype: int64

#Import NumPy
import numpy as np

#Display age range in a histogram
plt.figure(figsize=(12,6))
plt.xticks(rotation=0)
plt.title('How old are you? (in years)')
plt.xlabel('Age')
plt.ylabel('Number of respondents')

plt.hist(candy_survey_df.Age, bins=np.arange(0,105,5), color='#ff9936')

(array([  2.,  10.,  27.,  30.,  62., 151., 340., 391., 407., 326., 255.,
        161., 108.,  41.,  25.,   6.,   0.,   1.,   2.,   3.]),
 array([  0,   5,  10,  15,  20,  25,  30,  35,  40,  45,  50,  55,  60,
         65,  70,  75,  80,  85,  90,  95, 100]),
 )

The answers to the Age survey question were interesting because the data included ages that were too young to fill out the survey themselves. It led me to believe that an older parent/guardian answered in a younger person's place.

Country

What country do you live in?

With the knowledge that the respondents' countries weighed heavily amongst the two countries, I decided to narrow my analysis to only the top 10 countries.

#Creating a series of only the countries in the top 10 of responses
top_countries = candy_survey_df.Country.value_counts().head(10)

top_countries

United States     2037
Canada             225
United Kingdom      35
Germany             10
Australia            7
Netherlands          6
Scotland             5
Japan                5
Ireland              4
Mexico               4
Name: Country, dtype: int64

#Visualize the top 10 countries of responses in a bar chart
plt.figure(figsize=(16, 8))
plt.xticks(rotation=0)
plt.title('What country do you live in?')
sns.countplot(y = 'Country', data = candy_survey_df, order=candy_survey_df.Country.value_counts().iloc[0:10].index);

Respondents were primarily located in the United States which I found interesting since the UBC is a Canadian institution.

Candy Ratings: Joy or Despair?

The primary focus of the survey was to collect data on whether people liked, disliked, or had no strong opinion on candies. The ratings were: Joy, Despair, and Meh.

I started by creating a subset of the dataframe that only included candy ratings.

candy_df = candy_survey_df.iloc[0: , 5:]

candy_df

Then to get an idea of what data could look like, I created a subset of a single candy's ratings for visualization: the 100 Grand Bar.

#Create a subset of 100 Grand Bar data
grand_df = candy_survey_df.iloc[0: , 5:6]

grand_df

#Visualize the candy ratings

plt.figure(figsize=(12,6))
plt.xticks(rotation=0)
plt.title('Feelings on the 100 Grand Bar')
sns.countplot(x = '100 Grand Bar', data = candy_survey_df);

There were very few people who hated it and surprisingly people either loved it or had no strong opinion.

Since there are over 100 candy questions in the survey, I created a subset of 8 to explore. This survey had some really weird questions and it just so happened that this subset included "Generic Brand Acetaminophen."

#Created a data range of 8 candies 
candy_sample_df = candy_survey_df.iloc[0: , 26:34]

candy_sample_df

#Visualize ratings
candy_ratings_df = pd.melt(candy_sample_df) # .melt() function rotates data like a pivot table
plt.figure(figsize=(20,6))
plt.xticks(rotation=0)
plt.title('Feelings on a range of candies')
sns.countplot(x = 'variable', hue='value', data = candy_ratings_df);

Of this range, Dove Bars were an overwhelming favorite and I would have to agree.

Asking and Answering Questions

After visualizing some of the data I had a few questions to delve into further.

Q: How many adults (respondants 18 and over) answered the survey?

I created a new series of the cleaned data that contained only rows where Age was 18 and older, dropped rows with null values, and converted the float numerical value to integer.

adults_df = candy_survey_df.copy()
adults_df.drop(adults_df[adults_df.Age < 18].index, inplace = True) #drop rows for ages younger than 18
adults_df.dropna(subset = ['Age'], inplace=True) #drop rows where Age column has a NaN value
adults_df.Age = adults_df.Age.astype(int) #convert Age values from float to integer
adults_df['Age']

1       44
2       49
3       40
4       23
6       53
        ..
2455    24
2456    33
2457    26
2458    58
2459    66
Name: Age, Length: 2293, dtype: int64

plt.figure(figsize=(12,20))
plt.xticks(rotation=0)
plt.title('Respondants over the age 18')
sns.countplot(y = 'Age', data = adults_df);

We see from the data that of the 2,293 adult respondants most are in the mid-30s to mid-40s range.

Q: Who is going out versus staying in in the top 10 countries?

I created a dataframe of only the "Going Out" and "Country" columns for comparison.

#Reviewing the top 10 countries
top_countries

United States     2037
Canada             225
United Kingdom      35
Germany             10
Australia            7
Netherlands          6
Scotland             5
Japan                5
Ireland              4
Mexico               4
Name: Country, dtype: int64

top_countries_list = ['United States','Canada','United Kingdom','Germany','Australia', 'Netherlands','Scotland', 'Japan', 'Mexico' , 'Ireland']

top_countries_list

['United States',
 'Canada',
 'United Kingdom',
 'Germany',
 'Australia',
 'Netherlands',
 'Scotland',
 'Japan',
 'Mexico',
 'Ireland']

#Reviewing the column names

candy_survey_df.columns

Index(['Going Out', 'Gender', 'Age', 'Country', 'State, Province, County, Etc',
       '100 Grand Bar', 'Mary Janes', 'Any Full-Sized Candy Bar',
       'Black Jacks', 'Bonkers (The Candy)',
       ...
       'Three Musketeers', 'Tolberone', 'Trail Mix', 'Twix',
       'High Fructose Corn Syrup', 'Vicodin', 'Whatchamacallit Bars',
       'White Bread', 'Whole Wheat Anything', 'York Peppermint Patties'],
      dtype='object', length=108)

#Creating data frame of only Going Out and Country data
going_out_df = candy_survey_df[['Going Out', 'Country']].copy()
#Filtering data for only the top 10 countries in Country column
going_out_df = going_out_df.loc[(going_out_df['Country'].isin(top_countries_list))]
going_out_df

plt.figure(figsize=(24,16))
plt.xticks(rotation = 0)
plt.title('Going Out for Halloween by Country')
sns.countplot( y = 'Country', hue = 'Going Out' , data = going_out_df);

Of the 2,338 respondents, an overwhelming number were staying indoors for Halloween.

Q: What are the demographics (Gender and Age) per top 10 country?

I created a dataframe subset with only Gender, Age, and Country columns and then used the average age as a comparison.

candy_survey_demo = candy_survey_df.iloc[0:,1:4] #copy of survey dataset with only age, gender and country,
candy_survey_demo = candy_survey_demo.loc[(candy_survey_demo['Country'].isin(top_countries_list))] #subset data to only include top 10 countries 

candy_survey_demo

#average the age column per country and gender and then plot
candy_survey_demo.groupby(['Country','Gender'])['Age'].mean().unstack().plot.bar(figsize=(18,6), rot=0, title='Respondant Demographics per Top 10 Country', ylabel="Average Age");

I found it interesting that in the United States the respondents with the oldest average age chose to not respond with their gender and the respondents with the oldest average age in Germany were female.

Q: What are the top 10 candies that bring joy?

Now for the good stuff, unlike the subset of candy rating data that I used during analysis, I manipulated a dataframe subset of all of the candy ratings, and then pulled out the top 10 rated highest for "JOY."

#Recall data subset of just candy columns, create a copy using .melt() function to pivot data
candy_ratings_df = candy_df.melt(var_name='columns', value_name='index')
# use .crosstab() function to count total DESPAIR, JOY, and MEH values
candy_ratings_df = pd.crosstab(index = candy_ratings_df['index'], columns = candy_ratings_df['columns'])

candy_ratings_df

# use .iloc method to call only the "JOY" row, sort values in descending order, and take top 10 values
candy_joy_df = candy_ratings_df.iloc[1].sort_values(ascending=False).head(10)

candy_joy_df

columns
Any Full-Sized Candy Bar                1557
Reese's Peanut Butter Cups              1496
Kit Kat                                 1445
Cash, Or Other Forms Of Legal Tender    1439
Twix                                    1412
Snickers                                1397
Tolberone                               1319
Peanut M&M's                            1289
Lindt Truffle                           1275
Nestle Crunch                           1187
Name: JOY, dtype: int64

#visualize data
plt.figure(figsize=(14,9))
plt.xticks(rotation=75)
plt.xlabel('Candy')
plt.ylabel('Number of JOY responses')
plt.title('Candies that bring the most Joy')
sns.barplot( x=candy_joy_df.index, y=candy_joy_df.values);

Not very surprising that full-sized candy bars were the absolute favorite.

Q: What are the top 10 worst candies that bring despair?

As in the above question, I manipulated a dataframe subset of all of the candy ratings, and then pulled out the top 10 rated highest for "DESPAIR." Otherwise known as the 10 least liked candies.

# use .iloc method to call only the "DESPAIR" row, sort values in descending order, and take top 10 (or 10 worst) values
candy_despair_df = candy_ratings_df.iloc[0].sort_values(ascending=False).head(10)

candy_despair_df

columns
Broken Glow Stick                                                         1624
Real Housewives Of Orange County Season 9 Blue-Ray                        1479
Gum From Baseball Cards                                                   1461
White Bread                                                               1455
Kale Smoothie                                                             1434
Dental Paraphenalia                                                       1432
Creepy Religious Comics/Chick Tracts                                      1405
Candy That Is Clearly Just The Stuff Given Out For Free At Restaurants    1324
Whole Wheat Anything                                                      1289
Those Odd Marshmallow Circus Peanut Things                                1250
Name: DESPAIR, dtype: int64

plt.figure(figsize=(14,9))
plt.xticks(rotation=75)
plt.xlabel('Candy')
plt.ylabel('Number of DESPAIR responses')
plt.title('Candies that bring the most Despair')
sns.barplot( x=candy_despair_df.index, y=candy_despair_df.values);

The data retrieved with this question highlighted the odd questions included in the survey. Because of this, there wasn't a lot of meaningful data pulled.

Inferences and Conclusion

• This was a challenging dataset to analyze. It tested my patience when it came to cleaning up the countries and caused me to stretch when I knew about manipulating the math.

• The overwhelming majority of respondents said that they would not go out trick or treating. I wonder how the answers would have changed if instead, the question asked if the respondents were going out to Halloween parties.

• The demographics of survey respondents generally leaned male and were in the age range of the mid-30s to mid-40s.

• Full-size candy bars were a favorite for Halloween candy. I think that option was misleading because it wasn't a type of candy. Without that option, Reese's Peanut Butter Cups would be the true winner.

• There is bias in the survey data that since it's from a Canadian institute, the data is highly westernized and English-speaking focused. Even though there were responses from countries other than Canada and the US, there wasn't enough data for true learnings.

• I would suggest that the surveyors used a standardized set of location information (countries, provinces, states, counties, etc.) to eliminate the free for all with a text-field. For that reason, I didn't analyze the "State, Province, County" data column.

• There's an interesting infographic from Boing Boing of all of the day ratings sorted into additional categories.

References and Future Work

If I were to continue analyzing this dataset, I would look into these ideas:

• Clean up the data some more to dig into the different states and provinces.
• Analyze candy ratings across ages per country.
• Remove candy rating columns that weren't actual candy.
• Analyze ages per country.

References:

• University of British Columbia Candy Hierarchy Survey: https://www.scq.ubc.ca/so-much-candy-data-seriously/
• Pandas user guide: https://pandas.pydata.org/docs/user_guide/index.html
• Matplotlib user guide: https://matplotlib.org/3.3.1/users/index.html
• Seaborn user guide & tutorial: https://seaborn.pydata.org/tutorial.html
• GeekforGeeks: https://www.geeksforgeeks.org

My Jupyter notebook and the CSV file I used are saved to my GitHub for viewing.