jenn-learning/02-sql/SQL Wizard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76cc981d",
   "metadata": {},
   "source": [
    "# SQL Wizard\n",
    "\n",
    "This notebook is designed to teach you the language of SQL. SQL stands for Structured Query Language. It is a language designed to query databases. These queries allow you to pull data from the database, update rows in the database, and even delete data permanently. There are many different SQL databases, each of which can all have different plugins to enhance their capabilities even further. Remember that apartment searching tool I made for you? That's all using SQL :) It just also used a plugin meant for operating on geographic locations.\n",
    "\n",
    "The most common SQL database is SQLite. It is a lightweight library which any program can embed within itself to give it access to a SQL database. Most other databases are separate servers which must be run as an entirely separate software hosted on a sever somewhere. Because of this SQLite is very easy to get started with.\n",
    "\n",
    "Run the following cell to install SQL capabilities into Jupyter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ae40bab8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mDeploy AI and data apps for free on Ploomber Cloud! Learn more: https://docs.cloud.ploomber.io/en/latest/quickstart/signup.html\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%load_ext sql"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18ac848a",
   "metadata": {},
   "source": [
    "### Getting started with SQL\n",
    "\n",
    "Before we can get started with SQLite, let's first open a database! This is something you only need to do once in Jupyter since it will keep the database open for you. Run the following cell to open the database called \"properties.db\", which as you'll find out contains information about every Railey property at Deep Creek Lake.\n",
    "\n",
    "SQL databases all generally follow this model: Every database file can contain multiple \"tables\", and each table will contain rows and columns. If you know Excel this should be very familiar. Each Excel file can contain multiple sheet tabs at the bottom which allow you to store multiple spreadsheets in the same file. Those tabs at the bottom are equilavent to SQL tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "45e2c86e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span style=\"None\">Connecting to &#x27;sqlite:///properties.db&#x27;</span>"
      ],
      "text/plain": [
       "Connecting to 'sqlite:///properties.db'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%sql sqlite:///properties.db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ce9513-d9f9-4a1a-9d8d-8001ff98d285",
   "metadata": {},
   "source": [
    "### Your first query\n",
    "\n",
    "SQL queries are made of individual statements. These statements start with a word which specifies what you'd like to do, such as `SELECT` to retrieve data, `UPDATE` to update existing rows, and `DELETE` to delete records from the database. Let's practice a SELECT statement.\n",
    "\n",
    "The syntax of `SELECT` is as follows: `SELECT <the names of columns I would like returned separated by commas> FROM <table name>`. SQL also has a shorthand for selecting all columns: `SELECT * FROM <table name>`\n",
    "\n",
    "One last thing before we dive in! To make a Jupyter notebook cell run SQL, you must start the cell with `%%sql`. Without this, you'll likely get some errors since Jupyter will try to run it as SQL code. I will include this for you to make your life easier but you should know why it's there :)\n",
    "\n",
    "Additionally some queries will return a lot of data, so you may want to click on the area to the left of the returned data to shrink it "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fda3705-2abe-497d-9fd4-389d7f44b428",
   "metadata": {},
   "source": [
    "### Exercise 1: Getting started with SELECT\n",
    "\n",
    "As a first exercise, try selecting all columns from the table all_properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf28b6f1-c215-405c-afbf-c65cac1c2b7c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "# Add your select statement here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f2fc505-8403-401e-b409-c9dacfc40911",
   "metadata": {},
   "source": [
    "You'll notice that for your display convenience only ten results are displayed to you. Still, this query does select every row, but it would flood the notebook with results if it return all 500+ so the software doesn't allow that by default."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b36ab734-88f9-4db5-8a92-2d6d990362c9",
   "metadata": {},
   "source": [
    "## Basic SQL functions\n",
    "\n",
    "SQL comes with some functions which allow you to calculate various measures. The three we will look at are `COUNT`, `SUM`, and `AVG`.\n",
    "\n",
    "`COUNT` will return "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ad82b7",
   "metadata": {},
   "source": [
    "## Indexing review\n",
    "\n",
    "Great job with that exercise! Indexing is a valuable tool when working with sequences and we'll be relying on it heavily in the rest of the module. \n",
    "\n",
    "Let's learn a little bit more about indexing. The first problem we'll look at is how to get the last character in a string programatically. Above since you know the value ahead of time you can simply count and hard code the index of the last character. But what if you're not working on data which a known size? Here's a small demo - enter your name in the box below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "107d44de-1303-47e7-aa8b-170803de5812",
   "metadata": {},
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      " Christopher\n"
     ]
    }
   ],
   "source": [
    "student_name = input()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2b9c286e-0492-430e-914a-0125e975592c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "length = len(student_name)\n",
    "print(length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc123a72-0270-4371-b46e-34928fed1107",
   "metadata": {},
   "source": [
    "If you want to print the last character of a string you don't know the length of ahead of time you can use the length to index the last character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "02d46132-d407-47c5-ac5f-f81a5154d8b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r\n"
     ]
    }
   ],
   "source": [
    "last_index = length - 1\n",
    "print(student_name[last_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa2d7d9c-5c0c-460e-b692-cc7d15d31ebc",
   "metadata": {},
   "source": [
    "### Exercise 2: Why do we use `length - 1` for the index of the last character instead of `length`?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016ece47-878f-409d-9100-356634677c2a",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "812f8a3d-4df6-439f-b6c2-252bcdfb308e",
   "metadata": {},
   "source": [
    "### One More Indexing Trick\n",
    "\n",
    "This code is powerful because it works for strings of any length, not just 6-letter strings. Python also has a more idomatic method of indexing the last character in a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "edd384f0-af77-41ad-a09e-5df0024041bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "r\n"
     ]
    }
   ],
   "source": [
    "print(student_name[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ba81bcf-316a-47e0-addd-5c28c65bdfe7",
   "metadata": {},
   "source": [
    "Using negative indices starts from the end of a sequence and moves backwards, so `-2` is the second to last character and so on"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11957a5d-84fb-4f00-8c49-19f32567943c",
   "metadata": {},
   "source": [
    "## Iterating through sequences\n",
    "\n",
    "A very important part of working with data is iteration, or going through the items in a list one by one until you find one that you need or to do some analysis of each one. Let's examine a wordle letter by letter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f0de4d19-d951-42e7-9fb1-0d833aa5cfbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c\n",
      "a\n",
      "r\n",
      "r\n",
      "y\n"
     ]
    }
   ],
   "source": [
    "wordle = \"carry\"\n",
    "\n",
    "for letter in wordle:\n",
    "    print(letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e86f7ae2-eacf-4418-ad2a-9918488b10d0",
   "metadata": {},
   "source": [
    "The syntax for a `for` loop is:\n",
    "\n",
    "```\n",
    "for NEW_VARIABLE in SEQUENCE:\n",
    "    # Code which gets called once for each item in SEQUENCE with NEW_VARIBLE being updated to \n",
    "    # the next item in the list after the block inside the for loop is run\n",
    "```\n",
    "\n",
    "`for` loops are great for whenever you want to run a bit of code for each element in a sequence. Lets do somehting more interesting interesting with the block of code!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c8c761a-1f40-4bda-8513-7f814cd7eac2",
   "metadata": {},
   "source": [
    "### Exercise 1: Print a message each time you see an R in the wordle\n",
    "\n",
    "Hint: use an if statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dd2ca3a4-7477-46e2-ad02-3a2ed8319081",
   "metadata": {},
   "outputs": [],
   "source": [
    "for letter in wordle:\n",
    "    # Your code goes here\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c825e624-b6f9-4a34-b3e9-2dacfd921ac9",
   "metadata": {},
   "source": [
    "### Exercise 2: Count how many R's you see in the wordle and print the result after the for loop\n",
    "\n",
    "Hint: This builds upon your previous exercise. Don't be afraid to create a new variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c92147e9-0d55-43c0-a006-931fdafc70b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "for letter in wordle:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7736c145-d4e1-4c8d-be6d-b3ba6ef10250",
   "metadata": {},
   "source": [
    "### Advanced looping\n",
    "\n",
    "Brief reprieve: remember that loops can be broken out of with the `break` keyword. This is useful if you would just like to act on the first instance of something you find, you can just break after you act on it. This is useful for answering questions like \"Are there any R's in this word\". Since you only need to find one to answer that question, continuing to go through the rest of the word after you found on is redundant work and will save you time to stop as soon as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a123c63-f43c-4f4f-bafd-38342b09a55b",
   "metadata": {},
   "source": [
    "### Exercise 3: Print only one message for the first R in the wordle\n",
    "\n",
    "Hint: Copy your code from the first exercise and figure out how to modify it to only print once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3bebbabd-1d4d-4425-aed0-7be1c71187b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "for letter in wordle:\n",
    "    # Your code goes here\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd77f33-3563-452b-9988-2682bcf37c61",
   "metadata": {},
   "source": [
    "Did you know that if you just want to find if an element is in a list python has an easy shortcut for that?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7c51731e-567d-44e7-bd80-b40ec860f3bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have an R!\n"
     ]
    }
   ],
   "source": [
    "if 'r' in wordle:\n",
    "    print(\"We have an R!\")\n",
    "else:\n",
    "    print(\"No R in the wordle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3681535e-8317-4cef-96de-c01517ea07c2",
   "metadata": {},
   "source": [
    "Neat right? Feel free to change the code above and play around with it. The in operator with if statements will be very useful later in this notebook so keep it in mind"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a8e8603-1856-4d94-86f5-c2477cb5aedd",
   "metadata": {},
   "source": [
    "## List of lists\n",
    "\n",
    "A lot of data is multidimensional. If we have a list of words, to python that looks like a bunch of sequences inside one large sequence. We'll go through some simple examples to build some skills before we work with the full wordle dictionary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b0022962-fd8f-4d92-ab23-f0f0289d03d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = ['apple', 'butt', 'carp', 'dick']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "021c377d-ba82-4363-a1e8-934d44dbea63",
   "metadata": {},
   "source": [
    "If loop through dictionary with a for loop, each word will be accessed one after another.\n",
    "\n",
    "### Exercise 1: Print each word in the dictionary\n",
    "\n",
    "Hint: use a for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6bf8411e-bbaa-40b2-ab8b-4f7a1bc3f6c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879cb3c6-ac0e-41a7-8865-23bb861dcb89",
   "metadata": {},
   "source": [
    "But now, how are you supposed to access the letters in each word now, if you wanted to process each letter, not each word? Easy! Use a for loop inside of your first for loop :)\n",
    "\n",
    "### Exercise 2: Print each letter in the dictionary (in order)\n",
    "\n",
    "Hint: your code from the previous answer should be very useful here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6288c1ef-e057-4b6c-b4ef-e0459fc55dc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c064d435-2717-4296-98bf-802be9162cbe",
   "metadata": {},
   "source": [
    "### Multidimensional indexing\n",
    "\n",
    "Congratulations! You just did multidimensional indexing! What now? Multidimensional indexing is just a fancy way of saying we'll need to index a sequence which contains sequences multiple times to get one single item from our dictionary.\n",
    "\n",
    "Let's take a look at example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "eee51a96-90b3-44fa-b822-4a451f6a66e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "apple\n",
      "a\n"
     ]
    }
   ],
   "source": [
    "first_word = dictionary[0]\n",
    "first_letter_of_first_word = first_word[0]\n",
    "\n",
    "print(first_word)\n",
    "print(first_letter_of_first_word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202d4782-3fa6-4daa-81a5-ac82d2648773",
   "metadata": {},
   "source": [
    "Try modifying the example above to get the second letter of the first word, the second letter of the second word, or even the last letter of the last word. (Bonus points if you remember the trick from before)\n",
    "\n",
    "Here's another trick: indexing is an expression which returns a value. In the example above, we store that value in a temporary variable called `first_word`. We don't need to do that though, we can actually combine the indexing for a particular letter into one line without a temporary variable. Try playing around with the indexes below to get a feel for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bf1a7976-12bf-47f1-a9d1-28ba4da14fec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\n"
     ]
    }
   ],
   "source": [
    "a_cool_letter = dictionary[0][0]\n",
    "print(a_cool_letter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "269e56a4-ee9b-4875-9f27-19e5b1231bc8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\n"
     ]
    }
   ],
   "source": [
    "# Technically we don't even need the letter variable either. As the programmer its up to you to decide how explicit\n",
    "# you want to write your code. There is no solution or \"right\" amount of explicitness\n",
    "print(dictionary[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7d9799-a9e4-47d7-84ab-c70ad35126d6",
   "metadata": {},
   "source": [
    "## Indexing with numbers\n",
    "\n",
    "Instead of using a for loop to automatically go through each element in a sequence, occasionally it's useful to use the for loop to produce indices instead values. That's a lot of words which probably invokes a why but let's jump into an example to see why:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d3f9dceb-09ac-4bac-a127-4a2b1dd84b23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c\n",
      "a\n",
      "r\n",
      "r\n",
      "y\n"
     ]
    }
   ],
   "source": [
    "for index in range(5):\n",
    "    print(wordle[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbc79aec-123b-4fa5-a675-d317f36d2611",
   "metadata": {},
   "source": [
    "Well that didn't explain anything but trust me it will in due time. Really this probably just looks like a complicated and annoying way to print each letter. Before we get into why this is useful I'd like to highlight that index is a variable that gets assigned the values 0-4 (If you like an illustration, feel free to add a `print(index)`). The indexing operator, `[]`, can take any expression, not just integer literals. These facts are why this works.\n",
    "\n",
    "Now to answer the question I've been avoiding. Why is this ever useful? It's a niche thing but it's useful if you ever want to go through multiple sequences in lock step at the same time. Modify the example above to print each guessed letter alongside the actual letter in that position in the wordle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "25e12f2b-89b1-45b9-b158-d6a627e691ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c\n",
      "a\n",
      "r\n",
      "r\n",
      "y\n"
     ]
    }
   ],
   "source": [
    "# Here is a wordle guess a user input\n",
    "wordle_guess = \"carts\"\n",
    "\n",
    "for index in range(5):\n",
    "    print(wordle[index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98ca837-ee33-4a1d-9e96-918a894f3ba5",
   "metadata": {},
   "source": [
    "### Exercise 1: Print how many letter are in the correct position\n",
    "\n",
    "Building on top of your previous solution, if you have the guess's letter at position x and the wordle's letter at position x, now you can compare them and count how many are correct. This builds off of a lot of previous examples so don't be afraid to look back"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0129cef2-97c7-4fb5-8b98-27663625d3d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd04271b-c0ee-4574-9909-e72d526c8c3e",
   "metadata": {},
   "source": [
    "## Validating data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ac64228-7bbb-4ec0-b0d5-a4032d2c681e",
   "metadata": {},
   "source": [
    "Imagine you didn't get the wordle dictionary from a reputible site and instead got the wordle dictionary with a bunch of extra crap in it. Data cleaning and validation is a huge part of data analysis. Answering questions like \"Is this a valid word\", \"Is it the right size\". Or for other datasets, \"Does every person have an email address or is anyone's email address empty\".\n",
    "\n",
    "Sometimes you just need answers: \"Is the data ok\". Other times you'll need to decide how you want to fix problems in your data.\n",
    "\n",
    "Let's start with a simple example. Your sketchy wordle dictionary has words that are too short and too long in it. A very simple solution to this problem is to go through every entry in the sketchy dictionary and add it to a \"known good wordles\" list if it's the appropriate size. Let's try this below:\n",
    "\n",
    "Hint: `.append()` is a method to add values to a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "45605090-89dc-4661-b617-decb32db9172",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "sketchy_dictionary = [\"carts\", \"carry\", \"carriage\", \"ropes\", \"tights\", \"dude\", \"horse\"]\n",
    "validated = []\n",
    "\n",
    "# Your code goes here\n",
    "\n",
    "        \n",
    "print(validated)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34f6e526-1bd0-4bf5-ab3d-67646f7c86aa",
   "metadata": {},
   "source": [
    "Congrats! You successfully cleaned and validated that bad dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f46c02-9ba5-489e-9abc-9d6f5ab58749",
   "metadata": {},
   "source": [
    "## Analyzing some wordles\n",
    "\n",
    "Finally! The main event :) First thing's first, let's talk about CSV (comman seperated values) files. Our wordle datasets are stored in two different CSV files. Please take the opportunity to go look at them by clicking the folder icon on the sidebar and double clicking either of the two csv files to explore them. When you're done, click on Wordle Master tab to come back here.\n",
    "\n",
    "CSV files are a very simple \"spreadsheet\" format. They give you rows and columns. These files have one column called \"word\". If you were to open the file in a text editor, it would look like:\n",
    "\n",
    "```\n",
    "word,\n",
    "aback,\n",
    "abase,\n",
    "abate,\n",
    "abbey,\n",
    "abbot,\n",
    "abhor,\n",
    "abide,\n",
    "abled,\n",
    "abode,\n",
    "...\n",
    "```\n",
    "\n",
    "Each column in a CSV file is separated with a comma (Where it gets it's name from) and each row of the spreadsheet is separated with a newline character. It's a very simple text format but it's very easy to work with in any programming language and is therefore prety ubiquitous. Sometimes, but not all the times, the first row of a CSV file is a header row which provides a label for each column. If you are to process a CSV file it's important to know whether the first line is a header or part of the data so you can process it accordingly.\n",
    "\n",
    "Below is some python code which reads these csv files into multiple arrays, we can go through these after you take a look at them. See if you can take a guess as to what's happening:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "89f2ada2-a7bf-4423-a279-32a07858a387",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_solutions = []\n",
    "\n",
    "import csv\n",
    "\n",
    "with open(\"valid_solutions.csv\", \"r\") as csvfile:\n",
    "    csvreader = csv.reader(csvfile)\n",
    "\n",
    "    # This skips the first header row of the CSV file.\n",
    "    next(csvreader)\n",
    "    \n",
    "    for row in csvreader:\n",
    "        valid_solutions.append(row[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "270f638c-6aa0-4909-8e50-9fd8beda06d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_guesses = []\n",
    "\n",
    "import csv\n",
    "\n",
    "with open(\"valid_guesses.csv\", \"r\") as csvfile:\n",
    "    csvreader = csv.reader(csvfile)\n",
    "\n",
    "    # This skips the first header row of the CSV file.\n",
    "    next(csvreader)\n",
    "    \n",
    "    for row in csvreader:\n",
    "        valid_guesses.append(row[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58e93d6c-4468-4cef-978a-485f1c812482",
   "metadata": {},
   "source": [
    "Now the variables `valid_solutions` and `valid_guesses` are usable in other cells below! The block of code is pretty straightforward but introduces a lot of new things we won't dive into too much in this module. All this code does is use the built in python csv library to read a CSV file (opened with the `open` function). We use that reader to read each entry from the first column of each row into a list called `valid_guesses` and `valid_solutions`.\n",
    "\n",
    "Wordle has two word lists, one for all possible valid guesses (every 5 letter word in the dictionary) and a hand curated one from which the solution is chosen (so the player doesn't feel gypped because of a esoteric word they never heard of before. Since our word lists are in python lists now, lets use our data processing skills to see how long is each dictionary, and then after you figure that out, determine how many more words there are in the `guesses` list than there are in the `solutions` list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "9082a2e7-5737-49f0-8fc2-b175f11cf3b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29212900-28d3-43c5-86cf-fa43a3acd92d",
   "metadata": {},
   "source": [
    "Look at you! Answering interesting questions about the wordle dataset and we've barely got started! Let's try something harder!\n",
    "\n",
    "### Exercise 1: Count how many solutions start with the letter A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "773df9e2-523d-4f6b-9702-54c83a104a73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf344b4-8d73-43d0-9833-474662e5eef5",
   "metadata": {},
   "source": [
    "### Exercise 2: Count how many solutions contain an A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "df333e84-0fdc-40dd-8976-7c15e82fb913",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2db3b332-9e6a-47fe-a444-56f2f054f837",
   "metadata": {},
   "source": [
    "### Exercise 3: Pick a wordle from the solutions list at random\n",
    "\n",
    "Hint: You've done this before but this is a good time to break out google and see if you can figure it out again if you forgot ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c544cc7d-89ea-4df0-996f-d18bf76e6b53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34b52e3f-aca0-4efe-8713-e3d7000963b2",
   "metadata": {},
   "source": [
    "### Exercise 4: Determine if a given word is in the solution list\n",
    "\n",
    "Hint: There's an operator we've used before in this module which let's you test to see if a value is contained in a given sequence. If you can't find it or remember it there's always the trusty for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4414c453-d867-40f0-a95d-8a7ced923e45",
   "metadata": {},
   "outputs": [],
   "source": [
    "is_valid_solution1 = \"grass\"\n",
    "is_valid_solution2 = \"aiyee\"\n",
    "\n",
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53344a6-e612-451e-9757-4d76f4f56d54",
   "metadata": {},
   "source": [
    "### Exercise 5: Print all the words that start with \"sh\"\n",
    "\n",
    "This one is the first exercise which has a bit of practical value for you. If you have a word that starts with a green 'sh', this will let you see what they are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "0b2a35f6-2e52-4308-af55-cc10b03a5ad9",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee81005-a11f-4857-8707-a27eaf14cd4e",
   "metadata": {},
   "source": [
    "### Aside about comparing more than just one letter\n",
    "\n",
    "Slicing a list is something which goes handing in hand with indexing but we have not touched on it yet. If you remember, slicing is a way to return a subset of a list. This can make a problem like the one above easier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1b661897-031f-4781-9cdf-4b8942f2279a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s\n",
      "sh\n",
      "sh\n",
      "\n",
      "ell\n",
      "ell\n"
     ]
    }
   ],
   "source": [
    "to_slice = \"shell\"\n",
    "\n",
    "print(to_slice[0:1])\n",
    "print(to_slice[0:2])\n",
    "print(to_slice[:2])\n",
    "print()\n",
    "print(to_slice[2:5])\n",
    "print(to_slice[2:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7876848f-61e6-41bb-957c-ac3e063e92e4",
   "metadata": {},
   "source": [
    "Play around with the indicies above and try and get a feel for it! What happens if the starting index is after the ending index? What happens if the starting index is negative and you don't provide an ending index? Can you use two negative indexes? Can you use negative indexes to the the last two characters of the word?\n",
    "\n",
    "As a refresher, the slicing syntax appears inside the indexing operator and instructs python to return a subset of the list where the number before the colon is the starting index and the number after the colon is where python goes up to but doesn't include. If the starting number is left out, that means start from the beginning and go to the ending index. If the ending index is left out that means start at the starting index and go to the end. If they are both left out, it returns a copy of the list from the beginning to the end. Not the most useful thing but possible.\n",
    "\n",
    "The reason I bring all of this up is because of the previous exercise. An easy way to get the first two letters is with slicing. Take a look at the solution to Exercise 5 below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "183fdc2a-1110-486f-b134-6b9d6cd7ddbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shack\n",
      "shade\n",
      "shady\n",
      "shaft\n",
      "shake\n",
      "shaky\n",
      "shale\n",
      "shall\n",
      "shalt\n",
      "shame\n",
      "shank\n",
      "shape\n",
      "shard\n",
      "share\n",
      "shark\n",
      "sharp\n",
      "shave\n",
      "shawl\n"
     ]
    }
   ],
   "source": [
    "for solution in valid_solutions:\n",
    "    if solution[:3] == \"sha\":\n",
    "        print(solution)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5685b7ba-eca7-427a-b036-564c48c49fa2",
   "metadata": {},
   "source": [
    "The alternative would be writing a much larger if statement with and and explicit indexing. This is a fairly compact and readable solution, so long as the reader understands slicing syntax."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": ".venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "toc-autonumbering": true,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}