header image
Check this page for updates on upcoming classes, our learning goals, and the lesson modules we'll use to get there.

Session Calendar and session guides (Fall 2019)

CIT-129: Python 2

Weekly guides

Guiding questions, objectives, and lesson activities are available for each week of class. Click "toggle full session guide" to view the full guide.

Week 2: Loops and Dictionaries

Looping and methods

live_helpGuiding questions

  • What makes Python looping and iterating more convenient than in Java?

check_circleLearning Objectives

  • Use a for-loop to iterate over various data structures
  • Create dictionary-formatted strings

arrow_upwardback to schedule


Wednesday, 25-SEP-2019

File I/O

live_helpGuiding questions

  • How does Python interact with the operating system to navigate the file trees?

check_circleLearning Objectives

  • Harness python's file interace to read, write, and process data in files

bookResources

listLesson sequence

  1. Build a shared understanding of our file I/O tools on the white board using our linked resources

Exercise 1: Looping to a file

Study the following file contents. Generate a program in Python to reproduce this structure. NOTE: You'll need to use two for() loops, nested. You can try seeing if you can print the pattern out first to the console, then replace the calls to print() with calles to yourfile.write().

file example

Exercise 2: String formatting program specification

  1. Download the following text file of names: file link
  2. Your desired program output is a formal greeting for individuals whose names exist in a text file in the order: first last. The output should look like this:
  3. output
  4. The process of generating these greetings should take place in two methods: one should read in a file given a file name (assumed to be in the working directory) and extract the name in a single line. The second method should receive a line of text (i.e. a name in the form first last) and generate the greeting. Remember, you'll need to invert the string since we're using string formatting, not string concatenation.

Data-crunching with CSV files

What can we infer about the state of criminal justice in Allegheny County from their publicly released jail census?

  1. Download this CSV of jail census data by right-clicking the link and selecting something like "save link as..."
  2. Visit the WPRDC dataset page

Jail population statistics calculation example: recreate this program without using the CSV module at all! You'll need to use the split() function to break down lines of text from the file, and you'll want to use zip() to combine each row after the header with the values in the header row.

playlist_add_checkMid-week ToDOs

First things first: Please create a repository for your python code on github.com using this guide. (The guide is intended for Java programming students, but github is an equal opportunity code repo: Java and python are exactly the same.)


live_helpGuiding questions

check_circleLearning Objectives

bookResources

listLesson sequence

  1. Course oranization system: gitHub and upload index, folders, attendance
  2. Review work from last week: programming CSV parsers without CSV module
  3. JSON, XML, and serial binary format notes
  4. JSON parsing simple examples: opening and printing the Capital projects in PGH
  5. Mini task 1: Use a for loop to list all project info in a neatly formed set, like shown in the screen shot below:
  6. Mini task 2: Write a method called logMalformedProject that is called each time a project is visited by the main loop that does not contain a value for the key 'area'. This method should write the project id to a single line in a log file with an appropriate name.
  7. Mini task 3: Create a method that assembles a list of unique values for the project area. If you are feeling ambitious, do the same for 'status' and 'asset type' since this will come in handy during the search specification.

Screen shot of capital project print formatting

capital project analysis output

playlist_add_checkMid-week ToDOs

Start thinking about an API and an API key

An API key is a unique identifier that a coder provides to a remote server which unlocks the release of data to the requetsing client. Each server with an Application Programming Interface configures its data restrictions differently, but many require an API key to be sent with each request.

Visit programmableweb.com for a directory listing of thousands of servers with some form of API. Browse some that look interesting, click into that data source's profile, and make sure that the data is free and that the server returns JSON-encoded data.

arrow_upwardback to schedule


Wednesday, 2-OCT-2019

Keep developing our class JSON project

Write python code that conforms to the following specs:

purpose

Implement search criteria defined in the JSON format for searching for capital projects in PGH dataset, outputting resulting projects into a file in JSON format

Unified JSON-encoded search criteria:

{"fiscal_year": [-1], "start_date": [""], "area": [""], "asset_type": [""], "planning_status": [""]}

Search Notes:

  • For dates: We will throw out malformed dates that are not YYYY-MM-DD(This requirement was removed due to lack of connecetion to the primary data set)
  • A blank value in any specified query for a column/field will disqualify that record from inclusion in the results
  • Empty string: do not limit results by this criteria at all
  • Note: the "planning_status" key in the search JSON corresponds to the field named "status" in the csv

program requirement 1: searching

Write code that can read in a search criterion JSON file of your specification. You'll need to be prepared to share this specification with others in the class

Allow the user to specify search criteria for project fiscal year, start date, area, asset_type, and planning status

program requirement 2: management costs

Write a method that will calculate total project management costs for all the capital projects in Pittsburgh given a management cost scheme, encoded in JSON as specified by the class. Example: For all projects up to $10k, management costs are 8% of the total project budget. For projects between $10k and $100k, management costs drop to 5%, and over $100k, costs increase to 11%.

sendProducts

  1. Working code from mini-tasks 1,2, and 3
  2. A solid attempt at the full project specs in the green box
  3. A possible API source for the week after next

cakeExtension exercises

We can turn python objects into JSON files easily that can be digested by servers all over the world. We can also turn python objects into files that can only be eaten by other python interpreters. The library for this is called pickle. Try serializing (turning a python object into data inside a file, instead of in RAM) your project objects and resurrecting them using the pickle library.

311 Data Parsing challenge:

  1. Visit the WPRDC 311 data home page at the WPRDC. Study the data dictionary. Learn about the 311 program if you don't about their system.
  2. Extract a research question about the data that can be answered by processing the 300,000+ entries in their central data. Examples include:
    • Budgets cuts are coming! Which public works district seems to have the lightest load of 311 requests? The Heaviest? How would you redirect resources between districts?
    • Which submission mechanism is used most for which types of complaints?
    • Which neighborhood receives the most requests? Can we make any inferences between number of requests and the quality of life in that neighborhood?
  3. Write a program in python to answer these questions by ingesting the entire set of 311 data posted to the WPRDC. HINT: Start with a small subset of the data, like this random extraction of a few dozen rows.
  4. Come prepared to share your results!

arrow_upwardback to schedule


Wednesday, 9-OCT-2019

HTTP, Requests, and screen scraping

check_circleLearning Objectives

  • Explain the essential structure of HTML pages and how they store structured information. Write a basic HTML page about a life interest.
  • Employ the urllib and BeautifulSoup libraries to pull down an HTML page and find a particular element, and extract the data in that element.
  • Systematically store the result of a screen scraping endeavor into a text file that can be shared and used as inputs by other programs or tools.

languageResources

listLesson Sequence

  1. Structure of the WWW: Requests and responses in browsers
  2. Using request and response libraries in URL lib
  3. Structuring documents in trees! HTML basics
  4. Using Beautiful soup to parse a simple HTML File
  5. Run through screen scraping example on GoodReads.com. Learn how to use urllib and BeautifulSoup
  6. Work time on screen scraping project

cakeProducts to Produce

  • Code to the specification below. Then upload your a python files and any related documents to your GitHub account. I suggest creating a subdirectory in your git repository called "scraping" or something like that.

program objective

Create a program that uses the urllib and BeautifulSoup to grab HTML code from a public source, parses that source into meaningful bits of data, and spits those meaningufl bits of data into some form that could be transfered into anothre tool, such as a CSV for slurping up into a Database, a JSON file for use in the web, etc.

suggestions for good pages to parse

Choose a website whose page content is retrieved with some sort of query, such as a URL-encoded search query. This will allow your system to programatically tinker with the results you get back, and can be scaled to process lots more data than just a single, static page.

Pages with tables of data are great, since that allows us to loop over trees of page elements and procss their data one at a time

use methods!

When possible, please structure your code in discrete methods that accomplish a single task, returning useful values to the caller. This helps reduce code repetition, allows for modular re-use, and makes the code generally more readable than blobs of lines in a heap.


Special project in file/os tree visualization

live_helpGuiding questions

  • How can we help early computer learners visualize and interact with file trees?
  • What tools are embedded in the OS package that can help us with visualizing file trees

check_circleLearning Objectives

  • Harness critical methods of the os module

bookResources

listLesson sequence

  1. Discuss high level goals of the program and explore sample file trees
  2. Practice traversing the directory structure tree in Python
  3. Design specs for individual components of the project

playlist_add_checkMid-week ToDOs

We want to prepare for our API endeavor next week by doing the following steps:

Please create a program that calls an API of your choosing from the online index of available APIs called ProgrammableWeb.com. Use our sample code on GitHub and from last class to create a program that calls one of these APIs, retrieves a JSON file from the server, and parses that JSON file to produce some kind of output that is useful. Remember, this can be very simple: create an average of some quantitative measurement returned, etc.

Sample code for an API call to donorschoose.org.

arrow_upwardback to schedule


Wednesday, 16-OCT-19

Application Programming Interfaces

live_helpGuiding questions

  • How can we leverage the power of a remote database system programatically in python?

check_circleLearning Objectives

  • Use the requests library and supporting modules to request information from an API
  • Parse data returned from an API into useful products for our own purposes

bookResources

listLesson sequence

  1. Discuss course plan: project design for course
  2. API Fundamentals: Donors Choose API and sample tweaking according to the specification below
  3. API Design project and worktime

program objective

Use data from an API service to answer one or more compelling inquiry questions of your design

program requirement 1

Engineer a python script that uses the requests library to make calls to an API of your choosing. Assemble data that is returned from that API to answer the question or set of questions you designed for this project.

program requirement 2

Once you have data assembled that speaks to your research question, format the results of your API inquiry in a format that is understandable by a non-pythonista and can be posted on our course website and on the walls of our classroom 1136! Graphs, charts, figures, tables, images--all of these are welcome and invited.

playlist_add_checkMid-week ToDOs

Complete your API project and upload it to your github account.

arrow_upwardback to schedule


Wednesday, 23-OCT-2019

Review of API project; Tinkering with Pandas

For the first half of class, we'll review our progress with our APIs and begin our first look at numPy and Pandas

Peer teaching design

bookResources

listLesson sequence

  1. API output features: screen shots, comments, etc.
  2. Code comments and code portability
  3. Mini skill planning

playlist_add_checkPeer tutorial subjects

  1. 1: Using modules: (Lutz 2013, 5th ed: Part V chapters 22-24 are all on modules. We want to ingest the highest level concepts here with a short exercise and challenge.
  2. 2: Comprehensions and generators: List, dictionary, etc. (Lutz 2013, 5th ed: chapter 14 with emphasis on pages 441+ for comprehensions. Ch. 20 for comprehensions and generators)
  3. 3: Regexps: (Use python.org's regexp howto and the more detailed module documentation on re)
  4. 4: Object-oriented python: (Lutz 2013, 5th ed: The entire part VI is on classes and OOP--good luck!)
  5. 5: Exceptions: (Lutz 2013, 5th ed: Basics are in ch. 33 with the rest of part VII related to more advanced topics)
  6. 6: Functions and tools: global/nonlocal., lamda, yield, etc. (Lutz 2013, 5th ed: review ch. 16 on function basics; read chapter 19 on advanced functions. Steer clear of yield, since generators will use these.)

sendPeer tutorial specs

mini-project objective

Provide us pythons with 15-20 minute "crash courses" on topics that may not be worthy of an entire evening in python II

quick reference 1/2

Read the book and python.org documentation on your subject. Run some code. Distill down the example code provided in the references to the essential module and topics only. Also, boil down the reference content to diagrams, figures, or bulleted lists of core ideas or topics.

Assemble a 1/2 page (with the page in landscape orientation) quick reference for your topic that neatly arranges this information

prepare a sample activity

Drawing from examples of others, create a 10-15 minute coding exercise that you write entirely on your own using some sort of consolidating topic of your choosing (i.e. Eric using airline safety and metallurgy to discuss statistics). The example should include essential functionality of your topic and should be expandable.

Comment your example code and push it up to your GitHub account

Keep in mind that students in our course have a broad range of skills and the more you can provide an exercise with some more straightforward tasks and some other more complicated tasks, the more broadly engaging your code will be.

propose extension exercises

Create a short list of continuation exercises for somebody to guide their further explorations into your topic. Review one of Eric's extension exercise examples from Java.

Document these extensions either in a text file on GitHub or in comments in your sample code.

sendProducts

  1. Featured python code snippet and output banner
  2. Commented, packaged, and documented projects: 2x up to GitHub and link on this site

arrow_upwardback to schedule


The OS module for file system interaction & pandas practice

The OS module allows us to interact with our computer's underlying operating sytem methods, particularly those related to file system traversal and management. The activities in this section are intended to invite you to explore the key methods in the os module and build your understanding of the pandas library for data harnessing and processing.

Resources

python.org os module documentation

pandas documentation home at pydata.org

unzipping files in an archive

POSIX standard

CIT-115 student file tree cloud drive

Mini project: Navigating the treess

  1. Practice traversing this sample file tree: How many files are located in all these directories?
  2. Our goal is to create some basic metrics about the tree shape and size created by CCAC's CIT-115 students at West Hills and load this data into a Pandas dataframe object so we can determine average sizes, min, max, counts by file type, and more.
  3. Practice using the os.walk() method to extract information about the levels of files and directories within each tree. Choose a nice, small tree root to practice in, learning how to count directories, files, depth, and breadth of a given tree. Remember: depth starts counting at 1 with the first director inside the root.
  4. Create a nice, function-based program that ingests data about a whole chunk of CIT-115 trees in the archives linked here and writes this data to a Pandas data frame with columns for the following: (1)tree index, (2)tree's root name, (3)max depth, (4)max breadth, (5)total files, (6)total directories, (7)total size on disk. I suggest starting with a subset of these metrics and expand as you have the capability.
  5. Create a simple chart with pandas to visualize the results of the tree data: what can we say about the digital forest created by the CIT-115 students?

arrow_upwardback to schedule


Additional class specification (Not planned for use in Fall 2019

Developing python 2 entrance challenge

live_helpGuiding questions

  • Can we learn from/while "taking a test?"
  • What kinds of thinking lead to meaningful test questions?

bookResources

listLesson sequence

  1. Review sample python 1 questions from the resources above and those you find on your own
  2. Review the core content areas of our course: data structures, looping, and web/api interaction
  3. Study the specification below for creating a two-task instrument to gauge a student's readiness for entrance into our CCAC Python 2 course
  4. Jot down a list of what you think are the core skills students should have acquired in Python 1 to be prepared for python 2.
  5. With your list in mind, engineer the two-task instrument. For each task, create specs, sample output, success criteria, and a possible solution
  6. Upload your content to your git-hub account
  7. Create a printable PDF version and upload it to our shared one-drive space

listInstrument meta-specifications

These are the specifications for your python 2 readiness test specification

python 2 readiness assessment overview

Students engaged in programming stuff at CCAC exhibit a wide range of skills across languages and technical abilities. Some students with prior programming experience or a very tenacious work ethic may have skills sufficient to jump right into python 2. Design an assessment instrument that can be used to gauge a student's readiness for python 2

task specs

Using this very box as a model, engineer 2 program specifications:

  1. A more straightforward python 1 task, such as implementing nested if-logic or basic looping with simple break conditions
  2. A more challenging python 2 task, such as conducting basic file i/o and processing the results

Creating tasks that are mildly interesting will help students engaged

sample output

Include in your specification sample program output for each of your two programs which the student can view as they build their program

success criteria

Using Eric's Intro to Java assessment instrument and rubric as a guide, declare a standard for determining what kind of responses to your two program specs would constitute readiness for python 2.

This could take the form of a paragraph describing what kind of code structures should be present in the source code of each task and what they should do. Obviously, you'll need a criteria for each task

solution code

Code to your own specification and upload your source to your dedicated repository linked at technologyrediscovery.net. Don't include a link to your source in your PDF

sendProducts

  1. Relevant source code posted to your github code repo
  2. Your assessment instrument uploaded as a PDF to our shared drive

arrow_upwardback to schedule