Dictionaries, files, and looping

Dictionaries and files


resources	The python foundation maintains the most comprehensive and authoritative documentation on all built-in aspects of the python language. Start here: Map-based data structures (official documentation) The String built-in type (official documentation)

Warm-up 1: Looping to a file

Study the following file contents. Generate a program in Python to reproduce this structure. NOTE: You'll need to use two for() loops, nested. You can try seeing if you can print the pattern out first to the console, then replace the calls to print() with calles to yourfile.write().

Warm-up 2: String formatting program specification

Download the following text file of names: file link
Your desired program output is a formal greeting for individuals whose names exist in a text file in the order: first last. The output should look like this:

The process of generating these greetings should take place in two methods: one should read in a file given a file name (assumed to be in the working directory) and extract the name in a single line. The second method should receive a line of text (i.e. a name in the form first last) and generate the greeting. Remember, you'll need to invert the string since we're using string formatting, not string concatenation.

Group code along with CSV files

What can we infer about the state of criminal justice in Allegheny County from their publicly released jail census?

Download this CSV of jail census data by right-clicking the link and selecting something like "save link as..."
Visit the WPRDC dataset page

Jail population statistics calculation example: recreate this program without using the CSV module at all! You'll need to use the split() function to break down lines of text from the file, and you'll want to use zip() to combine each row after the header with the values in the header row.

JSON encoding

Guiding questions

What are the most widely used protocols for exchanging data between systems?

Learning Objectives

Parse JSON files into Python objects and manipulate those objects

Create JSON files from Python objects and transfer that data to other python systems

Resources

Lesson sequence

Course oranization system: gitHub and upload index, folders, attendance
Review work from last week: programming CSV parsers without CSV module
JSON, XML, and serial binary format notes
JSON parsing simple examples: opening and printing the Capital projects in PGH
Mini task 1: Use a for loop to list all project info in a neatly formed set, like shown in the screen shot below:
Mini task 2: Write a method called logMalformedProject that is called each time a project is visited by the main loop that does not contain a value for the key 'area'. This method should write the project id to a single line in a log file with an appropriate name.
Mini task 3: Create a method that assembles a list of unique values for the project area. If you are feeling ambitious, do the same for 'status' and 'asset type' since this will come in handy during the search specification.

Screen shot of capital project print formatting

JSON Search Criteria project specs

Write python code that conforms to the following diagram & specs. Note that this project was originally intended to require all students to process the same data set: the PGH capital projects. For FA21, you are using your own data set and adapting the specs. If you want to line up to the specs, use the PGH projects data set on the WPRDC.

Diagram


purpose	Implement search criteria defined in the JSON format for searching for records in your chosen CSV dataset and then writing the selected records into an output file also in JSON Unified JSON-encoded search criteria: For example, WPRDC has a capital projects dataset. We might want to only select a subset of these records. Which subset we select needs to be encoded in a nice, python-friendly format, namely JSON. So we could allow the searcher to say: "I only want records in 2019, that started in January and that are in the area of 'infrastructure'" {"fiscal_year": [-1], "start_date": [""], "area": [""], "asset_type": [""], "planning_status": [""]} Sample Search Notes: For dates: We will throw out malformed dates that are not YYYY-MM-DD A blank value in any specified query for a column/field will disqualify that record from inclusion in the results Empty string: do not limit results by this criteria at all Note: the "planning_status" key in the search JSON corresponds to the field named "status" in the csv
program requirement 1: searching	Write code that can read in a search criterion JSON file of your specification (meaning, you determine what the keys/values should be that will generate selected records). You'll need to be prepared to share this specification with others in the class Allow the user to specify search criteria for the records in your data set. IN the capital projects example, this included project fiscal year, start date, area, asset_type, and planning status

purpose

Implement search criteria defined in the JSON format for searching for records in your chosen CSV dataset and then writing the selected records into an output file also in JSON

Unified JSON-encoded search criteria:

For example, WPRDC has a capital projects dataset. We might want to only select a subset of these records. Which subset we select needs to be encoded in a nice, python-friendly format, namely JSON. So we could allow the searcher to say: "I only want records in 2019, that started in January and that are in the area of 'infrastructure'"

{"fiscal_year": [-1], "start_date": [""], "area": [""], "asset_type": [""], "planning_status": [""]}

Sample Search Notes:

For dates: We will throw out malformed dates that are not YYYY-MM-DD
A blank value in any specified query for a column/field will disqualify that record from inclusion in the results
Empty string: do not limit results by this criteria at all
Note: the "planning_status" key in the search JSON corresponds to the field named "status" in the csv

program requirement 1: searching

Write code that can read in a search criterion JSON file of your specification (meaning, you determine what the keys/values should be that will generate selected records). You'll need to be prepared to share this specification with others in the class

Allow the user to specify search criteria for the records in your data set. IN the capital projects example, this included project fiscal year, start date, area, asset_type, and planning status

Extension exercises

We can turn python objects into JSON files easily that can be digested by servers all over the world. We can also turn python objects into files that can only be eaten by other python interpreters. The library for this is called pickle. Try serializing (turning a python object into data inside a file, instead of in RAM) your project objects and resurrecting them using the pickle library.

311 Data Parsing challenge:

Visit the WPRDC 311 data home page at the WPRDC. Study the data dictionary. Learn about the 311 program if you don't about their system.

Extract a research question about the data that can be answered by processing the 300,000+ entries in their central data. Examples include:

Budgets cuts are coming! Which public works district seems to have the lightest load of 311 requests? The Heaviest? How would you redirect resources between districts?
Which submission mechanism is used most for which types of complaints?
Which neighborhood receives the most requests? Can we make any inferences between number of requests and the quality of life in that neighborhood?

Write a program in python to answer these questions by ingesting the entire set of 311 data posted to the WPRDC. HINT: Start with a small subset of the data, like this random extraction of a few dozen rows.

Come prepared to share your results!