Notes on: Python for Everybody

Links » Python Index

Getting Started

Reference Material

The course materials can be accessed on the course website. This is also where you find the textbook to go along with the course

Introduction

Why Program?

• become a creator of technology, don’t just be a consumer of it
• computers want to be helpful (What do you want to do next?)
• a programmer’s job is to intermediate between the hardware and the user

Hardware Overview

• the CPU is always asking “What next?”
• fetch-execute cycle (between CPU and main memory)
• main memory (deleted when computer is turned off) and secondary memory (remains)
• compiler and interpreter to the translation of the human-readable program code to machine code

Python as a Language

• invented by Guido van Rossum
• named after Monty Python (enjoyable but powerful)

Reserved Words

• you cannot use keywords as variable names

import keyword
print(keyword.kwlist)

['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break',
'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for',
'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or',
'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

• if it’s longer than three lines, make a script
• programs can be sequential, conditional (often nested) or repeated (often use iteration variables to make sure that the loop does not run infinitely)

The Building Blocks of a Program

The following are part of every programming language (even machine code):

• input: Data from outside; Read a file, sensor data, keyboard input
• output: The result of the computation displayed on a screen or stored in a file
• sequential execution: Perform statements one after another in the same order in which they are written in the script
• conditional execution: Execute or skip based on a condition
• repeated execution: Perform the same statements repeatedly, usually with some variation
• reuse: Write a set of instructions once and then reuse as needed throughout the program

Different Error Types

Syntax errors

These are the first errors you will make and the easiest to fix. A syntax error means that you have violated the “grammar” rules of Python. Python does its best to point right at the line and character where it noticed it was confused. The only tricky bit of syntax errors is that sometimes the mistake that needs fixing is actually earlier in the program than where Python noticed it was confused. So the line and character that Python indicates in a syntax error may just be a starting point for your investigation.

Logic errors

A logic error is when your program has good syntax but there is a mistake in the order of the statements or perhaps a mistake in how the statements relate to one another. A good example of a logic error might be, “take a drink from your water bottle, put it in your backpack, walk to the library, and then put the top back on the bottle.”

Semantic errors

A semantic error is when your description of the steps to take is syntactically perfect and in the right order, but there is simply a mistake in the program. The program is perfectly correct but it does not do what you intended for it to do. A simple example would be if you were giving a person directions to a restaurant and said, “…when you reach the intersection with the gas station, turn left and go one mile and the restaurant is a red building on your left.” Your friend is very late and calls you to tell you that they are on a farm and walking around behind a barn, with no sign of a restaurant. Then you say “did you turn left or right at the gas station?” and they say, “I followed your directions perfectly, I have them written down, it says turn left and go one mile at the gas station.” Then you say, “I am very sorry, because while my instructions were syntactically correct, they sadly contained a small but undetected semantic error.”.

Debugging

four basic strategies that complement each other (if one does not work, try the next):

1. Reading: Examine the code, read it back to yourself and check whether it was what you intended to say
2. Running: Experiment by running different versions of the program and try to display the intermediate steps. That sometimes requires some scaffolding
3. Ruminating: Think! What kind of error is it? What was the last thing you did before you encountered the error?
4. Retreating: At some point, if all the above don’t work, undo the most recent changes until you arrive at a program that you understand and that works as intended.

Variables, Expressions and Statements

Values and Types

print(type("I'm the value"))
print(type("2")
print(type("3.2")

<class 'str'> # This is the type
<class 'int'> # This is another type
<class 'float'> # This is another type

Variables

One of the most powerful features of a programming language is the ability to manipulate variables. A variable is a name that refers to a value. The relationship between variable and value is established through an assignment statement - must start with a letter or underscore (only use the underscore if you are writing library code for others though) - always choose mnemonic variable names

hours = 35.0 # this is an assignment statement
rate = 12.50
pay = hours * rate
print(pay)

• illegal variable names give a syntax error

Statements

A statement is just a unit of code that the Python interpreter can execute. Scripts are usually a sequence of statements.

Operators and Operands

Operators are defined as special symbols that stand in for computations such as addition, subtraction, multiplication and division. Operands are the values the operator is applied to.

20+32 # "20" and "32" are the operands in this case
hour-1
hour*60+minute
minute/60
5**2
(5+9)*(15-7)

Since Python 3.x, the result of a division (of two integers) is a value of the float type

result = 120/121
print(result)

0.9917355371900827

If you want a Python 2.x style result, i.e. truncated to the int, then you need to use //:

result = 120//121
print(result)

0

Expressions

An expression is a combination of values, variables and operators. But a value all by itself (or a variable - assuming it has a value assigned to it) are also valid expressions. Expressions are evaluated in interactive mode and the results are displayed. In a script, however, expressions by themselves do not produce output.

Order of Operations

The order of evaluation depends on the rules of precedence. Remember PEDMAS:

*P*arentheses *E*xponentiation *M*ultiplication *D*ivision *A*ddition *S*ubstraction

Modulus Operators

This operator works on values of the type int and yields the remainder when the first operand is divided by the second.

quotient = 7 // 3
print(quotient)

remainder = 7 % 3
print(remainder)

2
1

String Operations

The +-operator works with strings, it concatenates them, i.e. it joins them together.

part_one = "Hi, my name is "
part_two = "Linus"
print(part_one + part_two)
print(part_two*2)

Hi, my name is Linus
LinusLinus

Asking the User for Input

There is a built-in function called input which stops the program and waits for the user to type something. When the user presses Return, the program resumes and the function returns whatever was typed as a string. The \n is called a newline which is a special character that causes a line break (which is why, in the example below, the user input appears below the prompt)

prompt = "Is this love?\n"
input(prompt)

Is this love?
Yes!

A little program that prompts the user for a temperature in Celsius and outputs the same temperature in Fahrenheit:

prompt = "Input the degrees Celsius\n"
celsius = input(prompt)
fahrenheit = ( int(celsius) / (5/9) ) + 32
print(fahrenheit)

Conditional Execution

Boolean Expressions

Boolean expressions are expressions that are either True or False.

x = 5
y = 6

print(x == y)
print(type(x == y))

False
<class 'bool'>

There is also a new range of operators that produce boolean values when evaluated.

x != y   # x is not equal to y
x > y# x is greater than y
x < y# x is less than y
x >= y   # x is greater than or equal to y
x <= y   # x is less than or equal to y
x is y   # x is the same as y
x is not y   # x is not the same as y

Logical Operators

There are three: and (something is True only if both operands are True), or (True if either of the operands is True) and not (negation of the expression).

Any nonzero number is interpreted as True

print(17 and True)  #True
print(0 and True)   #0
print(17 or True)   #17
print(0 or True)#True
print(False and 17) #False
print(False and 0)  #False
print(False or 17)  #17
print(False or 0)   #0

Conditional Execution

We often need to check certain conditions, and then adapt our program to those conditions.

if x > 0 :
    print("x is positive")

Alternative Execution

A check of the condition leads down exactly one of either of two so-called branches

if x%2 == 0 :
    print("x is even")
else :
    print("x is odd")

Chained Conditionals

If I want to include more possible branches, I need the elif-statement. Each condition is checked after the last, if one of them is True, the branch executes and the statement ends. Even if more conditions are True, only the first true branch will execute.

if choice == 'a':
    print('Bad guess')
elif choice == 'b':
    print('Good guess')
elif choice == 'c':
    print('Close, but not correct')

Nested Conditionals

You can nest branches into one another as follows.

if x == y:
    print('x and y are equal')
else:
    if x < y:
            print('x is less than y')
    else:
        print('x is greater than y')

Catching Exceptions using Try and Except

try and except are Python’s built-in insurance policy against errors. Only if (any) error occurs in the try-block, Python jumps directly to the except-block. Handling possible errors through with a try-statement is called catching an error. It gives you the chance to fix the problem, try again or end the problem gracefully. See the following example for an illustration of the latter:

inp = input('Enter Fahrenheit Temperature:')
try:
    fahr = float(inp)
    cel = (fahr - 32.0) * 5.0 / 9.0
    print(cel)
except:
    print('Please enter a number')

Short-circuit Evaluation of Logical Expressions

Consider the following code:

# Example 1
x = 6
y = 2
print("Example 1: " + str(x >= 2 and (x/y) > 2))

#Example 2
x = 1
y = 0
print("Example 2: " + str(x >= 2 and (x/y) > 2))

#Example 3
x = 6
y = 0
print("Example 3: " + str(x >= 2 and (x/y) > 2))

Example 1: True
Example 2: False

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-41-7fc295d23b39> in <module>
    12 x = 6
    13 y = 0
---> 14 print("Example 3: " + str(x >= 2 and (x/y) > 2))

ZeroDivisionError: division by zero

We get a an error in the third but not in the second example because Python noticed that the overall expression in the second case cannot be anything but False after evaluating the first part, i.e. =x >= 2=. So, it short-circuited the rest of the evaluation to save its energy.

You can actually use this to guard parts of your evaluation just before the evaluation might cause an error.

x = 6
y = 0
print(str(x >= 2 and y != 0 and (x/y) > 2))

In this case, y ! 0= acts as a guard against evaluating (x/y) > 2 when y is equal to zero.

Functions

Function Calls

At its most basic, a function is a named sequence of statements performing a computation. After having specified the statements, you can call a (built-in) function as such:

print(type(32)) # both print() and type() are functions
max("Hello world") # "w" is the "largest" character
min("Hello world") # the space is the "smallest" character
len("Hello world") # gives the length of a string

<class 'int'>
w

11

Important Built-in Functions

Type conversion

int converts floating-point numbers and (the right kind of) strings to integers:

int('32')
int('Hello')  # this gives a ValueError
int(3.999999) # int() will not round but truncate
int(-2.3)

32
ValueError: invalid literal for int() with base 10: 'Hello'
3
2

float converts integers and strings to floating-point numbers:

float(32)
float("3.1415926")

32.0
3.1415926

str just converts everything to a string:

str(32)
str(3.1415926)

"32"
"3.1415926"

Math functions

Python ships with a math module that must be imported before it can be used:

import math
print(math) # get some information about the so-called module object

<module 'math' (built-in)>

The module object contains the functions and variables associated with the module. To call one of those, you need to use the name of the module and the name of the function, separated by a dot (a.k.a. as a period). This is called dot notation.

import math
signal_power = 200  # in microvolts
noise_power = 1     # in microvolts
ratio = signal_power / noise_power
decibels = 10 * math.log10(ratio)
print(str(decibels) + " dB")

23.010299956639813 dB

Another example involves getting a variable from the math module and using its trigonometric functions (sin, cos, tan, etc.):

import math
degrees = 45
# to convert from deg to rad, divide by 360 and multiply by 2π
radians = degrees / 360 * 2 * math.pi
print(math.sin(radians))

0.7071067811865475

Making Random Numbers

This turns out to be a pretty hard task for most computers as we generally want them to behave deterministically. When generating random numbers, this is a problem. But we can make it seem as if the computer is behaving non-deterministically by using algorithms to generate pseudorandom numbers using the random- module:

import random

for i in range(10):
    x = random.random()
    print(x)

0.4597169033073607
0.39433343645123353
0.9699872452986879
0.3886217989836309
0.713473451037861
0.05649189351989847
0.8393346778840809
0.37760550337740284
0.03950536181772901
0.7117717795167312

The program above produces ten (pseudo-)random numbers between 0.0 up to but not including 1.0. The randint-function takes the parameters low and high, and returns an int between low and high (including both):

random.randint(5,10)

9

To choose a random list from a sequence, use random.choice:

t = [1, 2, 3]
random.choice(t)

2

Adding New Functions

In order to add functions that we can reuse throughout our program, we need to define them using so-called function definitions:

def print_lyrics():
    print("I'm a lumberjack, and I'm okay.")
    print("I sleep all night and work all day.")

print(print_lyrics) # shows some information about the newly created variable
print(type(print_lyrics)) # this is function object with the type "function"
print(print_lyrics()) # this is how we call the function

<function print_lyrics at 0x7f50bc313290>
<class 'function'>
I'm a lumberjack, and I'm okay.
I sleep all night and work all day.

we can reference functions within functions:

def repeat_lyrics():
    print_lyrics()
    print_lyrics()
print(repeat_lyrics)

I'm a lumberjack, and I'm okay.
I sleep all night and work all day.
I'm a lumberjack, and I'm okay.
I sleep all night and work all day.

Flow of Execution

Functions can only be called after they are defined. Function definitions, on the other hand, do not alter the execution flow (statement after statement from top to bottom), but you need to remember that statements inside the function are not executed until the function is called.

When reading a program, try to follow the flow of execution rather than trying to read it top to bottom.

Parameters and Arguments

You can pass arguments to functions, e.g. when you call math.sin(some numeric argument). Inside the functions, the arguments are assigned to variables called parameters. Consider the following example to illustrate these concepts:

import math

def print_twice(anything):
    print(anything)
    print(anything)

print_twice(math.cos(math.pi))

-1.0
-1.0

Here, it is interesting to note, that the expression math.cos(math.pi) is only evaluated once (and then printed twice).

Fruitful Functions and Void Functions

In a script some functions are void, i.e. they do not return anything and when you try to assign them to a value you get a special value called None:

result = print_twice('Bing') # in a script, this does not return anything
print(result) # returns `None`

None

To return a result from a function, you need to use the return-statement within the function:

def multiply(a, b):
    multiplied = a * b
    return multiplied

x = multiply(3, 4)
print(x)

12

Why Functions?

• Grouping statements in your program into functional units makes it easier to read, understand and debug.
• Functions can make a program smaller by reducing repetitive code.
• Once debugged, well-designed functions can be repurposed within the same program and across other programs.

Iteration

The while statement

This statement first evaluates the condition. If it is false, it exits the while-statement and continues at the next statment. If the condition is true, the body is executed and the condition is evaluated again:

n = 5
while n > 0:
    print(n)
    n = n - 1
print('Blastoff!')

5
4
3
2
1
Blastoff!

Infinite Loops

If a the condition is always true, the loop will execute until your battery runs out - unless you make use of break to define a specific exit condition within the while-statement.

The code below, for instance, asks the user for input (and prints it back to her) until the user types done:

while True:
    line = input('> ')
    if line == 'done':
        break
    print(line)
print('Done!')

Finish an Iteration Early

If you want to exit an iteration early (but do not want to exit the entire loop), you can use the continue-statement. The following code illustrates that by not printing back lines to the user that start with the #-character.

while True:
    line = input('> ')
    if line[0] == '#':
        continue
    if line == 'done':
        break
    print(line)
print('Done!')

Definite Loops Using for

you can loop through a set of things constructing a definitive loop using the for-statement.

friends = ['Joseph', 'Glenn', 'Sally']
for friend in friends:
    print('Happy New Year:', friend)
print('Done!')

Happy New Year: Joseph
Happy New Year: Glenn
Happy New Year: Sally
Done!

In the code above, friend is the iteration variable, it steps successively through the items in stored in friends.

Loop Patterns

Counting and Summing Loops

In order to count the number of items in a list, the following for-loop might be used:

count = 0
for itervar in [3, 41, 12, 9, 74, 15]:
    count = count + 1
print('Count: ', count)

If you want to sum all the (numerical) items in a list, this code does the job:

total = 0
for itervar in [3, 41, 12, 9, 74, 15]:
    total = total + itervar
print('Total: ', total)

A variables such as total in the code snippet above is called accumulator. We won’t need either of the two programs above in practice as we have the built-in functions len() and sum().

Maximum and Minimum Loops

To emulate what the built-in function max() does, we can start with the following code:

largest = None
print('Before:', largest)
for itervar in [3, 41, 12, 9, 74, 15]:
    if largest is None or itervar > largest :
        largest = itervar
    print('Loop:', itervar, largest)
print('Largest:', largest)

Before: None
Loop: 3 3
Loop: 41 41
Loop: 12 41
Loop: 9 41
Loop: 74 74
Loop: 15 74
Largest: 74

None is used in the code above to mark as “empty”. To compute the smallest number (again, we have built-in min() to do the job in practice) in a list we can just change the > to a <:

smallest = None
print('Before:', smallest)
for itervar in [3, 41, 12, 9, 74, 15]:
    if smallest is None or itervar < smallest:
        smallest = itervar
    print('Loop:', itervar, smallest)
print('Smallest:', smallest)

Debugging by Bisection

When debugging loops always try to check in the middle of the code (if possible). For example, add a print statement in the middle of a loop and check its value. If it is already wrong, you know the bug hides in the first half of your loop body. This way you can cut down the number of lines you have to check quite significantly.

A bit of exercise code that puts lots of the concepts together:

while True:
    try:
        line = input('> ')
        if line == 'done':
            break
        list.append(int(line))
        print("current list items: ")
        print(list)
    except:
        print("Please enter a number")

# compute total
total = 0
for i in list:
    total = total + i
# compute count
count = 0
for j in list:
    count = count + 1
# compute avg
avg = total / count
print('total: ' + str(total) + "\ncount: " + str(count) + "\naverage: " + str(avg))

Data Structures

Strings

A string is a sequence of characters (all unicode in Python 3). Individual characters can be accessed using the bracket operator. Be aware that the index starts at 0 and not at 1.

So, for example, using the len() function to access the last letter of a string won’t work:

>>> fruit = 'banana'
>>> length = len(fruit)
>>> last = fruit[length]
IndexError: string index out of range

It only works if you substract 1 from length:

>>> length = len(fruit)
>>> last = fruit[length-1]
IndexError: string index out of range

Traversal through a string with a loop

You can traverse a string (stepping through it, looking at and possibly doing something with each character) with a while loop:

index = 0
while index < len(fruit): # <= would lead to IndexError
    letter = fruit[index]
    print(letter)
    index = index + 1

To do the same thing backwards, the while loop above must be adapted as follows:

index = len(fruit)-1
while index >= 0:
   letter = fruit[index]
   print(letter)
   index = index - 1

You can also use a for loop:

for char in fruit:
    print(char)

String Slices

If you only want to access a segment of a string, a so-called slice, you again use the bracket operator. The following image shows how that is done:

Strings are Immutable

This basically means that you cannot change a single character within the string without reassigning the entire string:

>>> greeting = 'Hello, world!'
>>> greeting[0] = 'J'
TypeError: 'str' object does not support item assignment

What you can do is:

>>> greeting = 'Hello, world!'
>>> new_greeting = 'J' + greeting[1:]
>>> print(new_greeting)
Jello, world!

Looping and Counting

The following function for instance loops through a string and counts the occurrences of a character given as an argument:

def count_char(word, letter):
    count = 0
    for l in word:
        if l == letter:
            count = count + 1
    print(count)

The in Operator

The in operator just return a boolean value if the first operand is a substring of the second operand:

>>> "a" in "banana"
True

String Comparison

Check whether two strings are equal:

if word == 'banana':
    print('All right, bananas.')

With < and > you can put strings in alphabetical order (beware though that uppercase letters always come before lowercase ones)

def word_sort(word):
    if word < 'banana':
        return('Your word, ' + word + ', comes before banana.')
    elif word > 'banana':
        return('Your word, ' + word + ', comes after banana.')
    else:
        return('All right, bananas.')

word_sort("Colibri")

'Your word, Colibri, comes before banana.'

String Methods

You can use the dir function to list the methods (i.e. built-in functions that are available to any instance of an object):

>>> stuff = 'Hello world'
>>> type(stuff)
<class 'str'>
>>> dir(stuff)
['capitalize', 'casefold', 'center', 'count', 'encode',
'endswith', 'expandtabs', 'find', 'format', 'format_map',
'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit',
'isidentifier', 'islower', 'isnumeric', 'isprintable',
'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower',
'lstrip', 'maketrans', 'partition', 'replace', 'rfind',
'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip',
'split', 'splitlines', 'startswith', 'strip', 'swapcase',
'title', 'translate', 'upper', 'zfill']
>>> help(str.capitalize)
Help on method_descriptor:

capitalize(...)
    S.capitalize() -> str

    Return a capitalized version of S, i.e. make the first character
    have upper case and the rest lower case.

To call (the correct therm is invoking) a method we append its name (delimited by space) to the object that we want to apply it to. There is a whole range of cool string methods, but the following examples only focus on some.

.upper() and .lower() make entire strings upper or lowercase.

>>> word = 'banana'
>>> new_word = word.upper()
>>> print(new_word)
BANANA

.find() can find substrings within strings. It can also take a start index as a second argument:

>>> word.find('na')
2
>>> word.find('na', 3)
4

.strip() removes all spaces, tabs or spaces from a string. .startswith() returns a boolean value if the string starts with the argument you give to it. If you want to make a case-insensitive search, you can chain .lower() and .startswith() together as such:

>>> line = "My name is Linus"
>>> line.lower().startswith('my')
True

Parsing Strings

You can use .find() to extract only the substrings of interest (like the hosts in an e-mail header):

>>> data = 'From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008'
>>> atpos = data.find('@')
>>> print(atpos)
21
>>> sppos = data.find(' ',atpos)
>>> print(sppos)
31
>>> host = data[atpos+1:sppos]
>>> print(host)
uct.ac.za
>>>

Format Operator

With the format operator, %, you are able to construct strings and dynamically replace values within it with data stored in other variables. An example:

>>> camels = 42
>>> 'I own %d camels' % camels
'I own 42 camels'

You can use different formatting like %d for integers, %g for decimals and %s for normal strings:

>>> 'In %d years I have spotted %g %s.' % (3, 0.1, 'camels')
'In 3 years I have spotted 0.1 camels.'

Files

Opening Files

When opening files, you are accessing (reading or writing) secondary memory. In Python, you use the open() function to do that. If it successfully opens a file, it returns the user a file hadle that can be used to access the data in the file:

>>> fhand = open('mbox.txt')
>>> print(fhand)
<_io.TextIOWrapper name='mbox.txt' mode='r' encoding='UTF-8'>

All the mentioned files should be available here.

Reading Files

As mentioned already, the file handle does not really contain the data, it is just reference to it. However, you can easily create a for loop to count the lines of a given text file.

fhand = open('mbox-short.txt')
count = 0
for line in fhand:
    count = count + 1
print('Line Count:', count)

Line Count: 1910

The advantage of the method above is that it does not require much memory, as each line is read, counted and then discarded before the next one is put into memory. If we know the file is small enough to be handled by (primary) memory, we can use the .read() method on the file handle.

>>> fhand = open('mbox-short.txt')
>>> inp = fhand.read()
>>> print(len(inp))
94626
>>> print(inp[:20])
From stephen.marquar

Searching Through a File

To print only the lines that start with “From:”, you can use the following code combining the patterns for reading a file with the string methods from the last section:

fhand = open('mbox-short.txt')
count = 0
for line in fhand:
    if line.startswith('From:'):
        print(line)

From: stephen.marquard@uct.ac.za

From: louis@media.berkeley.edu

From: zqian@umich.edu

From: rjlowe@iupui.edu
...

Why is there a new line between the lines of the output? Because the newline-character from the print() function is combined with the invisible newline-character from the file. You can use the .rstrip() method to ameliorate this problem:

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    if line.startswith('From:'):
        print(line)

From: stephen.marquard@uct.ac.za
From: louis@media.berkeley.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu
From: zqian@umich.edu
From: rjlowe@iupui.edu
From: cwen@iupui.edu
...

Next, you can structure the for loop using continue in order to skip “uninteresting” lines:

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    # Skip 'uninteresting lines'
    if not line.startswith('From:'):
        continue
    # Process our 'interesting' line
    print(line)

You can also use the .find() string method which returns the index of the searched substring or -1 if the substring was not found in order to show lines which contain “@uct.ac.za”:

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    # contracted version of the if-function
    if line.find('@uct.ac.za') == -1: continue
    print(line)

From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008
X-Authentication-Warning: set sender to stephen.marquard@uct.ac.za using -f
From: stephen.marquard@uct.ac.za
Author: stephen.marquard@uct.ac.za
From david.horwitz@uct.ac.za Fri Jan  4 07:02:32 2008
X-Authentication-Warning: set sender to david.horwitz@uct.ac.za using -f
From: david.horwitz@uct.ac.za
Author: david.horwitz@uct.ac.za
...

Letting the User Choose the File Name

The following code asks the user to input the file name:

fname = input('Enter the file name: ')
fhand = open(fname)
count = 0
for line in fhand:
    if line.startswith('Subject:'):
        count = count + 1
print('There were', count, 'subject lines in', fname)

Enter the file name: mbox.txt
There were 1797 subject lines in mbox.txt

Obviously, the code above does not know how to handle unexpected or faulty user input gracefully. To solve this, remember what try and expect can do for you.

Using try, except and open

We can use the aforementioned error handling structures to fix the flaw in the program:

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()
count = 0
for line in fhand:
    if line.startswith('Subject:'):
        count = count + 1
print('There were', count, 'subject lines in', fname)

Enter the file name: mbox.txt
There were 1797 subject lines in mbox.txt

Enter the file name: na na boo boo
File cannot be opened: na na boo boo

Writing Files

If you want to write a file, i.e. change it using Python, you have to open it with “w” as a second parameter:

>>> fout = open.('output.txt', 'w')
>>> print(fout)
<_io.TextIOWrapper name='output.txt' mode='w' encoding='UTF-8'>

You have to be careful though as opening a file in write mode clears out all the data stored in the file currently. The .write() method of the file handle object puts data into the file and returns the number of characters written:

>>> line1 = "This is cool,\n"
>>> fout.write(line1)
14
# you always need to close the file if we are writing files
>>> fout.close
<function TextIOWrapper.close()>

In IPython Notebooks you can use the %%writefile cell magic:

%%writefile output.txt
test
test2

Print the content of output.txt back:

with open('output.txt', 'r') as f:
    print(f.read())

test
test2

Dealing with the Invisible

Errors through whitespace can sometimes be hard to debug because, spaces, tabs and newlines are normally invisible:

>>> s = '1 2\t 3\n 4'
>>> print(s)
1 2  3
 4

The built-in repr() function can help by returning string representations of the object

>>> print(repr(s))
'1 2\t 3\n 4'

Exercises

The exercises in this chapter are the first ones interesting enough to be worked through in detail:

Exercise 1: Write a program to read through a file and print the contents of the file (line by line) all in upper case. Executing the program will look as follows:

python shout.py
Enter a file name: mbox-short.txt
FROM STEPHEN.MARQUARD@UCT.AC.ZA SAT JAN  5 09:14:16 2008
RETURN-PATH: <POSTMASTER@COLLAB.SAKAIPROJECT.ORG>
RECEIVED: FROM MURDER (MAIL.UMICH.EDU [141.211.14.90])
     BY FRANKENSTEIN.MAIL.UMICH.EDU (CYRUS V2.3.8) WITH LMTPA;
     SAT, 05 JAN 2008 09:14:16 -0500

Solution:

fname = input('Enter a file name: ')
try:
    fhand = open(fname)
    for line in fhand:
        line = line.rstrip().upper()
        print(line)
except FileNotFoundError:
    print('File cannot be openend: ', fname)

Exercise 2: Write a program to prompt for a file name, and then read through the file and look for lines of the form:

X-DSPAM-Confidence: 0.8475

When you encounter a line that starts with “X-DSPAM-Confidence:” pull apart the line to extract the floating-point number on the line. Count these lines and then compute the total of the spam confidence values from these lines. When you reach the end of the file, print out the average spam confidence.

Solution:

fname = input('Enter a file name: ')
confs = []
try:
    fhand = open(fname)
    for line in fhand:
        if line.startswith('X-DSPAM-Confidence:'):
            float_start = line.find(':') + 2
            confs.append(float(line[float_start:]))
    total = len(confs)
    avg = sum(confs) / total
    print('total:   ', total, '\naverage: ', avg)
except FileNotFoundError:
    print('File cannot be openend: ', fname)

Enter a file name: mbox-short.txt
total:    27
average:  0.7507185185185187

Enter a file name: mbox.txt
total:    1797
average:  0.8941280467445736

Exercise 3: Sometimes when programmers get bored or want to have a bit of fun, they add a harmless Easter Egg to their program. Modify the program that prompts the user for the file name so that it prints a funny message when the user types in the exact file name “na na boo boo”. The program should behave normally for all other files which exist and don’t exist. Here is a sample execution of the program:

python egg.py
Enter the file name: na na boo boo
NA NA BOO BOO TO YOU - You have been punk'd!

Solution:

fname = input('Enter a file name: ')
if fname == "na na boo boo":
    print("NA NA BOO BOO TO YOU - You have been punk'd")
    exit()
confs = []
try:
    fhand = open(fname)
    for line in fhand:
        if line.startswith('X-DSPAM-Confidence:'):
            float_start = line.find(':') + 2
            confs.append(float(line[float_start:]))
    total = len(confs)
    avg = sum(confs) / total
    print('total:   ', total, '\naverage: ', avg)
except FileNotFoundError:
    print('File cannot be openend: ', fname)

Lists

Similar to strings, lists are also sequences of values. While in a string the values are characters, they can be of any type in a list. The values of lists are called elements or items. The elements of a list don’t all have to be the same type; they can even be lists themselves (i.e. nested lists):

['spam', 2.0, 5, [10, 20]]

Lists are Mutable

Unlike strings, lists are mutable. Using the known bracket operator, we can access and change the elements of a list:

>>> cheeses = ['Cheddar', 'Edam', 'Gouda']
>>> numbers = [17, 123]
>>> numbers[1] = 5
>>> print(numbers)
[17, 5]
>>> numbers[-1] = 3
>>> print(numbers)
[17, 3]

The in operator also works on lists:

>>> 'Edam' in cheeses
True

Traversing a List

Most commonly, you will use a for loop:

for cheese in cheeses:
    print(cheese)

This, however, only works for reading and not for writing or updating the elements of the list; for that, you need the indices. For example you can combine the range (returns a list of indices from 0 to n - 1) and len (n, i.e. number of items in list) functions:

for i in range(len(numbers)):
    numbers[i] = numbers[i] * 2

Although a list can contain another list, the nested list will still count as a single element.

List Operations

You can concatenate lists using the + operator:

>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> c = a + b
>>> print(c)
[1, 2, 3, 4, 5, 6]

The * operator repeats the list n times

>>> [0] * 4
[0, 0, 0, 0]
>>> [1, 2, 3] * 3
[1, 2, 3, 1, 2, 3, 1, 2, 3]

List Slices

You can use the slice operator on lists:

>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t[1:3]
['b', 'c']
>>> t[:4]
['a', 'b', 'c', 'd']
>>> t[3:]
['d', 'e', 'f']

Omitting the first index means starting at the beginning and omitting the second means going until the end:

>>> t[:]
['a', 'b', 'c', 'd', 'e', 'f']

Due to the fact that lists are mutable, you can update multiple elements at a time. Sometimes its better to store the changed list in a new variable such that a copy of the unchanged list is kept:

>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t_new = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t_new[1:3] = ['x', 'y']
>>> print(t_new)
['a', 'x', 'y', 'd', 'e', 'f']

List Methods

One of the most important methods for list-objects is the .append() method which adds a new element to the end of a list.

>>> t = ['a', 'b', 'c']
>>> t.append('d')
>>> print(t)
['a', 'b', 'c', 'd']

.extend() takes another list as an argument and appends all of its items to the list-object that it operates on:

>>> t1 = ['a', 'b', 'c']
>>> t2 = ['d', 'e']
>>> t1.extend(t2)
>>> print(t1)
['a', 'b', 'c', 'd', 'e']

t2 remains unmodified in the example above.

Most list methods are void, i.e. they change the list object that they operate on and return None. So assigning them to variables won’t bring the desired result. For an example, see the .sort() method that sorts a list from high to low:

>>> t = ['d', 'c', 'e', 'b', 'a']
>>> t.sort()
>>> print(t.sort())
None
>>> print(t)
['a', 'b', 'c', 'd', 'e']

Deleting Elements

You can delete elements from lists in several different ways. If you know the index, use the .pop() method which, if no index is given, it just deletes and returns the last element of a list:

>>> t = ['a', 'b', 'c']
>>> x = t.pop(1)
>>> print(t)
['a', 'c']
>>> print(x)
b
>>> t.pop()
'c'

If there is no need to return anything, you can use the del operator which uses the following syntax:

>>> t = ['a', 'b', 'c']
>>> del t[1]
>>> print(t)
['a', 'c']

If you already know what to remove, but don’t know where it is in the list, use the .remove() method:

>>> t = ['a', 'b', 'c']
>>> print(t.remove('b'))
None
>>> print(t)
['a', 'c']
>>> t_new = ['a', 'b', 'c', 'd', 'e', 'f']
>>> del t[1:5]
>>> print(t_new)
['a', 'f']

Lists and Functions

There are a number of useful built-in functions that work on lists. max() and len() work with lists that contain elements of all (comparable) types. The sum() function only works with lists containing numbers.

>>> nums = [3, 41, 12, 9, 74, 15]
>>> print(len(nums))
6
>>> print(max(nums))
74
>>> print(min(nums))
3
>>> print(sum(nums))
154
>>> print(sum(nums)/len(nums))
25

Using these, we can rewrite the following program that takes user input and computes the average from this:

total = 0
count = 0
while (True):
    inp = input('Enter a number: ')
    if inp == 'done': break
    value = float(inp)
    total = total + value
    count = count + 1

average = total / count
print('Average:', average)

to this:

numlist = list()
while (True):
    inp = input('Enter a number: ')
    if inp == 'done': break
    value = float(inp)
    numlist.append(value)

average = sum(numlist) / len(numlist)
print('Average:', average)

Lists and Strings

Converting a string (sequence of characters) to a list (sequence of values) is easy using the built-in list function:

>>> s = 'spam'
>>> t = list(s)
>>> print(t)
['s', 'p', 'a', 'm']

If you need to break a string into multiple words, use the .split() method:

>>> s = 'pining for the fjords'
>>> t = s.split()
>>> print(t)
['pining', 'for', 'the', 'fjords']
>>> print(t[2])
the

If you want the .split() method to split not at spaces, but somewhere else, you have to provide the desired delimiter as an argument:

>>> s = 'spam-spam-spam'
>>> delimiter = '-'
>>> s.split(delimiter)
['spam', 'spam', 'spam']

You can think of the .join() method as the inverse of the .split() method. It takes a list of strings as an argument and concatenates them. It needs to be invoked on the delimiter:

>>> t = ['pining', 'for', 'the', 'fjords']
>>> delimiter = ' '
>>> delimiter.join(t)
'pining for the fjords'

Parsing Lines Using .split()

The .split() method is very helpful if you want to do something other than printing whole lines when reading a file. You can find the “interesting” lines and then parse the line to find the interesting part of the line. The following code prints the day of the week from our mbox-file from earlier:

fhand = open('mbox-short.txt')
for line in fhand:
    line = line.rstrip()
    if not line.startswith('From '): continue
    words = line.split()
    print(words[2])

Sat
Fri
Fri
Fri
...

Objects and Values

When assigning a and b to the same string, Python only creates one string object and both a and b refer to it:

>>> a = 'banana'
>>> b = 'banana'
>>> a is b
True

Doing the same with lists, however, creates two distinct objects, which are equivalent (have the same value) but not identical (because they are not the same object):

>>> a = [1, 2, 3]
>>> b = [1, 2, 3]
>>> a is b
False

Aliasing

However, if a refers to a (list) object, and you assign b = a, then both variables reference the same object:

>>> a = [1, 2, 3]
>>> b = a
>>> b is a
True

The association of a variable with an object is called a reference. If an object has more than one reference, the object is aliased. If the aliased object is mutable (e.g. a list), the changes made using one alias will affect the other:

>>> b[0] = 17
>>> print(a)
[17, 2, 3]

While sometimes useful, you should avoid aliasing mutable objects. Aliasing immutable object is not such a big deal as it hardly ever makes a difference.

List Arguments

The following function delete_head removes the first element from a list:

def delete_head(t):
    del t[0]

This is how it is used:

>>> letters = ['a', 'b', 'c']
>>> delete_head(letters)
>>> print(letters)
['b', 'c']

t and letters are aliases for the same object. There is an important distinction between operations modifying a list and those creating a list. For instance, the .append() method modifies a list while the + operator creates a new one:

>>> t1 = [1, 2]
>>> t2 = t1.append(3)
>>> print(t1)
[1, 2, 3]
>>> print(t2)
None

>>> t3 = t1 + [3]
>>> print(t3)
[1, 2, 3]
>>> t2 is t3
False

Consider the following function definition:

def bad_delete_head(t):
    t = t[1:]          # WRONG

This function leaves the original list unmodified, i.e. the list that was passed as an argument. Alternatively, you can write a function that creates and returns a new list:

def tail(t):
    return t[1:]

This function leaves the original list unmodified:

>>> letters = ['a', 'b', 'c']
>>> rest = tail(letters)
>>> print(rest)
['b', 'c']

Exercise 8.1:

Write a function called chop that takes a list and modifies it, removing the first and last elements, and returns None. Then write a function called middle that takes a list and returns a new list that contains all but the first and last elements.

Solution

t1 = ["a", "b", "c"]
t2 = ["a", "b", "c"]

def chop(t):
    del t[0]
    del t[-1]

def middle(t):
    return t[1:-1]

print(chop(t1))
print(t1)
print(middle(t2))

None
['b']
['b']

Pitfalls

List Methods Returning None

Most list methods return None, so the following does not make much sense:

t = t.sort()           # WRONG

Pick an Idiom (and Stick with it)

Pick one way to do things and stick to it. With lists there are often too many ways to do the same thing (e.g. =pop=, remove, del and even slice assignments can be used to remove an element from a list). To add an element, you can use the append method or the + operator. However, only the following way is correct if you want to modify an existing list by adding the value of x to it:

t.append(x)
t = t + [x]

and these are wrong:

t.append([x])          # Adds nested list containing variable to list
t = t.append(x)        # t is now None
t + [x]                # does not modify the list
t = t + x              # if x is not a list, this returns a TypeError

Make Copies

If you want to use a method like sort, but you want to keep the original (unsorted) list, you should make a copy:

orig = t[:]
t.sort()

Lists, split and Files

Consider the following code to parse the weekdays from a text file and the error message we get when running it:

fhand = open('mbox-short.txt')
for line in fhand:
    words = line.split()
    if words[0] != 'From' : continue
    print(words[2])

Sat
Traceback (most recent call last):
  File "search8.py", line 5, in <module>
    if words[0] != 'From' : continue
IndexError: list index out of range

Let’s add some print statements for the purposes of debugging:

for line in fhand:
    words = line.split()
    print('Debug:', words)
    if words[0] != 'From' : continue
    print(words[2])

Debug: ['X-DSPAM-Confidence:', '0.8475']
Debug: ['X-DSPAM-Probability:', '0.0000']
Debug: []
Traceback (most recent call last):
  File "search9.py", line 6, in <module>
    if words[0] != 'From' : continue
IndexError: list index out of range

the list words seems to be empty and a look into the text file betrays that there is an empty line when the code throws us an error. The index 0 is out of range because the list we constructed is empty. We can remedy this using a guardian condition:

fhand = open('mbox-short.txt')
count = 0
for line in fhand:
    words = line.split()
    # print('Debug:', words)
    if len(words) == 0 : continue
    if words[0] != 'From' : continue
    print(words[2])

Exercise 8.2

Figure out which line of the above program is still not properly guarded. See if you can construct a text file which causes the program to fail and then modify the program so that the line is properly guarded and test it to make sure it handles your new text file.

Solution

There is the possibility that a line just has the word “From” in it. Then our little program throws us another IndexError because words[2] will be out of range in a list that has a length of 1. In order to guard against that, the first if condition should be modified as follows:

...
if len(words) < 2 : continue
...

Exercise 8.3

Rewrite the guardian code in the above example without two if statements. Instead, use a compound logical expression using the or logical operator with a single if statement.

Solution

fhand = open('mbox-short-alt.txt')
count = 0
for line in fhand:
    words = line.split()
    # print('Debug:', words)
    if len(words) < 2 or words[0] != 'From' : continue
    print(words[2])

Exercise 8.4

Write a program to open the file romeo.txt and read it line by line. For each line, split the line into a list of words using the split function. For each word, check to see if the word is already in a list. If the word is not in the list, add it to the list. When the program completes, sort and print the resulting words in alphabetical order.

Solution

wordlist = []
fhand = open('romeo.txt')
for line in fhand:
    words = line.split()
    for word in words:
        if word in wordlist : continue
        wordlist.append(word)
sorted_words = sorted(wordlist)
print(sorted_words)

Exercise 8.5

Write a program to read through the mail box data and when you find line that starts with “From”, you will split the line into words using the split function. We are interested in who sent the message, which is the second word on the From line. You will parse the From line and print out the second word for each From line, then you will also count the number of From (not From:) lines and print out a count at the end.

Solution

fhand = open('mbox-short.txt')
count = 0
for line in fhand:
    words = line.split()
    if len(words) < 2 or words[0] != 'From' : continue
    count += 1
    # print("Debug:", words, count)
    print(words[1])
print("There were", count, "lines in the file with From as the first word")

Exercise 8.6

Rewrite the program that prompts the user for a list of numbers and prints out the maximum and minimum of the numbers at the end when the user enters “done”. Write the program to store the numbers the user enters in a list and use the max() and min() functions to compute the maximum and minimum numbers after the loop completes.

Solution

num_list = []
while True:
    try:
        num = input("Enter a number: ")
        if num == "done" : break
        num = float(num)
        num_list.append(num)
    except:
        print("Please enter a number")
print("Maximum:", max(num_list), "\nMinimum:", min(num_list))

Dictionaries

A dictionary is similar to a list, but less restrictive. While in lists, the indeces have to be integers, they can be of (almost) any type in dictionaries. Fundamentally, a dictionary maps keys (our indeces) to values. This association is called a key-value pair.

>>> eng2sp = dict()
>>> print(eng2sp)
{}

The curly brackets, {}, denote an empty dictionary. If you want to add items to the dictionary, use the following syntax:

>>> eng2sp['one'] = 'uno'
>>> print(eng2sp)
{'one', 'uno'}

The output format is equivalent to an input format, i.e. you can create a new dictionary with three items as such:

>>> eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'}
>>> print(eng2sp)
{'one': 'uno', 'three': 'tres', 'two': 'dos'}

Interestingly, the order of the key-value pairs changed. This is to be expected. It is not a problem because we need the keys to look up values anyways. If the key does not exist we get a KeyError.

>>> print(eng2sp['two'])
'dos'
>>> print(eng2sp['four'])
KeyError: 'four'

The len() function also works with dictionaries; it simply returns the number of key-value pairs.

>>> len(eng2sp)
3

The in operator works on dictionaries, too. It only tells you whether something appears as a key in the dictionary (if it just appears as a value, this is not good enough):

>>> 'one' in eng2sp
True
>>> 'uno' in eng2sp
False

If you want to know whether something exists as a value in a dictionary, you can use the following workaround:

>>> vals = list(eng2sp.values())
>>> 'uno' in vals
True

Exercise 9.1

Write a program that reads the words in words.txt and stores them as keys in a dictionary. It doesn’t matter what the values are. Then you can use the in operator as a fast way to check whether a string is in the dictionary.

Solution

word_dict = dict()
fhand = open('words.txt')
word_id = 1
for line in fhand:
    words = line.split()
    for word in words:
        word_id += 1
        if word in word_dict : continue
        word_dict[word] = word_id
print(word_dict)

Dictionaries as Sets of Counters

With dictionaries, we can now implement a more elegant solution to the problem of counting the occurrence of characters within any given string:

word = "brontosaurus"
d = dict()
for c in word:
    if c not in d:
        d[c] = 1
    else:
        d[c] = d[c] + 1
print(d)

{'a': 1, 'b': 1, 'o': 2, 'n': 1, 's': 2, 'r': 2, 'u': 2, 't': 1}

Effectively, this computes a histogram, which is the statistical term for a set of counters (or frequencies for that matter).

The .get() method takes both a key and a default value. If the key appears in the dictionary, .get() returns the corresponding values; otherwise it returns the specified default value:

>>> counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}
>>> print(counts.get('jan', 0))
100
>>> print(counts.get('tim', 0))
0

Utilising the .get() method of dictionaries allows us to write the code above more succinctly:

word = 'brontosaurus'
d = dict()
for c in word:
    d[c] = d.get(c,0) + 1
print(d)

Dictionaries and Files

You can use dictionaries to count the occurrence of words in a text file (For now, this uses a version of the romeo.txt file that has now punctuation):

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()

counts = dict()
for line in fhand:
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
        #   counts[word] = counts[word] + 1
            counts[word] += 1

print(counts)

Enter the file name: romeo.txt
{'and': 3, 'envious': 1, 'already': 1, 'fair': 1,
'is': 3, 'through': 1, 'pale': 1, 'yonder': 1,
'what': 1, 'sun': 2, 'Who': 1, 'But': 1, 'moon': 1,
'window': 1, 'sick': 1, 'east': 1, 'breaks': 1,
'grief': 1, 'with': 1, 'light': 1, 'It': 1, 'Arise': 1,
'kill': 1, 'the': 3, 'soft': 1, 'Juliet': 1}

Looping Through Dictionaries

As it is not very convenient to look through the output above, let’s write a for loop that traverses the dictionary and prints the key-value pairs.

counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}
for key in counts:
    print(key, counts[key])

jan 100
chuck 1
annie 42

However, as dictionaries are unordered (since Python 3.6+, they are insertion ordered), you need to find a way to order the output using a list. This is easy:

counts = { 'chuck' : 1 , 'annie' : 42, 'jan': 100}

# Make a list of the values that we can sort
lst = list(counts.values())
lst.sort()

# Invert the dictionary (use .iteritems() for Python 2.7)
counts_inv = dict((v,k) for k, v in counts.items())

for value in lst:
    print(value, counts_inv[value])

1 chuck
42 annie
100 jan

Advanced Text Parsing

In order to deal with the punctuation in the real romeo.txt file, you need string methods. They also allow you to not count “Who” and “who” as different words but as the same. Most importantly, you need the .translate() method. The documentation for that method reads as follows:

line.translate(str.maketrans(fromstr, tostr, deletestr))

Replace the characters in fromstr with the character in the same position in tostr and delete all characters that are in deletestr. The fromstr and tostr can be empty strings and the deletestr parameter can be omitted.

Additionally, Python already has a built-in concept of punctuation:

>>> import string
>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

Hence, you can adapt the code from earlier:

import string

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()

counts = dict()
for line in fhand:
    line = line.rstrip()
    line = line.translate(line.maketrans('', '', string.punctuation))
    line = line.lower()
    words = line.split()
    for word in words:
        if word not in counts:
            counts[word] = 1
        else:
            counts[word] += 1

print(counts)

Now, analysing the file romeo-full.txt with this code provides the following output:

Enter the file name: romeo-full.txt
{'swearst': 1, 'all': 6, 'afeard': 1, 'leave': 2, 'these': 2,
'kinsmen': 2, 'what': 11, 'thinkst': 1, 'love': 24, 'cloak': 1,
a': 24, 'orchard': 2, 'light': 5, 'lovers': 2, 'romeo': 40,
'maiden': 1, 'whiteupturned': 1, 'juliet': 32, 'gentleman': 1,
'it': 22, 'leans': 1, 'canst': 1, 'having': 1, ...}

Debugging Dictionaries

Scale Down the Input For instance, modify your program such that it only reads the first n lines. If there is an error, reduce n to the smallest value that manifests and error.

Check Summaries and Types Check the total number of items in a dictionary (and their types) or the total of a list of numbers (and their types).

Write Self-Checks Try to detect completely illogical outputs by checking for errors automatically. For example, check that the average of a list cannot be larger than the largest element of a list or less than the smallest.

Pretty Print Good formatting of your output can make it easier to spot an error. The time you spend building good scaffolding reduces the time you spend debugging.

Exercise 9.2

Write a program that categorizes each mail message by which day of the week the commit was done. To do this look for lines that start with “From”, then look for the third word and keep a running count of each of the days of the week. At the end of the program print out the contents of your dictionary (order does not matter).

Solution

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()

weekday_count = dict()
for line in fhand:
    words = line.split()
    for word in words:
        if word != "From" or len(words[2]) != 3 : continue
        weekday_count[words[2]] = weekday_count.get(words[2],0) + 1

print(weekday_count)

Enter the file name: mbox.txt
{'Sat': 61, 'Fri': 315, 'Thu': 392, 'Wed': 292, 'Tue': 372, 'Mon': 299, 'Sun': 66}

Exercise 9.3

Write a program to read through a mail log, build a histogram using a dictionary to count how many messages have come from each email address, and print the dictionary.

Solution

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()

email_count = dict()
for line in fhand:
    if not line.startswith('From ') : continue
    words = line.split()
    for word in words:
        if '@' in word:
            email_count[word] = email_count.get(word,0) + 1

print(email_count)

Enter the file name: mbox-short.txt
{'stephen.marquard@uct.ac.za': 2, 'louis@media.berkeley.edu': 3, 'zqian@umich.edu': 4, 'rjlowe@iupui.edu'
: 2, 'cwen@iupui.edu': 5, 'gsilver@umich.edu': 3, 'wagnermr@iupui.edu': 1, 'antranig@caret.cam.ac.uk': 1,
 'gopal.ramasammycook@gmail.com': 1, 'david.horwitz@uct.ac.za': 4, 'ray@media.berkeley.edu': 1}

Exercise 9.4

Add code to the above program to figure out who has the most messages in the file. After all the data has been read and the dictionary has been created, look through the dictionary using a maximum loop (see Chapter 5: Maximum and minimum loops) to find who has the most messages and print how many messages the person has.

Solution

fname = input('Enter the file name: ')
try:
    fhand = open(fname)
except:
    print('File cannot be opened:', fname)
    exit()

# Same as above
email_count = dict()
for line in fhand:
    if not line.startswith('From ') : continue
    words = line.split()
    for word in words:
        if '@' in word:
            email_count[word] = email_count.get(word,0) + 1

# Find largest using max-loop
largest = None
for i in email_count.values():
    if largest is None or i > largest:
        largest = i

# Make list of the keys and of the values
lst_key = list(email_count.keys())
lst_val = list(email_count.values())

# Denote index of largest value
ind_largest = lst_val.index(largest)

# Print
print(lst_key[ind_largest], largest)