Module -4

Chapter-1
Organizing Files
 The shutil Module
 Walking a Directory Tree
 Compressing Files with the zipfile Module
 Project: Renaming Files with American-Style Dates to European-Style Dates
 Project: Backing Up a Folder into a ZIP File
Chapter-2
Debugging
 Raising Exceptions
 Getting the Traceback as a String
 Assertions
 Logging
 IDLE‟s Debugger.
Chapter -1
Organizing Files

The Shutil Module:

 The shutil (or shell utilities) module has functions to let you copy, move, rename, and delete files in your
Python programs.
 Copying Files and Folders:
 The shutil module provides functions for copying files, as well as entire folders.
 Calling shutil.copy(source, destination) will copy the file at the path source to the folder at the path
destination. This function returns a string of the path of the copied file.
 Example:
import shutil, os
os.chdir('C:\\')
shutil.copy('C:\\spam.txt', 'C:\\delicious')
shutil.copy('eggs.txt', 'C:\\delicious\\eggs2.txt')
 The first shutil.copy() call copies the file at C:\spam.txt to the folder C:\delicious. The return value
is the path of the newly copied file.
 The second shutil.copy() call also copies the file at C:\eggs.txt to the folder C:\delicious but gives
the copied file the name eggs2.txt.
 While shutil.copy() will copy a single file, shutil.copytree() will copy an entire folder and every folder and file
contained in it.
 Calling shutil.copytree(source, destination) will copy the folder at the path source, along with all of its files
and subfolders, to the folder at the path destination. The function returns a string of the path of the copied
folder. The source and destination parameters are both strings.
 Example:
>>> import shutil, os
>>> os.chdir('C:\\')
>>> shutil.copytree('C:\\bacon', 'C:\\bacon_backup')
 The shutil.copytree() call creates a new folder named bacon_backup with the same content as the original
bacon folder. We have now safely backed up your precious, precious bacon.

Moving and Renaming Files and Folders:

 Calling shutil.move(source, destination) will move the file or folder at the path source to the path
destination and will return a string of the absolute path of the new location.
 If destination points to a folder, the source file gets moved into destination and keeps its current filename.
Example:
>>> import shutil
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')
 Assuming a folder named eggs already exists in the C:\ directory, this shutil.move() calls says, “Move C:\
bacon.txt into the folder C:\eggs.”
 If there had been a bacon.txt file already in C:\eggs, it would have been overwritten. Since it’s easy to
accidentally overwrite files in this way, you should take some care when using move().
 The destination path can also specify a filename.
Example:
>>> shutil.move('C:\\bacon.txt', 'C:\\eggs\\new_bacon.txt')
 The above line says, “Move C:\bacon.txt into the folder C:\eggs, and while you’re at it, rename that
bacon.txt file to new_bacon.txt.”

>>> shutil.move('C:\\bacon.txt', 'C:\\eggs')

  Here, move() can’t find a folder named eggs in the C:\ directory and so assumes that destination must be
specifying a filename, not a folder.
>>> shutil.move('spam.txt', 'c:\\does_not_exist\\eggs\\ham')
 With the above line finally, the folders that make up the destination must already exist, or else Python will
throw an exception.

Permanently Deleting Files and Folders

We can delete a single file or a single empty folder with functions in the os module, whereas to delete a folder and
all of its contents, you use the shutil module.
 Calling os.unlink(path) will delete the file at path.
 Calling os.rmdir(path) will delete the folder at path. This folder must be empty of any files or folders.
 Calling shutil.rmtree(path) will remove the folder at path, and all files and folders it contains will also be
deleted.
 Python Program to delete files with .txt extension:
import os
for filename in os.listdir():
if filename.endswith('.txt'):
os.unlink(filename)

Safe Deletes with the send2trash Module:

 Since Python’s built-in shutil.rmtree() function irreversibly deletes files and folders, it can be dangerous to
use.
 A much better way to delete files and folders is with the thirdparty send2trash module. We can install this
module by running pip install send2trash from a Terminal window.
 Using send2trash is much safer than Python’s regular delete functions, because it will send folders and files
to your computer’s trash or recycle bin instead of permanently deleting them. If a bug in your program
deletes something with send2trash you didn’t intend to delete, you can later restore it from the recycle bin.
 Example:
>>> import send2trash
>>> baconFile = open('bacon.txt', 'a') # creates the file
>>> baconFile.write('Bacon is not a vegetable.')
>>> baconFile.close()
>>> send2trash.send2trash('bacon.txt')
 In general, you should always use the send2trash.send2trash() function to delete files and folders. But while
sending files to the recycle bin lets you recover them later, it will not free up disk space like permanently
deleting them does.

Walking a Directory Tree

 To rename every file in some folder and also every file in every subfolder of that folder. That is, you want to
walk through the directory tree, touching each file as you go.
 Python provides a function to handle this process for you.
 The os.walk() function is passed a single string value: the path of a folder.
 You can use os.walk() in a for loop statement to walk a directory tree, much like how you can use the
range() function to walk over a range of numbers.
 The os.walk() function will return three values on each iteration through the loop:
 A string of the current folder’s name
 A list of strings of the folders in the current folder
 A list of strings of the files in the current folder

Example:
 import os
for folderName, subfolders, filenames in os.walk('C:\\delicious'):
print('The current folder is ' + folderName)
for subfolder in subfolders:
print('SUBFOLDER OF ' + folderName + ': ' + subfolder)
for filename in filenames:
print('FILE INSIDE ' + folderName + ': '+ filename)
print('')

• Since os.walk() returns lists of strings for the subfolder and filename variables, you can use these lists in
their own for loops. Replace the print() function calls with your own custom code.

Compressing Files with the zipfile Module

• ZIP files (with the .zip file extension), which can hold the compressed contents of many
other files. Compressing a file reduces its size, which is useful when transferring it over the Internet.

• And since a ZIP file can also contain multiple files and subfolders, it’s a handy way to package several files
into one. This single file, called an archive file, can then be, say, attached to an email.

• Python programs can both create and open (or extract) ZIP files using functions in the zipfile module.

Reading ZIP Files

• To read the contents of a ZIP file, first you must create a ZipFile object.
• ZipFile objects are conceptually similar to the File objects you saw returned by the open() function in
the previous chapter: They are values through which the program interacts with the file.

• To create a ZipFile object, call the zipfile.ZipFile() function, passing it a string of the .zip file’s filename
• Example:
>>> import zipfile, os
>>> os.chdir(‘E:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')

• A ZipFile object has a namelist() method that returns a list of strings for all the files and folders
contained in the ZIP file.
>>> exampleZip.namelist()
 • Strings can be passed to the getinfo() ZipFile method to return a ZipInfo object about that particular
file.
>>> spamInfo = exampleZip.getinfo('example/spam.txt')

• ZipInfo objects have their own attributes, such as file_size and compress_size in
bytes, which hold integers of the original file size and compressed file size,
respectively
>>> spamInfo.file_size
>>> spamInfo.compress_size
• The below line calculates how efficiently example.zip is compressed by dividing the
original file size by the compressed file size and prints
>>>'Compressed file is %sx smaller!' % (round(spamInfo.file_size /
spamInfo.compress_size, 2))
>>>exampleZip.close()

Extracting from ZIP Files

• The extractall() method for ZipFile objects extracts all the files and folders from a
ZIP file into the current working directory.
• Example:
>>> import zipfile, os
>>> os.chdir('C:\\') # move to the folder with example.zip
>>> exampleZip = zipfile.ZipFile('example.zip')
>>> exampleZip.extractall()
>>> exampleZip.close()
• In the above code, the contents of example.zip will be extracted to C:\. Optionally,
you can pass a folder name to extractall() to have it extract the files into a folder
other than the current working directory. If the folder passed to the extractall()
method does not exist, it will be created.
• The extract() method for ZipFile objects will extract a single file from the ZIP file.
• Example:
>>> exampleZip.extract('spam.txt')
>>> exampleZip.extract('spam.txt', 'C:\\some\\new\\folders')
>>> exampleZip.close()
• Optionally, you can pass a second argument to extract() to extract the file into a
folder other than the current working directory. If this second argument is a folder
that doesn’t yet exist, Python will create the folder. The value that extract() returns
is the absolute path to which the file was extracted.
Creating and Adding to ZIP Files
• To create your own compressed ZIP files, you must open the ZipFile object in write
mode by passing 'w' as the second argument.
• When you pass a path to the write() method of a ZipFile object, Python will
compress the file at that path and add it into the ZIP file. The write() method’s first
argument is a string of the filename to add. The second argument is the
compression type parameter, which tells the computer what algorithm it should use
to compress the files, you can always just set this value to zipfile.ZIP_DEFLATED.
(This specifies the deflate compression algorithm, which works well on all types of
data.)
 • Example:
>>> newZip = zipfile.ZipFile('new123.zip', 'w')
>>>newZip.write('spam.txt',compress_type=zipfile.ZIP_DEFLATED)
>>> newZip.close()
• This code will create a new ZIP file named new.zip that has the compressed contents
of spam.txt.
• Keep in mind that, just as with writing to files, write mode will erase all existing
contents of a ZIP file. If you want to simply add files to an existing ZIP file, pass 'a'
as the second argument to zipfile.ZipFile() to open the ZIP file in append mode.

Project: Renaming Files with American-Style Dates to European-Style Dates

Here’s what the program does:
1. It searches all the filenames in the current working directory for American-style dates.
2. When one is found, it renames the file with the month and day swapped to make it
European-style.
This means the code will need to do the following:
1. Create a regex that can identify the text pattern of American-style dates.
2. Call os.listdir() to find all the files in the working directory.
3. Loop over each filename, using the regex to check whether it has a date.
4. If it has a date, rename the file with shutil.move().

Program:
#! python3
# renameDates.py - Renames filenames with American MM-DD-YYYY date format
# to European DD-MM-YYYY.
import shutil, os, re
# Create a regex that matches files with the American date format.
datePattern = re.compile(r"""^(.*?) # all text before the date
((0|1)?\d)- # one or two digits for the month
((0|1|2|3)?\d)- # one or two digits for the day
((19|20)\d\d) # four digits for the year
(.*?)$ # all text after the date
""", re.VERBOSE)
# Loop over the files in the working directory.
for amerFilename in os.listdir('.'):
mo = datePattern.search(amerFilename)
# Skip files without a date.
if mo == None:
continue
# Get the different parts of the filename.
beforePart = mo.group(1)
monthPart = mo.group(2)
dayPart = mo.group(4)
yearPart = mo.group(6)
afterPart = mo.group(8)
# Form the European-style filename.
euroFilename = beforePart + dayPart + '-' + monthPart + '-' + yearPart + afterPart
# Get the full, absolute file paths.
absWorkingDir = os.path.abspath('.')
amerFilename = os.path.join(absWorkingDir, amerFilename)
euroFilename = os.path.join(absWorkingDir, euroFilename)
# Rename the files.
 print('Renaming "%s" to "%s"...' % (amerFilename, euroFilename))
#shutil.move(amerFilename, euroFilename) # uncomment after testing

Project: Backing Up a Folder into a ZIP File:

Steps to be followed:
Step 1: Figure Out the ZIP File’s Name
Step 2: Create the New ZIP File
Step 3: Walk the Directory Tree and Add to the ZIP File

Program:
import zipfile, os
def backupToZip(folder):
# Backup the entire contents of "folder" into a ZIP file.
folder = os.path.abspath(folder) # make sure folder is absolute
# Figure out the filename this code should use based on
# what files already exist.
number = 1
while True:
zipFilename = os.path.basename(folder) + '_' + str(number) + '.zip'
if not os.path.exists(zipFilename):
break
number = number + 1
# TODO: Create the ZIP file.
while True:
zipFilename = os.path.basename(folder) + '_' + str(number) + '.zip'
if not os.path.exists(zipFilename):
break
number = number + 1
# Create the ZIP file.
print('Creating %s…' % (zipFilename))
backupZip = zipfile.ZipFile(zipFilename, 'w')
# TODO: Walk the entire folder tree and compress the files in each folder.
for foldername, subfolders, filenames in os.walk(folder):
print('Adding files in %s…' % (foldername))
# Add the current folder to the ZIP file.
backupZip.write(foldername)
# Add all the files in this folder to the ZIP file.
for filename in filenames:
newBase/os.path.basename(folder) + '_'
if filename.startswith(newBase) and filename.endswith('.zip'):
continue # don't backup the backup ZIP files
backupZip.write(os.path.join(foldername, filename))
backupZip.close()
print('Done.')

backupToZip('E:\example\delicious')
Chapter-2
Debugging
• The chapter covers some tools and techniques for finding the root cause of bugs in your
program to help you fix bugs faster and with less effort.
• The debugger is a feature of IDLE that executes a program one instruction at a time, giving
you a chance to inspect the values in variables while your code runs, and track how the values
change over the course of your program.

Raising Exceptions
• Python raises an exception whenever it tries to execute invalid code.

• But you can also raise your own exceptions in your code.

• Raising an exception is a way of saying, “Stop running the code in this function and move the
program execution to the except statement.”
• Exceptions are raised with a raise statement. In code, a raise statement consists of the
following:
 The raise keyword
 A call to the Exception() function
 A string with a helpful error message passed to the Exception() function
Example:
>>> raise Exception('This is the error message.')
• If there are no try and except statements covering the raise statement that raised the
exception, the program simply crashes and displays the exception’s error message.
• Often it’s the code that calls the function, not the fuction itself, that knows how to handle an
expection. So you will commonly see a raise statement inside a function and the try and
except statements in the code calling the function.
• Example:
def boxPrint(symbol, width, height):
if len(symbol) != 1:
raise Exception('Symbol must be a single character string.')
if width <= 2:
raise Exception('Width must be greater than 2.')
if height <= 2:
raise Exception('Height must be greater than 2.')
print(symbol * width)
for i in range(height - 2):
print(symbol + (' ' * (width - 2)) + symbol)
print(symbol * width)
for sym, w, h in (('*', 4, 4), ('O', 20, 5), ('x', 1, 3), ('ZZ', 3, 3)):
try:
boxPrint(sym, w, h)
except Exception as err:
print('An exception happened: ' + str(err))
 • Here we’ve defined a boxPrint() function that takes a character, a width, and a height, and
uses the character to make a little picture of a box with that width and height. This box shape
is printed to the console.
• This program uses the except Exception as err form of the except statement . If an Exception
object is returned from boxPrint() , this except statement will store it in a variable named err.
The Exception object can then be converted to a string by passing it to str() to produce a user-
friendly error message .

Getting the Traceback as a String

• When Python encounters an error, it produces a treasure trove of error information called the
traceback. The traceback includes the error message, the line number of the line
that caused the error, and the sequence of the function calls that led to the error. This
sequence of calls is called the call stack.
• Example:
def spam():
bacon()
def bacon():
raise Exception('This is the error message.')
spam()

• The traceback is displayed by Python whenever a raised exception goes unhandled. But you
can also obtain it as a string by calling traceback.format_exc(). This function is useful if you
want the information from an exception’s traceback but also want an except statement to
gracefully handle the exception. You will need to import Python’s traceback module before
calling this function.
Example:
import traceback
try:
raise Exception('This is the error message.')
except:
errorFile = open('errorInfo.txt', 'w')
errorFile.write(traceback.format_exc())
errorFile.close()
print('The traceback info was written to errorInfo.txt.')
• The 116 is the return value from the write() method, since 116 characters were written to the
file. The traceback text was written to errorInfo.txt.

Assertions
An assertion is a sanity check to make sure your code isn’t doing something obviously wrong.
These sanity checks are performed by assert statements. If the sanity check fails, then an
AssertionError exception is raised.
In code, an assert statement consists of the following:
• The assert keyword
• A condition (that is, an expression that evaluates to True or False)
• A comma
• A string to display when the condition is False
Example:
>>> podBayDoorStatus = 'open'
>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'
>>> podBayDoorStatus = 'I\'m sorry, Dave. I\'m afraid I can't do that.''
>>> assert podBayDoorStatus == 'open', 'The pod bay doors need to be "open".'

• Assertions are for programmer errors, not user errors. For errors that can be recovered from
(such as a file not being found or the user entering invalid data), raise an exception instead of
detecting it with an assert statement.
Using an Assertion in a Traffic Light Simulation
• Say we’re building a traffic light simulation program. The data structure representing
the stoplights at an intersection is a dictionary with keys 'ns' and 'ew', for the
stoplights facing north-south and east-west, respectively.
• The values at these keys will be one of the strings 'green', ' yellow', or 'red'.
• The code looks like this:
market_2nd = {'ns': 'green', 'ew': 'red'}
mission_16th = {'ns': 'red', 'ew': 'green'}
• These two variables will be for the intersections of Market Street and 2nd Street, and Mission
Street and 16th Street.

• To start the project, you want to write a switchLights() function, which will take an
intersection dictionary as an argument and switch the lights.

• At first, we might think that switchLights() should simply switch each light to the next color
in the sequence: Any 'green' values should change to 'yellow', 'yellow' values should change
to 'red', and 'red' values should change to 'green'.

• Example:
def switchLights(stoplight):
for key in stoplight.keys():
if stoplight[key] == 'green':
stoplight[key] = 'yellow'
elif stoplight[key] == 'yellow':
stoplight[key] = 'red'
elif stoplight[key] == 'red':
stoplight[key] = 'green'
switchLights(market_2nd)

• But if while writing switchLights() you had added an assertion to check that at least one of
the lights is always red, you might have included the following at the bottom of the
function:
assert 'red' in stoplight.values(), 'Neither light is red! ' + str(stoplight)
Disabling Assertions

• Assertions can be disabled by passing the -O option when running Python. This is good for
when you have finished writing and testing your program and don’t want it to be slowed
down by performing sanity checks (although most of the time assert statements do not cause
a noticeable speed difference).

• Assertions are for development, not the final product. By the time you hand off your
program to someone else to run, it should be free of bugs and not require the sanity checks.

Logging

• If you’ve ever put a print() statement in your code to output some variable’s value while your
program is running, you’ve used a form of logging to debug your code.
 • Logging is a great way to understand what’s happening in your program and in what order its
happening.
• Python’s logging module makes it easy to create a record of custom messages that you write.
• These log messages will describe when the program execution has reached the logging
function call and list any variables you have specified at that point in time.
• On the other hand, a missing log message indicates a part of the code was skipped and never
executed.
Using the logging Module

• To enable the logging module to display log messages on your screen as your program runs,
copy the following to the top of your program.

• Example:
import logging
logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s %(levelname)s - (message)s')

• When Python logs an event, it creates a LogRecord object that holds information about
that event. The logging module’s basicConfig() function lets you specify what details
about the LogRecord object you want to see and how you want those details displayed.

• Example:
import logging
logging.basicConfig(level=logging.DEBUG, format=' %(asctime)s - %(levelname)s - %
(message)s')
logging.debug('Start of program')
def factorial(n):
logging.debug('Start of factorial( %)' % (n))
total = 1
for i in range(n + 1):
total *= i
logging.debug('i is ' + str(i) + ', total is ' + str(total))
logging.debug('End of factorial( %)' % (n))
return total
print(factorial(5))
logging.debug('End of program')
• Here, we use the logging.debug() function when we want to print log information. This debug()
function will call basicConfig(), and a line of information will be printed. This information will be
in the format we specified in basicConfig() and will include the messages we passed to debug().
The print(factorial(5)) call is part of the original program, so the result is displayed even if logging
messages are disabled.

• The factorial() function is returning 0 as the factorial of 5, which isn’t right. The for loop should
be multiplying the value in total by the numbers from 1 to 5. But the log messages displayed by
logging.debug() show that the i variable is starting at 0 instead of 1. Since zero times anything is
zero, the rest of the iterations also have the wrong value for total.
 • Logging messages provide a trail of breadcrumbs that can help you figure out when things started
to go wrong. Change the for i in range(n + 1): line to for i in range(1, n + 1):, and run the program
again.

Logging Levels:

• Logging levels provide a way to categorize your log messages by importance.

• There are five logging levels, described in Table 10-1 from least to most important. Messages can
be logged at each level using a different logging function.

Example
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG, format='%(asctime)s-%(levelname)s-%
(message)s')
>>> logging.debug('some debugging details.')
>>> logging.info('The logging module is working.')
>>> logging.warning('An error message is about to be logged.')
>>> logging.error('An error has occurred.')
>>> logging.critical('The program is unable to recover!')

Disabling Logging

• The logging.disable() function disables these so that you don’t have to go into your program and
remove all the logging calls by hand. You simply pass logging.disable() a logging level, and it
will suppress all log messages at that level or lower. So if you want to
disable logging entirely, just add logging.disable(logging.CRITICAL) to
your program.

• Example:
>>>import logging
>>> logging.basicConfig(level=logging.INFO, format=' %(asctime)s -%(levelname)s %(message)s')
>>> logging.critical('Critical error! Critical error!')
>>> logging.disable(logging.CRITICAL)
>>> logging.critical('Critical error! Critical error!')
>>> logging.error('Error! Error!')
 Logging to a File

• Instead of displaying the log messages to the screen, you can write them to a text file. The
logging.basicConfig() function takes a filename keyword argument.

• Example:
>>>import logging
>>>logging.basicConfig(filename='myProgramLog.txt', level=logging.DEBUG, format=' %
(asctime)s - %(levelname)s -%(message)s')
IDLE’s Debugger

• The debugger is a feature of IDLE that allows you to execute your program one line at a time. The
debugger will run a single line of code and then wait for you to tell it to continue.

• To enable IDLE’s debugger, click Debug▸Debugger in the interactive shell window. This will
bring up the Debug Control window.

• When the Debug Control window appears, select all four of the Stack, Locals, Source, and Globals
checkboxes so that the window shows the full set of debug information.

• While the Debug Control window is displayed, any time you run a program from the file editor,
the debugger will pause execution before the first instruction and display the following:
1. The line of code that is about to be executed
2. A list of all local variables and their values
3. A list of all global variables and their values

• The program will stay paused until you press one of the five buttons in the Debug Control
window: Go, Step, Over, Out, or Quit.
Go :Clicking the Go button will cause the program to execute normally until it terminates or
reaches a breakpoint.
 Step :Clicking the Step button will cause the debugger to execute the next line of code and then
pause again.
Over :Clicking the Over button will execute the next line of code, similar to the Step button.
However, if the next line of code is a function call, the Over button will “step over” the code in
the function.
Out :Clicking the Out button will cause the debugger to execute lines of code at full speed until it
returns from the current function.
Quit :If you want to stop debugging entirely and not bother to continue executing the rest of the
program, click the Quit button.

Debugging a Number Adding Program

print('Enter the first number to add:')

first = input()
print('Enter the second number to add:')
second = input()
print('Enter the third number to add:')
third = input()
print('The sum is ' + first + second + third)

Breakpoints

• A breakpoint can be set on a specific line of code and forces the debugger to pause whenever the
program execution reaches that line.

• Example:
import random
heads = 0
for i in range(1, 1001):
if random.randint(0, 1) == 1:
heads = heads + 1
if i == 500:
print('Halfway done!')
print('Heads came up ' + str(heads) + ' times.')