Python Pandas - Fixed-Width Text File

---

---

A fixed-width text file is a text file that stores the tabular data arranged in columns with a fixed number of characters, meaning each field occupies a specific width. Unlike delimited files (e.g., CSVs), these files rely on consistent column widths rather than separators like commas. This file format is commonly used in older systems or specific applications, where each piece of data starts and ends at specific positions.

In Python, the Pandas library offers a convenient method called read_fwf() for handling such file formats. This method allows you to read such data efficiently into DataFrames for analysis. In this tutorial, we will learn how to work with fixed-width text file using Pandas.

Pandas read_fwf() Method

The read_fwf() method in Pandas is designed for reading fixed-width text files. Its works like the Pandas read_csv() method, but instead of reading delimited data, it works with files where each column has a fixed width.

This method returns a Pandas DataFrame or a TextFileReader when iterator=True.

Following is the syntax of the this method −

pandas.read_fwf(filepath_or_buffer, *, colspecs='infer', widths=None, **kwds)

Key parameters are −

• colspecs: A list of tuples specifying the start and end positions of each column. Set to 'infer' by default, which tries to deduce column widths from the first 100 rows.

• widths: A list of integers specifying the width of each column. Use this for contiguous columns as an alternative to colspecs.

• delimiter: Characters considered as fillers in the file, e.g., spaces or special characters like ~.

• dtype: Allows manual specification of column data types.

• **kwargs: Additional optional keyword arguments passed to TextFileReader.

Example

Here is a basic example demonstrating reading a fixed-width file using the pandas read_fwf() method with column specifications. In this example we explicitly define the column intervals, using the colspecs parameter.

import pandas as pd

# Sample fixed-width data saved to a file
data = """
      Name   Gender   Age
    Braund     Male    30
   Cumings   Female    25
 Heikkinen   Female    35
       Tom     Male    28
       Ram     Male    25"""

# Writing data to a file
with open("sample_fwf.txt", "w") as file:
    file.write(data)

# Specifying column intervals
colspecs = [(0, 11), (11, 20), (20, 26)]

# Reading the fixed-width file
df = pd.read_fwf("sample_fwf.txt", colspecs=colspecs)

print("DataFrame from fixed-width file with specified column intervals:")
print(df)

When we run above program, it produces following result −

DataFrame from fixed-width file with specified column intervals:

	Name	Gender	Age
0	Braund	Male	30
1	Cumings	Female	25
2	Heikkinen	Female	25
3	Tom	Male	28
4	Ram	Male	25

Inferring Column Specifications

Pandas read_fwf() method can automatically infer column positions if the data is aligned and separated by white space or other delimiters.

Example

The following example demonstrates using the read_fwf() method for automatically inferring the column specifications with the default parameter settings.

import pandas as pd

# Reading the fixed-width file with default parameter settings
df = pd.read_fwf("sample_fwf.txt")

print("DataFrame from fixed-width file:")
print(df)

Following is an output of the above code −

DataFrame from fixed-width file:

	Name	Gender	Age
0	Braund	Male	30
1	Cumings	Female	25
2	Heikkinen	Female	25
3	Tom	Male	28
4	Ram	Male	25

Handling Data Types in Fixed-Width File

You can also manually specify column data types while parsing the data from a fixed-width file using the dtype parameter of the read_fwf() method.

Example

In this example, the Age column is explicitly parsed as a float type instead of the default int type.

import pandas as pd

# Reading the fixed-width file with specific data types for columns
df = pd.read_fwf("sample_fwf.txt", dtype={"Age": "float"})

print("DataFrame from fixed-width file:")
print(df)

print("\nData Types of Each Column:")
print(df.dtypes)

While executing the above code we get the following output −

DataFrame from fixed-width file:

	Name	Gender	Age
0	Braund	Male	30
1	Cumings	Female	25
2	Heikkinen	Femate	25
3	Tom	Male	28
4	Ram	Male	25

Data Types of Each Column: Name object Gender object Age float64 dtype: object

Reading the fixed-width file with a Custom Delimiter

The read_fwf() method allows you to read the fixed-width text file with custom delimiter using the delimiter parameter. This is useful for reading a fixed-width text file containing the delimiter other than white spaces.

Example

This example demonstrates reading a fixed-width file containing a custom delimiter other than the white space using the delimiter parameter. Here the "-" filler character(delimiter) will be ignored.

import pandas as pd

# Sample fixed-width data saved to a file
data = """
Name-------Gender---Age
Braund-----Male-----30
Cumings----Female---25
Heikkinen--Female---35
Tom--------Male-----28
Ram--------Male-----25"""

# Writing data to a file
with open("sample_fwf.txt", "w") as file:
    file.write(data)

# Reading the fixed-width file with Custom Delimiter
df = pd.read_fwf("sample_fwf.txt", delimiter="-")

print("DataFrame from fixed-width file with with custom delimiter:")
print(df)

Following is an output of the above code −

DataFrame from fixed-width file with with custom delimiter:

	Name	Gender	Age
0	Braund	Male	30
1	Cumings	Female	25
2	Heikkinen	Femate	25
3	Tom	Male	28
4	Ram	Male	25

Iterating the Fixed-Width File Data

By specifying a chunksize or iterator=True parameters of the read_fwf() method, you can get an iterable object of type TextFileReader for iterating the data of a fixed-width file data in chunk.

Example

This example shows parsing the fixed-width file data in chunks by specifying the chunksize parameter. Here chunk size represents number of lines to read per chunk for iteration.

import pandas as pd

# Reading the file in chunks
chunk_size = 1
chunks = pd.read_fwf("sample_fwf.txt", chunksize=chunk_size)

print("Iterating through Fixed-Width File Chunk by Chunk:")
for chunk in chunks:
    print(chunk)

When we run above program, it produces following result −

Iterating through Fixed-Width File Chunk by Chunk:

	Name	Gender	Age
0	Braund	Male	30

	Name	Gender	Age
1	Cumings	Female	25

	Name	Gender	Age
2	Heikkinen	Femate	25

	Name	Gender	Age
3	Tom	Male	28

	Name	Gender	Age
4	Ram	Male	25