I recently needed to convert text files created by one application into a completely different format, to be processed by a different application.

In some cases, the files were too large to be read into memory all at once (without causing resource problems). Here “too large” meant tens or hundreds of gigabytes.

The source files also did not always have line terminators (some did; most did not). Instead they had separately defined record layouts, which specified the number of fields in a record, together with the field terminator character.

I therefore needed to (a) find a way to read a source file as a stream of characters, and (b) define how the program would identify complete records, by counting field terminators.

The heart of the solution, written in Python 3, was based on the following:

1
2
3
4
5
6
7
8
9

with open('in.txt', 'r', encoding='windows-1252') as inFile, \
    open('out.txt', 'w', encoding='utf-8') as outFile:

    callable = lambda: inFile.read(4096) # characters
    sentinel = ''

    for chunk in iter(callable, sentinel):
        for ch in chunk:
            # conversion logic here

So, what is iter(callable, sentinel)?

The documentation introduces the functionality:

iter(object[, sentinel])

If the second argument, sentinel, is given, then object must be a callable object. The iterator created in this case will call object with no arguments for each call to its __next__() method; if the value returned is equal to sentinel, StopIteration will be raised, otherwise the value will be returned.

What is a “callable object” in Python? Anything which can be called - such as a function, a method and so on.

Why is a callable object needed? Because we want the Python built-in function iter() to iterate over something which is not naturally iterable - a file, in this case, with no line terminators. Another example, would be a file containing binary data.

Therefore we have to wrap our source data in something which can be treated as if it were iterable.

Using a lambda expression for our callable is a convenient and succinct piece of syntax. It lets us read the file one chunk at a time (each chunk containing 4,096 characters).

Regarding the sentinel, the technique used in the above sample code sets the sentinel to an empty string - and therefore the entire chunk of data is processed in one pass, since the sentinel is never encountered.

Each resulting chunk is a string of text, and a string is naturally iterable in Python. We can therefore use for ch in chunk to iterate over that string, one character at a time.

So, instead of reading the entire file into memory all at once, we only need to read in much smaller chunks, one after another.

Acknowledgement: The solution is based on information taken from the following SO question and its answers:

What are the uses of iter(callable, sentinel)?

Below is the full solution, which shows how the script counts field terminators.

Note that while the input file uses field terminators (a control character at the end of every field), the output file is written with field separators (a control character in between each field) together with a line terminator (the usual Linux ‘\n’ control character).

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109

#!/usr/bin/python3

#
# Convert file from one format to another for:
#  - line endings
#  - field separators
#  - double-quotes for strings
#  - reformatted dates
#  - file encoding
#

# ----------------------------------------------------

# Input file specs:

sample_spec = {
    'id': 'input_file',
    'fieldsPerRec': 6,
    'inFieldSep': '\x01',
    'inRecordSep': '\x0a',
    'stringFields': [2,3,4],
    'dateFields': [5]
}

# ----------------------------------------------------

inFileEncoding = 'windows-1252'
outFileEncoding = 'utf-8'

chunkSize = 4096 # characters

def convert(fileSpec, fileName):
    field = []
    record = []
    fieldCount = 0
    inRecordCount = 0
    outRecordCount = 0

    with open(inPath + fileName, 'r', encoding=inFileEncoding) as inFile, \
         open(outPath + fileName + outSuffix, 'w', encoding=outFileEncoding) as outFile:

        callable = lambda: inFile.read(chunkSize)
        sentinel = ''

        for chunk in iter(callable, sentinel):
            for ch in chunk:

                if ch == '\x02' or ch == '\n':
                    pass
                elif ch == fileSpec['inFieldSep']:
                    fieldCount += 1
                    if len(field) > 0 and fieldCount in fileSpec['stringFields']:
                        record.append('"' + ''.join(field) + '"')
                    elif len(field) > 0 and fieldCount in fileSpec['dateFields']:
                        record.append(formatDateTime(''.join(field)))
                    else:
                        record.append(''.join(field))
                    field = []
               else:
                    field.append(ch)

                if fieldCount == fileSpec['fieldsPerRec']:
                    outFile.write(outFieldSep.join(record))
                    outFile.write(outRecordSep)
                    record = []
                    fieldCount = 0
                    inRecordCount += 1
                    outRecordCount += 1

    print('')
    print('file: %s' % fileName)
    print('  source records: %d' % inRecordCount)
    print('  target records: %d' % outRecordCount)

def formatDateTime(date):
    return 'date[0:4] + '-' + date[4:6] + '-' + date[6:8] + ' 00:00:00'

def convertFiles(inFileSpec):
    for suffix in inFileSuffixList:
        convert(inFileSpec, inFileSpec['id'] + suffix)

#
# -----------------------------------------------------------
#

# these are common to all output files:
outSuffix = '_converted'
outFieldSep = '\x0b'
outRecordSep = '\x0a'

# ----------

#
# Output files are generated here using the file
# specs created above:
#

# ----------

inPath = '/home/infiles/'
outPath = inPath + 'converted/'
inFileSuffixList = ['.txt']

convertFiles(sample_spec)


# ----------

print('')