Naive approach: Manual Accumulator
The naive approach to chunking text or data streams often relies on a manual accumulator.
# The Bottleneck: High state, fragile edge cases
chunk, chunks = [], []
for word in words:
chunk.append(word)
if len(chunk) == chunk_size:
chunks.append(chunk)
chunk = chunk[-overlap:] # Manual overlap tracking is brittle
if chunk:
chunks.append(chunk) # Fragile final chunk handlingThis control flow is highly stateful, and breaks easily under edge cases.
Cleaner pattern: Sliding Window
Instead of building chunks manually, treat the sequence as a timeline and step through it using slicing. The overlap and the final partial chunk handle themselves naturally.
step = chunk_size - overlap
for start in range(0, len(words), step):
chunk = words[start : start + chunk_size]With chunk_size=3, overlap=1, step=2:
words = ["one", "two", "three", "four", "five", "six"]
words[0:3] # one two three
words[2:5] # three four five
words[4:7] # five six