How to Read a File in Chunks in Golang

by

To read a file in chunks in Golang, you can use the bufio.NewReader() function. This function takes an io.Reader object as an argument and returns a new buffered reader object.

Example

package main

import (
  "bufio"
  "fmt"
  "log"
  "os"
)

func main() {
  // Open the file for reading
  file, err := os.Open("data.txt")
  if err != nil {
    log.Fatal(err)
  }
  defer file.Close()

  // Create a buffered reader
  reader := bufio.NewReader(file)

  // Read the file in 4-byte chunks
  chunkSize := 4
  for {
    // Read the next chunk
    chunk := make([]byte, chunkSize)
    n, err := reader.Read(chunk)
    if err != nil && err.Error() != "EOF" {
      log.Fatal(err)
    }

    // Print the chunk
    fmt.Printf("%s", chunk[:n])

   // Check for EOF
    if err == nil && n < chunkSize {
      break
    }
  }
}

Output

This is a text file
We will read an entire file
Hello World

In this example, we first open the file for reading using os.Open() function and defer its closure. Then, we create a buffered reader using bufio.NewReader() and specify the size of the chunk we want to read.

Next, we enter a loop where we read the file in chunks using the reader.Read() function.

The Read() function returns the number of bytes read and an error. If the error is not nil and not equal to EOF, we use the log.Fatal() function to print the error and exit the program.

We then print the chunk using fmt.Printf() function and check if we have reached the end of the file by checking if err is nil and n are less than chunkSize. If we reach the end of the file, we break out of the loop.

Note that you can adjust the chunk size to read smaller or larger portions of the file at a time, depending on your needs.

That’s it.

Leave a Comment