How to split a file in specific size and pass that to the named pipe

Question

I have a file of size 2GB and want to read that in chunks and pass each chunk to named pipe.

Eg;

mkfifo /tmp/pipe
tail -c 200 myfile > /tmp/pipe &  #last 200 bytes of file in the pipe.
use the /tmp/pipe.

I want to pass some data of fixed known size to the named pipe and then pass that pipe to the lftp. Now, once this is done , then again load next bytes in the pipe and perform lftp operation.

How to read say 5MB(or any size, say x) from a file and then pass that to pipe and then use that pipe and then proceed to next 5MB and continue till end of file is reached.

Opening a pipe will be blocked until the other end is opened as well, and writing to the pipe will be blocked when the buffer is full. You should probably use a normal file instead of a pipe. — Bodo, Aug 10 '21 at 15:52

Kamil Maciorowski · Answer 1 · 2021-09-30T06:48:43.607

A straightforward solution is to use split --filter=….

--filter=COMMAND
write to shell COMMAND; file name is $FILE

^(source)

The --filter option is not portable, hopefully your split supports it. split from GNU coreutils does. The option takes a shell command. My tests with GNU split indicate the tool uses the shell specified in $SHELL (or sh if unset).

The shell command may be quite complex. Alternatively you can invoke an arbitrarily complex script (--filter=/path/to/script.sh). The command or the script will be invoked once for each piece, the piece will enter it via its stdin. If you can read from stdin then you don't need an additional (named) pipe.

If your OS provides /dev/stdin then I believe in your case the shell command may be:

lftp … -e "put /dev/stdin -o $FILE"

where you should replace … with the address of the server and whatever else is needed to match your needs. I admit I have only superficially tested the command, it comes from this answer.

Your split command will then look like this:

split -b 5242880 --filter='
   lftp … -e "put /dev/stdin -o prefix-$FILE"
' myfile

Note $FILE must not be expanded by the local shell, it should be single-quoted (and it is, see this question). split will run inner shells with the right respective value of $FILE in the environment. In the context of the inner shell $FILE must not be single-quoted.

The command will create files with names starting with prefix- on the server. Unless the locale there is weird, the names will sort as they should, so this command:

cat prefix-* > myfile

invoked in a shell on the server (after the local split successfully sends all the pieces) will create a copy of the original file there. If you suspect the locale is weird (or if you simply want to be on the safe side) then use:

(LC_COLLATE=C; cat prefix-* > myfile)

How to split a file in specific size and pass that to the named pipe

1 Answers1