Continuous RTSP stream recording with ffmpeg. Can I optimize it to not damage the SD card?

Question

I have got an IP camera with RTSP stream. I decided to record the stream using ffmpeg (version 3.2.14-1~deb9u1) on the Odroid-N2 device with Armbian. I have created a .sh script which is launched by Croon every minute. It checks if the ffmpeg recording is active for selected camera and also deletes files older than 7 days:

#!/bin/bash
RecordPathAndFind='/home/mona/CameraRecordings/Camera1/'
SegmentTime=900
MinutesAfterDeleteOldFiles=10080
DaysSecurityLimit=31

tempoutput=$(ps aux | grep ffmpeg | grep $RecordPathAndFind)
if [ ${#tempoutput} -ge 5 ];
then
      echo "FFMPEG with record command is already running. Skipping.\n"
else
      echo "FFMPEG with record command is not running. Starting now...\n"
      FFMPEGSTART=$(su - mona -c "cd /home/mona; /usr/bin/screen -dmS ffmpegcamera1 ffmpeg -rtsp_transport udp -i 'rtsp://admin:password@192.168.1.xxx:554/onvif1' -vcodec copy -c:a aac -map 0 -f segment -segment_time $(echo $SegmentTime) -segment_format mp4 -strftime 1 $(echo $RecordPathAndFind)%Y-%m-%d_%H-%M-%S.mp4")
fi

currenthourminutes=$(date +%M)
if [ "$currenthourminutes" == "00" ]; then
    echo "Current Minute is 00. Checking for old files to delete.\n"
    FILESDELETECOMMAND=$(find $(echo $RecordPathAndFind) -maxdepth 1 -type f -mmin +$(echo $MinutesAfterDeleteOldFiles) -mtime -$(echo $DaysSecurityLimit) -name '*.mp4' -ls -exec rm {} \;)
else
    echo "Current Minute is NOT 00. Skipping.\n"
fi

The script works fine, but I'm afraid of the life of the SD card in this device. I detected that ffmpeg is continuously writing the mp4 file (file size grows all the time). I think it would be better if ffmpeg waited for ~1MiB before flushing it to disk.

I tried different ffmpeg settings (adding "blocksize", "flush_packets", "reorder_queue_size" and few others I can't recall now), unfortunately it didn't change anything. The mp4 file size was increasing all the time (even by few KBs).

The first question is: Should I be worried about the microSD card life when ffmpeg is all the time writing the file (current environment)?

The second question is: Are there any other ffmpeg optimization settings which I missed?

If my fears regarding the SD card life are correct, can you please recommend some other things which I could add to my script (or change in system) in order to increase the life of micro SD cards (or USB sticks) used for recording purposes? I was thinking about using the ramdisk, and then manually move the files, however this could cause files loss in case of system reboot, hang, or power outage.

EDIT: Here is the command I use to check file size changes:

root@odroidn2:/home/mona/CameraRecordings/Camera1# printf "`date +"%m-%d-%y_%H:%S:%N"` `echo "Bytes:"` `stat --printf "%s\n" 2019-07-10_18-01-00.mp4`\n"
07-10-19_18:28:616623697 Bytes: 28443805
07-10-19_18:29:497492966 Bytes: 28453943
07-10-19_18:29:811378969 Bytes: 28458099
07-10-19_18:30:162532642 Bytes: 28724960
07-10-19_18:30:513455929 Bytes: 28730146
07-10-19_18:30:832042221 Bytes: 28734694
07-10-19_18:31:204934593 Bytes: 28739202
07-10-19_18:31:587997659 Bytes: 28744450
07-10-19_18:32:056139192 Bytes: 28750415
07-10-19_18:32:490959812 Bytes: 28755253
07-10-19_18:32:836352316 Bytes: 28757934
07-10-19_18:33:209513158 Bytes: 28762371
07-10-19_18:33:595689070 Bytes: 29021552
07-10-19_18:33:970968755 Bytes: 29026202
07-10-19_18:34:412742761 Bytes: 29031995
07-10-19_18:34:825309867 Bytes: 29037579
07-10-19_18:35:219852935 Bytes: 29043515
07-10-19_18:35:598878409 Bytes: 29046993
07-10-19_18:35:994067048 Bytes: 29051547
07-10-19_18:36:383952621 Bytes: 29055235
07-10-19_18:36:822644535 Bytes: 29331875
07-10-19_18:37:233137175 Bytes: 29336695
07-10-19_18:37:617277956 Bytes: 29343653
07-10-19_18:38:022776087 Bytes: 29348487
07-10-19_18:38:630347795 Bytes: 29357002
07-10-19_18:39:304204108 Bytes: 29364381

As I can see the file size increased by 10138 bytes (10 kilobytes) within 1 second.

ffmpeg should not be writing every few KB - it should be every 256 kB except when writing the header at the start and the closing writes at the end. — Gyan, Jul 10 '19 at 05:03
Thanks for your comment. I updated the question and added the file size changes log. Seems like it increased by 10 KB within 1 second. — Mona, Jul 10 '19 at 16:15

score 0 · Answer 1 · answered Jul 03 '22 at 19:40

I solved my problem by using a self-written script, executed by the croon:

* * * * * /home/mona/mona_ffmpeg_camera1_record.sh

#!/bin/bash
RecordPathAndFind='/home/mona/CameraRecordings/Tmps_Camera1/'
RecordPathCopyTo='/srv/dev-disk-by-uuid-XXX/Camera1_Recordings/'
#900 = 15 mins
SegmentTime=900
#7 days = 10080
MinutesAfterDeleteOldFiles=10080
DaysSecurityLimit=31
DeleteOldFiles=false
MakeRamDisk=true
CopyFilesToHDD=true

tempoutput=$(ps aux | grep ffmpeg | grep $RecordPathAndFind)
if [ ${#tempoutput} -ge 5 ];
then
      echo "FFMPEG with record command is already running. Skipping.\n"
      #Check if ffmpeg didn't hang:
      PIDOfFFmpeg=$(ps aux | grep ffmpeg | grep $RecordPathAndFind | grep -v 'SCREEN' | awk '{print $2}')
      echo "Found existing ffmpeg PID: $PIDOfFFmpeg.\n"
      #ModificationTime=$(stat -L -c %Y /proc/$PIDOfFFmpeg/fd/4);
      ModificationTime=$(stat -L -c %Y /proc/$PIDOfFFmpeg/fd/2);
      echo "Last File modification time: $ModificationTime.\n"
      if [ $ModificationTime -le $(($EPOCHSECONDS - 60)) ];
      then
        echo "Very old modification time. Restarting ffmpeg at next run.\n"
        kill $PIDOfFFmpeg
        #Backup existing files:
        FILES_TO_COPY=$(ls -lt "$RecordPathAndFind" | awk '{ print $9 }' | tail -n +2)
        FILES_COUNT=0
        for f in $FILES_TO_COPY; do
          echo "Currently processing file: $f"
          FILES_COUNT=$FILES_COUNT+1
          if [[ $FILES_COUNT -gt 0 ]]; then
             echo "File not skipped, copying..."
             cp "$RecordPathAndFind$f" "$RecordPathCopyTo"
             echo "Removing file..."
             rm "$RecordPathAndFind$f"
          fi
        done
      fi
    
      #test `stat -L -c %Y /proc/$PIDOfFFmpeg/fd/4` -ge $(($EPOCHSECONDS - 60)) || kill $PIDOfFFmpeg
else
      if [ "$MakeRamDisk" = true ] ; then
      echo "Mounting Ramdisk...\n"
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      umount -A $(echo $RecordPathAndFind)
      mount -t tmpfs -o size=1024m tmpfs $(echo $RecordPathAndFind)
      fi
      
      if ping -c 1 192.168.1.111 &> /dev/null
      then
      echo "FFMPEG with record command is not running. Starting now...\n"
      FFMPEGSTART=$(su - mona -c "cd /home/mona; /usr/bin/screen -dmS ffmpegcamera1 ffmpeg -rtsp_transport udp -i 'rtsp://admin:password@192.168.1.111:554/onvif1' -vcodec copy -c:a aac -map 0 -use_wallclock_as_timestamps 1 -f segment -segment_time $(echo $SegmentTime) -segment_format mp4 -strftime 1 $(echo $RecordPathAndFind)%Y-%m-%d_%H-%M-%S.mp4")
      #echo $FFMPEGSTART
      else
      echo "ERROR. Pinging 192.168.1.111 failed. Will retry in 1 minute...\n"
      fi
fi

currenthourminutes=$(date +%M)
if [ "$DeleteOldFiles" = true ] ; then
if [ "$currenthourminutes" == "00" ]; then
    echo "Current Minute is 00. Checking for old files to delete.\n"
    FILESDELETECOMMAND=$(find $(echo $RecordPathAndFind) -maxdepth 1 -type f -mmin +$(echo $MinutesAfterDeleteOldFiles) -mtime -$(echo $DaysSecurityLimit) -name '*.mp4' -ls -exec rm {} \;)
else
    echo "Current Minute is NOT 00. Skipping.\n"
fi
fi

if [ "$CopyFilesToHDD" = true ] ; then

SIZE=$(du -s -B1 "$RecordPathAndFind" | cut -f1)
echo "Folder size is: $SIZE"
#900MiB
COPY_WHEN_SIZE_BYTES=943718400
#COPY_WHEN_SIZE_BYTES=401219456

if [[ $SIZE -gt $COPY_WHEN_SIZE_BYTES ]]; then
echo 'Folder Size is greater than: '$COPY_WHEN_SIZE_BYTES' bytes. Copying...'
FILES_TO_COPY=$(ls -lt "$RecordPathAndFind" | awk '{ print $9 }' | tail -n +2)
FILES_COUNT=0
for f in $FILES_TO_COPY; do
  echo "Currently processing file: $f"
  FILES_COUNT=$FILES_COUNT+1
  if [[ $FILES_COUNT -gt 1 ]]; then
     echo "File not skipped, copying..."
     cp "$RecordPathAndFind$f" "$RecordPathCopyTo"
     echo "Removing file..."
     rm "$RecordPathAndFind$f"
  fi
done
fi

fi

It works in the following way:

The script detects if another instance is already running (this should be improved but is enough for me now).
It records 15 minutes parts to the ram disk (I set around 1GB, because my device has 4GB RAM).
When the ram disk is nearly full, it copies files from it to an old 2.5'' HDD (connected by USB), with power saving enabled, so it spins only for a few seconds, every few hours.
This might cause data loss in the ram disk, in case of electricity outage, but I prefer this method because it's very quiet and the spinning disk doesn't annoy me at nights :)

I've been using this method for around 2 years now, and I'm happy because I didn't damage any SD card.

Continuous RTSP stream recording with ffmpeg. Can I optimize it to not damage the SD card?

1 Answers1