0

Brief
We are running our workloads in kubernetes on AWS EKS. All our applications write the logs in /var/log/app_logs directory using hostPath. (yes it's not a best practice, this is due to legacy codebase, eventually we want to move to STDOUT).

Problems we face:

  1. We rely on the hourly logrotate to rotate the files under /var/log/app_logs
  2. During the hourly boundary, we tend to lose the loglines
  3. Upon investigating seems like the copytruncate is creating this datalos
  4. Looks like the application is not able to write to the logfile during the logrotation and we are losing those logs during the rotation.
  5. Since these services are running in kubernetes containers/pods, we won't be able to use the stop service, rotate and start the service approach.

The EKS worker nodes are running in Amazon Linux(centos).

The logrotate.conf we use:

"/var/log/app_logs/*.log" {
  create 664 foo foo
  daily
  missingok
  copytruncate
  rotate 7
  compress
  delaycompress
  notifempty
  sharedscripts
  postrotate
    /bin/bash /opt/scripts/upload-logs.sh
  endscript
  }

Is there anything that we could do to avoid this dataloss by tweaking the logrotate config? or is there any similar tools that solves this problem in kubernetes based environments.

dm90
  • 101
  • 3
  • https://serverfault.com/questions/189477/rotate-logs-of-a-dumb-non-interactive-application/189880#189880 – user9517 Feb 16 '22 at 12:36
  • @user9517 Thanks for your response! But the problem in this (our) setup is, we don't have just couple of process in linux server. All these logs are from kubernetes pods. We may not be able to send the kill signal to the pod and in that case it would make it more complex by killing the pod and eventually it will get rescheduled somewhere else. – dm90 Feb 16 '22 at 12:51

0 Answers0