Tuesday, July 8, 2008

Why its important to rotate logs.

Any of you who have been in the sysadmin business for more then a few days know how critical it is to rotate log files. When logs get to big your system will stop working. For example each file system has a maximum file size (http://www.novell.com/documentation/suse91/suselinux-adminguide/html/apas04.html#tab:maxsize) which if reached will cause problems. In the old days linux had a 2 gig ceiling and when your apache access log reached it apache would crash and not come back up. Some times worse.

Today I ran into a new one, in Solaris there is a limit to the number of sub directories that can be created. The magic number is none other than 32,767. We had a process that was storing log files for in there own sub directories. It had been doing this for multiple years and finally it had the lmit. We got this rather unhelp error, mkdir: Failed to make directory "test3"; Too many links.

The solution was rather simple. Tar up and archive the folders and then delete the originals. I set up a quick script to do this monthly and hopefully we will be all set. Going forward I would like to get a more robust solution in place like the standard logrotate. Dare to dream.

No comments: