Getting Data In

Limit on number of open files when reading syslog-ng logfiles

Parameter
Explorer

Hi, I like the method of indexing files as they appear in the syslog-ng log directory where each host goes to a different file. It makes it easier to search by source when setting source as a part of the path and saving each file in a directory names "something/$HOST".

The problem I see is if we end up having 10000 hosts logging on this syslog-ng, we may end up having an open file limit on the os or the app.

Anyone knows of a hard limit on open files OR some soft limits parameters (memory, throughput, etc) that would cause problems and what those limits are?

I'd like to have both the Linux and Windows point of view if you have any experience with one of these.

Thanks

Tags (1)

shaggybark
New Member

I have been using syslog-ng as the log collector and Splunk as the reporter for many years and it has worked well. Many years ago, I ran into a problem with too many open file descriptors from the 32 bit syslog-ng (64 bit would have more). I had to add this to the start up script for the syslog-ng service to get past a 256 bit FD limitation:

LDPRELOAD32=/usr/lib/extendedFILE.so.1 export LDPRELOAD32

Default file descriptors increased for syslog-ng

plimit -n 65535 $$ plimit -s unlimited $$

Linux would have a different solution & 64 bit would be a better way to go.

For the host extract, I just add this to the inputs.conf file: host_regex = ([^/]+).log$

0 Karma

Parameter
Explorer

Well, after much reading, here it is.

GOAL: Assigning the proper host value when reading from a directory containing syslog-ng logfiles which contain containing more than one host. This could also be used for a syslog input, suffice to change the source statement in props.conf to [syslog]

in $SPLUNK_INSTALLDIR/etc/system/local/transforms.conf

[syslog-ng_localfile]
DEST_KEY = MetaData:Host
REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s(\w)[.*$
FORMAT = host::$1


in $SPLUNK_INSTALLDIR /etc/system/local/props.conf

[source::/MY PATH TO LOGFILES/*.log]
TRANSFORMS-syslog_hostname=syslog-ng_localfile
SHOULD_LINEMERGE=false

Enjoy!

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Splunk will not have an issue with the number of file descriptors. It opens and closes files as necessary to be able to process the inputs. There are a couple of settings related to this.

http://www.splunk.com/base/Documentation/latest/Admin/Limitsconf

max_fd = <integer>
* Maximum number of file descriptors that Splunk will keep open, to capture any trailing data from 
files that are written to very slowly.
* Defaults to 100.

http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf

time_before_close = <integer>
* Modtime delta required before Splunk can close a file on EOF.
* Tells the system not to close files that have been updated in past <integer> seconds.
* Defaults to 3.

Splunk assumes that not every log file will be updated at a very high frequency. It also remembers its last file offset (seekptr) into the file - so that when the file gets "bigger" than the previous seekptr it can see to where it last read, and continue from there. If a file is not updated in time_before_close seconds, and there are more than max_fd files open, then some files will be closed so new ones can be opened and processed.

One important thing to note is that the number of files in the tree affects how quickly Splunk can 'get back around' and pick up updates to a file. For example, if you have a syslog tree with 100 hosts, each with 10 files per day, on a 30 day retention in the syslog tree - that's 30,000 files for Splunk to check for updates. Extrapolate that to 10,000 hosts and you see the issue. Splunk continues to work, but your updates to 'current' files are slowed down by rechecking 'historical' files.

Best practice here is to separate your 'active' files from your 'historical' ones. Some folks accomplish this with a symlink tree that gets built via a cron job daily. The symlink tree has a link to each 'current' file and one day's 'historical' files for yesterday. (The one day's worth of historical files is to help with midnight rollover.) You point splunk at the symlink tree, and it processes those current files much more quickly without the dead weight of rechecking historical files.

ajs07635
Explorer

There is a limit to the number of file descriptors/handles a process can have open. You can find out by running the follow cmd:

ulimit -Hn

You can raise the maximum value that way as well.

In windows, the value is set in the registry (of course!):

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session\Manager\Subsystems\Windows

However, I've got to ask, why would you log each host to its own file and not just one? Syslog will send the host name of the source host, and you can filter by hostname in splunk by using hostname=. Then you won't have to worry about running out of file descriptors (increasing the limit may cause other problems). Proper use of logrotate will help prevent this file from getting too large and splunk shouldn't skip a beat indexing from the files even when they are rotated.

0 Karma

Parameter
Explorer

Well, if I log to one file, the "source" is always the same. Hence searching using host= or src= is kind of useless. I may be wrong tough... I'd rather be.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...