Error in 'SearchParser': Hadoop Ops App

[manus2]

Hi,

We are getting this error after installing "Hadoop ops" app.

Error in 'SearchParser': Could not find macro 'hadoop_mr_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_mr_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_task_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_task_summary_table'.
Error in 'SearchParser': Could not find macro 'hadoop_job_summary_table' that takes 0 arguments. Expecting stanza name 'hadoop_job_summary_table'.

We had installed splunk_for_hadoopops & Splunk_TA_hadoopops in /opt/splunk/etc/apps

We had added /opt/splunk/etc/apps/splunk_for_hadoopops/local/inputs.conf with the help of inputs.conf.cdh3.example.

Edited /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf with correct and splunk port and password.

But still we are getting the error.

Any thoughts?

Thanks

[lyuanlai_splunk · Accepted solution]

a quickfix worth trying is running this search query in 'HadoopOps'->'Search'

"|savedsearch __generate_lookup_hadoop_host2hdfs"

You should see a message like "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'". If this works, same trick for host2mapred:

"|savedsearch __generate_lookup_hadoop_host2mapred"

[lyuanlai_splunk]

Please manually set "disabled = 0" for hadoopmon_cpu.sh and restart splunkd. Hopefully in about 5 minutes hadoop_host2maxcpu lookup would be generated

[manus2]

1. Getting CPU events
2. Getting cpu related o/p
3. Not enabled ./hopsconfig.sh --auth admin:pass@123 --enable-all throwing error

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh
enable failed

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_df.sh
enable failed

enabling /opt/splunk/etc/apps/Splunk_TA_hadoopops/bin/hadoopmon_dfsreport.sh
enable failed

[lyuanlai_splunk]

This one depends on the scripted input 'hadoopmon_cpu.sh' be invoked and run correctly.

1) check if 'source=cpu' returns events
2) check if executing the script 'Splunk_TA_hadoopops/bin/hadoopmon_cpu.sh' in a terminal returns data like

CPU pctUser pctNice pctSystem pctIowait pctIdle
all 1.52 0.00 2.02 0.00 96.46
0 1.01 0.00 2.02 0.00 96.97
1 2.00 0.00 2.00 0.00 96.00

3) check with 'hopsconfig.sh --list-all' whether hadoopmon_cpu.sh is enabled

[manus2]

I manually indexed the mapred-site.xml

When running '|savedsearch __generate_lookup_hadoop_host2mapred' we are getting:
"Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2mapred.csv"

Now it is showing
Lookup table 'hadoop_host2maxcpu' is empty.

[lyuanlai_splunk]

I suspect the mapred-site.xml that is needed to create the hadoop_host2mapred lookup is not being indexed by Splunk. The best advice I can give right now without spamming this thread anymore is to verify that mapred-site.xml is being indexed by Splunk.

[manus2]

1. it was generated automatically and I restarted splunk
2. same error when running '|savedsearch __generate_lookup_hadoop_host2mapred'

No results. Created empty file 'hadoop_host2mapred'
No matching fields exist

[lyuanlai_splunk]

The 3 enabling failures were probably caused by those files/directory not existed yet.

Now with local/inputs.conf generated,
1) do you see monitor:///usr/lib/hadoop/conf/*.xml enabled? If not, please manually add the corresponding entry (suitable for your cluster) to local/inputs.conf and restart splunkd. (see the example above)

2) is hadoop_host2mapred lookup still empty? if so, does running '|savedsearch __generate_lookup_hadoop_host2mapred' fix the problem?

[manus2]

1. deleted eventgen.conf
2. tried step 3 mentioned in that link by deleting .introspect file & inputs.conf
3. .introspect file has been created after restarting splunk
4. but ./hopsconfig.sh --auth : --enable-all gives output like: enabling /usr/lib/hadoop/logs/hadoop*jobsummary* enable failed

enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-datanode- user-ThinkCentre-M70e*
enable failed

enabling /usr/lib/hadoop-0.20/logs/hadoop-hdfs-secondarynamenode-user-ThinkCentre-M70e*
enable failed

1. I did restart whenever I modify inputs.conf

[lyuanlai_splunk]

1) You can safely delete eventgen.conf. It is not used.
2) Can you try the steps here http://docs.splunk.com/Documentation/HadoopOps/latest/HadoopOps/DeployandlaunchTA#Update_forwarders_... and see if it fixes the problem?
3) To be cautious... did you restart splunkd after making changes to inputs.conf?

[manus2]

There is some stanza related issue is coming up in every restart of splunk after installing this app.

            Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 40: splunkPort  =  8989
            Possible typo in stanza [sample.jmx.tasktracker.Hadoop] in /opt/splunk/etc/apps/Splunk_TA_hadoopops/default/eventgen.conf, line 42: splunkPass  =  $p1unK_Lab

I retried after changing the splunk port & password with 8089 and splunk password. But it remains same.

[manus2]

I added

[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

in /opt/splunk/etc/apps/Splunk_TA_hadoopops/local/inputs.conf

But no luck.

[lyuanlai_splunk]

For reference, below is a working example of local/inputs.conf of CDH3 namenode.

[monitor:///usr/lib/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

[monitor:///var/log/hadoop/hadoop-cmf-hdfs1-NAMENODE-cdh1.tw.splunk.com*.out]
disabled = 0
sourcetype = hadoop_namenode
index = hadoopmon_logs

[lyuanlai_splunk]

I should make a correction here, check if xml files in your HADOOP_CONF_DIR is being monitored in Splunk_TA_hadoopops/local/inputs.conf, something like

[monitor:///etc/hadoop/conf/*.xml]
crcSalt =
disabled = 0
sourcetype = hadoop_global_conf
index = hadoopmon_configs

However, I just remembered you have host2hdfs run correctly, which means it should be there.

[manus2]

I am reinstalling the app. Copied "SA-HadoopOps, splunk_for_hadoopops & Splunk_TA_hadoopops" to /opt/splunk/etc/apps/ and restarted the splunk.

./Splunk_TA_hadoopops/default/inputs.conf
./SA-HadoopOps/default/inputs.conf

in which inputs.conf we need to add monitoring lines for "mapred-site.xml"?

[lyuanlai_splunk]

check if mapred-site.xml is being monitored, if not, add it then re-run the savedsearch

[manus2]

There is no results

[lyuanlai_splunk]

Run this search (enclose with backquotes, this is a macro):
"hadoop_mapred_config"

Do you see any results?

[manus2]

saved search "__generate_lookup_hadoop_host2hdfs" gives a msg "Results written to file '/opt/splunk/etc/apps/SA-HadoopOps/lookups/hadoop_host2hdfs.csv'"

But saved search "__generate_lookup_hadoop_host2mapred" results:
"No results. Created empty file 'hadoop_host2mapred'
No matching fields exist"

[manus2]

Hi lyuanlai,

Thank you for your quick responses.

Now we are getting different error.

"Lookup table 'hadoop_host2maxcpu' is empty.
No matching fields exist
Lookup table 'hadoop_host2mapred' is empty.
No matching fields exist"