I need some help updating the mmdb file for the iplocation command. Ive read the other forum questions regarding this, as well as the docs, and i am a bit confused.    I initially uploaded the new mmdb file from MaxMind, the GeoLite2-City.mmdb. I uploaded it through the GeoIP panel through the lookups tab.    It uploads, but i cant seem to find the file afterwards. I am looking on the specific server that I uploaded the file to, we have a clustered environment, but that one specific server I uploaded it to should have it. I ran locate and find commands, but could not locate it. We still have the original under $SPLUNK_HOME$/share/dbip-city-lite.mmdb   Even though the dropbox for the mmdb file showed a successful upload, I can not find it anywhere.  I dont see any trace of the upload through splunkd, or through /export/opt/splunk/var/run/splunk/upload/ , or through any find or locate command.  I wanted to update the file path to include both databases, and i know i needed to change the limits.conf file, and update it to include both paths. But the question is, How do i change the limits.conf so that it replicates. We dont have any app named TA-geoisp or anything similar, and thats what these forums and docs reference.   Somewhere I saw that I could update the search app's limits.conf and just push that from the shcluster directory, as that will push a bundle change that will push out to all Search heads in the cluster. Since the search app is the default app, we could just use that app to point to the mmdb files. But we don't have the search app located under our /$SPLUNK_HOME$/etc/shcluster/apps/   We dont seem to have the search app under our Clustermaster/Deployer shcluster directory. I think i might be missing something. I would basically just like to update the limits.conf to point to the new dir path of both of the mmdb files. Id like to just edit the limits.conf to look like:     [iplocation] MMDBPaths = /path/to/your/GeoIP2-City.mmdb,/path/to/your/dbip-city-lite.mmdb       The question im trying to ask here, is when i upload the file through the gui, where does the file end up. And if i wanted to push these changes manually,  if i wanted to push to all SH and indexers from the deployer and deployment server, how do i go about replicating the folder that holds the mmdb as well as the limits.conf that hold the paths to the files.    Thank you for any assistance.    ... View more

I have a question regarding how to properly extract the time ranges between the Events to use as a field value for a Date-Range column. Im setting up the Chargeback app, and im making a specific report. Currently, Im tracking the total ingestion by the Biz_Unit. The main splunk query does fine, but there's a lot of time manipulation within the search, and im not sure how to properly set the date I need. here is an example of some of the output.    This is the query, i know its a large query, but this outputs all of the fields used in chargeback.    `chargeback_summary_index` source=chargeback_internal_ingestion_tracker idx IN (*) st IN (*) idx="*" earliest=-7d@d latest=now | fields _time idx st ingestion_gb indexer_count License | rename idx As index_name | `chargeback_normalize_storage_info` | bin _time span=1h | stats Latest(ingestion_gb) As ingestion_gb_idx_st Latest(License) As License By _time, index_name, st | bin _time span=1d | stats Sum(ingestion_gb_idx_st) As ingestion_idx_st_GB Latest(License) As License By _time, index_name, st `chargeback_comment(" | `chargeback_data_2_bunit(index,index_name,index_name)` ")` | `chargeback_index_enrichment_priority_order` | `chargeback_get_entitlement(ingest)` | fillnull value=100 perc_ownership | eval shared_idx = if(perc_ownership="100", "No", "Yes") | eval ingestion_idx_st_GB = ingestion_idx_st_GB * perc_ownership / 100 , ingest_unit_cost = ingest_yearly_cost / ingest_entitlement / 365 | fillnull value="Undefined" biz_unit, biz_division, biz_dep, biz_desc, biz_owner, biz_email | fillnull value=0 ingest_unit_cost, ingest_yearly_cost, ingest_entitlement | stats Latest(License) As License Latest(ingest_unit_cost) As ingest_unit_cost Latest(ingest_yearly_cost) As ingest_yearly_cost Latest(ingest_entitlement) As ingest_entitlement_GB Latest(shared_idx) As shared_idx Latest(ingestion_idx_st_GB) As ingestion_idx_st_GB Latest(perc_ownership) As perc_ownership Latest(biz_desc) As biz_desc Latest(biz_owner) As biz_owner Latest(biz_email) As biz_email Values(biz_division) As biz_division by _time, biz_unit, biz_dep, index_name, st | eventstats Sum(ingestion_idx_st_GB) As ingestion_idx_GB by _time, index_name | eventstats Sum(ingestion_idx_st_GB) As ingestion_bunit_dep_GB by _time, biz_unit, biz_dep, index_name | eventstats Sum(ingestion_idx_st_GB) As ingestion_bunit_GB by _time, biz_unit, index_name | eval ingestion_idx_st_TB = ingestion_idx_st_GB / 1024 , ingestion_idx_st_PB = ingestion_idx_st_TB / 1024 ,ingestion_idx_TB = ingestion_idx_GB / 1024 , ingestion_idx_PB = ingestion_idx_TB / 1024 , ingestion_bunit_dep_TB = ingestion_bunit_dep_GB / 1024 , ingestion_bunit_dep_PB = ingestion_bunit_dep_TB / 1024, ingestion_bunit_TB = ingestion_idx_GB / 1024 , ingestion_bunit_PB = ingestion_bunit_TB / 1024 | eval ingestion_bunit_dep_cost = ingestion_bunit_dep_GB * ingest_unit_cost, ingestion_bunit_cost = ingestion_bunit_GB * ingest_unit_cost, ingestion_idx_st_cost = ingestion_idx_st_GB * ingest_unit_cost | eval ingest_entitlement_TB = ingest_entitlement_GB / 1024, ingest_entitlement_PB = ingest_entitlement_TB / 1024 | eval Time_Period = strftime(_time, "%a %b %d %Y") | search biz_unit IN ("*") biz_dep IN ("*") shared_idx=* _time IN (*) biz_owner IN ("*") biz_desc IN ("*") biz_unit IN ("*") | table biz_unit biz_dep Time_Period index_name st perc_ownership ingestion_idx_GB ingestion_idx_st_GB ingestion_bunit_dep_GB ingestion_bunit_GB ingestion_bunit_dep_cost ingestion_bunit_cost biz_desc biz_owner biz_email | sort 0 - ingestion_idx_GB | rename st As Sourcetype ingestion_bunit_dep_cost as "Cost B-Unit/Dep", ingestion_bunit_cost As "Cost B-Unit", biz_unit As B-Unit, biz_dep As Department, index_name As Index, perc_ownership As "% Ownership", ingestion_idx_st_GB AS "Ingestion Sourcetype GB", ingestion_idx_GB As "Ingestion Index GB", ingestion_bunit_dep_GB As "Ingestion B-Unit/Dep GB",ingestion_bunit_GB As "Ingestion B-Unit GB", biz_desc As "Business Description", biz_owner As "Business Owner", biz_email As "Business Email" | fieldformat Cost B-Unit/Dep = printf("%'.2f USD",'Cost B-Unit/Dep') | fieldformat Cost B-Unit = printf("%'.2f USD",'Cost B-Unit') | search Index = testing | dedup Time_Period | table B-Unit Time_Period "Ingestion B-Unit GB"   The above image shows what im trying to extract. The query has binned _time twice:   | fields _time idx st ingestion_gb indexer_count License | rename idx As index_name | `chargeback_normalize_storage_info` | bin _time span=1h | stats Latest(ingestion_gb) As ingestion_gb_idx_st Latest(License) As License By _time, index_name, st | bin _time span=1d | stats Sum(ingestion_gb_idx_st) As ingestion_idx_st_GB Latest(License) As License By _time, index_name, st   Ive asked our GPT equivalent bot how to properly do it, and it mentioned that when im sorting the stats by _time and index, it was overwriting the time variable. it also kept recommending me change and eval time down near the bottom of the query, something like:   | stats sum(Ingestion_Index_GB) as Ingestion_Index_GB sum("Ingestion B-Unit GB") as "Ingestion B-Unit GB" sum("Cost B-Unit") as "Cost B-Unit" earliest(_time) as early_time latest(_time) as late_time by B-Unit | eval Date_Range = strftime(early_time, "%Y-%m-%d %H:%M:%S") . " - " . strftime(late_time, "%Y-%m-%d %H:%M:%S") | table Date_Range B-Unit Ingestion_Index_GB "Ingestion B-Unit GB" "Cost B-Unit"     Other instances it said that it wasnt in string format, so i couldnt use the strftime.    overall, im now confused as to what is happening to the _time value. All i want is to get the earliest and latest value by index and set that as Date_Range. Can someone help me with this and possibly explain what is happening to the _time variable as it keeps getting manipulated and sorted by.    This is the search query found in the chargeback app under the storage tab. Its the "Daily Ingestion By Index, B-Unit & Department" search query.  if anyone has any ideas, any help would be much appreciated.  ... View more

Hello, i am reaching out to ask if there is any way to make the chart that was generated with the scheduled PDF report option look any better.  We have this dashboard:     It looks fine. Everything looks nice and clean. When we use the schedule PDF, and we generate a pdf, it does not look good. It looks like  As an fyi, the above SS included both types for chart formatting, one had the values in the middle, and one had values above. Both look good in the dashboard, but neither look good in the pdf.  Is there a way to edit the chart bars so they have more space, or to edit the size of the numerals above the bars?    Is there an app that allows better editing of PDF within splunk? I feel like we have done everything we can to make the pdf look good, but we can not seem to be able to get the numbers to look good on the pdf.    Thank you for any guidance.   ... View more

I have a user that requested me to look into some of his reports. He wanted the permission of report 2 to match with report 1. Both are owned by two different people, but two people with similar roles and access.   After we tweaked the settings for the report, being shared in the app, having read access by all, and write permissions to those with the appropriate roles, they are still having issues viewing and editing.    The owner of report 1 is the owner/creator of the report. The report runs as owner, and is shared globally. He doesn't have permissions to edit the actual alert.  He created the report initially, how come he cant edit it. I even cloned it and reassigned ownership, to no avail.  Report 1  runs as owner, while report 2 has the option to run as owner or as the user. How come one report has that option while the other one is locked to running as owner? As far as user two goes, his roles include permissions to the used indexes, as well as access to the app, default search app, and he has even more roles and permissions than user 1. Yet, he receives an error when trying to view the link that splunk sends out that has the attached report.  My question is, is there anywhere else I should be looking at in order to find permission discrepancies. From everything ive seen, both users have access to the required indexes, have pretty much soft-admin on splunk, and i assume they have viewed these in the past. From roles to users to capabilities, they have everything in order, or at least it seems. Is there something I should check in the configs?    Thanks for any guidance.  ... View more

I have a question about filtering in data. We have a customer who is requesting a set of fields to be sent in from 0365. The issue is, we cant modify what we pull in because we are using  an API, not the universal forwarder. Currently I am trying to test out the search query to confirm that I am only pulling in the correct events with those fields.  The o365 data pulls in about 400+ fields. We are wanting about 40 of those events for a specific use case. My question is, what is the correct syntax for splunk to only search for those fields.  Original query that brings in about 400+ fields:   index=o365     New query for about 35 fields:   index=o365 "Operation"="*" OR "LabelAction"="*" OR "LabelAppliedDateTime"="*" OR "LabelIid"="*" OR "abelName"="*" OR "DlpAuditEventMetadata.DlpPolicyMatchId"="*" OR "DlpAuditEventMetadata.EvaluationTime"="*" OR "DlpOriginalFilePath"="*" OR "IrmContentId"="*" OR "PolicyMatchInfo.PolicyId"="*" OR "PolicyMatchInfo.PolicyName"="*" OR "PolicyMatchInfo.RuleId"="*" OR "PolicyMatchInfo.RuleName"="*" OR "ProtectionEventData.IsProtected"="*" OR "ProtectionEventData.IsProtectedBefore"="*" OR "ProtectionEventData.ProtectionEventType"="*" OR "ProtectionEventData.ProtectionOwner"="*" OR "ProtectionEventData.ProtectionType"="*" OR "ProtectionEventData.TemplateId"="*" OR "ProtectionEventType"="*" OR "RMSEncrypted"="*" OR "SensitiveInfoTypeData{}.Confidence"="*" OR "SensitiveInfoTypeData{}.Count"="*" OR "SensitiveInfoTypeData{}.SensitiveInfoTypeId"="*" OR "SensitiveInfoTypeData{}.SensitiveInfoTypeName"="*" OR "SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Confidence"="*" OR "SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Count"="*" OR "SensitivityLabelEventData.ActionSource"="*" OR "SensitivityLabelEventData.ActionSourceDetail"="*" OR "SensitivityLabelEventData.ContentType"="*" OR "SensitivityLabelEventData.JustificationText"="*" OR "SensitivityLabelEventData.LabelEventType"="*" OR "SensitivityLabelEventData.OldSensitivityLabelId"="*" OR "SensitivityLabelEventData.SensitivityLabelId"="*" OR "SensitivityLabelEventData.SensitivityLabelPolicyId"="*" OR "LabelName"="*" | fields Operation,LabelAction,LabelAppliedDateTime,LabelIid,abelName,DlpAuditEventMetadata.DlpPolicyMatchId,DlpAuditEventMetadata.EvaluationTime,DlpOriginalFilePath,IrmContentId,PolicyMatchInfo.PolicyId,PolicyMatchInfo.PolicyName,PolicyMatchInfo.RuleId,PolicyMatchInfo.RuleName,ProtectionEventData.IsProtected,ProtectionEventData.IsProtectedBefore,ProtectionEventData.ProtectionEventType,ProtectionEventData.ProtectionOwner,ProtectionEventData.ProtectionType,ProtectionEventData.TemplateId,ProtectionEventType,RMSEncrypted,SensitiveInfoTypeData{}.Confidence,SensitiveInfoTypeData{}.Count,SensitiveInfoTypeData{}.SensitiveInfoTypeId,SensitiveInfoTypeData{}.SensitiveInfoTypeName,SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Confidence,SensitiveInfoTypeData{}.SensitiveInformationDetailedClassificationAttributes{}.Count,SensitivityLabelEventData.ActionSource,SensitivityLabelEventData.ActionSourceDetail,SensitivityLabelEventData.ContentType,SensitivityLabelEventData.JustificationText,SensitivityLabelEventData.LabelEventType,SensitivityLabelEventData.OldSensitivityLabelId,SensitivityLabelEventData.SensitivityLabelId,SensitivityLabelEventData.SensitivityLabelPolicyId,LabelName     Basically, From my understanding and my research, if you just append a specific string in quotes, or outside of quotes, splunk searches all events for that string and pulls it in. Such as:   index=Test field1 field2 field3   That would bring in only events with field1 or field2 or field3 within it. Adding quotes to it, such as    index=Test "field1"="*" "field2"="*" "field3"="*"   Should filter the same way. I have tested it both way, with double quotes surrounding the field, as well as no quotes. Im also using | fields Which should only bring those fields in, but i dont know if its only showing those fields, but bringing in ALL of the events.    My question is, is this correct? With the base searches ive been testing with, searching all of the events in o365 for one day, full 24 hours, brings in  23,410,064 events. Filtering out with the query I pasted above, for the same day, same 24 hours, brings in 23,409,887 events. Ive tested this a couple of ways, and each time, searching over the same time period, the filtering query brings in about 1k less events. But I can still only view the first 1k events, 20 pages worth. But that may be another question.    My longwinded question boils down to, am I searching this data correctly? I know its a heavy index with millions of events, but filtering out to only 40 or so fields, some of which only appear .6% of the time, still brings in millions of events. Is there a way to fully validate it?    ... View more

I would like to add a label for the upper/lower 95. I was wondering how I could do that. Id like to have it the same color as the line as well, similar to --Upper|Lower95. ... View more

I am trying to create an alert or a report to track the number of deferred searches. We had an issue where the cluster captain deferred a massive amount of searches, and it messed up a few things. We are trying to create an alert to maybe mitigate that in the future. In addition to asking the best way to create a alert for this, id also like some more clarification on how to find the deferred searches.  Through the monitoring console,  either through the DMC or the Cluster Master,  I thought i had seen a panel for deferred searches, but i can not find one now. And when i run the search        index=_internal earliest=-24h "status=skipped" sourcetype=scheduler | stats count by host app | sort - count       I get results, but if i change status to deferred, which i assume is a status, I do not get anything. I was suggested to run        | rest /services/search/jobs | search status=deferred | table id, search, app, owner, earliest_time, latest_time, status, sid       but I do not get any status. Status is not a field.   The main question I have is How do I access the number of deferred searches? If i can find that, I can run stats count on it.    Thank you.  ... View more

So we have this alert set up to check to see if any hostnames that are being monitored havnt received any time monitoring data. The current search is as follows:   |inputlookup TimeServersV2.csv | search server="*" | eval HOST =lower(server) | fields HOST | where NOT [search (index=os sourcetype=test_stats*) OR (sourcetype=syslog ptp10 OR phc10sys) OR (index=windows sourcetype="Script:TimeStatus") OR (index=windows sourcetype=domtimec) OR (index=os sourcetype=time)| dedup host | eval HOST=lower(host)| fields HOST ]     The issue with this is, we believe, once it runs at 8 AM,  it takes a bit longer to run and process data, and it'll send out partial results after a min or so of running the query. We have a lot of saved reports/alerts/searches running at the top of most hours, so I think it may be sending out the incomplete search results after a bit of running, as splunk starts the next job. I moved its cron job schedule up an hour and a half to a lighter use hour, so that may help a bit, but i would also like to optimize this search so it runs faster. Currently, it runs about 40 seconds to a little over a minute: What would be the best way to optimize this search so it could possibly be run in under 30 seconds, if possible. Running it outside of the scheduled time runs in about 6 seconds, its just slow when it runs alongside all of the other searches. Itll send us an alert with a list of hostnames its found that were not on the list, yet when we run it manually, it will only spit out 4 or 5 results. That's why we think its not finishing the search when it sends out an alert. Any help would be appreciated.  ... View more

We have this dashboard that recently started alerting us on a risky command. We were using the fit command.    I followed the docs and added the following line to the newly created commands.conf that i had put in the apps local folder to push.  [fit] is_risky = false According to the docs, i assumed that this would just disable the warning for using that command. After i put it into the specified apps local folder, /export/opt/splunk/etc/shcluster/apps/<app-name>/local, I pushed the bundle and it seemed to have put it in the apps default folder on the search heads. But the biggest issue here is that once i pushed that bundle, splunk doesn't recognize the fit.py file any more. I tested putting that commands.conf in the apps default folder, same thing. I tested this a few times, and while im glad that the bundle pushes were working, im a bit confused as to why splunk no longer recognizes that command even though im only using the is_risky=false, which should only stop the warning.    Any help on this matter would be appreciated. Thank you. And if you could also answer as to why the local file in the app's directory is pushing to the apps default folder on the search heads, that would be a bonus. Thank you.  ... View more

So i am trying to compare bar graphs for event count for our indexes for two separate days. We are upgrading our environment, and I was wanting this query to show us the event count before and after we upgrade. I am have tried using the earliest=-<int>d and latest=-<int>d, but the query keeps using the time picker. I am using dbinspect, so i wasn't sure if that had something to do with it. Below is the working query that outputs the same results for both EventCount and EventCount_1     |dbinspect index=* | search index!=_* | fields bucketId eventCount index _time | stats sum(eventCount) as EventCount values(max(_time)) as Time by index | table index EventCount, | join type=outer index [| dbinspect index=* | search index!=_* | fields bucketId eventCount index | stats sum(eventCount) as EventCount_1 by index | table index EventCount_1] | table index EventCount EventCount_1       I have tried putting the the time periods in a few places, after the first index, in which the query runs, but returns the same results using the time from the time picker. If i place it after the search, I dont get any results.      |dbinspect index=* earliest=-4d latest=-3d | search index!=_* | fields bucketId eventCount index _time | stats sum(eventCount) as EventCount values(max(_time)) as Time by index | table index EventCount, | join type=outer index [| dbinspect index=* | search index!=_* earliest=2023-05-30T00:00:00 latest=2023-06-01T23:59:59 | fields bucketId eventCount index | stats sum(eventCount) as EventCount_1 by index | table index EventCount_1] | table index EventCount EventCount_1     ^  this is also a working query, but it still uses the time from time picker instead of the stated one in query ^ Am I supposed to be using a different type of time selection with the dbinspect? If i don't use dbinspect, I don't get the same results. Is there any other way to get these results? I'm just trying to get event count by index. Thank you for any help.    ... View more

I have this report that i received an error from. Ive seen the error from different searches, but i just started to look into them. In the email, it said the issue was  Error in 'where' command: The 'not' function is unsupported or undefined.   Im assuming the search ran fine before it started getting an error as it was turned into a report.  Current search:    |inputlookup X_servers.csv | search OS=*Windows* environment=Production OR environment="Disaster Recovery" | dedup host | rename host as HOST | table HOST environment OS application1 | sort +HOST |where NOT[|inputlookup Y_agent_managed.csv | table HOST]     I looked up that error, but I couldnt find anything useful out of them. We have our DMC, Both cluster Masters, and deployment servers all on Splunk 9.x.x, everything else is on 8.x.x. The posts I saw were talking about Splunk 6.x, so that one was a bit outdated and im at a loss as to the proper syntax. I tried replacing NOT with != but apparently splunk reads them both as NOT, which makes sense.      Id appreciate any help, thank you. We haev a few old reports that still use NOt like this .  ... View more