Re: Store search results in token and form token.

jprovenzale · ‎05-10-2022

Hello,

I have a dashboard that has a multi-select dropdown that contains a list of all database names.

When the dashboard is first run, the token that would hold the database name if a selection was made in the dropdown is set to * so all database events are read. Only the top 5 are returned.

My query looks like this:

index=whatever shard IN ("*")
| chart count as result by shard
| sort -result
| head 5

So say the display panel shows results for these databases.

229, 290, 112, 273, 242

I want to set the dropdown labelled Shards form token "form.shardToken" to the list of databases returned as well as updating the token shardToken with the same list of databases.

Hopefully that all makes sense.

ITWhisperer · ‎05-11-2022

shardToken will normally be set based on the selection from the dropdown. Do you want the full list of databases stored in another token (which isn't changed by the selection), or do you want the value of * to be the full (or partial / top 5) list?

jprovenzale · ‎05-11-2022

When the dashboard first runs, it checks all databases and returns the top 5.

I want the list of databases returned set to the dropdown form.shardToken as if they had been selected by the user and the actual token, shardToken, also set to the same returned list.

ITWhisperer · ‎05-11-2022

Create a search at the top of the dashboard (not associated with a panel)

index=whatever shard IN ("*") 
| chart count as result by shard
| sort -result
| head 5
| stats values(shard) as shard

Add this done handler to set the form token based on the results of the search

<done>
  <eval token="form.shardToken">$result.shard$</eval>
</done>

jprovenzale · ‎05-11-2022

Ok, I've done that but am receiving a warning as shown in the screenshot.

Is "init" a good place for the search and where should the done handler exist?

Thanks

ITWhisperer · ‎05-11-2022

No, an init block might contain token setting not searches

<form theme="light">
  <label></label>
  <description></description>
  <search>
    <query>
your search
    </query>
    <done>
done handling
    </done>
  </search>
  <init>
token setting
  </init>
  <fieldset submitButton="false" autoRun="false">
input definitions
etc.

jprovenzale · ‎05-11-2022

Thanks very much for your help so far.

I have made the changes and all warnings have cleared.
But no data is returned.

When I run the search in a search window, it DOES return the top 5 shards.
So something in the return assignment is not right. Code looks like this:

<form theme="dark">
<label>Troubleshooting Dashboard</label <description>=======================================================</description> <search>
<query>
index=whatever shard IN ("*")
| chart count as result by shard
| sort -result
| head 5
| stats values(shard) as returnshards
</query>
<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>
</search>
<init>
<set token="accountToken">"*"</set>
<set token="accountselectedToken">False</set>
<set token="shardToken">$form.shardToken$</set>
</init>
<fieldset submitButton="true" autoRun="true">

ITWhisperer · ‎05-11-2022

It is not clear what else might be wrong since you haven't provided the rest of the dashboard. Having said that, try using eval instead of set

<eval token="shardToken">$form.shardToken$</eval>

jprovenzale · ‎05-11-2022

I attempted to change set to eval with no luck.

I can provide the whole dashboard if you think it would help, but I see the issue in the <done> handler.

The query is fine as I can run it successfully in a search window, so there is a disconnect between the query statement :

| stats values(shard) as returnshards

and

<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>

if you need the full dashboard, please let me know.

Thanks

ITWhisperer · ‎05-11-2022

The full dashboard would be helpful or at least all the relevant parts.

jprovenzale · ‎05-12-2022

Ok.

Full dashboard below, with all sensitive bits changed and a couple of panels removed because of 20000 ch limit.

<label>Troubleshooting Dashboard</label>

<query>

index=whatever shard IN ("*") | search shard!=0 AND shard!=-1

| chart count as result by shard

| sort -result

| head 5

| stats values(shard) as returnshards

</query>

<done>

<eval token="form.shardToken">$result.returnshards$</eval>

</done>

</search>

<init>

<set token="accountselectedToken">False</set>

<set token="shardToken">$form.shardToken$</set>

</init>

<label>Shards</label>

<fieldForLabel>shardaccount</fieldForLabel>

<fieldForValue>shard</fieldForValue>

<query>| inputlookup ShardList.csv

| eval shardaccount=shard + " - " + account</query>

</search>

<set token="accountselectedToken">False</set>

</condition>

</condition>

</change>

</input>

<label>Time Range</label>

</default>

</input>

<choice value="1">Show Debug Panel</choice>

</input>

</fieldset>

<row>

<title>Token Debug</title>

<html>

<h2>Shard token = $shardToken$</h2>

<h2>Account token = $accountToken$</h2>

<h2>Account selected token = $accountselectedToken$</h2>

<h2>Debug token = $debugToken$</h2>

<h2>Form token = $form.shardToken$</h2>

<h2>Returned shards = $result.returnshards$</h2>

</html>

</panel>

</row>

<row>

<panel>

<title>Events Count By Shard Descending</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| chart count as result by shard

| sort -result

| head 5

</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.text">shards</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.text">Event Range</option>

<option name="charting.axisTitleY.visibility">collapsed</option>

<option name="charting.chart">column</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>

<option name="refresh.display">preview</option>

<set token="shardToken">$click.value$</set>

<set token="form.shardToken">$click.value$</set>

</drilldown>

</chart>

</panel>

<panel>

<title>Accounts</title>

<table>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| spath account

| search account=$accountToken$

| where account!=0

| top limit=5 account by shard

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<set token="form.shardToken">$click.value$</set>

<set token="shardToken">$click.value$</set>

<set token="accountToken">$click.value2$</set>

</drilldown>

</table>

</panel>

<panel>

<title>Endpoints</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| chart count by endpoint

| sort - count

| head 5</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.visibility">collapsed</option>

<option name="charting.axisTitleY.visibility">collapsed</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.chart">column</option>

<option name="charting.chart.stackMode">default</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">small</option>

<option name="trellis.splitBy">_aggregation</option>

</chart>

</panel>

</row>

<row>

<panel>

<title>Top 5 External IP Addresses</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1 NOT (original_ip=“123456789”) NOT (original_ip="-") NOT (original_ip=“123456789”)

| timechart limit=5 count by original_ip useother=f

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.text">@timestamp by 30 minutes</option>

<option name="refresh.display">progressbar</option>

</chart>

</panel>

<panel>

<title>Top 5 IP External Addresses Word Count</title>

<table>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1 NOT (original_ip=“123456789”) NOT (original_ip="-") NOT (original_ip=“123456789”)

| top limit=5 original_ip

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="refresh.display">progressbar</option>

</table>

</panel>

</row>

</form>

ITWhisperer · ‎05-13-2022

Try removing the assignments to shardToken - there are numerous places where you have done this.

shardToken is the output token from the multiselect input and gets set by the input when you change the input form.shardToken token.

jprovenzale · ‎05-13-2022

I don't see how this fixes my problem.

shardToken is only set in 3 places, one from the input dropdown and 2 in drilldowns, which is appropriate.

jprovenzale · ‎05-17-2022

So what is wrong with the below code that is not populating the form.shardToken?

<form theme="dark">
<label>Troubleshooting Dashboard</label>
<description>===========================================</description>
<search
<query>
index=whatever shard IN ("*") | search shard!=0 AND shard!=-1
| stats count as result by shard
| sort -result
| head 5
| stats values(shard) as returnshards
</query>
<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>
</search>

ITWhisperer · ‎05-17-2022

There doesn't appear to be anything wrong with this code snippet so form.shardToken is probably being overwritten elsewhere in your dashboard.

jprovenzale · ‎05-18-2022

That may be possible.

There is an initial pre-population of shards into the multi-select dropdown. This could affect the value of form.shardToken.

I guess it depends on what happens first, the initial pre-query or the pre-population.

Judging by current results I'm thinking the query runs first and the form.shardToken is overwritten by the pre-population.

If that's the case, I can store the query result in a temporary token, but where do I set the form.shardToken after the pre-population?
I attempted a done clause as part of the pre-population but this was not allowed and also attempted to assign in the init section.

Thanks

ITWhisperer · ‎05-18-2022

What do you mean by pre-population?

Why are you doing "pre-population" when the "pre-query" is doing the job you want?

jprovenzale · ‎05-18-2022

I am pre-populating the multi-select with.a list of all available shards.

The pre-query will simply assign the shards to that multi-select that it returns in the result.

In this way, initially, the multi-select will show the shards returned by the pre-query, however dropping the dropdown, they can add any other shards they may want to the list of shards to query.

ITWhisperer · ‎05-18-2022

Can you share the code that is "pre-populating" the dropdown?

jprovenzale · ‎05-22-2022

It's in the code I gave you earlier but here it is again.

</init>
<fieldset submitButton="true" autoRun="true">
<input type="multiselect" token="shardToken" searchWhenChanged="false">
<label>Shards</label>
<delimiter>,</delimiter>
<fieldForLabel>shardaccount</fieldForLabel>
<fieldForValue>shard</fieldForValue>
<search>
<query>| inputlookup ShardList.csv
| eval shardaccount=shard + " - " + account</query>
<earliest>@d</earliest>
<latest>now</latest>
</search>

ITWhisperer · ‎05-22-2022

OK this is a timing issue - create a new dropdown after your current dropdown which is hidden and sets your first dropdown

</init>
  <fieldset submitButton="true" autoRun="true">
    <input type="multiselect" token="shardToken" searchWhenChanged="false">
      <label>Shards</label>
      <delimiter>,</delimiter>
      <fieldForLabel>shardaccount</fieldForLabel>
      <fieldForValue>shard</fieldForValue>
      <search>
        <query>| inputlookup ShardList.csv
               | eval shardaccount=shard + " - " + account</query>
        <earliest>@d</earliest>
        <latest>now</latest>
      </search>
    </input>
    <input type="multiselect" token="doNotUseToken" searchWhenChanged="false" depends="$alwaysHide$">
      <label>Do not use</label>
      <delimiter>,</delimiter>
      <fieldForLabel>shardaccount</fieldForLabel>
      <fieldForValue>shard</fieldForValue>
      <search>
        <query>
          index=whatever shard IN ("*") | search shard!=0 AND shard!=-1
          | stats count as result by shard
          | sort -result
          | head 5
          | stats values(shard) as returnshards
        </query>
        <done>
          <eval token="form.shardToken">$result.returnshards$</eval>
        </done>
        <earliest>@d</earliest>
        <latest>now</latest>
      </search>
    </input>

How to store search results in token and form token?

other

Modern way of developing distributed application using OTel

Enterprise Security Content Update (ESCU) | New Releases

Archived Metrics Now Available for APAC and EMEA realms