How do I plot a per-operation success rate over a rolling 24 hour period?  As a point in time query producing a chart, I do index=kubernetes source=*proxy* api.foo.com OR info OR commitLatest | rex field=_raw ".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<raw_status>\d+) (?<details>[^\ ]+) " | eval status=case(details="downstream_remote_disconnect","client disconnect",match(details, "upstream_reset_after_response_started"),"streaming error",true(),raw_status) | eval operation=case(match(path,".*contents"),"put-chunked-file",match(path,".*info"), "get-file-info-internal", match(path,".*commitlatest"), "commit-latest-internal", true(), "get-chunked-file") | eval failure=if(match(status,"^(client disconnect|streaming error|[0-9]|400|50[0-9])$"),1,0) | stats count by operation, failure | eventstats sum(count) as total by operation | eval percent=100 * count/total | stats list(*) by operation | table operation, list(failure), list(percent), list(count) ... View more

I want to look for requests in a service mesh ingest log which have no corresponding application log entries. My first search is  index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect | rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})" | eval id=id1.id2.id3 | fields id my second search is  index=kubernetes source=*proxy* operation: | rex field=_raw "span_id:(?<id>[0-9a-f]{16});" | fields id and the obvious way of combining them yields no results index=kubernetes source=*envoy-proxy* (api.foo.com OR info) AND downstream_remote_disconnect | rex field=_raw "\[[^\]]+\] \"(?<downstream>[^\"]+)\".*\"(POST|GET) \"(?<host>[^\"]+)\" \"(?<path>[^\"\?]+)[\?]?\" [^\"]+\" (?<status>\d+).*\"(?<id1>[0-9a-f]{8})-(?<id2>[0-9a-f]{4})-(?<id3>[0-9a-f]{4})" | eval id=id1.id2.id3 | fields id | search NOT [ search index=kubernetes source=*proxy* operation: | rex field=_raw "span_id:(?<id>[0-9a-f]{16});" | fields id ] ... View more