Trace_Analysis_00_filter_groupby.py

"""
Note: There are several variations to do the same thing (see # or). The idea
is to choose the best (one or so) of the variations and implement that. I.E.
these are different designs possibilities for the API.
"""

"""
Filtering
"""

#
# find spans with a "db.statement" tag
#
db_spans = spans[model.tags.some[model.kvp.key == 'db.statement']]

# or
db_spans = spans[model.tags.some[model.key == 'db.statement']]

# or
db_spans = spans[model.tags.some['db.statement']]

# or
db_spans = spans[model.tags.some('db.statement')]

#
# filter db_spans to spans over 1 second in duration
#
slow_db_spans = db_spans[model.duration >= 1 * time.SECOND]

#
# conjuntions
#
slow_db_spans = spans[model.tags.some['db.statement'] & (model.duration >= 1 * time.SECOND)]


#
# filter to spans with process.service == 'customer'
#
customer_spans = spans[model.service == 'customer']

#
# filter to spans with process.service == 'driver' or process.service == 'route'
#
driver_route_spans = spans[(model.service == 'driver') | (model.service == 'route')]
# or
driver_route_spans = spans[model.service.isin(['driver', 'route'])]

#
# spans with an error tag
#
error_spans = spans[model.tags.some['error']]

#
# spans with the tag "error" == True
#
error_spans = spans[model.tags.some[(model.key == 'error') & (model.value == True)]]

# or - this approach is less flexible than the above
error_spans = spans[model.tags.some(key='error', value=True)]

#
# spans with the tag "error" == True or "error" == "true"
#
error_spans = spans[model.tags.some[(model.key == 'error') & ((model.value == True) | (model.value == 'true'))]]

#
# trace has an error
#
# save the predicate for reuse
is_error_span = model.tags.some[(model.key == 'error') & ((model.value == True) | (model.value == 'true'))]
trace_has_error = trace.spans.some[is_error_span]
# type is bool
isinstance(trace_has_error, bool)


"""
Predicates can be saved and used later
"""

is_error_span = model.tags.some[(model.key == 'error') & ((model.value == True) | (model.value == 'true'))]


"""
grouping
"""

#
# group spans by their service
#
# => {'customer': [span, span], 'driver': [span]}
spans.groupby(model.service)

#
# group spans by both service and operation
#
# => {('customer', 'greet'): [span, span], ('customer', 'bid-farewell'): [span]}
spans.groupby([model.service, model.operation])

#
# group spans by service then operation
#
# => {'customer': {'greet': [span, span], 'bid-farewell': [span]}}
spans.groupby(model.service, model.operation)

#
# group by presence of the span tag 'pod'
#
# => {True: [span, span], False: [span]}
spans.groupby(model.tags.some['pod'])

#
# group by presence of both span tags 'db.statement' and 'db.result_count'
#
# => {(True, True): [span, span], (True, False): [span], ...}
spans.groupby([model.tags.some['db.statement'], model.tags.some['db.result_count']])

#
# group by value of span tag 'as'
#
# => {None: [span, span], 'thrift': [span], 'http': [span]}
spans.groupby(model.tags['as'])

#
# group by the value of the process host tag
#
spans.groupby(model.process.tags['host'])

#
# group by data-center via a transform on process.tags.host
# assuming host has the form "someid-datacenter", e.g. "ab12c-us-east1"
#
get_dc = lambda host: re.sub('^[^-]+-', '', host)
by_dc = spans.groupby(model.process.tags['host'].transform(get_dc))

#
# group by parent service
#
spans.groupby(model.parent.service)

#
# group by has a child with an error
#
spans.groupby(model.children.some[is_error_span])

#
# group by has a ancestor has an error
#
spans.groupby(model.ancestors.some[is_error_span])

Exploring definitions of condensed digraphs.

Initial, raw trace:

In the above image, a 0 means

• a is the service:operation
• 0 is the span ID

ab 1

• ab is the client span from a to b
• 1 is the span ID

auth_0 14 [skip]

• auth_0 service:operation
• 14 is the span ID
• [skip] means it has the label "skip"

We want to arrive at a condensed digraph that looks like:

Have arrived at the following data structure:

node
	.members: null | node[]
	.members_roots: null | node[]
	.downstreams: { node: node, members_source: null | node }[]
	.upstreams: { node: node, members_target: null | node }[]
	.labels: Set<string>
	.data: *

Which allows for grouping of nodes. For a low-level node, like a span, which isn't actually a group, it would look like:

node
	.members: null
	.members_roots: null
	.labels: Set<string>
	.data
		.duration
		.logs
		.operation
		.process
			...etc
		.references
		.start_time
		.tags
	.downstreams
	.upstreams

From Ted Young's talk on Trace Driven Development: Unifying Testing and Observability, slide 14:

model = NewModel()
model(“Accounts cannot withdraw more than their balance”)
    .When(
        LessThan(
            Span.Name(“fetch-balance”).Tag(“amount”),
            Span.Name(“withdrawal”).Tag(“amount”)))
    .Expect( Span.Name(“rollback”) )
    .NotExpect( Span.Name(“commit”) )
    .Expect( Span.Name(“/:account/withdrawl/”).HttpStatusCode(500))

Check(model, testData)

What it might look like in the gist's boolean indexing API:

predicate = model.span['fetch-balance'].tag['amount'] < model.span['withdrawal'].tag['amount']

insufficient_funds = trace_test
    .when('The withdrawl amount exceeds the available balance', predicate)
        .it('Is not committed', model.span['commit'].is_empty)
        .it('Is rolled back', model.span['rollback'])
        .it('Results in a 500', model.span['/:account/withdrawl/'].tag['http.status_code'] == 500)


# In another file... use the trace test on a production stream

from .my_trace_tests import insufficient_funds

trace_test.check(trace_stream, insufficient_funds)