Repository navigation
Proper way to perform head sampling? #129
Description
How should I perform head sampling to ensure that only a subset of traces get sampled?
With tracing + tracing-opentelemetry, you can use Sampler::ParentBased combined with Sampler::TraceIdRatioBased to sample a percentage of root spans without any parent, or 100% of child spans if the parent is sampled.
Fastrace seems to set sampled: true on every new span created with SpanContext::random.
I was considering having a new_root function in our code which would be a wrapper around Span::root which changes .sampled depending on some criteria (e.g. coin toss). However, that wouldn't work since we don't have control over all usages of Span::root. Some are in fastrace-tonic for example.
A solution to this is a blocker for us to adopt fastrace, since on high QPS services we could only capture a miniscule percentage of traces.
Activity
andylokandy commentedon Aug 11, 2025
Seems that a proper solution is adding a sampling configuration to
fastrace-tonicto randomly setsampledtofalseregardless oftraceparent. Another way is to use tail-sampling which is highly recommended especially for high QPS scenario because we've found that the information provided by tail sampling (based on time or other criteria) is way more useful than random head sampling, especially given that tail-sampling has very little performance penalty in fastrace. You may try to use tail sampling first, and turn to the first solution if the performance is not satisfied.ArniDagur commentedon Aug 11, 2025
Would tail sampling be done with a custom
Reporter?I'd imagine it'd work like this: Given a list of spans, group by trace ID, find the root spans, and make a sample/no sample decision based on e.g. latency histograms and presence of failures in child spans.
I see a few problems though:
Vec<SpanRecord>, I'm worried about what happens if a single trace is split into multiple calls toreport. Then the later calls toreportwill have spans with a root-less trace ID. I guess you'd have to make yourReporterstateful (remember which trace IDs you decided to sample).fastrace-tonicis a "real" root span where notraceparentheader was given, or if it's simply the "process local" root span, where the real root exists in the calling service. In the former case, the sampling decision should happen in our process, but in the latter case the sampling decision has already been made and we need to follow its lead—ensure tail sampling doesn't exclude any spans which the caller expects us to sample (or include any spans where we don't have caller information, if you don't want that).Is "we" in this sentence a company? Was the tail sampling implementation you used open source or internal? If you have a working solution I'd love to see it :)
I know OpenTelemetry Collector also has a tail sampling processor, but I'm worried the QPS would be too high if we sent every span to OpenTelemetry Collector.
ArniDagur commentedon Aug 11, 2025
I don't think this will scale. If
fastrace-tonicis your only external dependency, then perhaps. But iffastracegains adoption you'll never be able to control all the places your library dependencies create root spans.In addition, some organizations may want to use custom head sampling strategies (e.g. adaptive sampling), while others want a simple ratio of all roots. I don't think baking sampling strategies into
fastrace-tonicor any other specific library is a good idea.andylokandy commentedon Aug 11, 2025
Tail sampling is a builtin feature of fastrace, which can be enabled in
Config. With tail sample enabled, you can callSpan::cancelto dismiss the entire trace based on your strategy.andylokandy commentedon Aug 11, 2025
Fastrace is initially developed for extensive traces in DBMS, which has crazy QPS and latency requirement. Anyway, fastrace has been adopted by various projects, which can be found here, and from the results and observation, we has such conclusion.
ArniDagur commentedon Aug 11, 2025
Manually calling
Span::cancelfrom within application code is too invasive, and would only allow simple tail sampling scenarios (e.g. based on duration of root span). I want developers to be able to just run an init function inmain, slap some#[trace]attributes onto the functions they want traced, set up span roots where needed, and have everything work. That's how it is withtracing+tracing-opentelemetry.Also, how is the second issue I raised tackled? A sampling decision from one service has to be able to propagate to all downstream services. I said:
Thank you for the help :)
andylokandy commentedon Aug 12, 2025
In fact,
tracingfalls short here. The random root is easy to get started but in real production will leave you massive disconnected and useless traces/spans. You can see detailed examples here. In the real production that tracing be really useful (only from several examples that I've witness or participated in, including positive examples and negative ones), propagation/sampling/root has to be carefully designed and maintained. If you'd like to slap some attribute on random functions and imagine it'll provide something useful when the system becomes abnormal online, you may be disappointed.Reasonable. I'd like to add a config to
fastrace-tonicto allow user to control the root span construction behavior.ArniDagur commentedon Aug 12, 2025
There's nothing preventing you from both carefully designing and maintaining trace propagation, while still using implicit root span creation. In fact, with implicit roots you have better visibility into where your actual trace roots are (they're obvious from seeing disconnected spans), giving you the opportunity to improve on trace propagation. When manually specifying root spans, you have to guess all the right places to instrument, and you'll almost certainly be missing some. My previous company used this approach to trace requests across dozens of micro-services.
However, that was not my actual point. In fact, root spans being explicit is in some way nice, as it allows you to
grepyour codebase for all the roots. My point was that tail sampling by manually callingSpan::cancelis not scalable in terms of developer time and effort. Likewise, I don't see how it solves most use cases other than just canceling based on duration.What kind of API are you considering?
This sounds like a step in the right direction. However, it doesn't address the entire issue. If
fastracebecomes popular, and downstream libraries start using it, there's no way to ever control all the placesSpan::rootis called. It'd only take one bad dependency capturing 100% of traces to completely DDoS your OpenTelemetry Collectors with traffic.What would solve the issue is a global mechanism similar to
opentelemetry::sdk::trace::Samplerandopentelemetry::sdk::trace::ShouldSampleto control head sampling, even when traces are initiated in libraries you don't control.andylokandy commentedon Aug 13, 2025
There are two feature requests:
There is workaround on current api: detect whether current parent exist and start span with the current span or create a root. But still it's not recommended. Because, by design, library could annotate
#[fastrace::trace]everywhere and let the application/downstream to decide whether to enable the spans by wrapping the calls to library withSpan::root. If all spans unconditionally start roots, the application/downstream has no change to ever shut the spans down.sampledflag onSpanContextforSpan::root.It's possible to implement such a mechanism in fastrace. But would it better if you can calculate the head sampling flag in the client side (i.e., the
sampledin traceparent)?