<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706706509839/05507f80-3502-4c78-a066-c609421c2714.png" alt class="image--center mx-auto" />
<h1 id="heading-mimir-for-metrics">Mimir: For Metrics</h1>
<h1 id="heading-loki-for-logs">Loki: For logs</h1>
<h1 id="heading-tempo-for-traces">Tempo: For Traces</h1>
<h1 id="heading-pyroscope-continuous-profiles">Pyroscope: Continuous Profiles</h1>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706706697689/3c2f2486-d564-4c79-8f64-e1daaa4ec096.png" alt class="image--center mx-auto" />
Grafana: Dashboard
Find anomalies in metrics
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706706823219/b5043430-875c-48a7-b89d-8e16e6e32afd.png" alt class="image--center mx-auto" />
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706706853607/bd5abd7d-5dbd-4ebd-9360-b2037bbde63a.png" alt class="image--center mx-auto" />
Find a needle in a haystack with log queries?
Loki : Log
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706706941746/33c3c009-499a-42c6-9346-f2eca1500b91.png" alt class="image--center mx-auto" />
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706707053993/5a156a4c-03e6-42bf-900e-36ccd26d7534.png" alt class="image--center mx-auto" />
Trade off: One log line searching time consuming.
Quey Language: LogQL
TSDB index same used in prometheus.
Line based deletes: As occasionally some secrets will be logged in logs :)
Find where latency and error originate across many services with traces?
Tempo
Distributed call graph.
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706707329166/4e198ded-f2ab-4c1f-b07c-45760e8a7d07.png" alt class="image--center mx-auto" />
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706707377036/d6e356d6-78d4-487c-b5dd-7f7fd969853b.png" alt class="image--center mx-auto" />
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706707447098/540127c7-7666-41b6-8109-f55b04c6bb44.png" alt class="image--center mx-auto" />
TraceQL ----- PromQL
<pre><code class="lang-bash">{resource.service="server"&amp;&amp;span.target="/account"}&gt;&gt;{status=error}
</code></pre>
Show me all the spans where the request was received by my server on the account endpoint.
Profile - Pyroscope
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706707981405/7895644f-f8c0-4b6a-84b6-b875e8435042.png" alt class="image--center mx-auto" />
Sampling Profile
Flame graph and icle graph
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706708048102/e5bdea1d-fcc7-4531-a8d9-bcf24947632e.png" alt class="image--center mx-auto" />
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1706708158592/8c2c70eb-56dc-4605-9853-65eaac72f4df.png" alt class="image--center mx-auto" />

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706706509839/05507f80-3502-4c78-a066-c609421c2714.png align="center")

# **Mimir**: **For Metrics**

# Loki: For logs

# Tempo: For Traces

# Pyroscope: Continuous Profiles

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706706697689/3c2f2486-d564-4c79-8f64-e1daaa4ec096.png align="center")

Grafana: Dashboard

Find anomalies in metrics

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706706823219/b5043430-875c-48a7-b89d-8e16e6e32afd.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706706853607/bd5abd7d-5dbd-4ebd-9360-b2037bbde63a.png align="center")

Find a needle in a haystack with log queries?

Loki : Log

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706706941746/33c3c009-499a-42c6-9346-f2eca1500b91.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706707053993/5a156a4c-03e6-42bf-900e-36ccd26d7534.png align="center")

Trade off: One log line searching time consuming.

Quey Language: LogQL

TSDB index same used in prometheus.

Line based deletes: As occasionally some secrets will be logged in logs :)

Find where latency and error originate across many services with traces?

Tempo

**Distributed call graph.**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706707329166/4e198ded-f2ab-4c1f-b07c-45760e8a7d07.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706707377036/d6e356d6-78d4-487c-b5dd-7f7fd969853b.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706707447098/540127c7-7666-41b6-8109-f55b04c6bb44.png align="center")

TraceQL ----- PromQL

```bash
{resource.service="server"&&span.target="/account"}>>{status=error}
```

Show me all the spans where the request was received by my server on the account endpoint.

**Profile - Pyroscope**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706707981405/7895644f-f8c0-4b6a-84b6-b875e8435042.png align="center")

Sampling Profile

Flame graph and icle graph

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706708048102/e5bdea1d-fcc7-4531-a8d9-bcf24947632e.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1706708158592/8c2c70eb-56dc-4605-9853-65eaac72f4df.png align="center")

OSS observability on AWS with Mimir Loki Tempo & Pyroscope

Mimir: For Metrics

Loki: For logs

Tempo: For Traces

Pyroscope: Continuous Profiles