Prometheus TSDB (Part 3): Memory Mapping of Head Chunks from Disk

Ganesh Vernekar

Ganesh Vernekar

Software Engineer @ Grafana Labs

Introduction

In the Part 1 of the TSDB blog series I mentioned that once a chunk is "full", it is flushed to the disk and memory mapped. This helps in reducing the memory footprint of the Head block and also helps speed up the WAL replay that we discussed in Part 2. We will be diving deeper into how this is designed in Prometheus in this blog post.

Prometheus TSDB (Part 1): The Head Block

Ganesh Vernekar

Ganesh Vernekar

Software Engineer @ Grafana Labs

Introduction

Though Prometheus 2.0 was launched about 3 years ago, there are not much resources to understand it's TSDB other than Fabian's blog post, which is very high level, and the docs on formats is more like a developer reference.

The Prometheus' TSDB has been attracting lots of new contributors lately and understanding it has been one of the pain points due to lack of resources. So, I plan to discuss in detail about the working of TSDB in a series of blog posts along with some references to the code for the contributors.

“Optimisations” to Avoid

Ganesh Vernekar

Ganesh Vernekar

Software Engineer @ Grafana Labs

Introduction

During the transition from writing code for the University assignments to writing real world softwares, I had to unlearn many things and learn to not overuse low level optimisations where it was not required. I had a big exposure to compiler optimisations at my University.

I am going to share a couple of “optimisations” to avoid that I see newbies often do, and I won't deny that I haven't done them myself (and still do). I have that in quotes because they are not really optimisations when all factors considered. I will follow up with more blog posts when I have more patterns to share.