8 posts tagged with "Prometheus"

Introduction​

In part 2 we saw that TSDB uses Write-Ahead-Log (WAL) to provide durability against crashes. But it also makes restarts of Prometheus slow when you hit a decent scale because replaying Checkpoint+WAL takes time.

In this post we will understand more about a new feature introduced in Prometheus v2.30.0: taking snapshots of in-memory data during the shutdown for faster restarts by entirely skipping the WAL replay.

Introduction​

When Prometheus has created a bunch of blocks, we need to regularly perform maintenance on those blocks to make efficient use of the disk and keep the queries performant.

In this blog post, we are going to look at 2 topics, compaction and retention, which happen in the background when Prometheus is running.

Introduction​

In the last four blog posts we saw the internals of how data is stored in the TSDB. It's now time to know how to query it. In this blog post we will be looking at 3 types of query that we do on the persistent blocks and briefly about the Head block.

Introduction​

In Part 1, Part 2, and Part 3, we have covered most of the things related to the Head (in-memory) block (i.e. at the time of writing this post, more things to come in Head). In this blog post, we will dive deeper into the persistent blocks which reside on disk.

Introduction​

In the Part 1 of the TSDB blog series I mentioned that once a chunk is "full", it is flushed to the disk and memory mapped. This helps in reducing the memory footprint of the Head block and also helps speed up the WAL replay that we discussed in Part 2. We will be diving deeper into how this is designed in Prometheus in this blog post.

Introduction​

In the Part 1 of the TSDB blog series I mentioned that we write the incoming samples into Write-Ahead-Log (WAL) first for durability and that when this WAL is truncated, a checkpoint is created. In this blog post, we will briefly discuss the basics of WAL and then dive into how WAL and checkpoints are designed in Prometheus' TSDB.

Introduction​

Though Prometheus 2.0 was launched about 3 years ago, there are not much resources to understand it's TSDB other than Fabian's blog post, which is very high level, and the docs on formats is more like a developer reference.

The Prometheus' TSDB has been attracting lots of new contributors lately and understanding it has been one of the pain points due to lack of resources. So, I plan to discuss in detail about the working of TSDB in a series of blog posts along with some references to the code for the contributors.

Introduction​

I successfully completed Google Summer of Code with Prometheus in the summer of 2018. I was mentored by Goutham Veeramachaneni

I did 3 independant addition/fixes as a part of my GSoC. All related to rules/alerting rules. Apart from my proposal, I also fixed some bugs in prometheus/tsdb during GSoC period.