Ingesting historical data into InfluxDB




A friend and I use InfluxDB (v2.7.1) to store time-series data for our joint side project. Every week, we find say a couple time-series that interest us, and ingest all data points spanning about 50-100 years into the past into InfluxDB.

Recently, we've noticed that our InfluxDB docker container crashes frequently on writes. Since the time-series database is a central dependency for our project, this is unacceptable and we set out to root-cause the issue.

Some details:

Problem overview

InfluxDB pod logs showed the following:

$ kubectl logs influxdb-stateful-set-0 --previous > /tmp/logs.out
$ cat /tmp/logs.out
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f1baa892f13 m=2 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f1baa892f13
stack: frame={sp:0x7f1b815f43f8, fp:0x0} stack=[0x7f1b80df4d58,0x7f1b815f4958)
0x00007f1b815f42f8:  0x0000000000000037  0x0000000000000000
0x00007f1b815f4308:  0x0000000000000000  0x0000000000000000
0x00007f1b815f4318:  0x0000000000000001  0x00007f1bacfc

The Go dump spans about 50 thousand more lines and I have omitted it here.

We see that a pthread_create failed. I then look inside minikube VM to see what went wrong.

First, I ssh-ed into minikube via minikube ssh, and then I ran sudo dmesg and the following line stood out:

cgroup: fork rejected by pids controller in /system.slice/docker-fcd16bcd491dc3e88fabe9cc504114a47218dbeaddcb7958db9d13c929823f2d.scope/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78140808_1a7c_4b38_b083_6b5a82381b44.slice/docker-86a3884408f0075dc171fd40444ba21f164045a8970eb8a56ab0168aa93ecab5.scope

So our InfluxDB process was hitting the cgroup limit that docker set for the process count.

I ran the following watch -n 0.01 systemctl status kubepods-besteffort-pod78140808_1a7c_4b38_b083_6b5a82381b44.slice while repro-ing the issue to see what the Task count was going to, but it only hit a couple hundred before InfluxDB died. The limit was around 2000 tasks.

Then, instead of checking for the thread count via minikube, I instead ssh-ed into InfluxDB container and ran the following script while repro-ing the issue:

$ cat script.sh

cat /sys/fs/cgroup/pids.current
cat /sys/fs/cgroup/pids.max
cat /sys/fs/cgroup/pids.events

echo "---------------"
$ chmod +x ./script.sh 
$ watch -n 0.01 ./script.sh

Indeed, I saw pids.current get 2000+ (around pid.max) before InfluxDB died.

So I just chalked up the systemctl Task count anomaly to systemctl maybe having some lag in its data and accepted that InfluxDB was for some reason spinning up 2000+ processes when we sent it write requests.

I then decided to look at logs from InfluxDB before it crashed to see why it might be spinning up so many processes. What stood out was how dense the log was. There were thousands of log line within a second. That didn't seem right. And a lot of log lines originated from the storage controller and seemed related to opening some shard file.

I looked at the number of shards we had in our bucker via ls /var/lib/influxdb2/data/$BUCKET_ID/autogen/ and it stood out. I dove deeper into docs [1] and [2] in the reference and also stumbled on doc [3].

Indeed, it seemed like by default, InfluxDB creates a shard per 7 day if your bucket has infinite retention, which ours did. So when we wrote data spanning the last 100 years into the database, it had to CRUD across a lot of shards.

You can run the following command to check your shard period

$ influxdb-stateful-set-0:/# influx bucket list --org $ORG_NAME --token $TOKEN
ID                      Name            Retention       Shard group duration    Organization ID         Schema Type
6a8934b488296ff3        _monitoring     168h0m0s        24h0m0s                 REDACTED                implicit
6ce28becfaed76d2        _tasks          72h0m0s         24h0m0s                 REDACTED                implicit
REDACTED                REDACTED        infinite        168h0m0s                REDACTED                implicit

As we see above, we only had 168h for our shard period.


We set out to update the shard of our existing bucket, but since it already had data, that didn't seem to really work. At least not 5 minutes after shard update.

$ influxdb-stateful-set-0:/# influx bucket update --token $TOKEN --shard-group-duration 80w --id $BUCKET_ID --retention 0

So then, I deleted the bucket to ensure all shards were deleted, and then re-created an empty bucket with proper shard period, and then verified that we no longer see the issue.

$ influxdb-stateful-set-0:/# influx bucket delete --name $BUCKET_NAME --org $ORG_NAME
$ influxdb-stateful-set-0:/#  influx bucket create --name $BUCKET_NAME --token $TOKEN --shard-group-duration 100w  --retention 12000w --org $ORG_NAME


InfluxDB defaults weren't lined up with our use-case. Hope this helps others using InfluxDB for a similar use-case.


  1. InfluxDB Internals
  2. InfluxDB Config Options
  3. InfluxDB Tech Tips