One day, our Redis instance started showing unusually high CPU usage.
At first, it looked like a simple traffic spike. DNS traffic had increased, Redis CPU was climbing, and timeouts were starting to feel real. There was only one thing using that Redis instance: CoreDNS with the Redis plugin.
The real problem was not that Redis was overloaded. It was that the client was re-authenticating on every Redis operation. This was not a Redis outage in the usual sense; it was repeated AUTH churn caused by poor connection reuse. The Redigo pool was effectively acting like a connection factory instead of a pooled client, so ordinary DNS lookups were creating far more Redis authentication calls than they should.
That narrowed the problem quickly. This was not a large distributed-systems mystery. It was a smaller failure mode hiding in plain sight.
The important lesson is simple: a CPU graph says Redis is busy. Command-level metrics tell you why.
The Symptom
The first signal was Redis CPU during a DNS request spike.
The useful signal came from Redis command metrics. AUTH was climbing with the application reads. That should feel wrong.
In a healthy Redis client, AUTH is tied to connection creation, not to each lookup. A client opens a connection, authenticates once, keeps it around, and reuses it. Under load, commands like HGET, HKEYS, SCAN, or DBSIZE should grow much faster than AUTH.
This is the normal behavior Redis clients aim for; the principle is consistent across client libraries, including the pooled connection design described in the official Redis client guidance and the connection lifecycle in the Redis protocol documentation. See Redis client connection pooling and redis-py connection pooling for the broader model.
When AUTH scales with request volume, the client is repeatedly creating authenticated connections instead of reusing them.
The Root Cause
The CoreDNS Redis plugin we investigated used redigo.Pool, but the pool was created without idle connection settings.
The original shape looked like this:
redis.Pool = &redisCon.Pool{
Dial: func() (redisCon.Conn, error) {
opts := []redisCon.DialOption{}
if redis.redisPassword != "" {
opts = append(opts, redisCon.DialPassword(redis.redisPassword))
}
return redisCon.Dial("tcp", redis.redisAddress, opts...)
},
}At a glance, it looks pooled. Technically, it is using a pool.
The missing detail matters. Redigo does not keep idle connections unless the pool is configured to do so. With MaxIdle left at its default zero value, a connection returned with Close() is not retained for later reuse. The relevant behavior is documented in the Redigo pool API. The Redis protocol itself also confirms that authentication is per connection, not per command: see Redis AUTH.
So this pattern:
conn := redis.Pool.Get()
defer conn.Close()can still create a new Redis connection again and again. With Redis authentication enabled, every new connection sends another AUTH.
Request traffic turns into connection churn. Connection churn turns into authentication churn. Authentication churn turns into avoidable Redis CPU pressure.
Why DNS Makes This Worse
DNS traffic is bursty by nature. A small operational event can create a sudden wave of lookups:
- pods restarting;
- clients retrying after a timeout;
- caches expiring together;
- a deployment changing service discovery;
- external traffic hitting the application at once.
If every DNS lookup performs Redis work, and every Redis operation opens a fresh authenticated connection, the multiplier becomes painful. This is the same burst pattern DNS systems are known for: a small trigger can amplify a fan-out effect, especially when each lookup turns into fresh connection setup work. See CoreDNS plugin architecture and CoreDNS DNS caching.
That is why this kind of bug survives basic testing. The code path works. The plugin answers queries. Then production traffic exposes the cost of missing connection reuse.
Reproducing The Shape
I reproduced the behavior locally with Redis protected by a password. The first reproduction used a small Go client that mirrored the plugin pattern: a Redigo pool with a Dial function, but no idle connection capacity.
The simulated lookup path performed Redis commands similar to the plugin:
DBSIZESCANHKEYSHGET
With 100000 simulated lookups against a Redis container limited to 2.00 CPUs, the bad path produced:
completed 100000 simulated requests at concurrency 25 in 48.035s
cmdstat_dbsize:calls=100000,usec=42948,usec_per_call=0.43,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=400003,usec=804301,usec_per_call=2.01,rejected_calls=0,failed_calls=0
cmdstat_hkeys:calls=100000,usec=233942,usec_per_call=2.34,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=100000,usec=201302,usec_per_call=2.01,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=100000,usec=314695,usec_per_call=3.15,rejected_calls=0,failed_calls=0The important number is AUTH: 400003 authentication calls for 100000 simulated lookups.
Giving Redis more CPU did not change the shape of the defect. It still authenticated roughly once per Redis operation.
Verifying With Real CoreDNS
A standalone client is useful, but I wanted stronger proof. So I built a local CoreDNS container with the patched Redis plugin source and ran it beside Redis in Compose. The CoreDNS plugin model and configuration format are documented in the CoreDNS plugins reference and the CoreDNS Corefile guide.
The working Corefile syntax for this plugin was:
.:1053 {
redis {
address redis:6379
password devpass
pool_max_idle 10
pool_idle_timeout 240000
}
errors
log
}For the bad run, I set pool_max_idle to 0, reproducing the original no-idle-reuse behavior.
CoreDNS answered from Redis:
192.0.2.10Then I sent 1000 real DNS queries to CoreDNS.
Fixed run, with pool_max_idle 10:
cmdstat_hget:calls=1000,usec=4554,usec_per_call=4.55,rejected_calls=0,failed_calls=0
cmdstat_hkeys:calls=1000,usec=6765,usec_per_call=6.76,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=1000,usec=2009,usec_per_call=2.01,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=4,usec=27,usec_per_call=6.75,rejected_calls=0,failed_calls=0Bad run, with pool_max_idle 0:
cmdstat_hget:calls=1000,usec=5564,usec_per_call=5.56,rejected_calls=0,failed_calls=0
cmdstat_hkeys:calls=1000,usec=6703,usec_per_call=6.70,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=1000,usec=1042,usec_per_call=1.04,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=3001,usec=13267,usec_per_call=4.42,rejected_calls=0,failed_calls=0That confirmed the fix through CoreDNS itself. The DNS query path still performed Redis reads, but AUTH no longer scaled with every command.
The Fix
The fix is to configure the existing Redigo pool so it retains reusable idle connections:
redis.Pool = &redisCon.Pool{
MaxIdle: redis.poolMaxIdle,
MaxActive: redis.poolMaxActive,
IdleTimeout: time.Duration(redis.poolIdleTimeout) * time.Millisecond,
Dial: func() (redisCon.Conn, error) {
// existing dial options
},
}I added Corefile options so the behavior can be tuned without recompiling:
pool_max_idle 10
pool_max_active 0
pool_idle_timeout 240000The defaults are intentionally conservative:
pool_max_idle:10pool_max_active:0, meaning no active connection limitpool_idle_timeout:240000milliseconds
In the local reproduction, the fixed path changed the result dramatically:
Before: request traffic turned into connection churn. After: the pool kept idle connections around, so Redis work stayed focused on lookups instead of repeated authentication.
Peak Redis CPU in a synthetic local test can still be high after the fix because the client can push Redis much faster. The meaningful change is the shape of the work: less authentication churn, fewer connection setups, and a much shorter pressure window during bursts.
The Monitoring Lesson
This is why Redis command metrics matter.
A generic CPU graph can tell you Redis is hot. It cannot tell you why. Command-level metrics show whether Redis is busy doing useful application reads, expensive key scans, authentication churn, connection churn, or something else entirely.
The important part is not just that Redis is under pressure. It is what it is spending that CPU on.
In this case, the important dashboard was not just Redis CPU. It was Redis CPU plus commandstats.
That distinction matters during an incident. If all you see is high CPU, the natural response is to scale Redis, increase timeouts, or reduce traffic. Those actions may buy time, but they do not explain the failure mode. In our case, a larger Redis instance still showed the same bad command shape: AUTH grew with the lookup workload.
That changed the question from “Why is Redis slow?” to “Why is this DNS path authenticating thousands of times?” It points toward client behavior, connection lifecycle, pooling, and retries instead of treating Redis as a black box.
Good monitoring also gave us a way to prove the fix. Before the change, 1000 real DNS queries through CoreDNS produced 3001 Redis AUTH calls. After the change, the same path produced 4. Without command-level metrics, the fix would have been a belief. With monitoring, it became measurable.
The useful signals were:
- Redis CPU increased during DNS traffic spikes;
AUTHincreased alongside lookup commands;- only CoreDNS was using this Redis instance;
- the plugin connection lifecycle explained the metric pattern.
Without command metrics, this could easily have turned into guesswork: scale Redis, increase timeouts, blame DNS traffic, or chase unrelated infrastructure noise. With the right metrics, the investigation had a direction.
Collecting This With Prometheus
For Redis, the easiest way to expose command-level metrics to Prometheus is to run redis_exporter beside Redis and scrape it from Prometheus. The exporter guidance and metric model are maintained by the project itself: Prometheus Redis Exporter and the Redis metrics reference.
A minimal Compose service looks like this:
services:
redis-exporter:
image: oliver006/redis_exporter:latest
command:
- --redis.addr=redis://redis:6379
- --redis.password=devpass
ports:
- "9121:9121"Then add a Prometheus scrape target:
scrape_configs:
- job_name: redis
static_configs:
- targets:
- redis-exporter:9121The exact metric names can vary by exporter version, but redis_exporter exposes command call counters in this shape:
redis_commands_total{cmd="auth"}
redis_commands_total{cmd="hget"}
redis_commands_total{cmd="hkeys"}
redis_commands_total{cmd="dbsize"}The first graph I would build is AUTH rate:
rate(redis_commands_total{cmd="auth"}[5m])Then compare it against useful Redis work:
sum by (instance) (rate(redis_commands_total{cmd=~"hget|hkeys|dbsize|scan"}[5m]))For this specific failure mode, the most useful alert is not “Redis CPU is high.” The useful alert is “AUTH is unexpectedly growing with application commands.”
Example PromQL:
rate(redis_commands_total{cmd="auth"}[5m])
/
clamp_min(sum by (instance) (rate(redis_commands_total{cmd=~"hget|hkeys|dbsize|scan"}[5m])), 1)
> 0.25That threshold is intentionally conservative and should be tuned to your environment. The point is the ratio. In a healthy pooled client, AUTH should be closer to connection creation. It should not track every lookup command.
I would pair that with Redis CPU:
rate(redis_cpu_user_seconds_total[5m]) + rate(redis_cpu_sys_seconds_total[5m])Together, those two graphs tell a much better story:
- Redis CPU shows impact.
- Redis command rates show cause.
- The
AUTHratio shows whether the client is re-authenticating too often. - The before/after dashboard proves whether the fix changed behavior.
Production Takeaway
This was not a Redis outage in the usual sense. Redis was doing exactly what the client asked it to do. The problem was that the client integration was asking it to authenticate far too often under load.
The fix was not just code. It was operational awareness:
- monitor Redis command-level metrics;
- alert on unexpected
AUTHgrowth; - compare connection churn against request volume;
- load test the integrations on hot paths like DNS;
- confirm that a “pool” is actually keeping reusable connections around.
Monitoring does not replace good engineering, but it makes bad assumptions visible before they become production incidents. For the broader operational model, the Redis and Prometheus docs are a good reference for how to instrument a system before it fails in production: Redis monitoring overview and Prometheus query language.
In this case, the graph told the story: DNS traffic went up, Redis reads went up, and AUTH went up with them. The code explained why. The reproduction made it repeatable. The CoreDNS test proved the fix.
That is the kind of evidence you want before touching production infrastructure.