Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CoreDNS Poor Performance Vertically #7111

Open
Hexen1988 opened this issue Feb 1, 2025 · 4 comments
Open

CoreDNS Poor Performance Vertically #7111

Hexen1988 opened this issue Feb 1, 2025 · 4 comments
Labels
question

Comments

@Hexen1988
Copy link

Hexen1988 commented Feb 1, 2025 •
edited by SuperQ
Loading

Hello Experts,

I am using CoreDNS v1.12.0 on an 8 core server with 32GB memory, I am getting poor QPS performance with UDP even when setting GOMAXPROCS=8 , the QPS at 90% CHR isn’t exceeding 2000 QPS with UDP. Multi socket has been set to be 4. Note its important for me to scale out vertically using a single CoreDNS instance due to some custom plugin that will be used later. I am not hitting any upstream limitation as I am directly forwarding to an upstream that can do 15K QPS at the same 90% CHR. So, its not an upstream limitation.

Corefile:

.:5353 {
    bufsize 4096
    multisocket 4


    cache {
        # cache specific configurations
        success 1000
        denial 200
        prefetch 5 1m
    }

    # Forward all queries to an upstream resolver
    forward . X.X.X.X:1153 {
        max_concurrent 100000
        policy round_robin
        prefer_udp
    }
}

I am using minimal packages:

root@XXXXXXXXXXXXXXXXX# ./coredns -plugins
bufsize
cache
forward
multisocket

root@XXXXXXXXXXXXXXXXXX# dnsperf -s 127.0.0.1 -d CHR_90.txt -l 60 -Q 10000 -m udp -p 5353  -S 2 -q 1000 -T 4 -c 100

  Response codes:       NOERROR 243338 (99.45%), SERVFAIL 25 (0.01%), NXDOMAIN 1312 (0.54%)
  Average packet size:  request 33, response 72
  Run time (s):         64.199124
  **Queries per second:   1980.234838888**

Any tips or guidance will be greatly appreciated.

Regards,
Hexen
@johnbelamaric
Copy link
Member

For a quick test, can you see what you get with near 100% CHR? That's a quick way to narrow down the issue.

You'll need to figure out if you are CPU or network bound, or if neither, it's likely some internal concurrency management issue.

Take a look at the testing @Shmillerov and others did during development of the multi-socket plugin. Look at this PR and dig through the links to see their techniques and tools.

@SuperQ
Copy link
Collaborator

SuperQ commented Feb 4, 2025

It's also worth taking some pprof samples and posting them to https://pprof.me to share here.

@Hexen1988
Copy link
Author

With 100% CHR I can get around 15k-20k qps. The weird thing is that I am using an upstream as forwarder that is capable to do 2000 qps at 90% CHR. Not sure why I am stuck at 2000 qps given upstream is scaling and I am able to reach 15k-20k qps with 100% CHR.

@SuperQ , appreciate letting me know what kind of data would be useful to upload ?

@SuperQ
Copy link
Collaborator

SuperQ commented Feb 4, 2025

You want /debug/pprof/profile.

https://coredns.io/plugins/pprof/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SuperQ @johnbelamaric @Hexen1988