We trained a small model to guess subdomains
On this page
Hey there,
When you enumerate subdomains, the passive sources give you what somebody already saw: certificates, old DNS records, links found by a crawler. The rest you have to guess. For twenty years, guessing has meant a wordlist. You try dev, staging, vpn and a few hundred thousand others, on every domain, in the same order.
The thing is, nobody names their machines at random. If you already know api-staging, grafana and fr-par-01, you have a fair idea of what else is in there, and it isn’t the same list as for the domain next door.
So we trained a model to do just that: read the names we already know for a domain and guess the missing ones. It’s in Profundis now, as AI Discovery. This article is about how it went, including the parts that went nowhere :)
What does it look like on an example?
Take a freight company we made up. We know 14 of its subdomains: www, mail, vpn, sso, api, api-staging, tracking, tracking-staging, grafana, gitlab, fr-par-01, fr-par-02, de-ham-01 and portal.
Ten more exist. We wrote them down before running anything, then gave the 14 known names to the model and let it make 10,000 guesses. That took 2 seconds on one GPU. Here is where each hidden name came out.
The company and its hidden names are invented. The ranks are the model’s real output on those 14 names. The scale is logarithmic.
Seven of the ten are in its first 112 guesses and an eighth comes much later. It never guessed the last two. We’ll get to why.
Would ChatGPT be better at generating candidates?
We get asked this a lot. From what we measured, no: a big general model finds fewer names than our small one, and it takes minutes where ours takes seconds.
We have to be precise about what we tested, because it wasn’t ChatGPT or Claude. To make a model guess, you give it the list of somebody’s subdomains, and we don’t send that to anyone. So we took the biggest open model we could run on our own machines: Qwen3.8 with 27 billion parameters, in a quantised build, on three GPUs, with a single prompt and sampling setting. Ours has about 18 million parameters and fits in a 71 MB file. It’s roughly 1,500 times smaller.
Everyone got the same 13 domains and the same known names as input. None of these domains was in the training data, but they come from the set we tune on, so this is home ground for us. We counted how many of the 289 hidden names came out in the first 1,000 guesses. (Names that are just an IP address written out are left out here. More on those below.)
Hidden names found in each model’s first 1,000 different guesses, on 13 domains, with the same known names given to each.
The big model isn’t bad at this. It knows what a company’s infrastructure looks like, and the names it found, it found for good reasons. But more than one line in four was a duplicate, and 1,000 different names took it about six minutes per domain. Ours gave the 10,000 guesses of the example above in 2 seconds, on a single GPU.
We also checked whether it was worth adding its guesses to ours. Of the 83 names it found, our own pipeline already had all but 2.
Speed matters more than it looks, because almost every guess is wrong, whoever makes it. In the example, 8 real names in 10,000 guesses is a good day. So you need a lot of guesses, and each one gets checked in DNS before anyone sees it. Nobody waits six minutes per thousand guesses on a search page.
Would ChatGPT or Claude do better than Qwen? Our guess is yes, a little: they’re bigger, and they answer faster than our three cards. We haven’t measured it, so we won’t give a number. Two things wouldn’t change, though. Our model has read 932 million names under 263 million domains from our index, and nothing else, while a general model has read the web. And you would be paying a provider for every guess, more than 99 in 100 of them wrong, while sending it the names of your target.
How does it compare with the other tools?
We looked for other trained models to put in this test. Outside subwiz, from Hadrian Security, there’s almost nothing you can actually run. The closest we found is a 2022 research paper that trains a GAN on subdomain names, with no released tool that we could find.
What people really use to build candidates are permutation tools. They aren’t trained. alterx and dnsgen combine the names you already know with wordlists and patterns, and regulator works out rules from the names themselves.
So we ran all of them on one test: 75 domains whose full list we know, 15 for each size, the same known names given to everyone (1,024 at most), and every guess scored against the full list with no DNS involved.
One thing isn’t equal, on purpose. The models rank their guesses, so we count their first 10,000. The permutation tools don’t rank anything: they print everything, for a resolver to try it all. So for them we count the whole output, however long it is.
Share of the 21,519 hidden names found on 75 domains. For the three models, the first 10,000 guesses per domain. For the permutation tools, everything they generate, however long: a few hundred names for regulator on most domains, millions for dnsgen on the big ones.
Our two models find the most names, 6,236 and 5,384. dnsgen gets to 20.5%, but it needs about 118,000 guesses on a typical domain, and millions on the big ones. Its output was cut at 5 million lines on 18 of the 75 domains, so that figure is a floor. alterx stays under 7%, even with its -enrich option, and subwiz finds 2.5%.
regulator needs explaining, because it finds 4,107 names (19.1%) with very few guesses. It has no wordlist. It reads the names you know, works out rules from them (ip-10-0-0-1 and ip-10-0-0-2 give ip-10-0-0-N), and generates only what those rules produce. On a typical domain that’s about 900 names. On an internet provider’s domain full of numbered names, it’s tens of thousands, and that’s where 3,985 of its 4,107 finds are.
The same test by number of names already known for the domain. Our model and subwiz: first 10,000 guesses. regulator and dnsgen: whole output.
By size, the picture changes, and it’s the fairer view: the totals above are pooled over names, so a few big domains weigh a lot (the smallest band holds 391 of the 21,519). On small domains, our model finds about half of the hidden names, and the best of the others is subwiz, with about a quarter. In the 201 to 1,000 band, dnsgen gets within a point of our model, at the cost of 1.4 million guesses per domain. On the giants, regulator almost catches up: 22.2% against 23.5%. Those domains are full of numbered names, and working out fr-par-03 from fr-par-01 and fr-par-02 is exactly what regulator does. It also finds 1,083 names that our bigger model doesn’t have in its first 10,000 guesses.
That’s the model alone, though. In production, numbered names and addresses are handled by plain loops before the model (more on that below), and this test doesn’t count those loops.
It isn’t a perfectly even match either. Our models were trained on other domains with a complete list, subwiz on its authors’ data, and the permutation tools ran with their default settings. None of the 75 domains was used to train ours, but they come from the validation set, the one we compare our own versions on.
On speed, the permutation tools win easily: they need no GPU, and the slowest of them took 82 seconds on its worst domain. Among the models, 10,000 guesses take subwiz 41 seconds on average and our small one between 2 and 15, on the same GPU.
AI vs brute force and permutation: what does a guess cost?
Finding names isn’t the only thing that counts. Every guess is a DNS query sent to somebody’s servers, so the cost is counted in queries and in minutes.
Same 75 domains, same 21,519 hidden names, and three figures for each method: how many queries it sends at one query per guess, how long they take at 1,600 queries a second (the pace of one of our production rounds), and how many hidden names it finds per 1,000 queries. One query per guess is a floor: in production, with its checks, a guess costs two to three. Brute force here is a public wordlist tried in its own order: the 114,000 names of SecLists’ subdomains-top1million-110000 and the 3 million names of the n0kovo list.
| Method | DNS queries | DNS time at 1,600 queries/s | Names found | Per 1,000 queries |
|---|---|---|---|---|
| Our full pipeline, production budget | 4.8 million | 50 min | 12,578 (58.5%) | 2.61 |
| Our bigger model, 10,000 per domain | 750,000 | 8 min | 6,236 (29.0%) | 8.31 |
| Our small model, 10,000 per domain | 750,000 | 8 min | 5,384 (25.0%) | 7.18 |
| regulator, whole output | 420,000 | 4 min | 4,107 (19.1%) | 9.77 |
| dnsgen, whole output | 79.9 million | 13h 52m | 4,406 (20.5%) | 0.06 |
alterx -enrich, whole output | 54.4 million | 9h 27m | 1,306 (6.1%) | 0.02 |
| alterx, whole output | 14.2 million | 2h 28m | 722 (3.4%) | 0.05 |
| SecLists wordlist, first 10,000 names | 750,000 | 8 min | 856 (4.0%) | 1.14 |
| SecLists wordlist, all 114,000 names | 8.6 million | 1h 29m | 1,090 (5.1%) | 0.13 |
| n0kovo wordlist, all 3 million names | 224 million | 38h 52m | 1,485 (6.9%) | 0.007 |
| subwiz, 10,000 per domain | 744,000 | 8 min | 537 (2.5%) | 0.72 |
At the same 10,000 queries per domain, our bigger model finds 6,236 names where the SecLists list finds 856. That’s seven times more for the same eight minutes of DNS.
A longer list doesn’t fix it. The whole n0kovo list sends 224 million queries, almost 39 hours at our pace, and ends with 1,485 names: 7 finds per million queries.
dnsgen does find a fifth of the names, but it takes 80 million queries and almost 14 hours to get there.
regulator has the best rate in the table, 9.77 names per 1,000 queries, better than our models. Nearly all of it comes from the big domains full of numbered names: 3,985 of its 4,107 finds. On the small ones it runs out of guesses quickly and finds 22 names out of 391, where our bigger model finds 197.
The first row is what we actually run: the model after the plain loops for numbers and addresses, at its normal budget. It finds 58.5% of the hidden names in 50 minutes for the 75 domains. This is home ground, though: these 75 domains come from our validation set. On the held-out test set below, which is larger and harder, the same pipeline is at 41%.
Generating the guesses is the small part of the cost. A wordlist costs nothing. The permutation tools take from 18 seconds to 12 minutes for the 75 domains, our models about 5 minutes on one GPU, and subwiz 51 minutes on the same GPU.
And against a passive tool?
It’s a different job, so take this one as an order of magnitude. On 12 domains, we ran subfinder with its free sources (no API keys), then subfinder with subwiz on top, then our full recon engine, which has its own passive sources and the model at the end. Every name from every tool went through the same DNS check.
Between them, the tools found 907 names that exist. subfinder had 558 of them, subfinder plus subwiz 613, our engine 872. The model was the first to find 13 of those. Of the names we missed, 29 came from one passive source we didn’t have.
One thing we didn’t expect: three out of four names that subfinder returned no longer resolved.
Where do the names come from?
From what we collect every day. Names that show up in certificate transparency logs, millions of them a day, are resolved by our DNS workers. So are the names from the TLD zone files and the ones our own recon runs find. What resolves goes into our DNS index and stays there, with its history: 8.4 billion DNS records today.
The model learned from the names of that index and from nothing else: 932 million names under 263 million domains.
How do you grade a guesser?
You need domains where you know the full answer. We have the complete list of names for more than 60,000 domains. How we get those lists would be an article of its own, so not today.
For each of them, we compared the full list with what our index already had. What’s missing is the exam: the model gets the names we knew, and we count how many of the missing ones it finds. Some of those domains were held out from the start and never used for training.
All the scores below are shares of hidden names found, and a change is given in points of that share. They don’t all come from the same exam, so compare them inside a section, not across the article.
How many does it find?
On the held-out test set, 41% of the hidden names, with about 10,000 guesses per domain.
The exam is 1,192 domains held out from the start, with 101,522 names we didn’t have. It ran once. What gets graded is the whole guessing pipeline: the model, plus the plain loops for numbers and addresses that come up below. In this offline test, a guess counts as one DNS query. The budget is 10,000 guesses for a domain, and more for the big ones: a hundred per name already known.
Share of the 101,522 hidden names found on 1,192 domains, by number of guesses per domain (the median: big domains get more). The grey bars are the same exam with our counting scripts alone, without the model.
From there, ten times more guesses buy ten more points, then six. And they get expensive. At 1,000 guesses per domain, 7 in 1,000 are names we didn’t have. At 10,000 it’s a bit more than 1 in 1,000, and at a million, 2 in 100,000. So the order of the guesses counts for more than their number, and every one of them is checked before anyone sees it.
What made no difference?
Most of what we tried. Each of these sounded like a good idea at the time.
Change in points of hidden names found. Each idea is measured against its own reference, on validation domains.
A model three times bigger finds 4 more names out of 289 at 1,000 guesses (the first comparison chart: 125 against 121), and 4 points more at 10,000 guesses on the 75 domains. It is better, for three times the compute. It isn’t the jump we hoped for.
Twice as much ground truth changed nothing we could measure. We retrained the model with only half of the domains with a complete list, and it lost 0.6 points, which is within the noise of the measurement.
Retraining it on its own finds gave one point the first time and 0.2 the second, so we stopped. A second model to reorder the guesses gave 0.14 points for twice the compute. Telling it more about the company (the software on its hosts, the page titles, the business sector, the organisation’s other domains) never moved the score enough to keep.
Teaching it what doesn’t exist
On a domain with a complete list, we know for certain which names don’t exist. We took the most plausible ones and trained the model to push them down.
The score dropped by 2.35 points, and small domains took most of the damage: the share of common names found there went from 93% to 70%.
The reason is almost funny. The most plausible names a domain doesn’t have are ftp, smtp and mail. We had taught the model that common names are a bad idea.
What helped?
Three things changed the numbers. Two of them aren’t about the model.
A counting script
Before training anything, we wrote the dumbest thing we could think of. It counts which names show up together across domains (a domain with grafana tends to have prometheus) and outputs the names that most often go with them.
Then our first trained model found 35.0% of the hidden names in 1,000 guesses, and the counting script found 34.3%. So the neural network beat a few hundred lines of counting by less than a point. That’s how we learned the first model had a problem.
Letting it see the domain
The problem was silly. The first version only read 16 of the names we already knew for a domain. That’s fine for a small one. On a domain with a thousand names, it was guessing with its eyes almost closed.
The second version reads much more. The test changed too between the two, so we can’t put one number on that fix alone. We ran both against each other again, on another exam: 1,997 domains where we had hidden part of what we knew.
Hidden names found in 1,000 guesses on 1,997 domains, by number of names already known for the domain.
The model is ahead by about ten points up to 200 known names. On the giants, the counting script still wins.
Names that are just an address
When we ran the held-out test set, the score came out about ten points below what the same measure gave on our validation domains. We split the names every way we could think of, and the whole gap came from one kind: names that are an IP address written out, like host-203-0-113-42. Internet providers create them by the thousand.
On the domains we had used to tune everything, those made up 40% of the names to find, and they sat in address blocks we already knew. In the exam they were 23%, in blocks that nothing pointed to.
No model is going to guess an address. So those names aren’t the model’s job anymore. A plain loop tries one address of a block and, if it answers, asks for the rest. Same thing for numbered series: once you know fr-par-02, you don’t need a neural network to try fr-par-03. That’s also why the counting script kept winning on the giants, where almost half of the hidden names are numbered ones.
What will no guesser ever find?
A bit under a quarter of the names.
We took 12,516 domains with a complete list where we already knew at least 6 names. They had 726,539 names we didn’t know. We sorted every one of them by what it would take to guess it.
What 726,539 hidden names are made of. In orange, the kind that can’t be guessed.
In the invented company, staging belongs to the first row and fr-par-03 to the second. tracking-api is made of two words the domain already uses, api-dev of one of its words plus a common one. A name like db.internal-tools sits under a parent we don’t know either.
The last row is a project name, a product, somebody’s cat. Nothing in what we know of the domain points to it, and it’s why the model never guessed heron in the example. nl-rtm-01 is the same story: a city code the company had never used anywhere we could see.
So the ceiling for guessing, with an unlimited budget, is about 77% of the hidden names. That counts the names under a hidden parent as guessable, which is generous: you have to find the parent first. We had 90% in mind when we started. Guessing alone can’t get there. The remaining names have to be read somewhere: in a certificate, in an old DNS record, on one of the company’s own pages.
How does it work in Profundis?
Search a domain on profundis.io and AI Discovery is offered next to the results. It reads the names we know, makes its guesses and checks every one of them in DNS. A run makes up to 20,000 guesses and stops after 60 seconds at most. It works in rounds: the names found in one round are given back to the model for the next.
Only the names that resolve are added to your results, with a “Found by Profundis AI” badge. The model runs on our own servers, and you’re charged only if it finds a new name.
Don’t expect a flood. On its first evening in production, we pointed the larger version, the one our own recon engine runs, at four big domains that we already knew well. A round there is 50,000 guesses, which makes about 133,000 DNS queries at about 1,600 a second, and takes about 100 seconds. The four runs found 17, 14, 13 and 3 new names. On a small domain where we already have everything that can be guessed, it often finds nothing.
What’s next?
The chart of the hidden names is also our to-do list.
Names made of one word of the domain and one common word are 4.5% of what’s hidden. We built a generator for them and measured it: 0.007 new names per 1,000 guesses, against 1.12 for the model. There are too many combinations and nothing to rank them with, so we dropped it.
regulator’s rules look better. Of the 1,083 names it found and our model didn’t, 914 are address names that our loops already catch. But 137 are found by nothing else we have, mostly numbered names under deep parents. That’s 0.33 new names per 1,000 guesses, five times what the pipeline’s own next guesses give. They sit in only 13 domains, so we ran it again on 1,289. It holds up: 809 names the pipeline misses, 0.28 per 1,000 guesses against 0.055 for the pipeline’s own next guesses, and the gap is widest on the biggest domains.
Names under a hidden parent are 20.7%. Finding the parent and going again underneath is what the rounds are for. We haven’t measured how far they get.
Then there’s the DNS budget. In a replay, we stopped early on the domains that had stopped answering and gave their guesses to the ones still finding names. That gave about 2 more points for the same total number of queries. It isn’t in production yet.
For the words that belong to one organisation, we don’t have a good answer. It’s the part we’d most like to crack. We checked our own certificate and host indexes for them: page titles give back a few (2.8% of those words), the certificates almost none.
We also keep feeding the model with our own data, and it doesn’t always help. The first training only saw the names that carry a DNS record. The names that records point to, mail servers mostly, are another 46.7 million. We trained a version with them next to a control without. On 279 test domains, the small model found fewer names with them than without, 0.5 points less. Most of those names are mail and mx under tiny domains, and they pull the model toward generic names. The bigger model’s result isn’t in yet.
And the exam itself has to change. The one above runs offline, one guess at a time. The next one will grade real production runs, with their timeouts and their DNS servers having a bad day.
We’ll come back with numbers when some of this has moved.
That’s it for today. If it finds something odd on one of your domains, tell us :)
Cheers, Sicarius from the Profundis Team
How the figures were counted
- A hidden name is a name present in the domain’s full list and absent from our index when the list was taken. Names under a wildcard are left out.
- The comparison with Qwen: 13 validation domains with 6 to 1,000 known names, 289 hidden names that aren’t an encoded IP address, first 1,000 different guesses of each model. Qwen3.8-27B ran as a 6-bit quantised file on two RTX 3080 Ti and one GTX 1080 Ti. Six minutes is the median time to 1,000 different names over the 13 domains. “Our own pipeline” is our bigger model at 10,000 guesses plus the rest of the guessing pipeline at its normal budget.
- The comparison of tools: 75 validation domains, 15 per size, picked by a fixed rule. Host names only, technical names (such as
_dmarc) left out. Every tool gets the same known names, 1,024 at most. Known names and duplicates are removed from the guesses. Measured on November 13 and 15, 2026, with no network access.- subwiz as published on November 13; it returned 743,580 of the 750,000 guesses requested. Timings on an RTX 4060; 41 seconds is the mean per domain.
- alterx v0.0.6 with its default patterns and words, then with
-enrich; dnsgen 1.0.4; regulator at commit2371a06. alterx and dnsgen print in an order that changes at every run: their figures at 10,000 would be meaningless, so only the whole output is given. It was read up to 5 million lines per domain, a limit reached on 18 domains for dnsgen and 2 for alterx-enrich.
- What a guess costs: a query is one distinct candidate, with the known names removed; production sends more than one DNS query per candidate, for its checks. DNS time is the number of queries divided by 1,600 a second. The wordlists are tried in their file order. dnsgen’s and alterx
-enrich’s figures are lower bounds on 18 and 2 domains, where the output was cut at 5 million lines. The generation time of our models is an upper estimate from 17 of the 75 domains. The full pipeline’s budget is 10,000 guesses per domain or 100 per known name, whichever is larger. - The 12-domain run: each tool’s names are checked the same way (the name must have an address, and a control name must not). subfinder ran with its default sources and no API key. Our engine’s result also holds names with no address (mail-only names, delegated zones), which this check doesn’t count.
- “What made no difference”: each change is against its own control or reference, on validation domains, paired by domain. The bigger model is the exception: it is the difference between the two rows of the comparison, 4 names of 289.
- The counting script against the first model: validation domains, 1,000 guesses. Against the second model: 1,997 domains and 33,847 names hidden from our own index, 1,000 guesses.
- The held-out test set ran once, on 1,192 domains held out until then, with 101,522 hidden names. Its budget is the larger of 10,000 guesses and 100 guesses per known name; the chart gives the median per domain (at the reference budget it is 10,000, and a tenth of the domains get 26,000 or more). The shares are of all hidden names, including those no generator of the pipeline can produce.
- The kinds of hidden names: 12,516 domains, 726,539 names. The kind of a name is decided from what we knew of the domain before guessing.
- The example: an invented company under
example.com. Nothing was sent to DNS. - The data behind the charts: /data/we-trained-a-small-model-to-guess-subdomains/v1/.