Blog Posts

Exploring Cisco’s Top 1 Million Domains Data

Cisco offers a daily list of the million most queried domain names from Umbrella (OpenDNS) users.    I had some time this weekend so decided to spend some time playing around with the data to see what I could find so I spun up a lightsail server and got to work.

Grabbing the file is as simple as:
wget http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip

You can retrieve a specific date like this:
wget http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-yyyy-mm-dd.csv.zip
(Looks like 2017-01-20 is the earliest they have online).

Once you get that downloaded and unzipped (unzip top-1m.csv.zip) you can start exploring.

You can pull out the top 10 domains with this command:
head -n 10 top-1m.csv

1,google.com
2,www.google.com
3,microsoft.com
4,facebook.com
5,doubleclick.net
6,g.doubleclick.net
7,clients4.google.com
8,googleads.g.doubleclick.net
9,apple.com
10,fbcdn.net

(Full Output)

You can search for keywords with this command:
cat top-1m.csv | grep "opendns"

437,opendns.com
719,hydra.opendns.com
720,sync.hydra.opendns.com
1314,disthost.opendns.com
2756,api.opendns.com
4565,cacerts.opendns.com
5569,ipf.opendns.com
5699,block.opendns.com
7024,updates.opendns.com
8482,bpb.opendns.com

(Full Output)

To count the domain levels use this command:
awk -F, '{count=split($2,a,"."); print count}' top-1m.csv | sort | uniq -c | awk '{print $2,$1}' | sort -k1,1n

1 1086
2 263509
3 469756
4 193802
5 54281
6 13698
7 2952
8 689
9 172
10 16
11 26
12 2
13 1
14 1
15 1
16 1
17 1
18 1
19 1
20 1
21 1
22 1
23 1

(Full Output)
Notice anything strange here? Hint: A domain name requires at least two levels to be valid.

To find the broken DNS names in this list this command works:
cat top-1m.csv | awk -F, 'BEGIN {file="top-1m.csv" ; while ((getline line < file) > 0) {if (line ~ /#/) continue; tld[tolower(line)] = 1}} {foo=split($2,a,"."); if (foo == 1) {if (!(a[1] in tld)) {print $0}}}'  

1200,home
1490,local
2082,za
3916,lan
6350,url
10173,belkin
10869,uop
11187,localdomain
12887,localhost

(Full Output)

Find domains added to the list for today.
I  wrote a script to download the last two days of files and compare them for new domains:

You can find the output for April 24, 2017 here.

Overall I am really impressed with this data and will be using it to do more research and to track trends across the internet.  They have some more to do but it is an amazingly valuable free tool.

Also recently I have feel in love with sprunge to push data to an ad free “pastebin” from the command line:

cat file.txt | curl -F 'sprunge=<-' http://sprunge.us

Continue Reading

Burp Settings File

I am a huge fan of Tim Tomes and his Burp Suite Configuration Suggestions blog post.   The problem is that I only use Burp a couple times a month and end up facing this screen and have to re-configure burp on every launch:

So I built burpsettings.json that:

  • Disables Browsers XSS Protection
  • Disables Burp Collaborator Server
  • Disables Intercept by Default
  • Changes Scan Mode to Thorough
  • Turns Off Anonymous Feedback

This will help make my burp startup time a lot faster and I thought I would share the config file so it could help someone else also.

Continue Reading

Newly Registered Domain Name Keyword Search

Today I was asked if it was possible to generate a list of domain names registered everyday with a keyword in the record (company name, city, trademark, etc).   There are a few paid services that do this and domainpunch.com has a web based tool that will do this but I wanted to automate it so I could use it with a slackbot so I put together this 4 line bash script:

Usage:
./domains.sh keyword

Output:
This is super simple script but as they say “simplicity is the ultimate sophistication“.

Continue Reading

Automating DigiCert Certificate Issuance

I am a big fan of DigiCert for TLS Certificates and CA/WebPKI services.   While they have amazing customer support and are an amazing company to work with, there are not a lot of automation scripts to interact with their API available. So over the weekend and with a lot of help from Clint Wilson I built a shell script that:

  • Creates a CSR/Key pair using OpenSSL.
  • Uses the Digicert API to:
    • Request a TLS certificate.
    • Approve the certificate.
    • Download the certificate in:
      • .zip
      • p7b
      • pem
      • pem (with no root)

Here is the script in action:

Here is the code:

I have tested it on OSX,  Ubuntu and CentOS7 and it is fairly cross platform friendly.  Extending this script to install it should be easy but we already had the automation built to do that so it was not necessary.

Let me know on twitter if you have questions.

Continue Reading

Leadership Quotes From My Mentors Dad

An amazing mentor and leader I work with has been talking to me recently about what real leadership looks like and shared with me a list of quotes he keeps on his desk that his dad who had a leadership role in the military collected and gave to him.  He gave me a copy and said I was free to share them.

My [Dad’s] Rules Of Leadership:
  • Develop a vision and live it.
  • Dont lie for your people and dont lie to your people.
  • Beware of RUMINT.  It’s faster than you are.
  • Dont back away from the hard decisions,  especially personnel decisions.
  • Bad news never goes down easy and it won’t get easier with time.  It’s best to get it over with.
  • Support your subordinate supervisors when they take the high road.
  • Tell people exactly what you expect of them, including the obvious.
  • Involve your people in decisions and action planning.
  • Give them credit when things work.  Give them top cover when things go awry.
  • Trust the experts.  That’s what you pay for them for.
  • Avoid Bullshit.  You may get past the fans, but you won’t get past the players.
  • Knowledge may be power, but knowledge shared is powered squared.
  • Set an example by taking on the hard jobs.
  • Listen-Decide-Explain-Act.
  • Old ship-driving rule: When you get in extremis, DO SOMETHING.  The worse thing you can do is nothing.  Make decisions smartly and dont vacillate.  If you are wrong, admit it, back up and turn right.
  • No, you are not always right.  Get over it.  You are not as smart as you think you are and you may not be as smart as others think you are either.
  • Never underestimate the power of the expression “Thank You”.
  • Don’t fight with your friends. You haven’t got the time.

These quotes are amazing and I will be reflecting on this list for the rest of my career and am really happy to be able to share them.

Here is a “leadership” picture so shared links look better:

Continue Reading

Site Footer