Before you get started, another long post.
Read fully before you do any edits, if you are going to do any edits. Any questions, speak up.

Well, I never really used any of the only mobile friendly testing site, but I recently put Google Analytics back on my site, (roughly a month ago), and I got an email today telling me that my site wasn't mobile friendly, so I used their test to run it and sure enough, it failed. I also ran a few others I found, and they all had the same result. So after putzing around, I found the culprit. Didn't take long but for those that run into the same issue, or, you're trying to make your site "Mobile Friendly", and trying to use one of these online testing system, then the fix is easy.

When I first tested it, this is what the result looked liked:


After doing the simple fix I am going to tell you, this is what it looked liked:


Googles mobile test:

Please login to see this link Get registered or Log in

It all comes down to one file in which you need to allow Googlebot to crawl your site. To do this, you will need to do the following:

OPEN: public_html/robots.txt

FIND:


REPLACE WITH:


Now, if you are using Google Ads, then you will probably want to add in two more User-agents;

Code: [ Select all ]

User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow:

User-agent: AdsBot-Google
Disallow:

User-agent: Mediapartners-Google
Disallow:

Now, me wanting to tidy up my site, I decided to combine them into a single grouping than having multiple. I also wanted to define where my site's sitemap was because it is suggested by both Google and Mozilla that it really should be there for crawlers to find it. But know, that if you do copy how I have mine set up, you will need to change your sitemap URL. If you don't have a sitemap, then go to this module: Your-Site-URL/modules.php?name=Site_Map. After visiting that module, then you should see a sitemap.xml file in your root of your site that has been just created/updated. I have that link in my menu on the left of my site for easy access.

Before I show you my code, I do have a warning.
!!WARNING!! - Don't go crazy with things in the robots file, for you can easily destroy what SEO you may have.

Now, here is my code that I use on my site.

Code: [ Select all ]

User-agent: Baidu
User-agent: *alexa*
Disallow: /

User-agent: Googlebot
User-agent: Googlebot-Image
Disallow:

User-agent: *
Disallow: /admin/
Disallow: /blocks/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /modules/
Disallow: /themes/
Disallow: /admin.php
Disallow: /config.php
Disallow: /ips.php
Disallow: /trap.php

Sitemap: https://www.headshotdomain.net/sitemap.xml

All I did was combine the User-agent: into single groupings instead of having separate. Went from:
User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow:

to

User-agent: Googlebot
User-agent: Googlebot-Image
Disallow:

But it really doesn't matter if you do that type of clean up. Also, I have the sitemap defined at the bottom. This can get a bit tricky, but since I have my site set up using the force WWW and HTTPS, which I go over in

Please login to see this link Get registered or Log in

, that is how I needed to put it. What I have read is you need to be absolute with the path, but I am not sure if you could get away with doing something like:

Sitemap:

Please login to see this link Get registered or Log in

Sitemap:

Please login to see this link Get registered or Log in

Which could cover both the HTTP and the HTTPS. But on my part, that would only be speculation.