Tuesday, 24 November, 2020

Creating Website Sitemaps

How to Create a Sitemap for your Website.

Today’s post is about Sitemaps. I’ll cover what sitemaps are, why you should use them, how you create one, how to submit them to search engines and where to find out more information about them.

In a later post I'll talk about how to configure a dynamic Sitemap for use with the Umbraco CMS.

What’s a Sitemap?

A Sitemap is an XML (computer readable) text file hosted on your website to provide search engines with information about the pages on your site.

They can provide additional information about your pages such as how often the page is updated, when it was last modified and what level of priority you give it in comparison to other pages on your site.

They’re not essential but since they help search engines index and categorise your site, it’s good idea to have one.

An Example Sitemap

Here's a snippet from the JAGC website's Sitemap to give you an idea what we're talking about:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
    <url>
        <loc>
http://jagc-live.azurewebsites.net/blog/archive/getting-the-ball-rolling/
        </loc>
        <lastmod>2017-06-27T13:58:21+00:00</lastmod>
    </url>
    …
    <url>
        <loc>
http://jagc-live.azurewebsites.net/blog/archive/how-to-download-and-back-up-microsoft-azure-web-applications-part-2/
        </loc>
        <lastmod>2017-06-27T13:58:24+00:00</lastmod>
    </url>
</urlset>

Ways to Create a Sitemap

You can create Sitemaps by hand with a text editor but most people choose to automate the process using a Sitemap generator or to do it programmatically.

Crafting a Sitemap by hand is fairly simple. The protocol https://www.xml-sitemaps.com/ is short and easy to comprehend and provided you have some web savvy know-how you should end up with a high-quality Sitemap. However, for larger sites or regularly updated sites, hand crafting Sitemaps isn't a realistic option.

The next approach is to automatically generate Sitemaps using an online tool. We'll go over that approach in a moment.

Creating a Sitemap by Hand

To create a sitemap by hand you need a text editor. On Windows that's usually Notepad, Notepad++ or Visual Studio; On a Mac it's going to be TextEditor or Sublime; on Linux it'll be Nano, Vi or Vim. Pick your favourite, I like Brackets at the moment, and create a new text file (UTF-8 encoded) called sitemap.xml.

Next copy and paste this boiler plate code inside:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://www.example.com/</loc>
   </url>
</urlset>

The XML format above forms the basic content of every Sitemap.

Each sitemap must have an XML header, be wrapped in a pair of <urlset></urlset> tags, and include a set of<url></url> tags for each page entry. In addition, each <url> tag must also include (encompass) a set of <loc> tags which specify the page's URL (given above as the landing page for www.example.com).

To add another page, say product1.html, you'd create another entry like so:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
      <loc>http://www.example.com/</loc>
   </url>
   <url>
      <loc>http://www.example.com/product1.html</loc>
   </url>
</urlset>

Just keep adding new entries for each page until you're done. There are other tags you can use too, like <lastmod> and <changefreq> but they're not essential. If you interested in adding more detail read the specification referenced earlier.

Finally, upload your new sitemap.xml to your website’s root directory (strongly recommended).

Technical side note: XML documents must have their entities escaped. This means any URL values you enter which contain &'"<> must be replaced with it's escaped equivalent. So:

  • & becomes &
  • ‘ becomes '
  • “ beciones "
  • > becomes >
  • < becomes @lt;

Creating a Sitemap using an Online Generation Tool

If writing the sitemap.xml by hand isn't for you you can always use a tool. I've used https://www.xml-sitemaps.com/ in the past to generate my Sitemaps and found it very useful.

All you have to do is enter your website's URL and the tool will crawl your site for you. Then download it and upload it to your site's root folder.

Making your Sitemap Visible to Search Engines

Once you've created and uploaded your sitemap most search engines will be able to find it. However, if you want to be sure you add an entry to your Robots.txt file pointing them in the right direction.

If you don’t have a Robots.txt file, you should think about creating one, you can read all about them at http://www.robotstxt.org/robotstxt.html

The entry you need to add is:

Sitemap: http://www.yourwebsite.com/sitemap.xml

Add it to file, any new line will do, and save it in the root directory of your website.

Submitting Sitemaps

The two most popular search engines, Google and Bing, provide web master tools where you can register your website's Sitemap and have them tested. You can find their respective tools at:

Note: You’ll need to create an account, login and prove you have control of your website before you can submit your Sitemap and test it.

At the time of writing Ask and Duck Duck Go don’t provide a method for submitting Sitemaps; they rely on their engines crawling your website and finding the pages. As for Yahoo and AOL they're powered by Bing, so will be autmatically updated once you submit the site to Bing webmaster tools.

Wrapping Up

Sitemaps are text based XML computer files designed for search engines to help them better understand and index your web content. They're not essential but useful in communicating what content you have to search engines and therefore in making it searchable on the web.

You should now have an idea of how to create, configure a Sitemap, and should you need to where to find more information.

All the links from this article, plus some more such as Sitemap validators, can be found on my Papaly links board at https://papaly.com/#board_id=7487681820fa4fa8b61257db633de1c1.

As always if you struggle, don't understand something or need help then feel free to get in touch.

Want to Thank Me?

Did you like this article? Was it helpful? Why not buy me a coffee on Paypal? Buy me a coffee?