Is Duplicate Content Bad?
First things first. You should know a few things before you stress out too much about duplicate content in Magento:
- Duplicate content happens all the time on the Internet and it can’t be avoided.
- Google will not penalize you in search results for having duplicate content on your Magento store if it’s done in good faith and not for spamming purposes.
- Instead, Google will decide which of the two pages with the same content it will show to the user. The second page will become invisible, though.
- As a result, you will not be maximizing your site’s potential to harvest organic traffic from Google.
- You should minimize the amount of duplicate content in Magento and we’ll show you how.
Now after we’ve established that duplicate content is not the end of the world, you should be able to understand how it appears on your website and what you can do to make some part of it go away.
How to Get Rid of Duplicate Content in Magento 1 & 2
What is duplicate content? According to Google, duplicate content is:
- Identical content on two or more different pages.
- Large chunks of content that are identical or extremely similar to each other.
- Identical headings (H1 – H6) and title tags.
- The same or slightly different meta tags (alt tags for images, meta description).
- Synonymized content (every year Google gets better at detecting this type of content).
Let’s divide duplicate content into two categories: user-generated and machine-generated.
User-generated duplicate content is simple to understand. It’s when the users don’t want to create original content for every product they sell or can’t make it different.
Some examples: similar pairs of socks, tablecloths with slightly different visual patterns, different sizes of bolts and nuts. Anything really. How do you create interesting and engaging content for thousands of boring and almost identical items?
Man-made duplicate content can be hard to beat. One of our recommendations is to incentivize users to regularly add new reviews of your products. This approach has two distinct advantages:
- Newly created reviews will fill your page with original content which will reduce the number of duplicates on your website.
- New users will read older reviews and show more trust towards your website. Websites with higher user trust have better user behavior patterns – longer view times, deeper navigation, tapping Call-to-action buttons, etc. Google loves these metrics and ranks such pages higher (browse our Magento Google PageSpeeds guide for additional tips). Ranking higher will lead to more visitors from organic search and more sales.
Magento is not perfect in handling content, too. Sometimes Magento will create duplicates that are hard to avoid. Let’s take a look at how your Magento can contribute to world spam.
Reasons Magento Creates Duplicates and How to Fix It
1. Your Pages Are Available at Both yourstore.com and www.yourstore.com
Magento is not strictly at fault here since if you don’t make sure you have a 301 redirect Magento will show both the “www” version and the “bare” version of the site.
Google treats every domain as a separate website. To Google, www.yourstore.com and yourstore.com are two different websites.
You need to make sure Google knows you want it to treat them as a single website. Our recommendation is to “glue” those two together and point to yourstore.com as the “better” version.
Add both websites to Google Search Console. The common misconception here is to only add a single version, usually the one that is used in marketing materials and customer information.
Add your website with the proper // or // designation since for Google it’s an important part of your address.
Now add the www version.
Make sure all your 301 redirects point to the right version of the website. We highly recommend you use yourstore.com as the “canonical” domain, so let your Magento developers and SEO specialists know about this.
1.1 Create the 301 redirect in Apache (Magento 1 and Magento 2):
For Apache-specific cases, we recommend editing .htaccess or httpd.conf file. Open .htaccess in your root directory, add a new line at the top of the other redirects if there are any so it looks like this:
RewriteEngine on
RewriteCond %{HTTP_HOST} ^www.yourstore.com [NC]
RewriteRule ^(.*)$ //yourstore.com/$1 [L,R=301,NC]
1.2 For Nginx (Magento 1 and Magento 2):
Edit server.rewrites in the following way:
if ($http_host ~* "^(?!www\.).*$") {
return 301 //www.$http_host$request_uri;
}
If you use Varnish, put this code in public.rewrites instead of a server.rewrites to avoid caching issues.
2. Magento Creates Duplicate Content With Search Results
Even though it’s a comparatively minor issue, search results can also interfere with your efforts to eliminate all duplicates.
2.1 How to Restrict Search Results for Magento 1
For Magento 1.9 we will add the following lines in robots.txt (found in your Magento root folder):
Disallow: /catalogsearch/result/
Disallow: /sendfriend/
These will block both Catalog search results and sharing from Google search results.
2.2 Restrict Search Results for Magento 2
Use robots.txt in your Magento root folder, and disallow search results from indexing. For Magento 2 it’s best to do that from Content > Design > Global Design Configuration, though, and not directly (click edit to start customization).
This is the section you are looking for:
Using robots.txt, we will fix duplicate content in search results for Magento 2. Add these lines to restrict crawlers access:
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
3. You Have the Same Product in Multiple Categories
Magento can have the same product in multiple categories. This is confusing for search engines and can result in thousands of duplicates. Adding a product to multiple categories means a single SKU will have 2 or even more unique URLs.
Well, the simple fix for both Magento 1 and Magento 2 would be to just make it your best practice to only use one category for one product but we all know life is never that easy.
Magento 1 Multiple Categories Fix
In Magento 1 you can fix this issue by disallowing catalog comparisons and categories from appearing in Google search index:
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Magento 2 Multiple Categories Fix
And here’s a SEO-friendly fix for Magento 2:
Go to Content > Design > Global Design Configuration and add new lines to your robots.txt:
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
4. Your Service Magento Paths Leak Duplicate Data
We are not finished with robots.txt just yet.
How to Remove Duplicate Content in Magento 1
Let’s remove Magento 1 filters and reviews from search results:
Disallow: /review/
Disallow: /*?category=*
Disallow: /*&category=*
Disallow: /*?price=*
Disallow: /*&price=*
Disallow: /*?mode=*
Disallow: /*&product_list_mode=*
Magento 2. Remove Duplicate Content for Tags, Reviews, and Filters
Magento 2 generates links to tags and reviews that will also result in duplicate content. Add these lines to your robots.txt to disallow them:
Disallow: /tag/
Disallow: /review/
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /*?mode*
5. Google Indexes Your Magento Store at Both HTTP and HTTPS
For both Magento 1 & 2, make sure that you use rel=canonical to indicate which page has priority. Rel=canonical will allow you to consolidate both HTTP and HTTPS pages and remove duplicates.
In contrast to Bing, Google recognizes and respects rel=canonical so it’s a valid tool to combat duplicate content. It is also the preferred method of dealing with duplicate content. So if you can manage duplicates with rel=canonical instead of robots.txt, Google will appreciate this.
Let’s see how it works.
When you go to any web page, its rel=canonical will look like this:
<link rel="canonical" href="//store.com/canonical-page.html"/>
5.1 How to Make Rel Canonical Pages for Magento 1
Magento 1 hides canonical settings in System > Configuration > Catalog > Search Engine Optimization.
Use Canonical Link Meta Tag for Categories and Use Canonical Link Meta Tag for Products are the options we’re looking for. Switch them to YES.
Please bear in mind that if you switch Use Canonical Link Meta Tag for Categories to YES you’ll delegate this option entirely to Magento. While this is fine most of the time, make sure you are OK with automating rel=canonical designations for Categories.
This will only matter, though, if you haven’t disallowed Categories in robots.txt in the fix above.
We also advise against going back from Yes to No. If it is a Yes, leave it. Switching back and forth might mess up your SEO rankings.
5.2 How to Make Rel Canonical Pages for Magento 2
Let’s configure canonical links in Magento 2.
Go to Stores > Configuration > Catalog > Search Engine Optimization.
Switch to YES for Use Canonical Link Meta Tag for Categories and Use Canonical Link Meta Tag for Products.
Now you have canonical links for your pages turned on.
You can also look into rel=next, rel=prev, and rel=all parameters to eliminate pagination issues.
Housekeeping Is Important. However…
Over the years Google claimed to have changed their content policy quite a bit. They supposedly shifted focus from keywords content issues to user behavior and social cues.
Even though Google’s standards for e-commerce website content are less strict than they used to be, avoiding duplicate content in your Magento store is essential for better performance on Google.
Having said all of this, we still recommend you to be a good webmaster and clean up your store from the most glaring duplicate issues.
This way you help search engines crawl and analyze your website correctly and make sure that the right pages become indexed and shown to your users.
So, find the time to go to Google Webmaster Tools and remove duplicates from your index.
Once you are done with your duplicate content, make sure you have your Magento store optimized for highest site load speed. You can follow our Magento 2 speed optimization guide as a reference or reach out to us and order Magento optimization services.
6. Your Short Product Descriptions in Magento Are Duplicates
Magento offers admins to fill in both short and long descriptions for the products. If you are wondering whether it’s worth the effort, our answer is a definite YES.
You’ll see the difference between a unique, custom-crafted short description and cookie-cutter generic stuff both in both your SEO rankings and customer engagement stats.
Also, don’t be fooled with the Short Description moniker. There is a lot of space here to get your creativity going. Where Long Description could talk about in-depth details – material, specs, and accessories – take the opportunity to sell your product in the Short Description.
7. Your Magento Multistore Hosts Lots of Duplicate Content
We know that sometimes regional targeting can be tricky. Splitting your store into two for UK and US audiences, Australian and New Zealand shoppers, or Spanish and Latin American visitors can be a piece of work.
It’s one thing when your store has content in a different language, but what if you have to use the exact same content for multiple regions?
Well, the hard truth is that you have to create unique content for both stores. Making unique content is a lot of work and will not be cheap. So look into the possibility to host same-language content in a single store.
While we’ve only taken a look at the issue of duplicate content because of the same language, Magento multistore duplicate content is an issue because of bad search engine optimization.
Same old here. Multiple categories for a single product, service links getting into the search index, lack of canonical and lang metadata, etc.
To eliminate duplicates on multiple Magento stores (single Magento installation), you’ll need to get rid of Magento duplicates inside individual stores (we’ve shown how to do this above). In addition:
- Use language metadata to indicate to Google that your stores target different users with similar content.
- Edit robots to exclude language-specific service link extensions from Google search index – so something like ?___from_store=es must be excluded from robots.txt with the following code:
Disallow: /*?___from_store=*
8. You Haven’t Removed Trailing Slashes in Magento Store URLs
Depending on your web server setup, your Magento store will either default to site.com/link1/ or site.com/link1 URL.
What you need to understand is that both these URLs are unique in the eyes of Google. This inconsistency will lead to duplicate content and blurring link juice between multiple URLs.
There is a lot of bad advice on the Internet about how you should deal with this issue. We certainly don’t recommend altering your core Magento files.
We’ll use 301 redirects to deal with duplicates. Open .htaccess file in your root Magento directory (this method is for apache only):
RewriteCond %{request_method} ^GET$
RewriteCond %{REQUEST_URI} !^/downloader.*$
RewriteCond %{REQUEST_URI} ^(.+)/$
RewriteRule ^(.+)$ %1 [L,R=301]
For Nginx web servers, we’ll need to edit server rules instead. Add to the nginx.conf the following code:
if ($request_method != POST) {
set $rew "A";
}
if ($http_x_requested_with != XMLHttpRequest) {
set $rew "${rew}B";
}
if ($rew = "AB") {
rewrite ^/(.*)/$ $scheme://$http_host/$1 permanent;
}
The new instructions will rewrite all site.com/link1/ to site.com/link1 URLs, saving your SEO juice for the right pages.
Housekeeping Is Important. However…
Over the years Google dev team have reviewed their content policy quite a bit. They shifted focus from keywords and text content to user behavior and social cues.
For Magento stores, this means that duplicate content is no longer a stumbling block for search rankings. It’s still important but it’s no longer an overwhelming challenge. Google standards for eCommerce-generated content are more relaxed and loyal than ever before.
If you don’t try to abuse or cheat the system, Google will not penalize your website for duplicate content. It understands why duplicates happen and why sometimes they are unavoidable.
Having said all of this, we still recommend you to be a good webmaster and clean up your store from the most glaring duplicate issues.
This way you help search engines crawl and analyze your website correctly and make sure that the right pages become indexed and shown to your users.
So, find the time to go to Google Webmaster Tools and remove duplicates from your index. Then make sure to optimize Magento images so that your media files are well-brushed up too.
So, to re-cap –
- Use 301 redirects when you move the page but want to keep its previous rankings
- Make sure your server redirects either to www.store.com or your top-level domain
- Be consistent in your links. Even if you can access your page from store.com/product, store.com/product.html, and store.com/product/, always stick to one version and never deviate from your choice
- Avoid huge repetitive footers such as copyright, other legal info, excessive warranty or delivery ads. Minimize stuff and use links more. Using banners can be a good alternative if you absolutely need to display that info on all pages
- Opt for rel=canonical and 301 redirects to robots.txt disallow tool but feel free to use all of them in moderation