What is Robots txt? It is the main part of SEO optimization. You need to learn how to settings here properly if not search engines such as Google or Yahoo will not find your site in their directories. So, your site is nothing and useless. Sounds scary right?

This is a fact.

The first time I dabbled in blogging, I just ignore it because I do not know to whom I want to refer. I do not learn what is it. This looks simple but it is a big mistake for a beginner who wants to start blogging. So please do not repeat what I have done.

Today I want to share with you about robots.txt.

What is robots.txt?

Briefly, it is a file included in every root directory of the web folder and it used to either allow or disallow access bots (search engine bots like Google bots or Yahoo bots) to access your files and content.

When a search engine starts crawler into your website, the first place it will detect is robots.txt and read which files are allowed or disallowed to access.

With it, you can control bots to access your folder of website.

For example, if you want to block search engines from accessing a particular
directory of the website all you have to do is write ‘Disallow: [directory name]’
like the example of ‘wp-admin’ below.

Based on the settings above, bots will access your admin-ajax.php and upload folder meanwhile it cannot access wp-admin and plugins folder.

How to check robots.txt?

To check it, just go to your web browser and type: yourdomain.com/robots.txt. It should show you how your settings for it.

Note: Whatever you have in robots.txt is accessible by everyone typing the
URL so this is not the place to add information that other people shouldn’t
know about.

In case it blocks other main WordPress directories like wp-content or wp-includes, simply you just delete the line and by default, bots will crawl all your directories except you disallowed it that you setting in robots.txt.

How to Edit robots.txt?

To edit your robots.txt, you can do it via cPanel or FTP client.

Here I would like show you both how you can edit it via cPanel and FTP client.

Method 1: Edit via cPanel

Step 1: Just login to your cPanel.

Step 2: In the cPanel dashboard go to File manager » public_html » your web folder » robots.txt. Then, click edit to start editing.

Here you can edit it as per you need.

Method 2: Edit via FTP client

Step 1: Log in into your FTP client.

Step 2: Same the path of cPanel but a little bit different. From FTP client login dashboard navigate to public_html » web folder » robots.txt

Step 3: Right-click on your robots.txt file and click edit. Now, you free to edit it here. After finish editing, just save it.

Conclusion

In this tutorial, I have learned what is robots.txt and how it can give impact on your SEO and you have learned how to edit it via cPanel and FTP login.