Fortran-lang no longer in Bing

[vmagnin]

I have also noticed at the bottom right of that last Bing page that it is copyrighted (C) 2020 Microsoft. The normal pages are copyrighted (C) 2023 Microsoft.

So is it an old page in cache?

[awvwgk]

Here are a couple of things I tried indexing our page manually via the Bing Webmaster Tool:

1. The main page https://fortran-lang.org/en/ is discoverable but not crawlable and can therefore not appear in the Bing index

2. The live inspection reveals that this is due to a "canonical" tag which refers to an alternative version for https://fortran-lang.org/en/, which is https://fortran-lang.org/index.html

3. However, https://fortran-lang.org/index.html is supposed to be a redirect to https://fortran-lang.org/en/, which is how we set it up to support translations

I guess that gives us a hint on what we can try to fix the indexing issue?

From the Webmaster Guidelines:

Use redirects as appropriate: If you move content on your website to another location, use a HTTP 301 permanent redirect for at least three months. If the move is temporary, i.e. less than one day, use a 302 temporary redirect. Avoid using a rel=canonical tag in place of a proper redirect when site content has moved from one location to another.

(any admin who would like to have access to the Bing Webmaster Tool, please send me your Microsoft account name and I can give you access)

[awvwgk]

Just merged the changes proposed by Henil @henilp105, we still get an error in the Bing Index

But the live inspection reveals now that the page is actually indexable

I also manually requested indexing for the main page, so let’s see whether this helps fixing the issue.

[certik]

@awvwgk any updates? I still can’t see fortran-lang in DDG or Bing.

I just asked the DDG CEO here if they could please unblock fortran-lang: If you are not a Bing frontend, do you think it would be please possible to unbl... | Hacker News.

[awvwgk]

Output of the URL inspection remains the same. But I noticed that the uploaded sitemap is currently broken, due to the language appearing twice in the URLs.

[certik]

@awvwgk thanks. Do you know how to fix the url?

[certik]

I tested Neeva, before they shut down, my understanding is that they have an independent crawler. This is what it returned:

So I think the front page is indeed broken, doing some kind of a redirection.

[henilp105]

Hi @certik , I have fixed the urls in a patch merged by @awvwgk last week, and Considering the urgency of the issue I have also opened a temporary patch to fix the bing directly via gh-pages as it seems the most likely cause could be HTML redirection , as we don’t have the functionality to add 301 Redirect in gh-pages, thus this PR is a temporary fix to this redirection bug , and this same method of redirection is also used in the fpm-docs repository and we seem to reproduce the same problem for the fpm-docs repository as well.

[certik]

Given that we have not figured out how to do this robustly, I would not do any redirection at all.

I recommend having the main English page at fortran-lang.org/ as is. Then translations can be done via the url prefixes, as we do now.

Let’s get it done, it eliminates one more possible reason why Bing delisted us. Let’s get us back in Bing and other search engines.

Only then we can experiment again, if we really need a redirect. We can see how sympy does it, see: sympy.github.com/index.html at 741aa29d3c35f27bb3371152d9d38a19df732533 · sympy/sympy.github.com · GitHub, so we can copy the page as is. Compare to us: webpage/index.html at bdd79d2a88c11cfac88ae45ae56dfa7232edf9cf · fortran-lang/webpage · GitHub, it could be that crawlers have special logic and can recognize the simpler “template” as redirection, but can’t recognize our custom redirection.

[certik]

Updates as of today (June 1, 2023):

fortran-lang.org does not redirect anymore (thanks @henilp105!).

Duckduckgo now shows fortran-lang at the top:

The little “(i)” next to it says “Official site identified by DuckDuckGo”, so I think they might have added a manual override when I asked the CEO (see above). Thank you! Much appreciated.

We still do not seem to show in Bing, but let’s give it some more time, now when the site does not redirect maybe their crawler will pick it up again.

[henilp105]

Thanks everyone for your help , I think we should be soon on the bing also ( @awvwgk can you please try re requesting with the bing webmaster tools).

Thanks and Regards,
Henil

[henilp105]

We are back on Bing , , Thanks Everyone for your Contributions in fixing this . ( it might take some time in indexing the entire webpage but, it has started indexing some parts of the webpage thus currently only some parts are visible in the bing ) .

CC @certik @awvwgk

[certik]

Indeed, I think we are back, this is what I see: a Wikipedia and right underneath is fortran-lang:

[vmagnin]

I also see the Learn page, in the second half of the first page in the fr Bing (but curiously, I needed to delete the Bing cookie in Firefox to make it appears in the results).

[Carltoffel]

The German Bing still doesn’t list our website. Instead, the first result after Wikipedia is this gem:

Deutsche Fortran WebSite

No need to understand German to see what is special about this site.

[awvwgk]

Got a reply from the Bing support yesterday as well

[certik]

@awvwgk thanks!

Now when it is resolved, what exactly was the problem(s), so that we can avoid this in the future?

[awvwgk]

That’s actually a good question, we use similar redirect logic in the fpm docs and there we have no issues. Maybe the migration from Jekyll to Sphinx was the culprit here?

[milancurcic]

We do know it was the migration from Jekyll to Sphinx but we don’t know what exactly about the migration caused it.

[henilp105]

@certik I think the problem would be both due to the migration from Jekyll to Sphinx and the redirection bug ( as the fpm docs is not discoverable in bing , there is another fpm on bing by jordansissel) , we would have to patch it with the similar method.