How to get around a roboto.txt block?

[Rick Lee]

I need to go onto www.mortgagequestions.com and cannot. I get a "Page cannot be displayed" message both on my work and personal laptops. When I do a Google search of the URL, it comes up with a robot.txt message as to why I can't get onto the site. I have no idea how to get around this and have never seen it before. Any ideers?

[john70t]

Javascripts might be blocked on the browser?
Not sure.

Checking to see if this indeed a legitimate website:
Whois.com - Domain Names & Identity for Everyone
=Bunch of hosting advertisements

www.allwhois.com
redirects to
https://www.markmonitor.com/
=blahblahblah"to protect the reputations and revenue of the world's leading brands"

[stomachmonkey]

Originally Posted by Rick Lee

I need to go onto www.mortgagequestions.com and cannot. I get a "Page cannot be displayed" message both on my work and personal laptops. When I do a Google search of the URL, it comes up with a robot.txt message as to why I can't get onto the site. I have no idea how to get around this and have never seen it before. Any ideers?

What browser?

[Rick Lee]

IE and Windows 7.

[masraum]

Originally Posted by john70t

Javascripts might be blocked on the browser?
Not sure.

Nope, that's not it. When I try to access the site, I get a login page, and that's with Java blocked. When I enabled java, it did the same thing.

I assume you're going to need to login?

Can you access the page using a smart phone?

You say that you've tried both your PC and your work laptop. Both through the same Internet connection, probably at your house? Can you VPN into work and then try to access it. I'm thinking that it's either your ISP or your home router.

[masraum]

If you need to log in to the website, I would absolutely NOT use this, but if you just want to do this for testing from your home network or PC to pull the page up without logging in, use this page

https://hide.me/en/proxy
or
https://www.filterbypass.me/
or
This one may or may not work for that website
Free web proxy – unblock any website – webproxy.net

Again, if you use one of the links above, and you login, I can't guaranty that those pages won't save your login info or be able to be hacked to give up your login info. But, I would say that they are fine/great for testing.

[masraum]

Originally Posted by Rick Lee

I need to go onto www.mortgagequestions.com and cannot. I get a "Page cannot be displayed" message both on my work and personal laptops. When I do a Google search of the URL, it comes up with a robot.txt message as to why I can't get onto the site. I have no idea how to get around this and have never seen it before. Any ideers?

Can you paste the message that you get when you try to type the page into a browser and also the message that you get when you try to google search for it?

[masraum]

Yeah, if I google search for www.mortgagequestions.com, it gives me lots of results. My guess is that something you've got is blocking access, ISP, home router, or possibly even AV (though that seems less likely unless your home and work PC use the same software.

If you try a smartphone, make sure your smart phone is on it's own data plan and not just surfing over your home wifi.

[Rick Lee]

I can't use my iPhone for it, as I need to type stuff in their templates and upload documents. WTF?

[stomachmonkey]

Originally Posted by Rick Lee

IE and Windows 7.

Well it's one of those stupid sites that is "optimized" for certain browsers, translation, we don't know how to make it work right for all browsers" or " we used Microsoft Frontpage and are too ****ing lazy to bother accommodating anyone not using XXXXX"

It wants IE 9 or better or Firefox.

Their robots.txt does not appear to be the real issue.

Only contents are

User-agent: *
Disallow: /info
Disallow: /home

What that says is for all bots do not allow them to read /home and /info pages.

Home is normally the primary landing page, not sure why someone would have it under a specific directory, industry standard is xx.com/index.html where index.html is the default page a browser will read upon hitting xx.com

Try hitting https://www.mortgagequestions.com/sso/mq/login.jsp

[Rick Lee]

That link doesn't work (for me) either. I'll switch over to Firefox tomorrow and report back. Thanks for the suggestions.

[Holger]

robots.txt has nothing to do with that issue.
robots.txt is only a RECOMMENDATION for the crawlers/spiders (they do not HAVE to follow that, but good ones do), it does not affect a browser at all.

On FF and Chrome it works for me. I get a HUGE link in the browsers URL bar, like this:

https://www.mortgagequestions.com/sso/mq/login.jsp?TYPE=33554433&REALMOID=06-69d40195-9749-498e-841a-977f492bac26&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=mortquest-secure@lb1b2cwb10p&TARGET=$SM$HTTPS%3a%2f%2fwww.mortgagequestions.com%2fsso%2fmq2%2flogin.jsp%3fTYPE%3d33554433%26REALMOID%3d06-bdc806c4-8355-4c2f-b4e0-ac3ecc5c7494%26GUID%3d%26SMAUTHREASON%3d0%26METHOD%3dGET%26SMAGENTNAME%3dmortquest-secure%40lb1b2cwb10p%26TARGET%3d$$SM$$HTTPS$%3a$%2f$%2fwww$.mortgagequestions$.com$%2fhome$%2flandscape$%3fjpid$%3dHome$%26cid$%3d15882

[motion]

PHH Mortgage.... GRRRRRRRRRR

[GH85Carrera]

Just as information I tried the link. It worked fine and I saw where to log in and a pic of a little girl at play.

I am using Firefox and WinXP. I love Firefox. I was slow to get away from IE and I am sorry I waited so long. Firefox is way faster and work.

[Rick Lee]

I was able to get past the login page yesterday on both computers. But after that I got the "page cannot be displayed" message. I can't put Firefox on my work computer, but will do it on home laptop tonight.

[Z-man]

Assuming the website answers questions about mortgages, with all the trouble you are going through to get to the website -- why not just call up a local real estate broker and a mortgage person?

Just sayin,
-Z

[Rick Lee]

We're a little beyond that, as I've had my mortgage with HSBC for 10 yrs. and PHH services it. I'm selling the house and need to upload some docs there. Calling their 800# just gets me some kid in India who tells me to go to that website for everything, which is slightly less frustrating than having a conversation with one of those guys.

[Z-man]

Originally Posted by Rick Lee

We're a little beyond that, as I've had my mortgage with HSBC for 10 yrs. and PHH services it. I'm selling the house and need to upload some docs there. Calling their 800# just gets me some kid in India who tells me to go to that website for everything, which is slightly less frustrating than having a conversation with one of those guys.

Gotcha. And I laughed... It could be worse -- it could be a Motor Vehicle Services Representative on the phone....


-Z

[Rick Lee]

Just downloaded Firefox, got past the logon page for mortgagequestions.com, got to the "register this computer" page and then it went to "unable to connect." WTF? I really need to get on this and nothing is working. Should I try Google Chrome next?

[masraum]

OK, so you get the home page up and then login, and then the failure occurs? That's different than not getting there at all.

Since you are having issues in the new browser as well as the old, try clearing your java cache.

Delete Temporary Files through the Java Control Panel

In the Java Control Panel, under the General tab, click Settings under the Temporary Internet Files section. ...
Click Delete Files on the Temporary Files Settings dialog. ...
Click OK on the Delete Files and Applications dialog.

The whole point of sending you to the page via your cell phone wasn't for you to do all of your work via your cell phone, but to see if bypassing your ISP and Internet router "fixed" the problem. If you can login with your phone via your data plan, then the problem may be your ISP or PC or Internet router. If you can't get there via your phone while on it's data plan, then maybe there's a problem with the website.

If using your phone works, then link your phone to your wifi and try again. If it still works, then you know the problem isn't your ISP or your Internet router. At that point you would be certain that the problem was with your laptop (both of them apparently).

If we are certain that it's the laptops, then the easiest and first thing to do would be to clear the browser cache (unlikely since you just installed a new browser) or your java cache (more likely since the new browser could be using the same old java cache as the other browser. Maybe the site changed something in their java and your old cached java is causing the problem.