How to get around a roboto.txt block?
 

I need to go onto www.mortgagequestions.com and cannot. I get a "Page cannot be displayed" message both on my work and personal laptops. When I do a Google search of the URL, it comes up with a robot.txt message as to why I can't get onto the site. I have no idea how to get around this and have never seen it before. Any ideers?

Javascripts might be blocked on the browser?
Not sure.

Checking to see if this indeed a legitimate website:
Whois.com - Domain Names & Identity for Everyone
=Bunch of hosting advertisements

www.allwhois.com
redirects to
https://www.markmonitor.com/
=blahblahblah"to protect the reputations and revenue of the world's leading brands"

What browser?

IE and Windows 7.

Nope, that's not it. When I try to access the site, I get a login page, and that's with Java blocked. When I enabled java, it did the same thing.

I assume you're going to need to login?

Can you access the page using a smart phone?

You say that you've tried both your PC and your work laptop. Both through the same Internet connection, probably at your house? Can you VPN into work and then try to access it. I'm thinking that it's either your ISP or your home router.

If you need to log in to the website, I would absolutely NOT use this, but if you just want to do this for testing from your home network or PC to pull the page up without logging in, use this page

https://hide.me/en/proxy
or
https://www.filterbypass.me/
or
This one may or may not work for that website
Free web proxy – unblock any website – webproxy.net

Again, if you use one of the links above, and you login, I can't guaranty that those pages won't save your login info or be able to be hacked to give up your login info. But, I would say that they are fine/great for testing.

Can you paste the message that you get when you try to type the page into a browser and also the message that you get when you try to google search for it?

Yeah, if I google search for www.mortgagequestions.com, it gives me lots of results. My guess is that something you've got is blocking access, ISP, home router, or possibly even AV (though that seems less likely unless your home and work PC use the same software.

If you try a smartphone, make sure your smart phone is on it's own data plan and not just surfing over your home wifi.

I can't use my iPhone for it, as I need to type stuff in their templates and upload documents. WTF?

Well it's one of those stupid sites that is "optimized" for certain browsers, translation, we don't know how to make it work right for all browsers" or " we used Microsoft Frontpage and are too ****ing lazy to bother accommodating anyone not using XXXXX"

It wants IE 9 or better or Firefox.

Their robots.txt does not appear to be the real issue.

Only contents are

User-agent: *
Disallow: /info
Disallow: /home

What that says is for all bots do not allow them to read /home and /info pages.

Home is normally the primary landing page, not sure why someone would have it under a specific directory, industry standard is xx.com/index.html where index.html is the default page a browser will read upon hitting xx.com

Try hitting https://www.mortgagequestions.com/sso/mq/login.jsp

That link doesn't work (for me) either. I'll switch over to Firefox tomorrow and report back. Thanks for the suggestions.

robots.txt has nothing to do with that issue.
robots.txt is only a RECOMMENDATION for the crawlers/spiders (they do not HAVE to follow that, but good ones do), it does not affect a browser at all.

On FF and Chrome it works for me. I get a HUGE link in the browsers URL bar, like this:

PHH Mortgage.... GRRRRRRRRRR

Just as information I tried the link. It worked fine and I saw where to log in and a pic of a little girl at play.

I am using Firefox and WinXP. I love Firefox. I was slow to get away from IE and I am sorry I waited so long. Firefox is way faster and work.

I was able to get past the login page yesterday on both computers. But after that I got the "page cannot be displayed" message. I can't put Firefox on my work computer, but will do it on home laptop tonight.

Assuming the website answers questions about mortgages, with all the trouble you are going through to get to the website -- why not just call up a local real estate broker and a mortgage person?

Just sayin,
-Z

We're a little beyond that, as I've had my mortgage with HSBC for 10 yrs. and PHH services it. I'm selling the house and need to upload some docs there. Calling their 800# just gets me some kid in India who tells me to go to that website for everything, which is slightly less frustrating than having a conversation with one of those guys.

Gotcha. And I laughed... It could be worse -- it could be a Motor Vehicle Services Representative on the phone....


-Z

Just downloaded Firefox, got past the logon page for mortgagequestions.com, got to the "register this computer" page and then it went to "unable to connect." WTF? I really need to get on this and nothing is working. Should I try Google Chrome next?

OK, so you get the home page up and then login, and then the failure occurs? That's different than not getting there at all.

Since you are having issues in the new browser as well as the old, try clearing your java cache. The whole point of sending you to the page via your cell phone wasn't for you to do all of your work via your cell phone, but to see if bypassing your ISP and Internet router "fixed" the problem. If you can login with your phone via your data plan, then the problem may be your ISP or PC or Internet router. If you can't get there via your phone while on it's data plan, then maybe there's a problem with the website.

If using your phone works, then link your phone to your wifi and try again. If it still works, then you know the problem isn't your ISP or your Internet router. At that point you would be certain that the problem was with your laptop (both of them apparently).

If we are certain that it's the laptops, then the easiest and first thing to do would be to clear the browser cache (unlikely since you just installed a new browser) or your java cache (more likely since the new browser could be using the same old java cache as the other browser. Maybe the site changed something in their java and your old cached java is causing the problem.

This is nuts. Can't get past the computer registration page on my iPhone either. It says "Safari cannot open the page because it could not connect to the server."

I don't know how to get to the Java Control Panel in Firefox, but I did a search for Java in my C drive and it came up with 21 items. Is it ok to delete them? New error message on laptop reads "Error the site cannot be accessed." Really getting tired of this BS. How can PHH have this set up like this such that so many people can't access it? I can't be the only one. Two weeks ago it worked fine for me.

That was my first thought too.. My old employer had PHH fleet management, and honestly, they just made everything worse. SO much worse.

Just so everyone is clear here, Java and Javascript are two completely different things.

Sorry, that is correct. I have javascript disabled on my PC and I was still able to get to the site (but I didn't login). Java packages may be downloaded when accessing the site, and that's what I was talking about clearing the cache for.

That's weird, two PCs, multiple browsers and your phone are all failing. It's starting to sound like a problem with their website, or it could possibly be a problem with your login specifically.

The java control panel isn't in firefox, it's in the windows control panel Java is an application installed on your PC. Go to the windows control panel, or if you search in the start menu for java and find an app, click that. What version of windows are you running?

Windows 7. 21 files came up when I did a start menu search for "java." I don't know which of those to delete and I don't think there was an option to disable.

Today I called them and they told me to set up a new ID while they deleted my old one. I got through that and a few more screens before same problem happened. Then they gave me an alternate URL that went to an HSBC-branded page and I was able to navigate that alright. But the one link I needed wasn't showing on my computer, while it was showing on hers. I can't believe what a chore this is becoming.

The problem I believe is Hitbox and the jscript they are using. Third party "call center" service in the form of a chat window.

Incompatible with certain browsers.

If you hit the site using chrome and turn on developer tools then check the console there are a bunch of errors and you don't get the Hitbox window.

Using Firefox I get no errors in the console and get the Hitbox chat window.

Chrome

http://forums.pelicanparts.com/uploa...1418864748.jpg


Firefox


http://forums.pelicanparts.com/uploa...1418864771.jpg

I get even further than that with Firefox and Chrome. But it still gives an error message two pages later, no matter which device or browser I use.

Well I don't have an account so that's as far as I can get.

And while I like you I'm not signing up for one either. SmileWavy