OT: 404redirect

Question

hi all. just looking for input from people who have been at
this longer than i have. there may be no answer to my question, but
it never hurts to ask, right?
i recently discovered the whole 404redirect thing and added
it to my websites. my boss uses Urchin for stats, and the last
couple days have seen a huge spike (like 500% or more) in pageviews
for the 404redirect page on more than one site. i haven't done
anything like delete a bunch of pages, or anything else i can think
of that would cause this to happen. and it doesn't seem very likely
that hundreds of people would all of a sudden try to go to the one
page i deleted, all in the same day.
so i was just wondering if anybody has any idea why this
might happen?
i was thinking one possibility could be that some search
engine(s) just re-crawled the sites for the first time since i
added it, so they're hitting the redirect every time they try to
hit a page that's been removed or something. does that make any
sense? i'm not really all that savvy about how search engines do
things.
another weird thing about it is the fact this hasn't caused
an increase in the pageviews to the redirect page. seems to me that
if the 404redirect is getting hit 2,000 times more than usual, then
the index page that it redirects to should also see an increase.
i just don't understand what's going on and thought that
maybe someone else has experienced this before and can tell me
whether it's something i should be concerned about. or at least
give me an explanation to tell my boss!
thanks!
-courtney

Answer 1

.oO(n h c)
>ok, so you're saying that i should do the 301 for pages
that have been
>moved/renamed
Yes. This way they will be kept available without any
interruption. They
can still be reached from a search engine and from external
sites, even
if it might still be the old link. Especially incoming links
from other
sites can last a very long time, until their webmaster
updates them, so
it's always a good idea to keep them working as long as
needed.
Actually you write such 301 redirects once in your .htaccess
(might take
some time for a hundred pages, but not that much anyway) and
then you
can forget about them. The users get the stuff they want and
the site
still simply works, as if nothing happened.
>and the 404 for pages that have been deleted completely?
Yes and no. There are some more 4xx status codes to describe
in more
detail what happened to a resource, for example 410 ("Gone"),
but most
of them are hardly used. So a 404 would be acceptable in this
case.
Better than nothing or an inappropriate response.
>but if
>i do the 404 redirect, the user will be taken to the home
page where i want
>them (instead of just leaving the site, as i'm sure many
people would do), but
>then the search engines won't know that the page has been
deleted and will
>still show it in search results? so basically i can keep
the old 404 error and
>risk losing visitors, or i can do the redirect and the
search engines will
>never stop showing pages that no longer exist in search
results?
More or less. In case of a really vanished page the server
should return
a 404, because that's what search engines and other caching
systems need
in order to clean up their caches and indexes.
But as I already said in another post - on that error page
you can do
whatever you want. You could even use a meta-refresh (not
instantly, but
with some seconds delay and a short notice) to forward the
users to the
homepage if you want. I wouldn't do that, but this is up to
you.
After all it sounds more complicated than it really is. Just
keep two
things in mind:
1) If a resource moves (URL changes), then this should be
handled with
a 301 redirect. Then SEs can update their indexes, incoming
links will
still work as expected and you won't lose any visitors, since
they can
still get straight to the content they want.
2) If a page really goes down for whatever reason, then the
response
should definitely be a 404/410. This usually won't happen to
often on a
normal website, so you won't lose any visitors there as well.
But it is
an important mechanism, that shouldn't be ignored.
A short story:
I've been through this all myself. After taking over my
stepbrother's
website (a lawyer) two years or so ago, it was necessary to
completely
rebuild the structure. The old site was build with FrontPage
and was
grown over many years (started in 2000). There were so many
problems,
that we (or better I as the programmer) decided to rebuild it
from
scratch with my own thoughts about the URL structure and my
own PHP
framework behind it. And the way we dealt with the old links,
that were
indexed in the SEs and used by many visitors, was pretty much
what I
described above.
We went through the server's logfile and the search results
on Google to
pick the most common pages from the old site. Those were
entered as
RedirectPermanent directives in the .htaccess file (if
possible, because
some pages were removed entirely), which just took about a
few minutes
for 20 pages or so. In the following months I looked through
the server
error log from time to time for 404s to find old still-in-use
links that
were not mentioned yet in our .htaccess. If I found one, I
just had to
add another 301 redirect to make it working again and that
was it.
All these redirects (26 at the moment) are still in place. I
don't know
if they are still necessary, but I don't care either. They
just make
sure, that if someone stumbles upon even such an old URL from
years ago,
it still works. For a company or - as in this case - a lawyer
this can
be very important.
> i was just reading that apache link you gave me, and
it's not making much
>sense to me, i'm afraid. it's either too far over my
head, or i'm just too
>tired to wrap my brain around it! maybe it'll make more
sense later.
OK, no problem. It was more or less an just addition to the
current
issue, because many of these problems could be completely
avoided with
some more thinking beforehand.
Micha