Designing your pages for search engines

Posted on November 18, 2002

in IA/Usability

by Ian Gregory (tull)

Rated 3.62 (Ratings: 15) (Add your rating)

Want more?

More articles in IA/Usability
More articles by tull

Ian Gregory

Member info | Full bio

User since: October 27, 2002

Last login: October 27, 2002

Articles written: 1

Search engine optimization is rapidly becoming a key part of the web development process. Making your client's sites look good is the easy part of the job - getting them traffic is much more of a challenge. There is lots of conflicting advice floating around, but I'm going to look at some of the techniques that have worked for me.

When?

The time to start thinking about optimizing your pages isn't when you've finished coding your pages and are ready to upload them - search engines need to be considered right from square one. It is infinitely easier to include many of the techniques described below in the page as you write it - and they are much likely to work better this way.

Content, content, content

The one key point to remember is that search engines are there to index content. If your pages don't contain useful content, you'll never attain a good ranking. But if you plan your pages to be content-rich, you're already a step on the way to a good search engine placement.

Think about what keywords you would search for if you were looking for a page like yours. Choosing good keywords is vital - try to include short phrases people might type in when searching. Steer clear of 'stop words' - these are words like 'the', 'or', and 'where' - search engines will ignore them because they are meaningless. And if you are stuck for ideas, have a look at the competition!

Getting your keywords in

Your page title is one of the most important things on the page to get right - include your most important keywords and make it snappy - remember it will be displayed on the search engine's results page and you want people to click it! But be aware of size limits - search engines will display only the first part of long titles, so try to aim for somewhere between 40 and 150 characters.

Don't forget to include your keywords in the META 'keywords' attribute, and write a short (150-400 character) description of the page in the META 'description' attribute. Again, the description is often displayed on results pages so make sure it is accurate and catchy.

Include your keywords throughout the page, but use them wisely. Aim for about 5% of the words on your page to be your top keywords - this is enough for the engines to recognize them, but not so many as to look like spamming! Make good use of the HTML tags in your page - heading tags are an effective way of giving search engines a 'hint' about what is on your page. The alt attribute of <img> tags and the title attribute of <a> are another place you can add keywords, but don't forget that they are there for those who are unable to see the images. Be wary of alienating your audience by filling the tags with lists of keywords!

Something that people often miss is that you can include keywords in the filenames of your images and other external files. In fact, it's a great idea to include all of your CSS and scripting in external files. Many search engine bots will only read the first x lines of your page, and the last thing you want them to see is all your page code! If you have to include scripting or styles inline, try and keep it to the minimum possible.

What to avoid

Try to steer clear of fancy tricks - frames, JavaScript menus, flash etc. Search engines will index frames, but will display them in their listings as individual frames - not what you want your visitors to see when they click your link! Bots (the software that visits your pages and adds them to search engine's indexes) won't follow links that work by using JavaScript tricks - so make sure your pages are linked with standard anchor tags.

Flash movies look great, but don't use them for the main content of your pages - the bots can't read them (and therefore won't be able to add anything from them into their indexes) nor can those without the capability to use flash, including the visually handicapped.

Spamming

Don't attempt to 'trick' the search engines into thinking your page is something it isn't. Their algorithms will spot it if you have used unsuitable keywords in an attempt to rank your page more highly, and will penalize your site. Another common trick is including 'invisible' text on your page that your visitors can't see - but search engine bots can. Again, if the search engines spot this it's curtains to your position.

Search engines are also starting to identify more advanced tricks such as 'cloaking' (using server side to show a different page to search engines than the one shown to ordinary visitors) and 'doorway pages' (pages specifically designed to be full of keywords, targeted at search engines). Don't take the risk. There is at least one major search engine that will drop you right down its rankings if you attempt to fool it.

Your turn...

So, if you're not already doing these things, why not? They are easy to do, requiring almost no effort on your part, and in my experience have caused increased rankings within the space of 6-8 weeks. You've got nothing to lose!

Ian Gregory has been developing web pages since he was a teenager. He's currently completing his final year of a Software Engineering degree at Manchester University, UK. He spent last year working with a major international company developing web based systems and providing advice and guidance on best practice in web development.

When he's not studying or in the pub, he can probably be found tinkering with something flashy at the local student radio station.

32 comments on this article. Log in to add your comment | Rate this article:

Meta tags?

Submitted by dshea on November 18, 2002 - 23:59.

Ian, small point of contention regarding usage of meta tag keywords. According to Danny Sullivan in an article last month, Inktomi is the only major crawler-based engine currently using them. Otherwise good article, and I'm glad you touched upon the accessibility concerns of hi-jacking things like ALT and TITLE for search engine placement.

Keywords?

Submitted by bearwalk on November 19, 2002 - 07:13.

Are META KEYWORDS really worth bothering with? I read the other day that Inktomi was the only SE that still cares about them.

Inktomi

Submitted by paulnattress on November 19, 2002 - 11:22.

There's no reason to stop using meta keywords just because Inktomi is the only search engine to use them. Inktomi provides results for MSN (amongst others, please correct me if I'm wrong). It makes no sense to alienate a major search engine.

Re: 724031

Submitted by tull on November 19, 2002 - 11:41.

dshea, thanks for the comment - that's an interesting article. It's a shame that search engines are disregarding tags added to HTML to aid searching, but understandable given the how they have been abused. However, I'd be inclined to agree with paulnattress - I think it's worth including them, if even alot of the search engines no longer use them - they have value with some search engines and every little helps!

Inktomi again.

Submitted by dshea on November 19, 2002 - 11:46.

According to the article, Inktomi's position is that "the meta keywords value is just one of many factors in our ranking equation, and we've never given too much weight to it."

My opinion is that the time and energy spent developing a list of comprehensive keywords is diminished. I'd much rather focus on developing good content and providing a valuable service that people come back to, than position myself higher and higher in (minor) search engine results to get those extra couple of percentage of hits. The latter idea is too 1998 for me.

Re: Inktomi again.

Submitted by tull on November 19, 2002 - 12:16.

I agree wholeheartedly that content is the most important part of a site. But I still think keywords have their place. Users still need to find the site in the first instance and the use keywords to search so it's important those keywords are reflected accurately in the content.

While explicit keywords in a meta tag are becoming less important, I think having a set of keywords appropriate to the text on the page and using these appropriately in the content is still important to getting those search engine rankings.

Search engine placement service

Submitted by paulpsd7 on November 19, 2002 - 13:28.

What about those search engine placement services? I know the ones that say they'll guarantee you placement in the top 10 results are bogus, but what about ones that just submit your listings to all the search engines? My clients frequently request this service, but I'm not about to go to each search engine and do it manually. Can any of you recommend a good service that's not outlandishly expensive?

old style?

Submitted by Fingland on November 19, 2002 - 18:15.

Whatever happened to the meta name="Robots" content="all" or the robots.txt file you used to be able to use to direct search engines around your site I haven't used that in a long time but see it again every once in a while, is that used at all?

Add URL

Submitted by c3o on November 19, 2002 - 18:34.

paulpsd7: Google, Alltheweb and Altavista are just about the only ones where submission is free and makes sense. An entry in DMOZ is vital, and the Yahoo directory can sure help as well, but the rest of the approximately 3429578234 engines that those scammers claim to submit you to can safely be ignored.

Fingland: Sure robots.txt is used. But like the rest of the robot meta tags (revisit-after, index/follow) it only matters if you specifically want to exclude search engine robots from certain pages.

DO use a meta DESCRIPTION

Submitted by luomat on November 19, 2002 - 23:51.

Meta keywords are pretty much dead, but meta DESCRIPTIONS sure are not. They are displayed when folks find search results for many search engines, including MSN. Opera 7 also has a great feature: when you bookmark a page in Opera7, it will automatically pull the meta-description into the description field for the bookmark (users can change it if they want, of course, but won't need to IF your M-D is good). Avoid having the same Meta Description for all your pages, as it makes the results seem fairly useless when they are found. Imagine all of Microsoft.com having the same M-D, what good would that do?

Let's not forget links!

Submitted by codepo8 on November 20, 2002 - 07:10.

Having links to the rest of the site and also links pointing to your site is also highly relevant especially for google. For example Google bombing was only possible by using loads of sites linking to one with the same link content. Professional search engine optimisers also use different domains to link to the page and sometimes set up loads of bogus domains linking to it for one month to make sure the search engine spiders have been around already.

Meta descriptions

Submitted by Martin Tsachev on November 20, 2002 - 08:28.

Using meta descriptions surely helps as Google uses them although they don't usually show them.

c3o, the Yahoo! directory just isn't what it used to be before, it doesn't look so appealing now.

Cloaking

Submitted by webqs on November 20, 2002 - 16:47.

Hello - Anyone have any links or input on "cloaking". This was mentioned in the article.
I use cloaking to provide plain text content to bots on Flash sites. Content in the movie that is sourced from the db is provided in a plain text format when a Bot user agent string is encountered. I believe this is a legit way to get Flash indexed....
Cheers
James

Re: Cloaking

Submitted by tull on November 20, 2002 - 17:06.

The best way to provide alternate content for Flash movies is to included the text in between the <object&gt tags... this will be seen by search engines but not rendered on browsers which support flash. It also has the advantage of providing textual content for non-flash browsers, which can have considerable advantages in terms of accessibility.

See the w3 guidelines for details.

But Do not Forget

Submitted by wharsono on November 20, 2002 - 19:27.

Do Not Forget About - Search Engine Friendly URL

Use canonical URIs

Submitted by Martin Tsachev on November 21, 2002 - 04:04.

Something a lot of people forget about is using canonical URIs - i.e. do not link to your pages like /foobar/ and /foobar/index.html, use only one of them preferably /foobar/.

Accessibility vs Search Engine trickery

Submitted by lloydi on November 21, 2002 - 06:58.

I often read that using hidden text is a 'bad thing' and that spiders will spot this trickery. Now, I understand why, and it's not something I do.

However, I often use CSS to hide 'skip navigation' links - these are links that allow blind users with screen readers to tab to the first link(the skip nav link) and jump straight over all the navigation into the main content of the page. This is a 'good thing'.

So we have a good vs bad thing going on now - by trying to create links that screen readers can use but that do not get in the way of the visual desgin for fully sighted users, we might fall foul of search engines. And to date I have not had any indication as to whether search engines are able to recognise a genuine well-intended (almost) hidden link and spam.

Re: Accessibility vs Search Engine trickery

Submitted by Martin Tsachev on November 21, 2002 - 07:12.

Hoydi, search engines don't parse CSS. Even if there is a search engines that is able to you can simply use external CSS and add something like Disallow: /css/ to your robots.txt.

Re: Accessibility vs Search Engine trickery

Submitted by paulnattress on November 21, 2002 - 10:04.

I think lloydi is trying to say that he uses a technique which is similar to cloaking, to allow people using access technology to skip links. Lloydi doesn't want this usage of hidden links to be classed as spam. To be honest, I would have thought that search engines could identify that these are there for accessibility reasons - just stick to "skip to navigation" or whatever. In either case, I would evangelise on the side of accessibility.

Shaggy, I had been wondering whether search engines looked at class names in css files to assign relevance to content tagged with that class. If they don't parse css files, then maybe they look at the class name in the HTML?

Re: Accessibility vs Search Engine trickery

Submitted by Martin Tsachev on November 21, 2002 - 10:57.

I don't think they look at class names at all but the chances are if they do they will start looking for simple things like class="header" (some people are using instead of h1, h2...). This would probably be to robots like the b/strong and i/em tags.

Re: Cloaking

Submitted by 4serendipity on November 21, 2002 - 13:42.

Webqs, if you are interested in cloaking you might want to check out the Open Directory category on this subject.

Please note that 99.9 percent of the time I find cloaking to be unnecessary and potentially dangerous to your site's ranking if you are ever cuaght. Google states in their Webmaster FAQ any site that employs cloaking may be banned from their index. Additionally, cloaking requires continual maintenance to make sure that your lists of spider IPs remain up to date. Using less extreme methods, such as Tull's suggestion to include text within the object tags, often produces the desired result without the danger of cloaking.

More on cloaking

Submitted by webqs on November 22, 2002 - 03:05.

Thanks for the links... made good reading. I think here I've mixed up the term "cloaking" as described in those links with what I'm trying to do with dynamic Flash sites served from a database.
Google states :
What is cloaking?

The term "cloaking" is used to describe a website that returns altered webpages to search engines crawling the site. In other words, the webserver is programmed to return different content to Google than it returns to regular users, usually in an attempt to distort search engine rankings. This can mislead users about what they'll find when they click on a search result. To preserve the accuracy and quality of our search results, Google may permanently ban from our index any sites or site authors that engage in cloaking to distort their search rankings.

So Google may ban sites that cloak and serve up content to bots that differs markedly from the content humans read. This in my opinion is a good thing, because it can as Google FAQ says "mislead users about what they'll find when they click on a search result.". In Google word's what I'm doing is not what they term cloaking.

What I'm endeavouring to do is serve up exactly the same content to bots in a plain text format that humans read and experience in Flash format on a site. To do this, I need to scan the UA string and determine if the visitor is a bot. What I'm doing here doesn't involve returning different content to the bot than what the human receives.

Here's some links to help illustrate it:

For the humans
For the bots
What Google cached - hopefully the link will work.

The code to do this isn't that extreme (simple if-else conditional in PHP). I looked into the object method which is also interesting and just as simple as this.

Cheers
James

Re: More on cloaking

Submitted by bearwalk on November 22, 2002 - 03:28.

So Google may ban sites that cloak and serve up content to bots that differs markedly from the content humans read.

I don't think they mean what human eyes see, but what code their browsers recieve. More likely, Google visits the same page twice with different HTTP_USER_AGENT strings; one saying "Googlebot" and one saying "Mozilla" (thus emulating a user's browser) and then compares the HTML they get. If the two pages are too different, it assumes cloaking was involved.

Re: More on cloaking

Submitted by Martin Tsachev on November 22, 2002 - 05:47.

Google surely don't visit your pages twice, that would be a waste of bandwidth as most sites do not cloak. The fact is that there is bad cloaking and good cloaking (like the example above), only a human can decide which one is good and which is bad.

Re: More on cloaking

Submitted by dbindel on November 24, 2002 - 01:47.

"Only a human can decide which [type of cloaking] is good and which is bad." True, but the Google bot is not a human, is it? Therefore it abides by its own rules, which were made by its programmers, and it cannot distinguish between which type of cloaking is going on... it would take way too much time to do that. So to Google, there is absolutely no difference between "bad" cloaking and "good" cloaking..... they are one and the same.

Re: More on cloaking

Submitted by Martin Tsachev on November 24, 2002 - 03:17.

Googlebot doesn't decide on cloaking at all because it can't, they can probably detect cloaking on geographic location by spidering from different data centers but that would be a waste of time. Generally Google bans sites because of other bad things not cloaking.

future

Submitted by vive on November 26, 2002 - 10:39.

Here's an interesting and amusing site http://www.scumware.com. If your site is going to be full of content just remember that programs will try to sell things based by the words shown on the page. Here's an example from one of the creators of the idea http://ezula.com/TopText/Features.asp and in aol's latest software some text navigational links become clickable advertisements all in the name of increasing companies bottomline sales by using others handwork and creativity.

Re: future

Submitted by tull on November 26, 2002 - 11:15.

Microsoft originally considered implementing functionality such as this in Windows XP/IE 6, but the feature was withdrawn, officially because of a delay. Many people attribute the widthdrawal to resistance to the idea, both from within Microsoft and from the tech media & other groups.

I don't think similar programs will stay around for long (except as spyware) as they simply don't add value to the customer.

more observations...

Submitted by benjer on December 4, 2002 - 04:24.

In the past I used to use class="header" but now I modify my tags etc. with an external css file. Apparently if your 'TITLE' and are equal it will be of greater importance.

h1 {
	font-family: Arial, Helvetica, sans-serif;
	font-size: 16px;
	line-height: 24px;
	color: #FF3300;
	padding-top:25px;
	padding-bottom:25px;
}

Its also good ptractice to repeat the page DESCRIPTION in the first paragraph below the heading. When creating textual links, try to make the links a keyword. Keyword links will garner much more credit than 'Click Here.'

Link Popularity seems to be the new craze - try and get links from as many *relevant* websites as possible. For example, search Google for "link:www.evolt.org" - about 6,200 - enough to be very envious of :)

But as stated its generally the copy itself that will make/break your SEO

RE: more observations...

Submitted by jayZ on March 18, 2005 - 06:27.

Good point benger about using CSS to modify the tag itself . The trick with search engine optimisation is Keep It Simple - it really is. Search Engine Spiders are not very sophisticated HTML parsers. They have a very specific set of things to look at, and are easily thrown off. The more your web page (code - wise, anyway) resembles a page from circa 1994, the better. As for the imporance of link text - well sometimes it makes you want to bang your head against the wall when! I remember demonstrating to a search engine optimisation client the error of their ways by showing them how they were number 2 in Google for the phrase "Click here" against 500 million competitors...

Also

Submitted by cianuro on November 29, 2006 - 15:16.

Also, it is important (And difficult) to write your content for your users with search engines in mind and vice-versa. On search engine optimisation Ireland we have struggled to "pepper" keywords into the content as much as possible without making the actual content unreadable. Another easy to implement search engine marketing idea is to use keyword rich anchor text when INTERNALLY linking. Not only is it descriptive for your users, it describes to search engines the content of the page being linked to. It's a shame to see some webmasters still using hidden text and hidden iframes these days but it's nicer to see the search engines are catching on. Dave.

dmoz problem

Submitted by piju2111 on December 6, 2007 - 17:41.

ya thats correct i also try lot of times to be listed on dmoz, but as you said, summit and forget about it. i dont know why they are not replying. thank you

Start of page header

Submit

Article Categories

Highest rated articles

Recent comments

Designing your pages for search engines

Want more?

Ian Gregory

When?

Content, content, content

Getting your keywords in

What to avoid

Spamming

Your turn...