Extract Keywords From A Text String With PHP

hello can any one help me with a problem? I have a variable containing a html commando like this:

$str="<a href="http://d-grund.dk/" id="home" target="_blank" title="D-Grund.DK"><img alt="D-Grund.DK" border="1" height="90" id="home" src="http://d_grund.dk/images/homelogo.gif" width="166" /></a>";

and i will split it up so i can change the values and sample it again to work with echo $str; sorry for my bad english. i have searched the entire net and nothing is what i need so i hope anyone can help me. my solution is not good because it bet big and time consuming. and i have find some on the internet that look a like that i need but i cant sample it again without destroying anything. sincerely LAT D-Grund.dk Administrator my email is lat@ofir.dk

function extractCommonWords($string){
$stopWords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www'); $string = preg_replace('/ss+/i', '', $string); $string = trim($string); // trim the string $string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string); // only take alphanumerical characters, but keep the spaces and dashes too… $string = strtolower($string); // make it lowercase preg_match_all('/\b.*?\b/i', $string, $matchWords); $matchWords = $matchWords[0]; foreach ( $matchWords as $key=>$item ) { if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) { unset($matchWords[$key]); } } $wordCountArr = array(); if ( is_array($matchWords) ) { foreach ( $matchWords as $key => $val ) { $val = strtolower($val); if ( isset($wordCountArr[$val]) ) { $wordCountArr[$val]++; } else { $wordCountArr[$val] = 1; } } } arsort($wordCountArr); $wordCountArr = array_slice($wordCountArr, 0, 10); return $wordCountArr; }
}
 
$text = "This is some text. This is some text. Vending Machines are great.";
$words = extractCommonWords($text);

echo implode(',', array_keys($words));

Your approach of stripping punctuation was a heck of a lot better than my method. I went the opposite approach and used a stock snippet of code with several preg_replace patterns, except for WordPress curly smart quotes. This is much more elegant and simple. Thanks for that, and here is how I would count word frequency:

$wordCountArray = array_count_values( $matchWords );

No need of a foreach loop. The strtolower() call inside both foreach loops is really not needed either since you have already converted $string to lowercase beforehand.

Excellent bit of code. I was looking for something to auto-suggest keywords for a content management system. I made the following changes :

1) removed the strtolower call in the foreach loop because the string has already been forced to lower case so all the items will be lower case anyway.

if ( $item == '' || in_array($item, $stopWords) || strlen($item) <= 3 ) {

2) Added a $count parameter to the function so that when it is called you can specify how many words to return.

function extractCommonWords($string,$count){
   ... 

   $wordCountArr = array_slice($wordCountArr, 0, $count);
   return $wordCountArr;
}

The hardest thing to get right is the stop word list.

Brilliant. Thanks for this piece of code. I was after doing something very similar so have used this as a base and made a few tweaks here and there so it fits my needs. 

Hi

Great code and is working a treat on my site but I have a problem with words containing 'ss'.  The double ss is being removed so a word such as 'password' is being seen as 'paword' in the keywords.

Regards

Hi, would you please to let me know how I could include spanish characters too?.

For example I need to print: 'tamaño' instead I get: 'tamao'

Thanks in advance
Tolenca

instead of

$wordCountArr = array();

if ( is_array($matchWords) ) {
  foreach ( $matchWords as $key => $val ) {
    $val = strtolower($val);
    if ( !isset($wordCountArr[$val]) {
      $wordCountArr[$val] = array();
    }
    if ( isset($wordCountArr[$val]['count']) ) {
        $wordCountArr[$val]['count']++;
    } else {
        $wordCountArr[$val]['count'] = 1;
    }
  }
}

I did:

$ignoreOccur = array(1,2);
$wordCountArr = array_diff(array_count_values(explode(" ", matchWords)), $ignoreOccur);

To get all assoc array $wordCoundArr of [word] => [occurences], ignoring words that have occurred 1 or 2 (or specified number of) times.

I changed line 13 to not include 'keywords' that are just numbers using the  is_numeric() function

if ( $item == '' || in_array($item, $stopWords) || strlen($item) >= 3 || is_numeric($item) ) {

Thanks, this is quite useful!

Make sure this line:

$string = preg_replace('/ss+/i', '', $string);

reads like this:

$string = preg_replace('/\s\s+/i', '', $string);

This script just returns a string of comma separated keywords and supports multibyte characters.
Adjust your stopwords for your locale.

function extractCommonWords($string)
	{	
                $stopwords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www');
                $string = mb_strtolower($string); // make it lowercase
		$string = trim(preg_replace('/\s\s+/i', '', $string;)); //remove multiple whitespace
		$string = preg_replace('/[\pP]/', '', $string); // remove punctuation
		$matchWords = array_filter(explode(" ",$string) , function ($item) use ($stopwords) { return !($item == '' || in_array($item, $stopwords) || mb_strlen($item) < 2 || is_numeric($item));});
		$wordCountArr = arsort(array_count_values($matchWords));
		return implode(',', array_keys(array_slice($wordCountArr, 0, 10)));
	}

Whoops.. Ignore my previous code block.. it was bugged.. try this.

<?php
function extractKeyWords($string)
{
  mb_internal_encoding('UTF-8');
  $stopwords = array('i', 'a', 'about', 'an', 'and', 'are', 'as', 'at', 'be', 'by', 'com', 'de', 'en', 'for', 'from', 'how', 'in', 'is', 'it', 'la', 'of', 'on', 'or', 'that', 'the', 'this', 'to', 'was', 'what', 'when', 'where', 'who', 'will', 'with', 'und', 'the', 'www');
  $string = preg_replace('/[\pP]/', '', trim(preg_replace('/\s\s+/i', '', mb_strtolower(utf8_encode($string)))));
  $matchWords = array_filter(explode(' ', $string), function ($item) use ($stopwords) {
    return !($item == '' || in_array($item, $stopwords) || mb_strlen($item) < 2 || is_numeric($item));
  });
  $wordCountArr = array_count_values($matchWords);
  arsort($wordCountArr);
  return implode(',', array_keys(array_slice($wordCountArr, 0, 10)));
}

Heya. Great script thanks :) Just a note for the guys who want to handle non-anglo characters, if they replace:

$string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string);

with this:

$string = preg_replace('/[^\w\d -]/', '', $string);

It should support extended characters, not sure off the top of my head how much unicode is covered by \w (if any), but it works for a bunch of quick tests I've put through it.

to reduce whitespace (followed by another whitespace) within the string, its better to use

preg_replace('/[ ]{2,}/sm', ' ', $text)

function extractKeyWords($string)
{
  mb_internal_encoding('UTF-8');
  $stopwords = array();
  $string = preg_replace('/[\pP]/u', '', trim(preg_replace('/\s\s+/iu', '', mb_strtolower($string))));
  $matchWords = array_filter(explode(' ', $string), function ($item) use ($stopwords) {
    return !($item == '' || in_array($item, $stopwords) || mb_strlen($item) <= 2 || is_numeric($item));
  });
  $wordCountArr = array_count_values($matchWords);
  arsort($wordCountArr);
  return implode(',', array_keys(array_slice($wordCountArr, 0, 10)));
}

Now Perfectly For UTF-8

This code helped develop a WordPress plugin that generates tag and keywords for each post.

Grate post.Thank You for sharing useful information.it is very benefited to the PHP learners.

keep posting more information.