Next: Experimental results
Up: Description of the corpora
Previous: Preprocessing
Description of the FSTs used
The first two unsupervised techniques we are presenting in this sub-section have shown their value in the clustering [14] and categorization area [25]. Particulary, the document frequency technique is an effective and simple
technique, and it is known that it obtains comparable results to the classical supervised techniques like and Information Gain [26]. With respect to the transition point technique, it has a simple calculation procedure, which has been used in other areas of computational linguistic besides clustering of short texts: categorization of texts, keyphrases extraction, summarization, and weighting models for information retrieval systems (see [19]). Therefore, we consider that there exists enough evidence to use this technique
as a term selection process.
- Document Frequency (DF): This technique assigns the value to each term , where means the number of texts, in a
collection, where ocurrs. This technique assumes that low frequency terms will rarely appear in other documents, therefore,
they will not have significance on the prediction of the class for this text.
- Term Strength (TS): The weight given to each term is defined by the following equation:
with
Besides, both texts, and must be as similar as a given threshold, i.e.,
, where must be tuned
according to the values inside of the similarity matrix. A high value of
means that the term contributes to the texts and to be more similar than .
A more detailed description can be found in [25] and [18].
- Transition Point (TP): A higher value of weight is given to each term , as its frequency is closer to a frequency named the transition point () which can be found by an automatic inspection of the vocabulary frequencies of each text, identifying the lowest frequency (from the highest frequencies) that it is not repeated; this characteristic comes from the formulation of Booth's law for low frequency words [6]
(see [19] for a complete explanation of this procedure). The following equation shows how to calculate the final value:
where is the frequency of the term in the document .
The DF and TP techniques have a temporal linear complexity with respect to
the number of terms of the data set. On the other hand, TS is computationally more expensive than DF and TP, because it requires to
calculate a similarity matrix of texts, which implies this technique to be in , where is the number of texts in the data set.
Next: Experimental results
Up: Description of the corpora
Previous: Preprocessing
David Pinto
2007-05-08