Matthias Friedrich's Blog

Scikit-learn: Feature Extraction From Text

I've been playing with scikit-learn recently, a machine learning package for Python. While there's great documentation on many topics, feature extraction isn't one of them. My use case was to turn article tags (like I use them on my blog) into feature vectors.

[Update: Ported the code to scikit-learn 0.11 which is incompatible to 0.10 and older.]

To get some data to play with, let's first extract tags from an RSS feed using feedparser:

>>> import feedparser
>>> feed = feedparser.parse("http://blog.mafr.de/feed/")
>>> tags = [[t.term for t in e.tags] for e in feed.entries]

To keep the example manageable, we'll use the following data:

tags = [
  "python, tools",
  "linux, tools, ubuntu",
  "distributed systems, linux, networking, tools",
]

We want to turn our tags into a matrix with articles (samples) on the rows and tags (features) on the columns. With three articles and 6 different tags, we expect to get a 3x6 matrix.

After looking around, I found the CountVectorizer class that does almost what I want. All I have to do is to provide a tokenizer that takes a string and turns it into a list of tokens. For this particular problem, it's easy to implement:

import re
REGEX = re.compile(r",\s*")
def tokenize(text):
    return [tok.strip().lower() for tok in REGEX.split(text)]

A quick test shows that it seems to work:

>>> tokenize("foo Bar, baz")
['foo bar', 'baz']

Now let's plug it into CountVectorizer and run it:

>>> from sklearn.feature_extraction.text import CountVectorizer
>>> vec = CountVectorizer(tokenizer=tokenize)
>>> data = vec.fit_transform(tags).toarray()
>>> print data
[[0 0 0 1 1 0]
 [0 1 0 0 1 1]
 [1 1 1 0 1 0]]

In some cases it's useful to restrict the number of features. CountVectorizer has a max_features constructor argument that limits the vocabulary to the top K features only. Also, note that the fit_transform() method returns a sparse matrix, but we're turning it into a numpy array because it's easier to work with.

So, the shape of the matrix looks right, but is it really correct? Let's dump the column headers to make sure:

>>> vocab = vec.get_feature_names()
>>> vocab
[u'distributed systems', u'linux', u'networking', u'python', u'tools', u'ubuntu']

My samples don't contain duplicate tags, but if they did, I could weed them out in tokenize() or by modest numpy magic:

>>> import numpy as np
>>> np.clip(data, 0, 1, out=data)

This sets all array entries greater than 1 to 1 (and those smaller than 0 to 0). While we're at it, we can also calculate tag distributions by calculating column sums:

>>> dist = np.sum(data, axis=0)
>>> print dist
[1 2 1 1 3 1]

And finally join with the vocabulary to make it readable:

>>> for tag, count in zip(vocab, dist):
...     print count, tag
1 distributed systems
2 linux
1 networking
1 python
3 tools
1 ubuntu

That's it. Turning tags from an RSS feed into features with very little code.

Other articles

Basics of Near Duplicate Detection

Published: Thu 06 January 2011
By mafr

In misc.

tags: computer science python

Finding duplicate files is easy, anyone can do it. Finding files that are almost identical is more difficult, but it's useful for use cases like detecting plagiarism. In this article, I'll present a simple python program that calculates the textual similarity of two documents.

The basic idea is to reduce …
read more
Software Developers: You Need Computer Science Education!

Published: Wed 18 August 2010
By mafr

In misc.

tags: computer science opinion

Computer science and software development are two entirely different things. The former is a science, the latter is mostly craftsmanship, still struggling to become an engineering discipline in its own right. Being a good computer scientist doesn't make you a good software developer and vice versa, but as a software …
read more
Are Link-Sharing Services Irrelevant?

Published: Sun 25 April 2010
By mafr

In misc.

tags: computer science opinion web

You can use RSS to easily follow a few high-profile websites and link sharing services like Slashdot or Digg to discover popular web content. But that's like reading a classic newspaper and some magazines: The information provided may have a higher chance of being relevant to you, but there's still …
read more
Finding the Majority Item in a Stream

Published: Sun 21 February 2010
By mafr

In misc.

tags: computer science python

Going through old CACM issues I discovered a paper (PDF) on stream processing. A common problem in this field is to find frequent items in a data stream when you only get one pass through the data and you need answers in real time. This is interesting in situations where …
read more
CAP, Consistent Hashing, etc.

Published: Sun 22 March 2009
By mafr

In misc.

tags: computer science databases distributed systems

I've been reading up on distributed systems again. For quite a while, my monthly copy of CACM has been my only connection to computer science topics. This time, I followed a few references and came across interesting concepts (most of them familiar from back in university).

Amazon doesn't just sell …
read more