Words by c.z.robertson

Features in bayesian spam filtering

2009-04-17T05:01:24GMT/BST

I've recently switched from using CRM114 as my spam filter to SpamAssassin. I wanted to take advantage of systems like Razor and Pyzor and I wanted to apply some whitelisting. On the other hand, I'm inclined to think that a bayesian classifier approach to content filtering makes a lot more sense than SpamAssassin's collection of weighted tests. I don't know how the weights of SpamAssassin's tests were calculated. They seem very precise, but I wonder about their accuracy. Furthermore, you might expect their accuracy to vary from person to person.

So here's what I was thinking: The weights of the tests should be calculated in a bayesian way. Run each test over the email and if it triggers then add it as a feature for bayesian consideration. Currently all the bayesian spam filters that I'm aware of only use words (or some tokens based on words) in the email as the features they consider. But I'd like to know how much spaminess is implied by an email coming from a machine with no reverse DNS. SpamAssassin gives that a score of 0.1 (with a score of 5 indicating spam, by default), so it's got some sort of implicit notion of probability, but I'd like to see that probability calculated using bayesian techniques.

Funnily, while writing this post I realised that I'm not the first person to come up with this idea:

Specific spam features (e.g. not seeing the recipient's address in the to: field) do of course have value in recognizing spam. They can be considered in this algorithm by treating them as virtual words. I'll probably do this in future versions, at least for a handful of the most egregious spam indicators. Feature-recognizing spam filters are right in many details; what they lack is an overall discipline for combining evidence.

That's from Paul Graham's A Plan for Spam, the essay that brought the world's attention to bayesian spam filtering in the first place. If anyone knows of a system that does this then please let me know.

Comments: 0

ORCHID chopstick

2008-08-25T15:57:17GMT/BST

From the elaborate packaging of some very pretty chopsticks I was given last week:

Just as we can conceive a word in a grain of sand, we can perceive the universe from a pair of chopsticks. As a top company, ORCHID chopstick, which specialise in chopsticks, sale of chopsticks as craft work. The unique managing concepts of ORCHID chopstick bring up an enterprise from a pair of chopsticks.

Chopsticks are not only a kind of dispensable tableware in our daily life, but also a perfect present for our relatives and friends. The significance of chopsticks as a present is lies in the homophonies of chopsticks in Chinese with the implied meanings of happiness and luckiness for newly-married couples, chopsticks also expresses the sincere wishes to persons who are becoming one family and having babies as soon as possible. For children, it means fast growth and for aged, it brings happiness, health and longevity. Furthermore, chopsticks come up as a pair which is indispensable of each other, therefore, chopsticks can be a symbol of solidarity and friendship. As a present for friends, chopsticks means good things come as a pair and a long lasting friendship.

Comments: 0

Learn from the mistakes of others

2008-03-24T16:32:39GMT/BST

If you're going to get involved in a pillow fight, don't do it in a black woollen greatcoat. a) It's black, so all the little white feathers show up, b) it's wool and all the little white feathers stick to it very well, and c) there's a lot of surface area to spend the next few days picking the little white feathers off.

Comments: 0

Hands of Ruin

2008-03-21T12:36:53GMT/BST

Last year I decided that I should get back into music again. Hands of Ruin is the result.

I'm sticking with the dark, atmospheric electronica aesthetic that you might have heard with my earlier work. My tools are a bit different though. I'm now working almost exclusively with ZynAddSubFX and Seq24. I'm trying not to spend too much time experimenting with new bits of software. Both of those tools are nice because they're very simple and it's easy to get musical results out of them. Unlike Csound, you don't have to spend weeks building your instruments before you can create some sound.

I'm also trying to sell my music now, which is an interesting departure for me. Too much time spent reading about entrepreneurialism, probably.

Comments: 0

Weekend reading

2008-02-02T22:44:52GMT/BST

Comments: 0