Saturday, October 13, 2007

Cresting that hill

Now I'm mostly ignoring a lecture in my Software Engineering class. This weekend is the midpoint of my final semester. I've done one midterm, and I have one more scheduled for the afternoon. My report draft was finished earlier this month. After tomorrow, all I'll have left is one or two homework, finalizing my thesis, and the finals. I feel like I'm getting over the top of a very long, slow rollercoaster, seeing the track ahead, and getting ready for the downward ride.

My graduation ceremony is December 8. I don't expect anyone to come except my family, but you can email me if you want to be there.

Wednesday, October 10, 2007

Be all that you can be in the mercenary reserves

Conservatives and libertarians alike generally claim that anything that "The Government" does is bad more or less by definition, and that a smaller, leaner government which manages fewer services is better.

One notable exception, at least where Republicans are concerned, is the military. The one truly legitimate function of government, as far as they're concerned, is in maintaining a kick-ass defense force that can take on the world. In fact, according to this source, the U.S. currently spends more of its budget on the military than the next fourteen countries combined, and that accounts for about 43% of all the military spending in the whole world.

So conservatives hate government but love the military, which should come as no surprise to anyone who's had a pulse at any point in the last 25 years. And I'm not going to pass a value judgment on the relative amount we spend on the military, at least not in this post. All I'll say is that there are lots of things in the world that can kill people, and foreign armies are only a few of them. Like, say, hurricanes. Collapsing bridges. Poverty. Those kill people too.

That's not to say that people don't die from military attacks as well. I'm just saying, you know, if the purpose of spending money on the military is to prevent Americans from dying, then what you've got is a minimax problem: save the most lives for the least dollars. And I find it hard to believe that for our current spending rate, you couldn't save a few more lives by doing something else with the cash we could save by reducing our total military spending a bit. Say we used only 40% of the total world expenditures on military. Say we only spent as much as the next 12 countries combined. That's a lot of repaired bridges, know what I mean?

I'm not a pacifist, though. I believe that maintaining a certain level of military is necessary to the survival of a country. You need to keep the Visigoths from sacking the city, as it were. It's just that reasonable people can disagree on what the ideal level is, and I happen to think the level is substantially less than ten times that of our nearest ally. Speaking of the Roman Empire, bear in mind that they maintained a badass army for a long time before they experienced a financial collapse due to bad management. It was only after that, that the barbarians who had waited politely outside the gates for centuries got to just stroll in the crumbling front door.

Anyway, if there's one thing conservatives like more than spending money on the military, it's privatizing functions that are currently managed by government. After all, everyone knows that when you leave things up to the Stupendous Free Market, you guarantee that they will be done much better and more efficiently than if you, um, took concrete steps towards actually getting them done. Really. Just trust us on this one.

So anyway, under the Bush administration, apparently the military no longer gets a special exemption from this rule. That's why we are now paying for Blackwater, a private security contractor that is now in charge of guarding diplomats. We pay approximately $445,891 per year for the privilege of hiring a Blackwater security specialist. By contrast, a military sergeant costs us around $69,000 per year on the high end, including room and board. See how efficient the private market is?

I was never tempted to join the army. Call me a coward or a spoiled brat, I'll probably own up to it. I'd also probably make a lousy soldier, because I'm not good at unquestioningly following orders. I question tasks a lot at my job, not because I'm trying to be a pain, but because I feel that I can execute a task more effectively when I understand what the intention is behind the task. I'm not saying this to brag or show that I'm in any way "better" than a big tough military guy. I'm just saying that my mindset is somewhat different from what's required in a military role, and they'd have to beat a lot out of me to get me there. I recognize that having an army that is willing to follow order is pretty important, however.

I'm taking a class in Project Management right now -- hooray for the LAST class I need to complete for my Master's! One of the issues that is discussed in the text is that the larger your project gets, the more important it is to maintain a strict structure in your organization. The proverbial "two guys in a garage" can accomplish a small project very well, but a hundred people who all demand creative control over the same project is a disaster waiting to happen. Now apply that to the military, which is arguably one of the biggest freakin' projects anybody could ever undertake. I mean, post-surge Iraq currently has around 175,000 soldiers on the ground, to say nothing of the people at home who are connecting them with the civilian leadership.

I can't really imagine what's involved with organizing so many people, but it's pretty clear to me that if you have a general telling his underlings saying "We need to accomplish X, see that that gets done," and then a colonel below him says "All right men, let's figure out how to do Y, which is a sub-task on the way to accomplishing X," and then some clown of a Lieutenant way down the line says "No, screw this task, I don't want us to accomplish X anyway!" then you've got a problem. It's not because that lowly Lieutenant is necessarily wrong about the quality of the final objectives, or because he couldn't necessarily do a better job of managing the war than the general. The problem is that everybody can't be a general, and if you let all 175,000 soldiers make the decisions then you've got a fine mess, and people are fighting against each other instead of working toward a common goal.

Of course, a military that works as a single unit can do great evil. Of course, a single individual can also do great evil. But there's MORE of the military, so they can do MORE evil because they're acting as a single body. We need a military, because there is this one big task that is necessary to accomplish. Adequate national defense is absolutely vital to the health of a nation in a non-utopian world. So is executing wars, in such cases where war is necessary and right. By which I mean, not this war.

The problem with the military right now is not that it is a military; it's that it's a military in the hands of people who are hell-bent on using it towards nefarious ends. No, strike that. They don't set out saying "Let's all do bad things now." Really. Not even Bush. The key to understanding the Bush presidency is personal cronyism. Bush didn't pick "Heckuva job Brownie" to head FEMA because he intended to put an incompetent boob in the job. No, the chain of events is: first Joe Allbaugh was given the job because he helped run Bush's 2000 campaign, and then Brownie was picked to succeed Allbaugh because he was Allbaugh's old college buddy.

By and large, the members of the Bush administration see government jobs as an opportunity to make money, and to kick more money over to their friends. Other people may get hurt, killed, or impoverished as a result of this focus, but that is a by-product of the official policy, not the intention of the policy itself. What's happening here is that the president and his cabinet all have an attitude that, basically, who gives a crap if government is effective, as long as I'm helping out my friends? And that attitude gets trickled down through the ranks, because people with that perspective will pick friends who often have the same perspective, who in turn pick THEIR friends who have that perspective, and so on.

The military privates aren't part of this tree of mutual back-scratching. Demographically speaking, if they were well-connected then they probably either wouldn't be signing up (like me), or else they would be signing up as lieutenants rather than privates. But the privates are following the orders of people in the back-scratching tree. Orders to do annoying things like "treat prisoners humanely" and "follow the Geneva conventions" and "Please don't kill civilians if you can help it" ultimately need to come from the top of the command chain, and the problem is that those orders aren't coming. It's not because the kids signing up for the military are bad people. Rather, doing those things to improve international relations is harder than not doing them, and everyone has the capacity to do evil and then rationalize it, given the right circumstances and enough peer pressure. I believe that this probably includes many of the very fine and moral people here, myself as well. But for people who voluntarily joined the military, gave up many of their civil rights, and accepted a program of unquestioningly following orders, maybe it's especially true. I don't know.

But really, the problem is the lax attitude at the top, and nowhere is this better exemplified than in Blackwater. Blackwater is given that sweet deal that nets them over 400 grand a year. They are paid by the US government as if they were a military force, but they are not subject to the military code of justice. In essence, there is no law governing them, which goes some of the way towards explaining why they keep getting involved in incidents like this recent one where they gunned down 14 Iraqi citizens in cold blood.

If the Blackwater employees were soldiers, they would be directly accountable to the their commanding officers and the US government, who still at least have SOME standards of behavior, and have the authority to court martial and imprison the offenders in extreme cases. But no one has any such authority over Blackwater. Even though we're paying them with our tax dollars, they are not required to do anything we ask them to.

And astonishingly, the Bush administration is standing behind them and refusing to give anyone the go-ahead to investigate this incident. Nuri al-Maliki, the US-picked prime minister of Iraq, now wants to ban Blackwater from his country. Only Bush doesn't want to let him.

It's worth pointing out that the head of Blackwater is a member of an evangelical political group called "Christian Freedom International", and is well-connected in politics by virtue of being related by marriage to Amway magnate and Republican gubernatorial candidate Dick DeVos. See? It pays to be connected. Ahhh. Good old Amway.

So, you know, don't blame the military. While lots of bad things are done by military grunts who are required to do whatever they are asked to, even worse things are being done by people who have no orders to follow. They're given free reign of the place, they're not required to follow any international codes of conduct whatsoever, and their only real objective is to do their job in such a way as to loot as much money as possible. Both from Iraqis and from Americans.

The fact that military personnel do bad things does not mean that the institution of the military is the problem. Just like the fact that corrupt crony governments doing bad things does not mean that "government" is the problem and should be abolished. The problem is that we have some really bad leaders right now, who are turning a blind eye towards violations of ethical conduct, if not outright endorsing it.

Monday, October 08, 2007

First draft completed

Yesterday I brought my thesis report up from 23 pages to 45 pages before calling it a night at midnight.

It's not the most spectacular writing I've ever done; it'll need lots of proofing and major details are still missing. But a friend of mine told me "It's better to have a thesis report that's DONE than one that's GOOD." So, I'll reread it a bit tonight and then send it to the appropriate people.

Saturday, October 06, 2007

Longest Saturday ever...

And it's not over yet.

I thought I'd just wrap up a few things this morning with the program and then spend the rest of the day writing. As it turns out, the steps to analyze the data are actually non-trivial and require some thought and more programming. Who'd have thought it?

The good news is that I'm done collecting all the results I want for this draft. I have a big spreadsheet made of sites going in one direction and topics going in the other, and each cell has a "weight" given to that topic by a site. That way I can compare the weights and see if there are any interesting patterns.

To be honest, nothing about the table is as interesting as the results I commented on from Digg. For example, USA Today, which has sort of a "Newspaper of the common idiot" vibe about it, does indeed have an excessively high amount of stories on Britney Spears and Paris Hilton. But the point I was making on Digg is that if Digg users accurately represent ordinary newspaper readers (which, you might reasonably argue, they don't) then taking "entertainment" and calling it "news" is not really an appropriate strategy.

A few other unusual results I found: New York Times gives a relatively high amount of attention to Paris Hilton also. Not in their top few stories, but distinctly in the top half.

Also, Fox News gives a surprising amount of coverage to serious news. When I included "Blackwater" in the results, I found that Fox News is the only paper that has recently given that topic higher priority than all the others. However, when I looked deeper into the individual stories they reported, I noticed that mostly they weren't by Fox reporters: they were stories that originated with the Associated Press, and then were just relabeled as "Fox News" and pasted on their site. I don't think anyone even filters it. In fact, Fox News has a much higher presence on Google News than more serious news organizations does, and it seems to be because they just automatically repost anything that comes their way.

I dunno, maybe I shouldn't stretch too hard to look for excuses to bash Fox. If I throw out theories like this then the paper won't seem very objective.

The paper today stands at 26 pages, of which 12 are actual substantive text. I have a long way to go - my eventual target is around 50 pages (including the padding), but I'll be happy with 30 or so pages for this draft. Luckily, I have a lot of tables and graphs to paste in; I have lots of material to steal from this blog; some philosophical discussion of user taxonomies that I can borrow from my term paper this summer; and I can always throw in code samples when I'm describing the program. I think I'll make it, but it's going to be a long night and another long day.

And after that, I get to start studying for my two midterms! Yay!

More 9/11 about 9/11 Rudy 9/11 Giuliani

As I mentioned in yesterday's post, the data analysis for my Master's Report indicated that some of the most well-beloved topics on Digg.com are about how much Rudy Giuliani sucks. But why does he suck so much?

One of the top recommended stories on that front search page, currently sitting at 2,796 diggs, is a blog entry entitled "Mr Giuliani Please Stop Mentioning 9/11". But give Rudy a break, he can't stop mentioning 9/11 because he has absolutely nothing else to run on. He wasn't a particularly popular leader until, like Bush, the photo opportunity of a lifetime fell in his lap. (The photo opportunity also fell in Bush's lap, I mean. I don't mean that Bush fell in Giuliani's lap.)

Since then he has exploited 9/11 in virtually everything he says and does. Like most of the other Republicans, Rudy talked about 9/11 constantly during his address at the 2004 convention. One of his supporters recently hosted a fundraiser for him asking for donations in the amount of $9.11. When told about asbestos hazards at the site of the World Trade Center, he told reporters "I was at ground zero as often, if not more, than most of the workers ... I was there working with them. I was exposed to exactly the same things they were exposed to. So in that sense, I'm one of them."

A couple of weeks ago Rudy gave a speech for the National Rifle Association. Trying to convince NRA members to support him even though he was a strong gun control advocate when he was mayor, he explained: "I also think that there have been subsequent intervening events — September 11 — which cast somewhat of a different light on the Second Amendment and Second Amendment rights. It doesn't change the fundamental rights, but maybe it highlights the necessity for them more."

But wait, that's not all. During that speech, he pulled a ham-handed political stunt by (in my opinion) pretending to take a phone call from his wife while he was talking. He said some cutesy stuff to her at the podium, and then said "I love you honey," receiving approving applause from the assembled crowd. Later, when Giuliani was asked in an interview why he took that call, he explained: "quite honestly since Sept. 11 most of the time when we get on a plane we talk to each other and just reaffirm the fact that we love each other."

That's adorable. Even more so when you consider the fact that on September 11, 2001, Rudy Giuliani was married to a different woman. (He married his current wife, Judith Nathan, on May 24, 2003.)

And Rudy is the Republican front runner. Right now he's the favored candidate to win the nomination by a significant margin. As much as people apparently love to read stories that bash Giuliani, that means the other candidates are considered even worse.

Speaking as a partisan Democrat, I think that's totally awesome.

Friday, October 05, 2007

Starting to collect results

So far I've managed to scan every Digg.com story that relates to my chosen topics. When you search Digg for a story, you get a list of all the stories that were ever submitted to Digg with the search words appearing in the title. First somebody submits a page, and then other people come along and recommend the page. The page winds up with a "score" that indicates the total number of people who recommended the same site.

Collected below is output from my analysis that shows how many stories appeared, followed by the average score of all the stories.

Welcome to News Miner 1.0.
The database has been opened.
What shall we do today, hmmm?
1. Get monthly news clusters (one topic)
2. Get monthly news clusters (all topics)
3. Explore current clusters
4. Get new Digg scores (one topic)
5. Get new Digg scores (all topics)
6. Generate results table
7. Analyze results
0. Quit
7

There were 14 topics
topic "Anna Nicole Smith": size 752, mean 3.5226063829787235
topic "Barack Obama": size 756, mean 6.994708994708994
topic "Blackwater": size 358, mean 9.997206703910615
topic "Britney Spears": size 3019, mean 3.1570056310036434
topic "Giuliani": size 1468, mean 18.582425068119893
topic "Gulf Coast": size 61, mean 3.459016393442623
topic "Harry Potter": size 2502, mean 5.09912070343725
topic "Hillary Clinton": size 996, mean 7.945783132530121
topic "John Edwards": size 566, mean 7.80565371024735
topic "Mitt Romney": size 426, mean 7.030516431924883
topic "New Orleans": size 858, mean 5.970862470862471
topic "Paris Hilton": size 2497, mean 3.644373247897477
topic "Rupert Murdoch": size 127, mean 7.52755905511811
topic "Tiger Woods": size 390, mean 3.546153846153846

I've highlighted a couple of interesting numbers in bold italics. Notice that "Paris Hilton" and "Britney Spears" both have a very high numbers of pages, indicating that many people found stories about those people that they considered were worth submitting. (Most of them, by the way, are jokes or empty promises of smut.) But other people either don't read those stories, or don't like them enough to recommend them.

On the other hand, look which topic is far and away the clear winner of the Digg scoring game: Rudy Giuliani with an average of 18.6. In fact, Rudy's average score is more than twice as high as the average of his next political competitor Hillary Clinton.

So that means that people love Rudy Giuliani, right? Ermmm... not exactly. Look at the headlines on the page showing the all-time highest rated stories about Giuliani.

  • Mr Giuliani Please Stop Mentioning 9/11
  • Rudy Giuliani Constitutionally Ineligible To Be President
  • Anger at Giuliani 9/11 fundraiser "$9.11 for Rudy" in poor taste
  • America's Worst Nightmare: President Giuliani
  • Giuliani: "For Me Every Day Is An Anniversary Of Sept. 11" GET OFF IT!
  • Rudy Giuliani: "Freedom is Slavery"
  • Rudy Giuliani's daughter is supporting Barack Obama
  • DIGG this! Soldier to Giuliani: Have you done your foreign policy homework?
  • Reporter Arrested on Orders of Giuliani Press Secretary
  • Giuliani Closed Off Streets to Avoid 9/11 Victims' Families

Um, yeah. Are you noticing a pattern here? They're all negative. Not some of them. All of the top nine stories. Apparently people love to read about Giuliani so they can reinforce how much they hate him.

To be fair, Hillary Clinton's page has a lot of negative stories too, but certainly not all of them. Many of the hits are anti right wing coverage, and people apparently reacted well to her TV spoof of "The Sopranos" with Bill.

Disclaimer: Digg users are not a representative sample of the general public, but they are my stand-ins for them. Digg users are a self-selected group of active news readers. What Nixon referred to as "The silent majority," aka "People who don't pay attention" are not represented here, and I have no way of knowing which stories they would recommend if they were asked.

Monday, October 01, 2007

Google captured me again

D'oh. I've been CAPTCHA'D.

Sitting here in Texspresso after work. I decided to let my program run at top speed. I wasn't sure whether it would take a fixed amount of time to catch me, or whether it's mainly based on the number of page hits. I reduced my sleep time so that I get a new web page every two seconds. It only took them twenty minutes to make me stop, so the speed at which I hit them is definitely a big factor.

Oh well. In that time I managed to collect 1100 new clusters, which finishes off the month of September 2006 (the month that Paris Hilton got arrested, which make some entertaining analysis). But I only managed to pick up 100 stories, so I've got more to do.

Nephlm mentioned a program called Tor that hides your IP address, so maybe I'll try that and see if it works.

Update: Tor works! It works like a charm! Nephlm, I owe you a beer. Come to Austin sometime and I'll pay up.

Tor is a product of the Electronic Freedom Foundation, and what it does is rout your web requests through various remote servers so that the Google server can't tell where you're really coming from.

But an amusing side effect is: When I logged in to blogger, everything was in German. I must be sending requests through a host in Germany somewhere, and now Blogger sees my destination and thinks I want the German version of Google.

Oh well, who cares, as long as I'm getting my data. :) "Post veröffentlichen" means "publish this post," right?

Sunday, September 30, 2007

Paper abstract

For anyone who's interested. I want to take this opportunity to repeat my thanks to those people who suggested directions to go in when I asked for help earlier this year.

In recent years, major news corporations seem to dedicate an increasing amount of time and space to "fluff," reporting on celebrities, entertainment and crime stories, rather than more essential national and international news. As such news content is increasingly gathered online, it has become feasible to aggregate large amounts of data from a wide range of sites. This report proposes a model for collecting information from news agencies, then applying the techniques of Data Mining to organize this reporting in a way that identifies the priorities of individual organizations.

In addition, the rise of user-based taxonomies has made it possible broadly to evaluate the interests of people who actively read and recommend news. In the final analysis, data collected from users of Digg.com are compared with data collected from media sites. This provides a benchmark for determining whether the delivery of "fluff" news is delivered is a fair response to popular demand, or whether typical news readers are dissatisfied with the level of serious event coverage found in the media.

Thesis saga continues: It's working

As per my last post, I'm now sitting here in Schlotzsky's Deli, which has free wi-fi, and my data mining program is just blazing along. I have my flag set to yell at me immediately if I get the "Are you human?" warning.

Maybe it's just because I've been listening to Harry Potter and the Deathly Hallows on tape, but my current frame of mind is that this is kind of exciting. Sort of like I'm sneaking around to my safe houses in order to avoid being apprehended by the authorities. It's the nerdiest cloak-and-dagger story you've ever heard, I bet. And by comical coincidence, I just checked my progress and it's looking at a story about Harry Potter from March '06 right now.

In a weird kind of way, this has actually helped me refocus my attention on how to attack the problem, a bit. Previously I was just indiscriminately grabbing all kinds of data, without regard to whether it was useful or not. Now that I know that my time is limited and I could be "captcha'd" at any moment, I've tightened my focus in a way that makes a lot of sense. I'm focusing on stories only within a specific time range, and only bothering to look at clusters of approximately average size. This way, I know that even if I'm interrupted in the middle and can't collect any more data at all, I'll still have plenty of information to work with.

This has also given me some new ideas on how to interpret the data, and I'm looking forward to analyzing it later. Eventually I won't need to worry about what Google thinks of me, because I can just read their stuff from my own private database.

Saturday, September 29, 2007

An unexpected hazard of mining other people's websites for information:

Sorry for the deluge of long computer sciency posts. The thing is, it's helping me to blog about my thesis. Earlier this week when I posted some comments about my research, I pasted the whole post into my paper and got another three pages out of it. Awesome! It needs some editing, but there's plenty of solid material in there. So, let's see if I can get away with writing the whole paper just by blogging. You, dear readers, will just have to decide whether to suffer through these posts or skip them. Unfortunately, tonight's commentary is about a big setback.

My web skimming program has been having a field day with the Google news archive. I'm currently pulling stories from back to a year and a half ago. Before dinner tonight, I picked up 2000 new Google clusters on "John Edwards." I was pretty cheered by this progress.

When I got home, I fired up the program again and started searching the year for clusters of "Anna Nicole Smith"... and got nothing. Not a single hit.

This was kind of bewildering to me. I tried a few more times, digging through it with the debugger. Nothing. So finally I pulled out the URL of the search page my program was looking at, and pasted it into my browser. I got this message:

403 forbidden

We're sorry...

... but your query looks similar to automated requests from a computer
virus or spyware application. To protect our users, we can't process
your request right now.

We'll restore your access as quickly as possible, so try again soon.
In the meantime, if you suspect that your computer or network has been
infected, you might want to run a virus checker or spyware remover to
make sure that your systems are free of viruses and other spurious
software.

We apologize for the inconvenience, and hope we'll see you again on
Google.
To continue searching, please type the characters you see below:
[Typical captcha text returned]

UH-oh. I experienced a bit of temporary jumpiness as I realized that Google noticed I've been hitting their server really hard and really fast. I typed in the confirmation text, of course, and it let me view the page. But I tried the program again, and it still didn't work.

I did some research, winding up at this post. I don't really get the details, but it sounds like Google has been targeted by malicious spyware programs in the past, which do tons of web searches that somehow uncover target servers that are vulnerable to attack. Then they install copies of themselves on those target servers, which in turn do more malicious searches on Google's site.

So, yeah, that's pretty neato that they catch bad guys. Unfortunately, they also catch me. That's bad. I have a thesis that needs finishing.

I decided to wait a few hours, and in the meantime I put in some code that makes it pause for five seconds before it gets a web page. I don't want to annoy them.

A few hours later, the spam catcher stopped harassing me. I let the program run for a while longer, and it managed to walk through a couple thousand more clusters, all from the month of March. But then it stopped again, with the same message. This time I had a break in there to kill the program before it started failing a bunch more challenges.

This is going to be a slow process. I want my data. Now. I might consider bumping the delay up to thirty seconds in the morning.

Also, I suspect that Google is making a note of my ISP to determine that I am an evildoer. If that's the case, then maybe I can get around it by wandering around town with my laptop. I'll go from one wireless hotspot to the next, grabbing a few thousand entries here and there, until I've got the whole year's worth of material.

Monday, September 24, 2007

Data mining the news (ongoing work)

My thesis is about using data mining to analyze the relative emphasis that traditional media outlets give to various types of stories. Then I'll be comparing this data to the emphasis that actual news consumers who inhabit Digg.com give to the same stories. My point is to discover which types of stories are overplayed or underplayed, and come to some sort of conclusions about which types of news sources best reflect the pubilc interest.

To that end, I've written a big Java program around an online MySQL database. In the last few days I've cataloged about 22,000 news pages, although only a small number of them will ultimately turn out to be important to the study. I've labeled roughly a dozen web sites and a dozen news topics as "interesting." The sites are:
  1. www.washingtonpost.com
  2. www.nytimes.com
  3. www.foxnews.com
  4. www.guardian.co.uk
  5. online.wsj.com
  6. www.usatoday.com
  7. www.cnn.com
  8. www.townhall.com
  9. www.washingtontimes.com
The stories are:
  1. Rudolph Giuliani
  2. Anna Nicole Smith
  3. Harry Potter
  4. Tiger Woods
  5. Rupert Murdoch
  6. Barack Obama
  7. Gulf Coast
  8. Mitt Romney
  9. New Orleans
  10. Hillary Clinton
  11. Britney Spears
  12. Blackwater
  13. Ron Paul
Crazy lists, aren't they? There is some method to this madness. With the stories, I tried to get a reasonable sample of popular topics, some of which are serious and some of which are decidedly unserious. I have a lot of presidential candidates in there since I'll be especially interested to compare who's being covered vs. who people WANT to be covered. For instance, my hunch is that expecting that Ron Paul is a topic of interest much more for Digg readers than for media outlets. Ron Paul seems to have some kind of word of mouth campaign going on where libertarian fans of his call shows like C-Span and post on blogs all over the place, whereas the news seems to be largely ignoring him. I'm not a Paul support, except to the extent that I think he's clearly the least evil Republican in the race.

With the web sites, the idea is to have a variety of media sources. Some are considered serious news sites; some are "fluff" news (I picked USA Today specifically for that reason, and it's possible that CNN will tend to fall in that category as well); and several are explicitly right wing rags. To be fair, I really would like to have included left wing rags, but the only ones I can identify are blogs, which are not treated much as news sources. The news is all pulled off of news.google.com. I search for the topics of interests, then read the resulting stories more or less indiscriminately and identify which site each one comes from.

Based on this, I have a total of nearly 2000 "news" sources, ordered by the number of stories found in searches since I started collecting data. In the stories I've pulled so far, after about three days of serious searches on the 13 topics, the New York Times and the Washington Post (my main "serious news" sites) each account for 104 stories. But dailykos.com has shown up zero times, so I guess there's a master list that they're clearly not on. TPM Muckraker and TPM Cafe both show up, and those are both explicitly liberal sites, but there are only 8 stories from them. "The Nation": 9 stories. So, liberal sites = small sample size. No use.

By contrast, townhall.com, whose "about" page proudly announces that they were founded as a "conservative web community," accounts for 123 stories. Yes, you read that right: for the topics I picked, townhall is treated as "news" more often than either the New York Times or the Washington Post. So, bottom line, I get to pick on right wing news sources more than left wing news sources, simply because left wing news isn't "news."

Almost time for the Daily Show now, so I've managed to procrastinate this long. Go me!

If anyone would like to make further contributions, feel free to suggest other story topics that are in the news. Anna Nicole Smith and Harry Potter aren't actually generating very many headlines these days, so I need more unserious topics that the media uses as padding these days. Suggestions? And if you have more right-wing, left-wing, or "mainstream" news sources that I should be looking at, make some suggestions. I'll check my database and see if there are enough stories represented to get something useful out of them.

This is it. I'm officially in grad school hell.

Bless me father, for I have sinned. It has been two weeks since my last blog post.

So -- ha ha -- did I think that semesters like this one and this one were tough? Bugger that, this one takes the cake. The first draft of my 50-ish page Master's Report is supposed to be done in early October, so I've been focused on that for the week since my last class. Meanwhile, in my next class weekend I have one homework and two midterm exams.

I spend an entire weekend working non-stop on my thesis, then I got to enjoy going back to work fresh on Monday. My boss gave me Friday afternoon off, which was a nice gesture, except of course for the fact that I used it to do schoolwork.

I spent most of Saturday at a coffee shop on campus. Actually driving to campus was a stupid plan, because apparently there was this little football game going on that I wasn't thinking about. I was originally planning to go to the library and renew my TexShare card, but parking turned out to be impossible. So, coffee shop. Nice thing about UT is that it's so wired you can actually get wireless internet from everywhere, included some parking lots.

My work's really taking shape now. I've filled out the 14-page template for my report, which feels like I've accomplished some real work even though only two pages of actual double spaced text are written.

I meant to start working on the homework tonight; however, I've been so brain-fried that I mostly just ran the data collection program, stared at the news for a while, and did a whole lot of nothin' else. Blogging is just another form of procrastination, which I think I will continue to do until the Daily Show starts, at which point I will concede defeat for the evening. There's always tomorrow.

I was going to write more about my thesis in this post, but I'd rather keep this one strictly a post wherein I bitch about the trials of being a grad student, and cleanly separate the stuff about what kind of work I'm doing into a separate post. I think blogging will help me overcome writer's block in adding more detail to the report, so humor me, dear readers. See you in the next post.

Monday, September 10, 2007

I sure did fall for that one!

This weekend I got email from a "Paulraj JY" in India. It said:

Subject: Greetings from a New Friend fm India

Dear Sir and Mr.Russell,

Greetings from India. It was a surprise for me to read your blog and it
is full of surprises. Though you addressed yourself as an athiest, you are full of human virtues and you are a nice person to befriend.
Though I'm a Christian Missionary, I'm interested in your views,
thoughts and way of life.
When you consider yourself to be an athiest it may not be meaningful for me to ask you to pray for our service among children those who are left uncared. These Children are very special to us and we enjoy working with them. You, as I estimate, full of good values for human values and relationship, would be happy to hear more about our work among such children. We would be very happy to win your heart and have you as one of our wellwishers of our program.
You have a beautiful family. Please convey our greetings to them. Bye for now....
With special love and regards
Paulraj JY

Now, it's actually not all that uncommon that I get email from other countries. I am a regular contributor to two podcasts that have some small measure of notoriety, and I get people I never heard of commenting on my blog from time to time.

Nevertheless, there WAS that small voice that said to me, "Hey, this sounds a little bit like the stilted language in some of those Nigerian scam emails." But then I thought, "Nah, those guys mostly shotgun form emails when they're looking for new suckers. This guy was very specific about my blog and my atheism. Be a good atheist emissary and answer him."

So I wrote:

Dear Paul,

Part of the reason why I openly identify myself as an atheist is because theists rarely encounter people who are willing to say that they don't believe in God, and so they may have a lot of misperceptions. While I don't believe that atheists are better people than Christians, I do think that we are just as likely to care about humanity and have compassion.

In any case, thank you for your friendly email and have a good day.

Then he wrote:

Dear Friend Russell,
Thanks for your prompt response, which makes me happy. I just appreciate your openness. Please receive our special love and we really feel proud about your heart full of compassion for mankind....
Since you have a great concern for the betterment of mankind, I think it may not be improper to let you know that we are working with AIDS orphans and we've a formal inagural function our Grace Foster Home on the fifteenth of this month. Please remember us on this special day....
Kindly find an attached picture of our special children with we love to fellowship with and care for their better future.... I'm sure that you will appreciate our work... Bye for now...
lovingly yours friend from India
Paulraj JY

Well, that's a shame. So finally I wrote:

Dear Paul,

I have to give you credit for making the extra effort to personalize your scam email. However, since I now believe that you are a Nigerian con man attempting to perpetrate a 419 fraud, I would like to invite you to kindly go to hell. Of course, I don't believe in hell. But since you are in Nigeria, I reckon that's close enough.

Sincerely,
Russell Glasser

Presidential candidate or Buffy villain?

Yeah, we already know I'm a sucker for those side-by-side similarity pictures. The similarity in these pictures at AmericaBlog really is impressive.

If you're not a Buffy fan, you can look here to see who the creepy guys are.

Saturday, September 08, 2007

Paradox of omniscience and free will

Lots of theological debates center around the religious idea of free will. Some varieties of theists, i.e. Calvinists, don't believe in free will at all. Some atheists (like my friend Denis Loubet) don't believe in free will either, believing that the notion is incompatible with a completely materialistic universe.

Those are all interesting topics, but one issue I find equally interesting is whether "God," as Christians define him, can have free will. I think I'm borrowing this line of reasoning from an old Raymond Smullyan book, although I can't remember exactly where.

God is supposed to be omniscient. He knows everything about the past, present, and future. In fact, his knowledge is so complete that he must know every action that he himself will take in the future.

Now, suppose you yourself were granted the power of omniscience -- not omnipotence or any of the other useful attributes, but you know everything. Suppose it comes time to make a fairly mundane decision, like what you will eat for breakfast. You can have scrambled eggs or oatmeal. So you wonder, what am I in the mood for? Scrambled eggs, or oatmeal? But this is an easy decision: you are omniscient! Simply use your unlimited knowledge to peer a few minutes into the future, and see what it is that you will have for breakfast. And when you look at your future self, you know, as a matter of absolute certainty, whether you will be eating eggs or oatmeal.

But wait a minute. What if you are in a perverse frame of mind and wish to exercise your free will? So you say to yourself "Okay, here's what I'll do. I'll check the future, but I won't do what it says. If I see myself eating oatmeal, then I'll pick scrambled eggs. If I see myself eating eggs, it'll be oatmeal."

Now what does that mean for your powers? If your vision is guaranteed to be accurate, then you don't have the free will to change your decision. But if you can change your decision, then your vision was wrong, and you are no longer omniscient.

This is one reason why I conclude that no being can be both omniscient and free.

Wednesday, September 05, 2007

Surprise, we have bigots for neighbors

We have a couple of agave plants decorating the sidewalk on both sides of our driveway. They're sharp spiky plants, but that's not so unusual; there are a number of neighbors around the block who have a cactus or two.

We also have a couple of very prickly neighbors. They're an old retired couple living two doors down from us. We've been living at our current residence for nearly six years, and those folks haven't spoken a word to us in about five. Ginny says she smiles and waves at them and they scowl back at her.

When we first moved in, they were friendly and invited us to church, which we politely declined. We used to host a regular gaming night with our mostly atheist friends. They started asking "Why are there so many cars here on Mondays, and what are all those bumper stickers about?" So my wife told them. And that's about the time they stopped talking to us. I never felt like it was outright hostility, but she did. In any case, we haven't had much contact.

We have a couple of our own bumper stickers. She has a Darwin fish and a "Freedom from religion" sticker. Mine is more humorous; it says "Knowledge is Power. Power corrupts. Study hard, be evil."

This weekend one agave was cut. I don't mean carefully trimmed, I mean completely hacked up all across the front. Ginny has some pictures on her blog. We found pieces of spiny leaves in another neighbor's trash can on trash day, but we knew that they had been away on vacation so it wasn't them. Ginny was sure it had been this unpleasant couple. She was angry about it. Since I tend to have a bit of a more diplomatic approach to people than she does, she asked me to go over there and talk to them. I wasn't looking forward to it, but I wanted to hear their side of the story without prejudging them, hoping it was perhaps a big misunderstanding.

So I rang the bell and greeted them in as friendly a manner as possible, all smiles. I reintroduced myself to the woman and asked if she perhaps knew anything about the chopped plant. Despite giving me a fairly frosty reception, she invited me in and called her husband down. I had a seat on their couch, they took positions opposite me, and the husband had his arms folded the whole time and a very sullen scowl on his face.

Yes, he cut down the agave. I received a lecture on how dangerous it is to the neighborhood kids, and all sorts of gruesome scenarios about eyes being poked out. But what struck Ginny and me as weird later was when we realized that they hadn't cut any of the spines facing the sidewalk -- only the side on the street. (Again, see the picture.)

They then went on to lecture me about the general awful nature of our yard. Now, our yard may not be the most beautiful and well-kept in the neighborhood, but it is mowed regularly and there are quite a few houses that look worse than ours. I'm not a gardener myself, and I'm really busy with school, but I think Ginny does a reasonable job with it.

I took all this politely and said I understood their concerns, and is there anything else? Then we got into the bumper stickers. The wife said several times that they "make her sick" and she is very angry that we disrespect her religion. That she could never be friends with someone who doesn't "share her values." That she is firmly set in her beliefs and would never change them.

I said I don't want or expect her to change her beliefs, I have never asked her to. I don't proselytize to people who haven't approached me about the subject. And while I sympathized with her feelings, the very fact that she is willing to announce that the bumper stickers sicken her is unfortunately one of the chief reasons why we feel the need to express ourselves in this way. That Christians -- not you, I stated -- feel that it's acceptable to go door-to-door inviting people to their religion, and that we are expected to keep quiet about our opinions because they are supposedly offensive. We are sad that you view our bumper stickers that way, but we see it as a small but legitimate exercise of our free speech.

I then went on to state that while I understand the safety concerns regarding the spikes, it would have been polite if he had come over and brought them up with my wife. Then perhaps they could discuss the appearance and come up with an effective way of trimming it, or let her handle it. His wife restated the fact that they could never be friends with us. I said "I would never refuse to be friends with somebody just because of their beliefs. Only their attitude would make it difficult." Then I said I am not asking to be their friend; I'm only asking them to be friendly as neighbors and be a little more willing to open up lines of communication with us before taking it upon themselves to redecorate our property. I nicely asked him to come over some morning and discuss his concerns with my wife so that she can understand them as well. He agreed, but I'm pretty sure he didn't liked it.

As I mentioned before, I'm the more diplomatic one in the family. Just for good measure, this morning Ginny called the local police to talk about the incident, describing it as trespassing and vandalism. Before I left for work we were visited by a very cheerful and friendly cop, who got to hear all about the history and laughed at the notion that our yard would be an eyesore to anyone.

We didn't want to file charges. He offered to go over there and talk with them, even give a warning that they could be arrested if they were on our property again. We declined that too. I said I'm still hoping that the husband will come over and work things out amicably.

But I did happen to glance over at the neighbor's house while the cop car was in our driveway, and I saw the window blinds being pulled up. It was bright outside and I didn't get to see the expression on her face as she watched us talk to the policeman, clearly discussing our plant. But I have a pretty good imagination and I have to admit, it was kind of satisfying.

Saturday, September 01, 2007

Long live the laptop

I bought a new laptop computer at Fry's just over a month ago. My old one was enough to surf the web and take notes, but impossible to use for any programming work. (Or, let's be honest, games either. But I didn't focus on that. Honest.)

Anyway, I've been fairly happy with my new laptop, an HP Pavilion. But last week one of the buttons on the touchpad stopped working -- to be precise, it was the equivalent of the left mouse button. Now, this was not a huge issue, because you can click on something by tapping the touchpad itself, and also I bought a small wireless mouse that does the job better anyway. But it was annoying, especially since the button felt like it should be working just fine. I thought it might even be a software problem.

So I brought it back to Fry's and asked them take a look at it. They took a few minutes and then said "We'll get you a replacement." That's it. So I quickly wiped all my personal data (they say they wipe the hard drive immediately, but I figure you ought to be careful) and then they just walked up, pulled a replacement fresh out of the box, and slipped it in my carrying case. It took about 30 minutes of paperwork, but not too bad.

Now on the one hand, I appreciate Fry's replacement policy, and think that was extremely handy. On the other hand, this episode doesn't make me very confident in the quality of my purchase. I spent much of today reinstalling all my essential software (Eclipse, MySQL data manager, Firefox, Thunderbird, Google Earth... and yeah, World of Warcraft) and that was a huge pain in the butt. I hope that I don't just need to keep returning to Fry's for replacements every month.

On the other other hand, this incident does make me appreciate the new decentralized way of managing data that I've gotten used to. I didn't have to go home and back up my work, because every document I need is in a briefcase or source control program on my desktop. My contacts are online in Plaxo; my bookmarks are in Del.icio.us; my web feeds are on reader.google.com. All the work was to get the programs running, and mostly you can quickly download the latest versions of everything straight from the web without inserting any discs. That's awfully convenient.

Thursday, August 30, 2007

Beautiful sentiments about programming

Wrapped up in grad school as I am, it's easy to lose sight of the big picture, and why I got involved in this career path in the first place.

For my classes in Software Engineering and Management, I have to read The Mythical Man-Month by Frederick Brooks. I know the book by reputation; as it was first published in the 70's, I presume that the material is very old news to many people who share my interest in programming. Even so, this is new to me, so I wanted to share a passage from the book that I personally found very inspiring.

"The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.

Yet the program construct, unlike the poet's words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

Programming then is fun because it gratifies creative longings built deep within us and delights sensibilities we have in common with all men."

Oh yeah.

Monday, August 27, 2007

Speaking of schooling...

Ben started kindergarten today. Ginny and I have very different feelings about it. Ginny has had Ben staying at home with her for the last five years, and she's undergone some separation anxiety. She's worried about whether he'll adapt to the new environment, whether he'll behave, whether he'll eat well when he's not at home, and whether he'll be homesick.

I have different feelings about it: I'm really excited. Of course that's easy for me to say: I work all day, and I'm used to not having him constantly there anyway. So I mostly only see the good side of this new milestone in his life. I mentioned most of the reasons for my positive feelings in my earlier post about homeschooling. He'll be expanding his horizons, meeting lots of kids his own age, having a teacher with a different perspective on the world than ours. He'll spend time learning to read and draw and use numbers (which he's already very good at for his age). He'll have his time divided between two very different environments, and have raised expectations about how he spends the weekdays.

I met his teacher and some classmates last week. They all seem great. Mrs. Snyder struck me as a very cheerful and sweet young lady with a lot of teaching experience. We watched him go in this morning, and she took charge of the class right away.

I talked to him on the phone when Ginny picked him up. He sounded very tired but said he had had lots of fun. He couldn't be even a little bit specific about what kind of fun, so I had to pry it out of him with leading questions. :) But overall, I think it was a positive experience.

Thursday, August 23, 2007

A parable

Hapless Citizen: "DARN IT! Oooo, curse my AWFUL luck. This rotten dagnabbertiblabbit car of mine will not start. Again! I'm going to be late for work."
Masked Superhero: "Fear not, Citizen!"
Hapless: "Who are you?"
Hero: "I'm the Ayn Rand Crusader!"
Hapless: "Are you here to solve my problems?"
Ayn Rand Crusader: "No, even better! I'm here to motivate you to solve them yourself!"
Hapless: "Well that does sound useful, I guess."
ARC: "Now, what seems to be the trouble?"
Hapless: "Well, it's this stupid car of mine. It's got some years on it, it doesn't run very well, and I've had to spend a fair bit of money on a regular basis to keep it in running condition."
ARC: "Aha! This looks like a job for... the Ayn Rand Crusader! Citizen, your problems are as good as ended. Observe!"
[ARC whips out a comically large sledgehammer from the pockets of his colorful tights.]
Hapless: "You're going to solve my problems with a hammer? What are..."
[ARC brings down a mighty blow on the car, proceeding to pulverize it into a metal pancake.]
Hapless: "OH MY GOD, what the fuck??? You just totally smashed my car!"
ARC: "Yes, and now all your problems will be solved!"
Hapless: "...Oh. Really? Does that mean you're going to get me a new car?"
ARC: "Of course not! I don't do handouts! But fear not, you will have a new car soon enough."
Hapless: "I don't follow you."
ARC: "Well, now that your old car has been destroyed, I have created a market demand for a new car. Before you know it, the Amazing Free Market will surely be knocking on your door, begging to replace it with a much better car."
Hapless: "But... there aren't any decent car dealerships around here for miles. And it's going to cost me a lot more to get a new car than it did even to keep my old car in working order."
ARC: "Never fear! Thanks to the Incredible Free Market, new car dealerships will soon open up within walking distance! And not only that, they are sure to make you a car that is both excellent and affordable! And besides, even if that doesn't happen, you can easily make a new car for yourself that is just as good."
Hapless: "But I'm a software engineer. I don't know how to make a car."
ARC: "What are you, lazy? You said your old car was bad. Anyone can build a car that's better than a bad one! Get off your butt and learn how to do it properly."
Hapless: "Let me get this straight. You 'helped' me by destroying my old car, and now you're just going to leave me with nothing."
ARC: "LIAR!!! Have you not heard a word I've just said? Why do you misrepresent my position so egregiously? As I have been trying to explain to you, the Magnificent Free Market will replace your car for you."
Hapless: "But right now I don't even have the car I did before. How am I supposed to get to work?"
ARC: "Perhaps some enterprising private charity will come along and give you a lift."
Hapless: "Thanks for nothing."
ARC: "No need to thank me, citizen, I'm just doing my job. Thank the Incomparable Free Market for the generous bounties that will soon be yours. Ayn Rand Crusader, AWAY!!!!!"


...and as for the rest of Johngalt666's "points", I'll have to get back to those a bit later.