There are many independent reports coming in that as of last night, YouTube is accessible again in China via the big providers China Netcom and China Telecom. I can confirm Beijing China Netcom can access it.

It may be that blocks are unwinding after the CPC 17th National Congress, or it could be a hiccup. Thomas Crampton, who has been unable to do video blogs from Beijing, will be happy to hear the good news.

Well it appears the Olympic Ticketing Phase 2 debacle is complete. This is one case where China Daily doesn’t (or can’t) plaster over the ugly details:

An official from the BOCOG Ticketing Center said Tuesday night that the Games organizers had decided to temporarily halt domestic ticket sales to improve the technical plan and will announce new ticketing information on November 5.

“Because of the overwhelming volume of page visits, the technical system was unable to perform the tasks well enough, and many applicants were unable to successfully submit their applications,” the official said, adding: “We sincerely apologize to the public.”

The first-come-first-serve scheme had 1.85 million tickets on sale through the booking website, a hotline and designated branches of Bank of China.

But demand was much higher than organizers anticipated: According to the Beijing Olympic Ticketing Center, the official ticketing website (www.tickets.beijing2008.cn) saw 8 million hits in the first hour starting 9 am, while the ticketing hotline received 3.8 million calls.

Only some 9,000 tickets were sold in two hours; and the ticketing center confirmed that successful orders will be valid.

Some other interesting stats and reading:

Another day, another ticketing server crash. Following the debacle last week with hacked servers selling Colorado Rockies World Series tickets, the servers offering tickets to the Beijing Olympics crash earlier today, the first day tickets became available. — TicketNews.com [link]

“Can’t get on the Web site to buy tix,” she wrote. “And I didn’t bring my passport today to go to the bank.” Without her passport as ID, she couldn’t buy the tickets in person at Bank of China. Andrew Lih, a blogger in Beijing, preserved a picture of the Beijing 2008 error screen for posterity.” — BusinessWeek [link]

More than 200,000 Internet ticket applications were submitted each minute during the first hour of sales, jamming the site, the Beijing Organising Committee for the Olympic Games (BOCOG) said in a statement. The committee’s website received eight million hits in the first hour alone, while its phone bank was swamped by more than two million calls, the statement said. “The website and phone lines are jammed. Even I can’t get through by phone to the ticketing centre,” Wang Yue, a BOCOG spokeswoman, told AFP. — AFP [link]

The ticket Web site was viewed 8 million times in the first hour after it opened at 9 a.m. Beijing time and a ticketing telephone line received 2 million calls, according to a statement on the official Beijing Olympics Web site. — Bloomberg [link]

This is likely the most demanding OLTP application in China’s history. Trying to transact 1.85 million items first-come first-served across three different portals (Web, phone and physical POS), requiring the use of a backend payment system is demanding for any organization, much less the China BOCOG that’s doing this as a one-off project. (And without a real electronic clearinghouse for banking in China, there are even fewer experienced outfits domestically that can do this.)

Supposedly TicketMaster and a local partner were responsible for the system, but it seems they made a terrible miscalculation in having everything jacked into the same public Web-based system.

For those living in China, this can’t be a real surprise. Developing Asian countries suffer from the “First world hardware, third world software,” problem.

So you will have gleaming towers, beautiful stadiums and luscious greens that will be done on time for the Olympics. But it will take far more effort to stop public spitting, getting people to queue in a civilized manner and deliver customer service at international standards.

One World, One Dream, One massively frooked OLTP system…

The screen presented to folks today trying to order Olympic tickets.

Phase 2 of Olympic Ticket sales in Beijing started today, and it’s not exactly going smoothly. I didn’t expect much from a first-come-first-serve national offering with 130 million+ Internet users hammering one web site, so I can’t say I was surprised. Here’s a tally of how it went.

There are three methods of trying to get tickets, from the BOCOG Web site:

The tickets will be sold through the following three channels: the Beijing Olympic Official Ticketing website; 1,000 BOCOG-designated Bank of China (BOC) branches; and the Beijing Olympic Ticketing Call Center, +8610 952008.

Web site: It was completely jammed up, and took many attempts to even log in. A friend said that after a while he was able to see the event page, and he thinks he might have got an order in, but cannot tell. I never got passed logging in and browsing events, and even that took 30 minutes. Lots of timeouts and resets.

Phone number: One friend called and said it registered a wrong number. I tried at 11 am and it gave the standard busy message in China, “Sorry the subscriber you called is busy now. Please try again later.” So much for any kind of modern call queueing.

Bank of China (BOC) branch sales: This was our method of last resort — show up in person in hopes they have the most reliable way of getting tickets. Well, around 10:45am at the Bank of China branch in FullLink Plaza in Chaoyang, they said they were only able to sell tickets to one person the entire morning. Why? Because their point of purchase terminal is exactly what the rest of us use — the Web site. So even though they had a line of customers in the morning to buy tickets, and there were still folks strolling in with cash and Visa cards to try and buy, they only sold 10 tickets the entire time. They had the same Web site woes as everyone else. They told us the best thing to do is just keep hammering the Web site to get in.

So there’s Phase 2 in a nutshell for you. Looks like the best bet is a ticket scalper.

There is an interesting column in USA Today comparing the Wikipedia-Britannica debate to what is happening with Christianity in the US.

Do-it-yourself Christianity

Independent congregations are slowly chipping away at the ‘trusted brands’ as the Christian faith becomes more like Wikipedia and less like Encyclopedia Britannica.

I’ve often thought that Wikipedia contains a dynamic similar to evagelical congregations that build on authority by consensus. And I’m glad I’m not the only one that sees it that way.

In nondenominational churches, there are certain people and activities that get by without full critical scrunity because community norms of “Love thy brother” make anyone even slightly skeptical seem like mean spirited nonbelievers. The result is that dissenters leave, and those who stay join a spiral of silence and submit themselves to the power of a forceful minority.

In Wikipedia the equivalent is “Assume good faith.” There is no faster way to kill (or divert) a debate in Wikipedia-space than by declaring “AGF!” Anyone accused of not assuming good faith winds up looking like a nonbeliever and a heretic. In that sense, yes, Wikipedia can exhibit cult-like tendencies.

That’s why every few months I browse through one of the more fascinating repositories in Wikipedia — Missing Wikipedians. It’s like leafing through an old church directory. “Oh yes, I remember Thelma, the one who played the organ for the choir,” or “Wow, that RickK was one badass vandal fighter.” On certain visits, I imagine the tribute from the Academy Awards where they honor actors who have passed away that year with a dignified lament.

It brings on some nostalgia and contemplation when seeing the edit history of that page. What if Wikipedia could have retained some of those veterans? What could the community have done different? Will we ever get anyone like that again? Are we better or worse for having that person leave?

After a two year moratorium, it appears that anonymous (or more precisely, unregistered) users will be able to create new articles/pages in Wikipedia again.

The “clamp” was lowered in December 2005 after the Seigenthaler incident proved embarrassing to the Wikipedia community and started the march towards quality over quantity.

The move was announced boldly on the WikiEN-L mailing list by Greg Maxwell, one of the more resepected developers in the community and chief research coordinator for the community.

In December 2005 during the John Seigenthaler biography controversy (http://en.wikipedia.org/wiki/John_Seigenthaler_Sr._Wikipedia_biography_controversy) it was decided to require that users create an account and log in before starting a new article. The ability of people to make changes without logging in remained unchanged. […]

In the time since late 2005 the English Wikipedia community has grown substantially. The nearly exponential growth rate in articles we previously experienced has stopped. Even if disabling anon page creation was beneficial then, there is no current evidence suggesting that the change continues to be beneficial. As such, barring complications, anonymous page creation will be re-enabled on English Wikipedia on Friday November 9th.

After a one month period, on December 9th, we will re-evaluate this decision using previously established methods (average article lifespan, rate of deletion, manual quality classification, random samplings of newly created articles, and most importantly, community discussion). If there is evidence of harm, anonymous page creation will be disabled to collect more data and provide time for discussion. If there is no significant evidence of harm, the issue will be evaluated again after six months. Further milestones and actions may be proposed at that time.

It seems like a sensible trial given that many folks, including myself, have noted that the article growth has slowed substantially.

The announcement presented as fait accompli bothered some folks. But this is a classic example of Wikipedia’s community dynamics — being bold, with a person’s established social capital, rather than any strict hierarchical mandate, providing the authority for making things happen. It will be an interesting experiment.

Congratulations to the Chinese Wikipedia, which just hit 150,000 articles, despite being blocked in China.

The latest stats showed that Hong Kong and Taiwan contributors make up over 50% of the contributors. Dedicated PRC users who can get to zh.wikipedia.org by proxy and overseas Chinese make up the rest. A quick spot check also shows lots of the activity in zh: relates to pop culture and current events, such as Harry Potter (哈利波特－死神的聖物), Rain Man (雨人), Heart of Greed (溏心风暴), GiGi Lai (黎姿). This does not seem so different than English Wikipedia, and may well be what gets folks introduced to Wikipedia first.

Welcome Comcast USA users to the club of Internet blocking. You now share a bond with millions of Internet users in China! It may seem annoying, but with the right tools and some perseverance, you too can keep downloading without any hiccups.

The story

It seems the US Internet service provider has been using Great Firewall-style tactics to prevent customers from running P2P protocols like BitTorrent. Some sleuthing by the EFF found that TCP reset packets (RST) are sent to kill connections related to P2P file transfers by Comcast customers. This clandestine connection sniping is pretty hard to diagnose without geeky tools like Wireshark or ethereal, but the shutdown technique is used by more and more ISPs. It’s what the Great Firewall here in China depends on for blocks triggered by keywords.

This revelation comes at a particularly bad time for ISPs in the US, when the network neutrality debate had died down. But this will re-energize the Internet purists, as it directly hurts the credibility of ISPs who say the US does not need regulation of “neutrality.” If Comcast had given fair notice to customers via service agreements about proper and improper use of their connections, that would be one thing. But users had their IP connections shut down mysteriously for unstated reasons. That’s something that usually happens in other places. Like China.

There is a solution

While there is widespread piracy over P2P networks, there are absolutely legitimate uses for them. Comcast seems to have classified any BT P2P file transfers as something that should be shuttered for copyright infringement. That would be a bad assumption.

The other day I downloaded NeoOffice (open source) for the Mac at 140 Mbytes using BitTorrent because it’s much faster than FTP. I was able to get 120 kilobytes/second on P2P versus 15 kilobytes/second via straight download. Many folks download Linux distributions and operating system patches via BT for exactly this reason.

Is there a solution for customers? Well EFF is considering a legal challenge, as this seems ripe for a class action lawsuit. In the meantime, there are ways to circumvent RST-based tactics of firms like Comcast.

Here, China Netcom also frowns on P2P by slamming shut transfers and tracker communication. A combination of techniques, like using BT clients supporting encrypted connections, can solve the problem. The following is what works for me, and it should work for most nearly anyone that has to deal with a firewall/fitlering system with BT.

BACKGROUND: There are two different “channels” that BitTorrent uses — tracker communication and peer communication. Tracker communication is basically what the BT client needs to connect to a tracker server, which has the particulars of the transfer: what file is being transferred, which peers have it, and the progress of the client. It’s basically the coordination center for the entire session and is the only real vulnerable hub of a P2P system, becoming a single point of failure/blocking. The other part is peer communication. This is what takes place between your computer and the multitude of other computers on the Internet. This makes up the big bulk of traffic on P2P, when your computer is perhaps chatting with 100+ other clients to transfer little chunks of the file you want.

So the tactic of ISPs is to block either or both of these types of communication. In days of old, when BitTorrent was new (or ISPs didn’t care or notice) all peer communication happened on port 6881 and tracker communication happened on 6969. For a long time this worked fine. But since these port numbers are well known, to block BT the ISP could simply block all packets to those ports. Game over for the client.
So people started changing port numbers to high numbered random values (37412 for example) used for peer communication and to less known port number for tracker communication. That worked for a while. But in this escalating game of cat and mouse, ISPs started putting in systems to actually inspect packets across all ports to see if they had telltale BitTorrent “headers,” and shutting down those connections. Thus high numbered, randomly selected ports were not good enough. The power swung back to the ISPs.

SOLUTION. What’s fascinating is the furious software arms race the P2P open source community engaged in to solve this problem. Programmers have upped the ante by using encryption and de-centralizing the tracker function to the point where BT is now nearly unblockable. But it’s not for the average user, since you do need some special configuration with the right clients.

The basic solution is to use encrypted peer communication, and a proxy server for the tracker communication.

Newer clients like uTorrent (Windows) and Azureus (nearly every platform), now support encrypting all traffic between peers using RC4 encryption, and setting an arbitrary port number. The only thing ISPs see then are IP packets with encrypted gibberish going from one random port number to another computer’s random port number. They cannot tell whether it is VoIP traffic, a file transfer, VPN, MMORPG data, or anything else. It is completely opaque to them, and filtering cannot work on the packets. Because the two peers do a handshake to establish a unique session key that no one else knows, the ISP is out of luck.

The RC4 encryption used by clients, while not the state of the art, is hard enough to crack that it isn’t practical to inspect those packets without major horsepower (like supercomputer horsepower). Comcast, China Netcom, or anyone else as intermediary ISP have no real options but to pass it along as an ordinary IP packet.

Tracker communication needs a different treatment. It’s much easier for ISPs to block this, because there are only a few dozen popular BitTorrent trackers in the world. By simply blocking all traffic to them, or watching each packet, they can just shut down those connections. The simplest way to circumvent this is to use a proxy. Azureus supports the use of a SOCKS proxy server. As a China Internet user, I always have an SSH tunnel open and in use for my proxy communication. It’s just a normal part of my day to get to blocked sites like BBC, Blogger, YouTube, etc.

However, SSH is not something everyone has. I happen to have it as part of a hosting plan, but it’s fairly easy to get one as part of a $5.95/month plan like on BlueHost. There are also sites that give free SSH accounts, like silenceisdefeat.org. In the Azureus options, you can simply instruct the client to use a proxy for tracker communication. That way, the ISP you are using cannot even tell any P2P is happening since your proxy server is doing all the tracker communication on your behalf, and it’s encrypted in the SSH tunnel. (There is a full tutorial about this technique for Windows here and here).

With this whole thing setup — high numbered random ports, encrypted peer communication and proxy tracker communication — your local ISP is none the wiser to what’s going on, even when employing basic surveillance techniques like packet inspection. I’ve been able to max-out my connection speed using this arrangement for torrents that have lots of peers. There are some small caveats — not all clients support RC4 encryption, so not all the seeds/leechers listed will be available to you. Also, if your SSH connection breaks off for some reason, it will likely stall your transfer. (I use a command line tool like “autossh” to keep a persistent SSH connection.)

As I warned though, this is not for the average person. The most exotic part of the solution is an SSH tunnel, which only real hard-core Internet users would have.

The final tally

What this arms race means in the long run is more interesting. If the US government will not regulate the maintenance of “neutrality” into the operation of ISPs, users can demand it in part by encrypting everything and preventing operators from discriminating against (or currying favor towards) certain types of traffic.

This has always been the problem with the perennial hope of ISP-supported Quality of Service (QoS) because it depends on the operator being a relatively fair or accommodating intermediary. This assumes there is a telco/ISP being purely a “common carrier” whose job is to expeditiously relay traffic efficiently and for the benefit of the customer. The problem is one side of the connection is the ISP’s customer, the other end is usually not. Also, more and more ISPs have a vested interested in pushing their own VoD, VoIP or walled garden product over the exact same lines that Google, Facebook and Joost are using for their multibillion dollar ambitions.

It is natural, though problematic, for a “common carrier” to mix its own product’s fortunes into its relaying policy. And that’s where the heart of the debate is.

When you need to know about gaming in China, you turn to Frank Yu. The former Microsoftie is one of the most informed people on electronic gaming in Asia, and was recently interviewed by the China Business Network. Well worth the listen.

Frank was the one who told me about Zhengtu Network, an MMORPG peculiar to Asian and, specifically, Chinese tastes. Its melee-style battles and concept of “fair play” is rather different than Western norms of game design. From last week, there was a video blog by Thomas Crampton where I talked a bit about Zhengtu, and Kaiser Kuo talks about P2P video.

Zhengtu’s free-to-play and virtual merchandising is consistent with other Asian online revenue models (like MapleStory and Cyworld). My not-even-teenage nieces in Singapore were completely addicted to MapleStory’s cartoonish game play, and they wind up stumping me each visit when they demo the latest hot game with the youth market I’ve never even heard of. I increasingly turn to them for trendspotting as I’ve gone from “Cool Trendy Uncle” into “You’re So Yesterday Uncle.” They taught me about Habbo.com, even though they’re not allowed to have an account on it. Mom’s orders.

At least I can still crush them to smithereens in Wii Tennis, which going by news reports, absolutely proves I’m an old fogey.

I usually talk to at least a reporter a week on background concerning Wikipedia’s community and associated shenanigans. Writing a book on the subject will attract that attention.

This week, however, Telegraph UK’s article about inclusionism/deletionism put the Pownce issue up front and center again. I talked to reporter Ian Douglas about a lot of concerns, and he pretty much came out with the right set of facts. But this was perhaps worded too strongly:

Submission of new articles is slowing to a trickle where in previous years it was flood, and the discussion pages are increasingly filled with arguments and cryptic references to policy documents.

Creation of new articles is hardly a trickle, it’s just down from previous highs, so that we are seeing perhaps the latter part of the S-curve of growth. Though we got into the lack of dumps, missing statistics and other tools that might help to diagnose this phenomenon, but that did not make it into the article. My mention is but a single data point in the debate, so it would have been good to have better stats on this.

Andrew Lih was a well-known deletionist until recently when he became embroiled in the row over the entry for Pownce, a messaging and bookmarking website from Kevin Rose, the founder of the popular site Digg.com. The entry for Pownce, which had been written up in Business Week, was deleted as advertising until Lih resurrected it. He wrote about the row on his blog and has become a de facto spokesman for the inclusionists, and says he feels like an old hand.

“The old timers remember the early days when we used to say ‘ignore all rules’ and ‘assume good faith’, but people tend not to emphasise that now. The third or fourth generation of Wikipedians has only heard Jimmy Wales talk about the problems.

“So now, mixed in with the euphoria and positive energy it’s a lot of cutting, fighting, referencing, cutting back while leaving the good stuff in. New priorities are arriving. Newer folks feel like they’re wielding a machete, not planting new trees.

“A lot of the veterans see established articles nominated for deletion. They try not to be arrogant, try to be inclusive, but it’s tedious after six, seven or eight times.”