A personal account

Carlos Justiniano
Founder and Co-developer

Wednesday, January 28th - We arrive in Copenhagen

We encountered a number of challenges during our ChessBrain World Record attempt. Our troubles began when we arrived in Copenhagen on Wednesday January 28th, only to discover that our luggage was lost! Being without a fresh wardrobe wasn't nearly as bad as not having our primary development machines.

Not all of our equipment happened to be among the missing luggage. I broke my own personal record by carrying a total of five laptop computers in carry-on luggage. Unfortunately, none of the five machines were sufficient to do the necessary setup work prior to the event. The machines were primarily used as test equipment. Our wireless bridge, switch, and US-to-European power adapters and cables were with the missing luggage.

The impact of not having our equipment meant that we would already lose one of three precious days of setup and test time.

Cedric Griss of the Distributed Computing Foundation was my travel companion and more importantly the event coordinator on the ChessBrain side. I've known Cedric for well over a decade and his presence during the trip was reassuring. Cedric kept in constant contact with Air-France to track and coordinate the return of our equipment.

By evening we had our luggage, some cables and the network switches we needed. However, our primary development machines were still missing in action.

Later that evening we realized that the hotel room lacked sufficient electrical outlets to power our equipment. We spoke with the hotel staff to see if they had additional power strips. No such luck. We were forced to venture into the downtown area in hopes of finding a computer store open late in the evening. Again, no such luck. We returned to the hotel and started wandering around.

The hotel was under reconstruction, so we thought there might be a possibility of finding extra cables we might "borrow". Our scavenger hunt paid off when we found a box with extension cables. The items appeared to be have been pulled from rooms under reconstruction. We took apart the items in order to check the insides and remove the sharp nails that protruded from each box.

All we had left to do was test each cable. Cedric advised me to test each item outside my hotel room in the opposite side of the building. This was just a precaution in case we blew a power circuit. Fortunately, each adapter worked perfectly. We found a few other items such as a desk and a knife. We procured enough to turn a hotel room into a small office. By 2am, we managed to setup a few laptops and try the hotel's wireless Internet access.

By morning our development machines had been located and Cedric had arranged for them to be sent by cab from the airport directly to our hotel.

Thursday, January 29th - ChessBrain team meets for the first time!

Just a day before the actual event another group of our ChessBrain team arrived in Denmark. We had a meeting scheduled with the Danish Unix Users Group (DKUUG) and conference programme chair Kenneth Geisshirt. Later that evening we all met for the very first time. Despite having worked with Colin Frayn (ChessBrain's co-author) for well over a year, I had never spoken with him on the phone, nor had I ever seen his photograph. I recognized Peter Wilson and Kenneth Geisshirt from their photos. Peter Wilson was the former chairman of The World Chess Federation's (FIDE) Internet and Computer Committee and now a key member of our ChessBrain team.

After a discussion in the hotel's lobby area, we went up to my hotel room where I prepared a demonstration of ChessBrain's software.

From top left to right
Kenneth Geisshirt, Mary Mcdermott, Peter Wilson
Carlos Justiniano and Colin Frayn

Mary Mcdermott arrived with Peter Wilson in support of our efforts. She is an International Chess Arbitrator and helped ensure that the chess side of the event was handled in accordance to official procedures.

Cedric and his business contact, Martin Gorm Pedersen, were also present. Martin has an impressive computing background and was a standby validator for the event. After our meeting was over, Martin did a food run and brought back dinner, while Colin and I examined ChessBrain.

Before that time, Colin had never seen ChessBrain actually running. During our trial runs, that evening, ChessBrain reached a peak of 1100 machines. By early morning, the count had reached 1350. Our conservative goal was to see 1000 machines active during the event. It had now become clear that we could see far more machines active than we had ever imagined.

Friday, January 30th - Show time!

The team decided to meet for breakfast early in the morning. By noon we were packing equipment into a van and leaving for the Symbion Science Park. Although the event wasn't scheduled to start until six in the evening, we wanted to have as much time as possible to work out any potential issues.

This turned out to be a good plan as we noticed delays in the wireless network not long after setting up our equipment. Cedric contacted IT-Kompagniet A/S the Internet providers for Symbion Science Park. Within what seemed like minutes, there was a person onsite to examine the problem.

By this point in time, we were already falling behind in our preparations. DKUUG's Benny Kjærgaard stepped in to assist, while Colin and I met with the event validators to explain how ChessBrain works.

We discussed the requirements set forth by Guinness and assured the validators that they would have access to everything they needed to validate the event. During the event, the validators had access to program source code, database tables and could carefully monitor the ChessBrain team.

Both validators are professors within universities in Denmark and both specialize in Distributed Computing.

With network issues resolved we proceeded to start the event. Event validator Professor Jens Clausen (of the Technical University of Denmark) stood directly behind me as we reset database statistics and restarted the ChessBrain SuperNode server.

Grandmaster Peter Nielsen played his first move and we sent it to ChessBrain. Within seconds we realized that we had a very serious problem on our hands. ChessBrain was taking far too long to make its first move! Later we joked that ChessBrain was trying was trying to solve the game of Chess before announcing a win in some number of moves. Naturally, this isn't possible...

The stress level in the room was absolutely tangible. We scrambled to make small modifications by adjusting values in a configuration file. After several attempts and restarts the problem became clear. ChessBrain was being attacked by its very own PeerNode clients! The worldwide interest in the event was so great that thousands of machines were all trying to communicate with ChessBrain. Technically this is very similar to a network distributed denial of service (DDoS) attack.

Gavin Roy was present at EHPG, the co-location site in the United States that houses the ChessBrain servers. While we were making adjustments in Denmark, Gavin treated the situation as a denial of service attack and logged onto the servers to work from the inside.

I asked Gavin what he actually did via an IM session. He replied, "basically I tweaked down syn backlogs so that they would timeout more quickly so that people would have more of a chance to get into the server as all the clients basically acted as a syn ddos attack they would syn, but the server didn't have any open connections to give an ack."

Gavin went on to say:

First I doubled the /proc/sys/net/ipv4/tcp_max_syn_backlog from 1024 to 2048. giving more clients the ability to connect and wait in line as it were since you were, at the time that I first logged in hitting the 1024 syn backlog then I reduced /proc/sys/net/ipv4/tcp_synack_retries to 1, that way it would only retry once, in case it was indeed a syn attack.

Gavin tried a few other things while we modified the SuperNode server to change the reconnection behavior of the PeerNode clients. The combined effort seemed to do the trick as we were able to get ChessBrain to make its first move!

The ability to make real-time changes speaks volumes to the power and flexibility of *nux (Unix/Unix like) systems.

Gavin has made the switch logs available for viewing:

• Traffic Analysis for 38 -- switch.ehpg.net
• Packets Per Second for 38 -- switch.ehpg.net

Once the game was underway we watch nervously, uncertain whether ChessBrain would stop playing again because of the overwhelming number of machines trying to participate.

The ChessBrain team was in three separate locations. Gavin Roy was in Northern California in the United States. In Denmark, at the event, Colin and I were in one room, while Peter Wilson who sat directly across from GM Peter Nielsen was in another room. Because we were all in different locations we needed a way to communicate with one another during the game. We used a secure communication product called ICS. I was one of the key developers of ICS before it was known as ICS and sold to Y3K Secure Communications Inc. Y3K became a key sponsor supporting the Distributed Chess Foundation, and so we were pleased to use their product during the event. Personally I felt comfortable using a product I could trust!

Peter Wilson (on right) is shown communicating with the ChessBrain team in the next room. The point here was to ensure that GM Nielsen was free from unnecessary distractions as he focused on the actual game.

During the game the communication between the ChessBrain team members was projected onto a wall so that visitors could examine the team's dialog in real-time.

Event programme chair, Kenneth Geisshirt is shown above examining Y3K's ICS screen.

ChessBrain's Peter Wilson (left) shaking hands with GM Peter Nielsen after the game.

After the game we considered various possibilities. GM Nielsen explained his thoughts and rationalized his play.

Each ChessBrain member proceeded to sign the official game score sheet which is being sent to the Guinness World Record office in London. Event validator, professor Jens Clausen is shown below adding his signature to the official score sheet.

In all honesty our first world record attempt was filled with many technical issues. Because of the overwhelming worldwide interest, a large number of our members were unable to contribute during the event.

During the next few months we'll work hard to improve ChessBrain in preparation of toping our current record. I'm particularly honored by the hard work and dedication of the entire ChessBrain team. We did it!