ChoJin's Quarter

Ruby and Web Scraping/Automation

ChoJin — Sun, 06 Sep 2009 22:08:08 +0000

Sometimes you encounter a web task that is soooo boring you'd wish you had robot doing it for you. I encountered a few of those during this summer so I figured out it was time to acquire some Web Scraping and Automation Ninja skillz.

Turned out it's fairly easy, thanks to very useful libraries made for this very same reason. I'm therefore going to give you a quick overview of its simplicity, show you how much you can achieve with only a few lines of code and I'll conclude this blog post with a real-world script example proving med' school students should learn programming

Ruby and Mechanize to the rescue

Nowadays, when it comes to scripts I would suggest two languages: Ruby or Python. They are so powerful and so convenient, it's very hard to argue for anything else unless you have strong constraints.

Ruby or Python?
It's mostly a matter of taste, and my heart leans toward Ruby for various reasons (that I won't develop here to prevent from an epic troll), hence the use of Ruby here. In the Ruby land, I found two libraries to be quite valuable for web scraping, automation and pentesting : Mechanize and WWMD.

In this post, I'm going to illustrate Mechanize, WWMD will be used in a later post. With Mechanize it's fairly easy to connect to an HTTP or HTTPS web page, retrieve its content, parse it, fill some forms, click a button or all these kind of things.

A basic introduction to Mechanize

For example, let's say you have a HTTPS web page, with a form named "connexion", a text field for the user and pass information and a button to login. You could use the following code to authenticate:

PLAIN TEXT

agent = WWW::Mechanize.new { |agent|
agent.user_agent_alias = 'Mac FireFox'
}
page = agent.get('https://www.example.com/')
connform = page.form('connexion')
connform.user = "user1"
connform.pass = "pass1"
page = agent.submit(connform, connform.buttons.first)

Easy enough, isn't it?

Mechanize has a lot of syntaxic sugar, for example if the form has multiple buttons with assigned names you could do:

PLAIN TEXT

connform.button_with(:name => 'login')

the "_with" sugar applies to pretty much anything the page object contains, such as links, forms, buttons etc.

With Mechanize it's as easy to check a radiobox or select an option from a selection list:

PLAIN TEXT

form.radiobuttons_with(:name => 'box').first.check
form.field_with(:name => 'list').options[0].select

Or 'clicking' on a link:

PLAIN TEXT

agent.click page.link_with(:text => 'News')

Mechanize also supports standard xpath scraping, so let's say you want to find a bold text in a paragraph inside a div with an id='foo':

PLAIN TEXT

value = page.search("//div[@id='foo']/p/b").text

You got the idea, everything is fairly straight forward and I strongly urge you to read Mechanize's documentation and a general xpath guideline.

A real world example

In a later post, I'll show you how to apply these principle to implement a game bot/trainer, but for this small introduction, let's take a simple example.

At my school, the first year students have to choose a schedule (among 8 of them) at the beginning of the year. Some of them are more popular than others, and only a few places are released for each schedule at random hours every day. Because of this, you have at any given time approx. 100 students frenetically pressing the 'F5' key to try to catch a spot in the popular schedules (yeah, first year of med' school students are under high pressure and become robots ). When places are added, they are all gone in approx. 2s or less, so you need either of the following features to secure a spot:

Being Lucky
Being patient
Being a robot in front of your computer 24/7
Having Ninja speed and skills
All of the above

Or... You can yet prove again than brain 0wn$ muscles and learn Ruby programming

The following script is fairly simple and should be a good example of multiple features of Mechanize. It basically authenticates with a chosen user credential to the website, then try to select a given schedule. If it fails, it sleeps 1s (remember places are all gone in less than 2s for the popular ones...), and try again.

When a schedule is available, the script still has to choose a sub-group. So it enters the Ninja auto-fire mode, and quickly tries all groups available from the select list in the form. As soon as it succeeds, it validates the choice and stop. You also have a few sanity checks in the script to prevent from wrongly selecting a schedule/group in case of failing attempt from the server to fool automation...

I introduced a few bugs on purpose in this script to prevent script kiddies from my school to re-use it as is, insofar as it is given here solely to illustrate the subject matter. I'm making a wild guess that if one of them is smart enough to figure out how to install the ruby environment and dependencies, fix the bugs I introduced and run it successfully, he would probably be skilled enough to write it himself. So no real harm is done

Here is the script (with anti-script kiddies bugs), I hope it will be self-documented enough to understand it. Drop me a comment otherwise.

PLAIN TEXT

#! /usr/bin/env ruby
require 'rubygems'
require 'pp'
require 'mechanize'
def print_curchoice(agent, link)
page = agent.click link
curchoice = page.search("//div[@id='page']/p/b").text
puts "Current choice: #{curchoice}"
end
def select_sp(agent, link, sp)
page = agent.click link
form = page.forms.first
form.section = "SP#{sp}"
puts "Selecting SP#{sp}..."
page = agent.submit(form, form.buttons.first)
error = page.search("//div[@id='page']/p[@class='erreur']").first
return false unless error
form = page.forms.first
# Iterate through the group until we can find a successful one
for option in form.field_with(:name => 'group').options do
option.select
grp = form.groupe
puts "Selecting group #{grp}..."
result = agent.submit(form, form.buttons.first)
section = result.search("//div[@id='egap']/p/b").text
# Sanity check, just in case...
unless section && section == "SP#{sp}#{grp}"
puts "Ooops..."
next
end
puts "Validation..."
result = agent.submit(result.forms.first, result.forms.first.button_with('bdeclare'))
msg = result.search("//div[@id='page']/p[@class='msg']").text
unless msg && msg == "Votre inscription au groupe SP#{sp}#{grp} a été enregistrée."
puts "Ooops..."
next
end
puts "Epic Win! "
return true
end
return false
end
if ARGV.size != 3
puts "========================="
puts "SPSelect by ChoJin © 2009"
puts "========================="
puts "./spselect.rb "
puts "example: to register to the section SP1 with username 'user1' and password 'pass1':"
puts "./spselect.rb user1 pass1 1"
exit 1
end
agent = WWW::Mechanize.new { |agent|
agent.user_agent_alias = 'Mac FireFox'
}
page = agent.get('https://www.biomedicale.univ-paris5.fr/scola/scola/')
connform = page.form('connexion')
connform.user = ARGV[0]
connform.pass = ARGV[1]
page = agent.submit(connform, connform.buttons.first)
link_choice = page.link_with(:text => 'Choix semestre 1')
link_cur_choice = page.link_with(:text => 'Mon groupe semestre 1')
print_curchoice(agent, link_cur_choice)
while true
ret = select_sp(agent, link_choice, 8)
break if ret == true
sleep(1)
end

That's all folks!

Kinesis’s Customer Service

ChoJin — Sun, 06 Sep 2009 14:55:21 +0000

Couple of years ago I bought a Contoured Kinesis Keyboard due to a carpal tunnel syndrome (or at least my thumb was starting to feel numb on a regular basis).

I can't tell you, in spite of its very strange look, how wonderful this keyboard is. Not only it solved my pain/sensation issues, but it's really a very comfortable keyboard. Most people would expect it very hard to get used to it (or switch to normal keyboard afterward), but it couldn't be farther from the truth. It only took about a week to get used to it, and after 2 weeks I was even typing faster than before. After a month or so, I even decided to switch to dvorak mapping to further deepen the ergonomic experience. Since then (so roughly 6 years now), I've been writing with a dvorak mapping on this very same keyboard (maybe I should replace the pads btw...). If you want a great keyboard, I strongly recommend this one.

But 2 years ago or so, two keys started to behave strangely. Not working or getting stuck (the alt/option and command keys on my left thumb). So I opened my keyboard, and the key switches, and quickly realized the best would just be to replace the key switches. Reading from the constructor's specs I could see they were selling different models/versions with slight dimension variations.

Thus, I emailed Kinesis to ask them for the exact key switch references, figuring I could just replace them myself since the soldering skills involved were not too high ^^ (and my warranty had expired). Someone promptly answered me, asking me whether or not I was sure it was just a key switch issue. And finally, to my surprise, I was astonished couple of days later to receive by mail, free of charge (no cost at all, not even shipping), without even asking for it (I was just asking for the technical references so I could order some from the constructor) three key switches. That is what I would call a great customer service experience

I'm only talking about it now, because it actually took me 2 years to get my motivation together to actually use them and repair my keyboard (I was using a workaround involving hard-pressing the key, or punching it when stuck ). So anyway, prop' to Kinesis for its great customer service!

First Stitches in the OR!

ChoJin — Thu, 23 Jul 2009 19:10:41 +0000

Just a small post to remember forever that today was the day I did my first stitches in the OR (in pediatric surgery)

It probably sounds stupid from an outside point of view, and I'll probably feel the same in a couple of years, but right now I'm just happy I finally got to do it.

Baby steps, baby steps... but it was worth working toward them.

On Influenza A

ChoJin — Sun, 21 Jun 2009 16:02:14 +0000

Just a small post to point to an article bunnie [1] wrote about the virus Influenza A : On Influenza A by bunnie.

This article should be of interest for people with a background in computer sciences but little knowledge in biology. It offers some funny/interesting computer sciences comparisons which should be of pedagogic interests for computer savvy people.

By the way, if you like it, I strongly urge you to watch the following video from the 24th CCC Congress:
Programming DNA - A 2-bit language for engineering biology

[1] for those who don't know, bunnie is an amazing reverse engineer and hardware designer. Among other things, he's the first hacker who managed to break the XBox's security, as told in his book "Hacking the XBox", which is still, up to this day, one of my favorite computer sciences book. I could not emphasize enough about how you really should read it.

Back to Basics: Reverse Engineering of a Non-Standard Magstripe — Part One

ChoJin — Wed, 17 Jun 2009 00:31:45 +0000

Hello,

I think it's time I go back to basics and post something a little bit technical on this blog, which was the original intent of this blog anyway, should I remind everyone .

So we're going to talk about magnetic cards. We all have at least half a dozen of them, if not more, and although this technology is probably as old as the dinosaurs, for most people, how to read, how to write and what those cards contain still remain a mystery.

Pretty much everything has already been written about magnetic stripes, I will therefore focus this article on a well-used photocopier card, which has the interesting feature of using a non-standard protocol to prevent from tempering and counterfeiting.

Disclaimer

Let's be clear right from the beginning.

I won't give away the brand of this photocopier card, nor will I disclose the hardware used to properly read or make such cards (so don't bother asking). I'm only writing this article because I feel there is a lack of information about non-standard magstripe and copy/tempering protection schemes in this (magnetic ) field. This article is not about breaking the law, it's about knowledge and inner curiosity. I strongly urge you to avoid ending up in jail just because you wanted to save a few cents from a photocopier machine...

I warned you, and I will not be held responsible for your actions, should you decide to use this knowledge badly.

After this over-dramatic disclaimer, let's move on with the real article

Introduction

I won't go into the details about all the existing standards and so on. I'd suggest you read the very well known Count Zero's Card-O-Rama article from Phrack magazine about this subject matter. I will only write a quick summary about the standard modulation technique used on magstripes and then move on to talk about our photocopier card.

Generalities

Ok so, to make things simple, you can see a magnetic stripe as a sequence of tiny North and South oriented magnetics. When read through a solenoid, the induced current created by a given sequence of North and South polarities generates a square analog signal (yes, it's not that simple in reality, but it's good enough for our purpose).

As we will se through an example later, this square signal is then interpreted as a sequence of binary data (0 and 1) given the modulation scheme used. At this point, bits are usually grouped as chunk of 5 (BCD encoding) or 7 (ALPHA encoding) bit and interpreted using simple look-up tables.

As an example here is the BCD encoding table we'll be using later on:

Data Bits				Parity
b1	b2	b3	b4	b5	Character	Function
0	0	0	0	1	0 (0h)	Data
1	0	0	0	0	1 (1h)	"
0	1	0	0	0	2 (2h)	"
1	1	0	0	1	3 (3h)	"
0	0	1	0	0	4 (4h)	"
1	0	1	0	1	5 (5h)	"
0	1	1	0	1	6 (6h)	"
1	1	1	0	0	7 (7h)	"
0	0	0	1	0	8 (8h)	"
1	0	0	1	1	9 (9h)	"
0	1	0	1	1	: (Ah)	Control
1	1	0	1	0	; (Bh)	Start Sentinel
0	0	1	1	1	< (Ch)	Control
1	0	1	1	0	= (Dh)	Field Separator
0	1	1	1	0	> (Eh)	Control
1	1	1	1	1	? (Fh)	End Sentinel

Aiken Biphase Modulation

Now let's focus on how the square analog signal is converted to a sequence of binary data. On standard cards, the scheme used is called Aiken Biphase Modulation, or BMC encoding or often F2F (frequency/double frequency) as well.

In this simple encoding scheme, bit "1" are encoded using twice the frequency than bit "0". A picture is worth thousand words:

As said earlier, let's take a simple example, a sequence of 5 bit, and let's see how we can visually decode it step by step (assuming the encoding scheme is indeed F2F).

To ease the understanding of the encoding, and emphasize how "0" and "1" are encoded using frequency variations, we're going to highlight each bit separately.
The first bit (b1):

As you can see, a bit begins with a phase shift, no matter what the original phase is, and since we are here encoding a bit "1", the phase shifts again in the middle of the clock cycle.
The 2nd bit (b2) is quite the same since we are encoding a bit "1" again:

A phase shift to begin the bit, and another one in the middle of the clock cycle. With the 3rd bit (b3), you should begin to understand the process. We are here encoding a bit "0":

Again, the bit begins with a phase shift, but since we are encoding a bit "0", we don't shift the phase again during the clock cycle. That's basically how we know, and the reader knows, the bit should be interpreted as a "0" and not a "1".
The 4th bit (b4) is a "1" again:

And finally the 5th bit (b5), a "0":

As you see it's fairly easy to understand, but you have to pay attention to when a bit begins and ends. This will become very important when we'll analyze the non-standard encoding scheme used by our photocopier card.

You may wonder how the decoder knows how long is a "clock cycle", its time base. The answer is easy : we usually have a very long sequence of a constant square signal at the beginning of the magnetic stripe, interpreted as "0"s, and which therefore gives the decoder a time reference for further decoding.

Since we're on our way to decode this signal, we could now go all the way. Assuming the card is using a BCD (5bit chunks) encoding scheme, and using the look-up table given earlier, this give us:

"11010" decodes as the char ";", which is for standard magnetic stripe cards, what we call a "start sentinel", i.e. a char which announces the beginning of the real data. As you can also see in the look-up table, only the first 4 bits are really used, the 5th one is a parity bit. This means the 5th bit is set such as we always have an odd number of bit "1" in a char. This is used as a consistency check in standard encoding schemes, to double-check whether or not the decoding is corrupted.

Exercise to the reader

To conclude with this common standard encoding, and as an small exercise, you could try to decode this signal (assuming a F2F analog encoding scheme, and a BCD digital encoding scheme):

End of part one. The next part will finally focus on our mysterious photocopier card.
Stay Tuned!

End of Act I Scene 2

ChoJin — Wed, 10 Jun 2009 13:11:30 +0000

That's it, my 2nd year of med' school just finished, the 3rd year will begin this coming September.

I didn't have much time during this year to give any updates to those who still cares, so here is a small overview of this past few months:

I've been busy helping the association which mentors first year students
As said earlier, I indeed implemented a Simple Quiz Recognition Tool (using some morphological mathematics), named... SQRT to process all the first year students' training exams, compute grades and generate ranking, histograms and so on... It uses C++ and Ruby on Rails, but I'll tell you more about it later.
I've almost completed my Master in Medical Ethic Research as well (the only thing left is my master thesis oral presentation on June 16th). My Master thesis title is (sorry in french) "Ingénierie, Médecine et Éthique -- Introduction aux régulations des appareils médicaux aux États-Unis et en Union Européenne". If you're crazy enough to read it, drop me an email and I'll send you the pdf.
Of course, I've learnt medical stuff, I went to a couple of Hospital for my internships... Which is cool. I still love it, and I don't regret my decision a bit, but I do miss film making very much along with all my friends still in California as well...
And the new "me" went to a few crazy med school parties, bars and so on... (sorry I can't speak about those parties, they are R rated ). New friends coming, old ones gone... Life as bitchy as it can as usual...

Otherwise, nothing much to say. There are a lot of things I wanted to do but couldn't, mostly because of time constraints, such as working on a humanitarian project, doing some photo shoots... and unfortunately I also had to drop from my sign language class... Well, maybe next year... We'll see

That's it ! Next posts will be about computer stuff, you'll see, my dark side is coming back, it should be fun

Back to School

ChoJin — Sun, 14 Sep 2008 21:36:52 +0000

Small update.

My vacations are coming to an end. The internship was a blast.

There's nothing like learning how to cut the plaster with a saw on yourself...

I went to the OR a couple of times for some crazy surgeries (even an 8h surgery! I almost had an orgasm )

And I even managed to recycle my old magician skills making animals with balloons for the kids at the hospital.
It was truly a blast. I loved every single minute of it (yeah, even when I had to clean some Poop )

Otherwise, not much to say, except we (the tutoring association at my med' school) are preparing the back-to-school season for the first year students. I'm implementing a quiz OCR system for it, because complete solutions are around 10k euros! Can you believe that?! WTF seriously... And finally I've been accepted to the Master of Biomedical Ethic course (which I would be doing at the same time than my 2nd medical year).

That's pretty much it, enough of my life... Next time I'll try to post a few technical posts again. Maybe about some reverse engineering work I'm currently doing, or about the quiz OCR system... We'll see. Or, as Pierre pointed out, I should probably post again some stuff about cooking

First Medical Internship

ChoJin — Thu, 03 Jul 2008 20:12:18 +0000

Today was the day, the day they called us one by one to ask us whether we wanted to do medicine, dental... Then we had to choose our first 3 week internship.

I'm happy to report my first internship will start next Monday and I managed to get pediatric surgery.

Wish me luck not to kill someone or along those lines

ChoJin, MD

ChoJin — Sat, 28 Jun 2008 09:01:42 +0000

I'm lacking words to express myself, it's almost surreal...

So much happened in my life in the past few years, especially this past year. I took a chance and gambled my life with a great deal of uncertainties and doubts. Was I crazy? Did I really want too much? And after all.. Wasn't my brain too old already to be up for the challenge?

I got the results yesterday, and I still can't believe the results... but I passed. Among approx. 2400 people with 377 places for the medical field my final rank is 102nd which means I passed, and with a large margin. Hopefully this will enable me to get the internship I want this summer, I'll know on Thursday.

Some say you see the movie of your life just before you die. The same happened to me yesterday, except I think it was my re-birth. Sorry for the cliché and for lacking words to describe it right now, but I just wanted to thanks my friends for their support and I'm relieved I didn't have to disappoint them.

I'm very happy to say that Pauline passed too, but sadly, a lot of my other friends didn't make it, which gives to this "victory" a really bitter flavor...

Anyway, here are my final results:

First Semester:

Anatomy: 17.1 / 20 (weight: 1)
Biology: 10.9 / 20 (weight: 2.6)
Chemistry: 13.6 / 20 (weight: 2.3)
Physics: 15.9 / 20 (weight: 2.3)
Average Grade: 13.816
Rank: 252

Second Semester:

Biochemistry: 15.6 / 20 (weight: 2.6)
Biophysics: 14.7 / 20 (weight: 2.1)
Histology/Embryology: 12.5 / 20 (weight: 2.1)
Physiology: 15 / 20 (weight: 1)
SHS (Ethics, Philosophy, Sociology...): 18.5 / 20 (weight: 4)

Final average grade: 15 / 20
Final rank: 102
Numerous Clausus: 377 (for approx. 2400 candidates)

After "House, MD" ("Dr. House" in France), soon we should have "ChoJin, MD"

Steadicam tests for Asdin/X-otik

ChoJin — Fri, 20 Jun 2008 20:55:44 +0000

Just some random steadicam tests using my Flyer and Asdin's Sony camera. I'll probably help him shooting the next Adrone's music video. We'll see...

We can see Asdin giving a try with the steadicam near the end of the video

PS: Yes I know I should add weights to the sony camera, but I didn't have anything to do so