The problem in a nutshell is that the MySQL Ruby lib is not respecting the encoding setting defined in the database.yml file and does not encode the strings, that are pulled from the database in that encoding. Here is an example database.yml, which defines the encoding of the database connection:

1
2
3
4
5
6

development:
  adapter: mysql
  database: gsm_dev
  username: root
  password:
  encoding: utf8

Fortunately there is a nice solution I found: use the ruby-mysql lib. It is on gemcutter so you can do:

1

gem uninstall mysql && gem install ruby-mysql

This is going to uninstall the buggy mysql lib and install the working ruby-mysql. Hope that helps!

P.S.: I am not sure that this is a production-ready solution, but at least it provides a way to run things without hacks until the MySQL C API is fixed.

Debugger still not working

I just discovered something interesting. It is not really a big deal, but it is something to keep in mind:

$ gem1.9 install ruby-debug
Building native extensions.  This could take a while...
ERROR:  Error installing ruby-debug:
    ERROR: Failed to build gem native extension.

/usr/local/Cellar/ruby/1.9.1-p376/bin/ruby1.9 extconf.rb
Can't handle 1.9.x yet

Obviously the debugger is still not working on ruby 1.9.

Using the rails console with ruby 1.9

There is a slight catch if you want to use the console. If you just do

./script/console

you will notice many strange behavior, like broken unicode characters and even core dumps like the one below if you have unpacked native gems in your project.

(3830) malloc: *** error for object 0x1027ec038: pointer being reallocated was not allocated
*** set a breakpoint in malloc_error_break to debug
Abort trap

The problem is that the console is actually using the system default irb, which was using ruby 1.8 on my machine. In order to use the 1.9 irb you should do the following:

./script/console --irb=irb1.9

This is if your 1.9 irb is called irb1.9

What I really liked in this approach are several things:

• I can easily blog from TextMate or any other editor supporting markdown highlighting
• I can keep my blog in source countrol, like github
• I have the flexibility to do whatever I want with the text files of the posts (like exporting them to PDF or LaTeX later)
• I avoid the complexity of WordPress and the WTFiness when the posts render weirdly from the database (like escaped ‘<’ and ‘>’ symbols)

As a whole I think that heavy engines like WordPress are just too complex for most of the blogs. Keeping your posts in a database makes them difficult to be exported to other formats like PDF and also the HTML code tends to become too messy in a very short time. Especially if you are using WYSIWYG editors.

Why I decided to switch from my initial custom engine?

The sinatra app that I implemented was basically parsing a bunch of markdown files with some meta data embedded in them and was rendering them in the blog. A lot of the functionality was offloaded to javascript.

There was a major flaw in that approach: SEO sucked!

Because of the javascript code, which was doing most of the work the search engine crawlers were not able to detect the proper titles of the posts and the titles are something really important for the SEO.

So, I had to make a change in order to make my blog discoverable. I decided to use jekyll because a lot of people are using it already. For example github uses it for its project pages. Also quite a lot of people are using it as a blogging engine.

It is quite powerful and also it is doing preprocessing and effectively translates your blog into a static site. This is a great thing, because it makes the blog very fast.

Comments? Use a JS solution like disqus and you will be fine ;-)

You want a twitter widget? Javascript is your friend. The twitter widget on my blog is a jQuery plugin, which integrates very easily.

What about compass?

May be you heaven’t heard of compass, but tt is really great! The idea is that you have that special language to write CSS called SASS. The good thing about SASS is that it gives you things like mixins, partials and variables, which makes reusing CSS easier. It integrates with already existing CSS frameworks like blueprint and makes writing layouts a peace of cake.

For example, imagine that you are developing a site and you want to have 2 columns layout with header, content and footer. Here is how your CSS will look like with compass:

// defines a class for a body tag for 2 columns
body.two-col
  #container
    +container // includes a partial called 'container'
  #header, #footer
    +column(24) // the header and the footer are 24 columns wide
  #sidebar
    +column(7, true) // the sidebar is 7 columns wide
  #content
    +column(16) // the content part is 16 columns wide
    +append(1) // and it as 1 column spacing from the sidebar

This is really convenient, because you don’t have to handle a lot of standard CSS and you just throw out your ideas.

Source code

You can find the source code of this blog here: http://github.com/valo/valentinmihov.com. Any comments and suggestions are more than welcome.

What’s next?

I plan on posting a video on getting started with jekyll, compass and heroku for your own blog. It will be a several steps procedure, which will allow most of the people to have their own jekyll blog, hosted for free on heroku (which uses the Amazon cloud)

Setup

I am working on a MacBook Pro with Mac OS X 10.6.2. I installed homebrew so I can install ruby 1.9. Because I don’t want to override my ruby 1.8 install I had to make a change in the formula for ruby. I added a –program-suffix=1.9 to the configure script. Here is how it will look:

system "./configure", "--prefix=#{prefix}",
                      "--disable-debug",
                      "--disable-dependency-tracking",
                      "--enable-shared",
                      "--program-suffix=1.9"

After that I did:

gem1.9 install rails mechanize nokogiri formtastic

After some glitches, because I had a previous install of ruby 1.9 on my machine everything ran well.

Creating the rails app. Problems start to appear

I created the rails project and here my first serious problem struck.

When I try to run

ruby1.9 ./script/server

I receive

Loading development environment (Rails 2.3.5)
Missing the Rails 2.3.5 gem. Please `gem install -v=2.3.5 rails`, update your RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.

A workaround for this is to do

rake1.9 rails:freeze:gems

It is going to freeze the rails framework into the vendor folder and the server is able to run from there on. I noted to myself that I will have to unpack all the gems that I use in the project. This is really not a big deal for me, so I just continued from there on.

Unicode templates. The biggest pain so far.

I wrote my first template with some unicode symbols in it (the project is in Bulgarian) and when I tried to render the template I received an error:

invalid byte sequence in US-ASCII

This was quite frustrating, but after some research I figured out that the reason for this is the default encoding of the ruby VM. It is using the LANG environment variable to determine that. My LANG was empty so it assumed US-ASCII. An easy way to test what is your default encoding is to do:

$ irb1.9
>> __ENCODING__
=> #<Encoding:US-ASCII>
>> 

This means that ruby will load all files with this encoding by default, which also includes the templates. The way to fix this is to set the LANG var:

$ export LANG=en_US.UTF-8 && irb1.9
>> __ENCODING__
=> #<Encoding:UTF-8>
>> 

This fixes this error, but causes a new one:

incompatible character encodings: ASCII-8BIT and UTF-8

This was a tough one. It turned out that there is a bug about this in the rails’ lighthouse: here

Using the workaround of Hektor fixed the issue, although it is not perfect, because it forces the encoding of the erb templates to utf-8, which will not work if you use some other encoding.

Mechanize and SSL pages

After going through this obstacle I encountered another one. I am using mechinize and nokogiri in my project to scrape some web pages. In one of the scenarios I am accessing a SSL page. Here is what I got after trying to parse this page:

Net::HTTP::Post: <some_ssl_web_page_address_here>
request-header: accept => */*
request-header: user-agent => Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.3
request-header: connection => keep-alive
request-header: keep-alive => 300
request-header: accept-encoding => gzip,identity
request-header: accept-language => en-us,en;q=0.5
request-header: host => my.globul.bg
request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
request-header: content-type => application/x-www-form-urlencoded
request-header: content-length => 110
Rescuing EOF error
Rescuing EOF error
Rescuing EOF error

EOFError (end of file reached):
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/protocol.rb:135:in `sysread'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/protocol.rb:135:in `block in rbuf_fill'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/timeout.rb:52:in `timeout'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/timeout.rb:82:in `timeout'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/protocol.rb:134:in `rbuf_fill'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/protocol.rb:116:in `readuntil'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/protocol.rb:126:in `readline'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/http.rb:2136:in `read_status_line'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/http.rb:2125:in `read_new'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/http.rb:1117:in `transport_request'
  /usr/local/Cellar/ruby/1.9.1-p376/lib/ruby1.9/1.9.1/net/http.rb:1103:in `request'

It turns out that this is a kind of a known problem, which is described here. You can have a look at my comments and here is a commit, which is fixing the problem. It is still not merged into the mechanize code, because I should come up with a unit test for it.

Bottomline

Lol! That was a tough fight! I think that ruby 1.9 and particularly the libs in the ruby community have a long way to go until they are usable by the regular developers. At least the community is driving forward. I haven’t tried rails 3, but I heard that it has fixed the issues with the encodings in the templates. For the rest of the libs it is just a matter of time to polish all the issues. Until then: give 1.9 a try and report any issues you have.

Context of the problem

One of the side projects I am working on right now is a grading system for high school programming problems. It is hosted on a low cost sharing hosting. A specific thing about this is that the database server is also shared and also is quite distant from the application server. This means that every query is adding an overhead of at least 20ms to the execution time (sometimes even 200ms), even if the query takes 0.1ms on the database server. Now imagine that you have an action which makes a linear number of queries depending on the number of the users in the system.

One other limitation is that the memory of the ruby VM is limited to something like 200MB. This means that loading a lot of ActiveRecord objects as a method for reducing the memory usage is not an option, because sooner or later the watchdog of the server will start to kill our ruby processes.

Some of you will say that hosting rails on a shared server is just not serious. My answer is that because we have a fairly low number of hits it is possible to use such hosting, you just have to make some tweaks.

The schema of the database

We have the following simplified schema:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class User < ActiveRecord::Base
  has_many :runs

  column :name, :string
end

class Problem < ActiveRecord::Base
  has_many :runs

  column :name
end

class Run < ActiveRecord::Base
  belongs_to :problem
  belongs_to :user

  column :total_points, :integer
end

I am using a fictional column method to describe the attributes of each table.

What is the problem we want to solve?

We want to generate a ranklist of all users into the system. The ranklist is generated as following:

1. Total score: for every user we must compute the maximum score for every problem and sum the maximum scores for all the problems to compute the total score. For example: if there are problem A and problem B, and the user John has 2 submits for problem A with score 50 and score 60, we are going to take a maximum score 60 for problem A. If John has 3 runs for problem B with score 20, 60 and 100, we will take maximum score of 100 for problem B. Given that, John will have a total score of 60 + 100 = 160.

2. Problems solved for 100 points: we compute the number of problems for each user, which has a max score of 100 points. For example in the previous setup only problem B is counted, because the max score John has on it is 100, so John has only 1 problem solved for 100 points.

3. Total runs: for each user we count the number of runs for that user.

Solution 1

It is very easy to solve the above problems using the ActiveRecord associations. Here is how they could be solved with code:

1. Total score:

1
2
3
4
5

  User.all.map do |user|
    user.runs.group_by(&:problem) do |problem, runs|
      runs.map(&:total_points).max
    end.sum
  end

1. Problems solved for 100 points:

1
2
3
4
5

  User.all.map do |user|
    user.runs.group_by(&:problem) do |problem, runs|
      runs.map(&:total_points).max == 100 ? 1 : 0
    end.sum
  end

1. Total runs:

1

  Run.count(:all, :group => :user)

The problem with all these approaches is that they either issue a lot of queries or they load a lot of objects into the memory. Each of these problems makes our application useless in a shared hosting environment.

Solution 2

Let’s try to pack everything into one query and wrap it into ActiveRecord code. We want to compute everything into the database. Making the right indexes should make the things pretty fast.

Adding pure SQL into the code is ugly and I don’t encourage that at all, but sometimes we have to get our hands dirty. Let’s at least try to reduce the ugliness by wrapping this into the User module, so that it is neatly packed and easy to refactor.

The following query is computing all I described and also it gets some info about the users. We want to return an array of tuples, where there is a tuple for each user with the needed data and also his/her name. Here is the query:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

SELECT
  users.id,
  name,
  SUM(max_points_per_problem) as score,
  SUM(runs_per_problem) as runs_count,
  SUM(CASE max_points_per_problem WHEN 100 THEN 1 ELSE 0 END) as full_score
FROM users
LEFT JOIN
  (
    SELECT
      MAX(total_points) as max_points_per_problem,
      user_id,
      COUNT(runs.id) as runs_per_problem
    FROM runs
    JOIN problems ON problems.id = problem_id
    GROUP BY user_id, problem_id
  ) as problem_points
ON problem_points.user_id = users.id
GROUP BY users.id
ORDER BY score DESC

Now the question is: how to integrate that into our rails code? Here is the answer: using select_all:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

class User < ActiveRecord::Base
  class << self
    def generate_ranklist
      query = <THE_UGLY_THING_I_WROTE_A_FEW_LINES_UPWARDS>

      User.connection.select_all(query).inject([]) do |ranklist, row|
        ranklist << [
                      User.send(:instantiate, row),
                      row['score'],
                      row['runs_count'],
                      row['full_solutions'],
                      row['full_score']
                    ]
      end
    end
  end
end

Conclusion

Using the described method we were able to reduce significantly the number of queries we do, which reduced the execution time on the shared server from several seconds to under a second and we were also fitting into the memory limit, because we are loading only things we really need.

This is also going to help in larger projects running on dedicated servers. It is just a matter of time when scalability issues emerges for an web app. Optimizing the database queries is one of the most beneficial ways for optimization.

WARNING: I am not advocating in any way that you should custom craft all your queries. Most of the cases you don’t have to do so! Most of the cases only several objects are loaded by the app and a few queries are issued. In the case I describe we are making some very complex aggregations across large amounts of data, which makes ActiveRecord magic very inefficient!

8809 = 6
7111 = 0
2172 = 0
6666 = 4
1111 = 0
3213 = 0
7662 = 2
9312 = 1
0000 = 4
2222 = 0
3333 = 0
5555 = 0
8193 = 3
8096 = 5
7777 = 0
9999 = 4
7756 = 1
6855 = 3
9881 = 5
5531 = 0
6782 = ?

Send me a private message in twitter or via email and I will tell you if your answer is correct :-).

I decided that it will be great to make a benchmark of how the blog runs with the two interpreters. Here are the description and the result.

Configuration

• Machine: Apple MacBook Pro, 2.5 GHz Intel Core 2 Duo, 4GB 667 MHz DDR2, 250 GB HDD, 5400 RPM, Mac OS X 10.5.7
• Ruby 1.8.7 p72, 64 bit
• Ruby 1.9.1 p0, 64 bit
• Sinatra 0.9.2
• rdiscount 1.3.4
• haml 2.0.9
• thin 1.2.2
• mongrel 1.1.5
• nginx 0.7.61 as reverse proxy

The logic behind the test

The page that I tested with was doing the following actions:

• Open around 18 files from the filesystem and read their first 3 lines
• Parse the lines using regex
• Parse some dates from the data read from the files
• Read 5 of the files in-memory
• Render some HAML templates with content got from the read files

There is a git repository with the code. It is my blog.

The test candidates

I tested the following configurations:

• 5 Thin instances running Ruby 1.8 behind nginx reverse proxy
• 5 Mongrel instances running Ruby 1.8 behind nginx reverse proxy
• 5 Thin instances running Ruby 1.9 behind nginx reverse proxy

I use thin because this is the only web server, that compiles against ruby 1.9 on my machine. I even had to compile it myself, because of the 64 bit interpreter that I have (also custom compiled).

The tests were performed with concurrency 10 and 1000 requests. Here is the command line I used:

# ab -c 10 -n 1000 http://127.0.0.1:8080/

The results

Here are some graphs of the results

As you see using ruby 1.9 boosts the performance of the application more than 2 times.

Here are links to the extended info generated by ab:

• ruby 1.8 + mongrel
• ruby 1.8 + thin
• ruby 1.9 + thin

I will be glad to hear your comments about the test and your experience with ruby 1.9. For the real enthusiasts this site provides info about what gems are 1.9 compatible.

<% if true %>
  Hi from Rails!
<% end # hi from rails %>

The solution: if you want to put comments in ERB templates use this syntax:

<% if true %>
  Hi from Rails!
<% end %>
<%# hi from rails %>

• The marketing people need time to generate the changes
• The developers need time to understand the changes. Usually emails are exchanged, because the idea is not clear in all the places
• The developers end up making a lot of find/replace, copy/paste changes, which are nothing more than text editing

Sounds familiar, huh?

I was thinking about this and I believe that the problem in the whole process is in the developers and the code base of the web site. In the perfect world the code of the website will be clean XHTML with very well separated CSS style and unobstructive javascript. What this means is that the code of the website will be readable by ordinary people, who don’t understand any of the above things. Therefore these people will be able to easily make simple changes of the text without fearing that they will break something.

You can’t expect the non-programming people to understand what is javascript, why are these empty div-s here and why do we need them in order to have rounded corners. This reminds me again about the saying “Source code is intended to be read by humans and sometimes to be able to run on machines”.

Here is a real argument, why we should not bloat our code. It will save us time in the future and will make our company more effective.

Bottom line

If you write clean and understandable code you will not only save time during future maintenance, but your team could be more effective, because non-programming people can make small tweaks.

I will be speaking on the conference about stubbing and mocking with mocha. I have made an example project on which I will explain my ideas. You can find the latest source here.

We will see there :-).

I decided to call this engine blogxter. It relies purely on the filesystem, which makes blogging very easy task. I just fire up a Textmate and create a new file. The posts are written in markdown. I decided to use it because it is very understandable. In the future I am thinking about trying with AsciiDoc.

The site is hosted on Heroku and runs with sinatra. I use HAML for defining the layout and soon I will switch to Sass, when the styles start to get complex.

This is for now. I will soon post the code of the engine on github. Stay connected.

Our repository is SVN and I am using git-svn to track it. So I decided to maintain a branch called “stable”, which has everything in a state that I am sure that is working. I am working on this branch and every time I want to push something to the trunk, I switch to the master and execute:

$ git checkout master
$ git merge stable
$ git svn dcommit

This will merge my changes from the stable branch (which is a trivial merge) and will push my changes on the server. After that I switch back to the stable branch and I don’t care what is happening with the trunk.

Every several days I update my stable version with the trunk.

This whole thing is saving me a lot of hours or reconfiguring and tracking regressions and I can invest these hours into real development. Git rulz :)

To be honest, first it was hard. The main problem was to get the ideas behind git and to find the right commands for doing the everyday stuff. After several days of working with it I really fell in love. Now I am using it as my main version control system. What I really like is that I can easily branch the code and start working on something experimental and at the moment I decide that it is ready to be integrated in the product I just merge it in the main code branch with the whole history of developing it. The other great thing is that I have the whole history on my machine and I can work offline.

But, there are some weaknesses, particularly in the git-svn bridge, which definitely need some more work. The major one is the support for external repositories. Right now there is no support for that in git-svn, so I had to make a separate repository for each external one. After that I wrote a script for updating the whole repo. Here is the code:

#/bin/bash

DIRS="git-repo-1 git-repo-2 git-repo-3"

for d in $DIRS; do
pushd $d > /dev/null

# Search for sub repositories
for subd in `find . -iname .git`; do
echo "**** START ****"
pushd $subd
cd ..

CHANGES=`git status | grep 'working directory clean'`

if [[ $CHANGES ]]; then
  git svn rebase
else
  git stash && git svn rebase && git stash apply
fi

if [[ ! $? ]]; then
  echo "Error updaing $d!"
  exit 1
fi

popd
echo "**** END ****"
done

popd > /dev/null
done

This bash script will go through a list of repositories (in the example these are git-repo-1, git-repo-2…) and will search for sub-repositories in their tree. If it finds any, it will stash the current changes (if any) and will rebase the code with the latest one. I requires some mode polishing, but as a whole it is working well.

The other weaknesses are that the GUI tools for working with git are not as many as the ones for SVN and also they are not so pretty-looking. This is a problem for people, who are not developers and are used to work with nice visual tools (in our company there are lots of them). For people, like me, who are used to work from command line this is not a problem.

The other problem, which I am concerned about is the windows support for git. I didn’t have the time to investigate that, because it will require me to install windows, but I see that there are a lot of effort thrown in that direction. I hope that the support is good, because many people are using windows as a development platform. Unfortunately I am a little bit doubtful about that and I will expect to have problems with the windows port.

Well, the company is dreambox.com and our main goal is to revolutionize the way the on-line learning is functioning. The idea is to provide personalized on-line learning. Right now we are focused on building on-line learning platform, which is targeted at teaching math to small children. Expect more interesting information to come.

I also have posts in the company blog. Right now it is only one with the idea of increasing. Check it out here.

1. Get a Linksys WRT54GL wireless router (it runs linux!!!)

2. Install DD-WRT firmware on it

3. Go to the web interface of the router (usually http://192.168.1.1). Go to Administration->Commands and enter the following in the Commands textbox:

    iptables -t mangle -I PREROUTING -i vlan1 -j TTL --ttl-set 128
   

4. Hit “Save Firewall” and you are ready to go!

This will set the TTL of all incoming packets to 128 ;-)

I don’t have anything against people with strange English accents (I guess that I also have one), but to make a video showing your product like this it not one of the greatest things a company can make.

1
2
3
4
5
6
7

# ./sing_ten_bottles_song
NULL green bottleshanging on the wall,
NULL green bottleshanging on the wall,
And if one green bottle
Should accidentally fall

Segmentation fault#

The big question is on which line exactly the segmentation fault will be thrown? ;-)

I lost around 1:30 hours for figure out how to compile the mysql gem. The installer was not able to find the mysql libraries and includes. The magic line turned out to be:

# gem install mysql -- --with-mysql-config=`which mysql_config`

This makes the install just a piece of cake! :)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#include <cstdio>
#include <map>
#include <string>

using namespace std;

map<string, string> gmap;

void test(bool use_global_map)
{
  map<string, string>& m = use_global_map ? gmap : map<string, string>();
  m["A"] = "This is a changed value";
}

int main()
{
  gmap["A"] = "Initial value";
  test(false);
  printf("Value in map with false arg: %s\n", gmap["A"].c_str());
  test(true);
  printf("Value in map with true arg: %s\n", gmap["A"].c_str());
}

In this code there is an error and it is an initialization of the reference in the test function with a map, which is local for the conditional expression. This means that the reference will be initialized with an object, which will be destroyed after the line is executed.

The problem is that VC++ allows you to write this. The more interesting thing is how this code will behave. It turns out that whatever value you pass to the test function it will always behave like you created a new map and initialized the reference with it. So, gmap will never be changed!

Compiling this code with GCC gives an error, which is the correct behavior for this kind of expressions.