Tuesday 26 April 2016

ruby on rails - poor Redis performance inside Sidekiq worker

I have a Rails app, that runs 50 concurrent Sidekiq workers, that receive data from external APIs, manipulate it and store it in a Database.
In order to prevent to store the same data multiple times, I create unique strings, that I store into Redis.



r = Redis.current
r.pipelined do
options.each do |o|
r.setex(o.generate_index, 1.hour, o.value)
end
end



Before saving a new object to the Database I check if it's unique string exists in Redis:



r = Redis.current
option_ids = r.pipelined do
new_options.each do |o|
r.get(o.generate_index) # check if unique string already exists
end
end



I also tested without pielining with the same result:
Redis GETs/SETs are extremely slow.
Some GET-Request take more that one second!



weird thing: While running 50 Sidekiq workers, that had a bad Redis performance, I benchmarked Redis from the shell, with very good results:



r = Redis.current
10000000.times { puts Benchmark.realtime{ 1000.times{|i| r.get(i)} }}



with output like:



0.197379041
0.192059888
0.196165358
0.207617425
0.198095963
0.195844917
0.211404108

0.203188759
0.208119298
0.184018683


No Benchmark took longer than 0.5 second.



I also tried Redis.new instead of Redis.current with the same result.



I can't figure out what's the problem




EDIT:
After some more testing and benchmarking I noticed that the benchmarks in my code seemed to block other workers and slow them down much more than expected. Without benchmarking the workers are up to 10 times faster.
Furthermore it seems that my vserver just cant handle that many workers. My development laptop can take 100 workers and still performs very fast.
Moreover when the load is heavy, database queries seem to be very slow on the ruby side.
I benchmarked simple Mongoid lookups like



time = Benchmark.realtime do
existing_option = Option.find_by(index: o.generate_index)
end



These benchmarks sometimes took longer than one second, but the query did not show up in the mongodb slow log (which means the actual query took less than 100ms on the database).
However pure rubycode (without database hits) seems to perform ok.
I wonder if it performs better when I implement the HTTP-requests with Typhoeus Hydra instead of sidekiq.

No comments:

Post a Comment

c++ - Does curly brackets matter for empty constructor?

Those brackets declare an empty, inline constructor. In that case, with them, the constructor does exist, it merely does nothing more than t...