Monday January 24, 2011

[Time] NameMessage
[09:04] tecnalia Hellow
[09:04] tecnalia I have a quetion related with ZMQ and the checkSum or CRC at the end of the ZMQ datagrams?
[09:05] tecnalia Someone cpuld tellme where can I find info about this topic
[09:12] tecnalia Hellow
[12:13] benoitc mmm how would you handle a publisher in a daemon that forkk process ?
[12:14] benoitc I have some workers launched in python that should send message and different listeners . number of workers can be dynamically changed
[13:35] kabs Hi
[13:35] kabs Hello, I created one sample pub-sub model in c using ZeroMQ library, my pub runs while(true) and sends 1 2 3 4 5 in each loop , my sub code spawns 5 thread each thread filter one number send by pub, I finish the thread once it gets two instance of each number.
[13:37] kabs I run pub and then sub. Now sub don't exit though pub is sending infinitly but once I do ctrl c for sub, pub gets all the messages
[13:37] kabs Can someone tell me why I need to do control c to pub so that sub gets all the messages ??
[13:59] sustrik benoitc: you have to create new context in the forked process
[14:00] sustrik kabs: strange
[14:01] sustrik can you create a minimal test case?
[14:02] kabs benoitc: I am using same context for each thread
[14:02] benoitc sustrik: yes figured it :) I've changed my design now , i've my workers doing REQ to all connected consummers and consumerers putting their uri in a conf checked each time by workers
[14:03] benoitc i'm actually using redis to save temporary tasks waiting a better design
[14:03] kabs benoitc: So I create sockets corresponding to each thread using same context and pass this socket to each thread
[14:03] benoitc i see
[14:03] sustrik kabs: that works for different threads in a single process
[14:04] sustrik when you for new process, you need a new context
[14:05] kabs sustrik: I am using same main to spawn these threads , so it is a same process I guess so I needn't have new context for each thread right?
[14:07] sustrik kabs: right
[14:09] kabs sustrik: then why my subs( threads) are not able to get all the messages. Some of the subs ( threads ) get all messages while others don't . But when I press ctrl c on pub, all subs( threads) gets messages and they exit as expecte
[14:12] kabs sustrik: I tried different context for different threads and it worked!
[14:14] sustrik you have to provide a minimal test case
[14:14] sustrik so that people can look at what's going on
[14:26] benoitc mmm what happen when socket we connect is closed at the end ? in case i do multiple connect ?
[14:26] benoitc is this cleanly closed ?
[14:28] toni hi there. I am using the pyzmq binding. I have to pack an auto-generated envelope as part of the payload. My payload format is json. The encode raises a "'utf8' codec can't decode byte". Any hints how I could solve the problem?
[14:28] toni I know, this is more python or encoding specific than it is zmq-specific, but maybe someone can help me?
[14:29] benoitc ther is an encode_string option in json.dumps
[14:29] benoitc or you can do json.dumps(..).encode('utf-8')
[14:29] benoitc smth like it
[14:32] sustrik benoitc: it's cleanly closed
[14:32] benoitc cool, thanks
[14:34] toni benoitc: the real problem is the envelope ("...��8���.."). I tried to encode it as utf-8, but as I do so, I get a UnicodeDecodeError t0o.
[14:37] benoitc ah
[14:38] benoitc if you have some binary in your json you will have to base64 it
[14:55] toni benotitc: thanks, Ill try this
[15:07] kabs Hello, I tried pub-sub model with both pub and sub on same machine,so, one pub is running infinite loop and sending data, I tried 50 subs first and they were able to get data, then I tried increasing number of subs to 500 and it gave "to many files open error", I changed the ulimit to 65535 and it worked, now I want subs to be around 5000. How can I scale my pub-sub model??
[15:09] kabs with 5000 subs , it took lot of time for even a single subscriber to get data and once subscribers started getting data, it hanged after sometime. I am running subs in threads. Any help in increasing scalability??
[15:13] spht kabs: 5000 threads? Have you lowered your thread stack size?
[15:14] kabs Yes 5000 threads, no didn't lower that, don't know about it
[15:14] spht kabs: on linux IIRC the default thread stack size is 8mb....
[15:15] spht kabs: but if you want to run a lot of clients on a single machine I would probably use an async mechanism instead of threading and then have worker threads for any computation needed
[15:15] kabs spht: actually I want to know is as subscriber increases how can I increase the scalability of pub-sub model
[15:16] kabs spht: since in my application , number of subs can increase to any number
[15:16] kabs spht: can you give me more info on "async mechanism instead of threading and then have worker threads for any computation needed" or some link where I can read this stuff you mentioned
[15:17] spht kabs: what are these clients, processes on the same machine? Remote machines?
[15:17] kabs spht: same machine, I am making test programs to learn this model using zeroMQ
[15:18] sustrik kabs: the error means there are too much connections open
[15:19] sustrik check your OS settings for the max value
[15:19] kabs sustrik: yes, so how can I go for scalable system, increasing max value everytime doesn't seems to be right way of doing it
[15:19] sustrik shrug, if your system limit is 1000 sockets, you cannot handle 5000 connections
[15:20] spht kabs: it's extremely hard to write good test cases for c10k-problems, especially on a single machine. That said, on quick solution to minimize the # of threads used would be to create several sockets from every thread and monitor them with zmq_poll
[15:22] kabs spht: didn't understood you solution well, can you please elaborate or send some link that mentions this??
[15:23] spht kabs: For general c10k info: // for zmq_poll see docs
[15:24] spht the point is, for a large number of threads, the overhead of the threads themselves are significant, both for the kernel context switching and memory use
[15:30] kabs spht: thanks! will read about zmq_poll and will see if I can think of scaling my system
[19:59] drbobbeaty Question about ZeroMQ 2.1.0 (from the git repo) on Ubuntu 10.04.1... I have a program that's running fine on CentOS 5, but on Ubuntu 10.04.1 when I try to open the URL epgm://bond0;;77777 I get the message: The protocol is not compatible with the socket type.
[20:00] drbobbeaty Is this some kind of IPV6/IPV4 issue? Or something that's been seen.
[20:00] drbobbeaty I don't see anything in my Google searches on this.
[20:28] sustrik drbobbeaty: it's that mutlicast makes sense only with PUB/SUB sockets
[20:31] drbobbeaty sustrik: but when I run the exact same code - as a SUB, in this case, on CentOS5 it's fine. Have you heard of any CentOS vs. Ubuntu differences?
[20:32] sustrik it's a SUBV socket and epgm transport and you get "The protocol is not compatible with the socket type" error?
[20:34] sustrik last version of 0MQ from github?
[20:35] drbobbeaty It's a SUB socket and it's the latest on github for Ubuntu. For CentOS, it's a week or so ago off github, but still "post-2.1.0".
[20:36] drbobbeaty Works great on CentOS, fails with that error on Ubuntu. I have boxes running side-by side.
[20:52] sustrik there were some recent changes that may have caused the behaviour
[20:53] drbobbeaty Really? OK... what should I do?
[20:53] sustrik can you provide a minimal test case?
[20:53] sustrik it should be easy
[20:53] sustrik probably just create a socket and bind
[20:53] drbobbeaty Sure. I'll try to write one up.
[20:53] sustrik thanks
[21:41] drbobbeaty sustrik: if you use this gist: , you can even delete lines 69 to 94, you'll get the same error I see. On CentOS5 it's fine. On Ubuntu 10.04.1 it fails on the connect() call (line 68).
[21:46] drbobbeaty I'm going to send it to the mailing list as well... just in case you're gone for the day.
[22:31] benoitc hum /win 8
[22:36] pythonirc101 can i use zeromq to communicate between two machines behind NATs?
[23:56] mikko pythonirc101: what do you mean?