ZeroMq IRC Log

Monday January 24, 2011

[Time] Name	Message
[09:04] tecnalia	Hellow
[09:04] tecnalia	I have a quetion related with ZMQ and the checkSum or CRC at the end of the ZMQ datagrams?
[09:05] tecnalia	Someone cpuld tellme where can I find info about this topic
[09:12] tecnalia	Hellow
[12:13] benoitc	mmm how would you handle a publisher in a daemon that forkk process ?
[12:14] benoitc	I have some workers launched in python that should send message and different listeners . number of workers can be dynamically changed
[13:35] kabs	Hi
[13:35] kabs	Hello, I created one sample pub-sub model in c using ZeroMQ library, my pub runs while(true) and sends 1 2 3 4 5 in each loop , my sub code spawns 5 thread each thread filter one number send by pub, I finish the thread once it gets two instance of each number.
[13:37] kabs	I run pub and then sub. Now sub don't exit though pub is sending infinitly but once I do ctrl c for sub, pub gets all the messages
[13:37] kabs	Can someone tell me why I need to do control c to pub so that sub gets all the messages ??
[13:59] sustrik	benoitc: you have to create new context in the forked process
[14:00] sustrik	kabs: strange
[14:01] sustrik	can you create a minimal test case?
[14:02] kabs	benoitc: I am using same context for each thread
[14:02] benoitc	sustrik: yes figured it :) I've changed my design now , i've my workers doing REQ to all connected consummers and consumerers putting their uri in a conf checked each time by workers
[14:03] benoitc	i'm actually using redis to save temporary tasks waiting a better design
[14:03] kabs	benoitc: So I create sockets corresponding to each thread using same context and pass this socket to each thread
[14:03] benoitc	i see
[14:03] sustrik	kabs: that works for different threads in a single process
[14:04] sustrik	when you for new process, you need a new context
[14:05] kabs	sustrik: I am using same main to spawn these threads , so it is a same process I guess so I needn't have new context for each thread right?
[14:07] sustrik	kabs: right
[14:09] kabs	sustrik: then why my subs( threads) are not able to get all the messages. Some of the subs ( threads ) get all messages while others don't . But when I press ctrl c on pub, all subs( threads) gets messages and they exit as expecte
[14:12] kabs	sustrik: I tried different context for different threads and it worked!
[14:14] sustrik	you have to provide a minimal test case
[14:14] sustrik	so that people can look at what's going on
[14:26] benoitc	mmm what happen when socket we connect is closed at the end ? in case i do multiple connect ?
[14:26] benoitc	is this cleanly closed ?
[14:28] toni	hi there. I am using the pyzmq binding. I have to pack an auto-generated envelope as part of the payload. My payload format is json. The encode raises a "'utf8' codec can't decode byte". Any hints how I could solve the problem?
[14:28] toni	I know, this is more python or encoding specific than it is zmq-specific, but maybe someone can help me?
[14:29] benoitc	ther is an encode_string option in json.dumps
[14:29] benoitc	or you can do json.dumps(..).encode('utf-8')
[14:29] benoitc	smth like it
[14:32] sustrik	benoitc: it's cleanly closed
[14:32] benoitc	cool, thanks
[14:34] toni	benoitc: the real problem is the envelope ("...ï¿½ï¿½8ï¿½ï¿½ï¿½.."). I tried to encode it as utf-8, but as I do so, I get a UnicodeDecodeError t0o.
[14:37] benoitc	ah
[14:38] benoitc	if you have some binary in your json you will have to base64 it
[14:55] toni	benotitc: thanks, Ill try this
[15:07] kabs	Hello, I tried pub-sub model with both pub and sub on same machine,so, one pub is running infinite loop and sending data, I tried 50 subs first and they were able to get data, then I tried increasing number of subs to 500 and it gave "to many files open error", I changed the ulimit to 65535 and it worked, now I want subs to be around 5000. How can I scale my pub-sub model??
[15:09] kabs	with 5000 subs , it took lot of time for even a single subscriber to get data and once subscribers started getting data, it hanged after sometime. I am running subs in threads. Any help in increasing scalability??
[15:13] spht	kabs: 5000 threads? Have you lowered your thread stack size?
[15:14] kabs	Yes 5000 threads, no didn't lower that, don't know about it
[15:14] spht	kabs: on linux IIRC the default thread stack size is 8mb....
[15:15] spht	kabs: but if you want to run a lot of clients on a single machine I would probably use an async mechanism instead of threading and then have worker threads for any computation needed
[15:15] kabs	spht: actually I want to know is as subscriber increases how can I increase the scalability of pub-sub model
[15:16] kabs	spht: since in my application , number of subs can increase to any number
[15:16] kabs	spht: can you give me more info on "async mechanism instead of threading and then have worker threads for any computation needed" or some link where I can read this stuff you mentioned
[15:17] spht	kabs: what are these clients, processes on the same machine? Remote machines?
[15:17] kabs	spht: same machine, I am making test programs to learn this model using zeroMQ
[15:18] sustrik	kabs: the error means there are too much connections open
[15:19] sustrik	check your OS settings for the max value
[15:19] kabs	sustrik: yes, so how can I go for scalable system, increasing max value everytime doesn't seems to be right way of doing it
[15:19] sustrik	shrug, if your system limit is 1000 sockets, you cannot handle 5000 connections
[15:20] spht	kabs: it's extremely hard to write good test cases for c10k-problems, especially on a single machine. That said, on quick solution to minimize the # of threads used would be to create several sockets from every thread and monitor them with zmq_poll
[15:22] kabs	spht: didn't understood you solution well, can you please elaborate or send some link that mentions this??
[15:23] spht	kabs: For general c10k info: http://www.kegel.com/c10k.html // for zmq_poll see docs
[15:24] spht	the point is, for a large number of threads, the overhead of the threads themselves are significant, both for the kernel context switching and memory use
[15:30] kabs	spht: thanks! will read about zmq_poll and will see if I can think of scaling my system
[19:59] drbobbeaty	Question about ZeroMQ 2.1.0 (from the git repo) on Ubuntu 10.04.1... I have a program that's running fine on CentOS 5, but on Ubuntu 10.04.1 when I try to open the URL epgm://bond0;225.1.1.1;77777 I get the message: The protocol is not compatible with the socket type.
[20:00] drbobbeaty	Is this some kind of IPV6/IPV4 issue? Or something that's been seen.
[20:00] drbobbeaty	I don't see anything in my Google searches on this.
[20:28] sustrik	drbobbeaty: it's that mutlicast makes sense only with PUB/SUB sockets
[20:31] drbobbeaty	sustrik: but when I run the exact same code - as a SUB, in this case, on CentOS5 it's fine. Have you heard of any CentOS vs. Ubuntu differences?
[20:32] sustrik	it's a SUBV socket and epgm transport and you get "The protocol is not compatible with the socket type" error?
[20:34] sustrik	last version of 0MQ from github?
[20:35] drbobbeaty	It's a SUB socket and it's the latest on github for Ubuntu. For CentOS, it's a week or so ago off github, but still "post-2.1.0".
[20:36] drbobbeaty	Works great on CentOS, fails with that error on Ubuntu. I have boxes running side-by side.
[20:52] sustrik	there were some recent changes that may have caused the behaviour
[20:53] drbobbeaty	Really? OK... what should I do?
[20:53] sustrik	can you provide a minimal test case?
[20:53] sustrik	it should be easy
[20:53] sustrik	probably just create a socket and bind
[20:53] drbobbeaty	Sure. I'll try to write one up.
[20:53] sustrik	thanks
[21:41] drbobbeaty	sustrik: if you use this gist: https://gist.github.com/634738 , you can even delete lines 69 to 94, you'll get the same error I see. On CentOS5 it's fine. On Ubuntu 10.04.1 it fails on the connect() call (line 68).
[21:46] drbobbeaty	I'm going to send it to the mailing list as well... just in case you're gone for the day.
[22:31] benoitc	hum /win 8
[22:36] pythonirc101	can i use zeromq to communicate between two machines behind NATs?
[23:56] mikko	pythonirc101: what do you mean?