| [Time] Name | Message |
|
[00:27] sig
|
It seems like I'm missing messages with a PUSH/PULL socket. It only is the first message too. I recall reading about some gotcha, is this close?
|
|
[01:03] indygreg
|
sig: have you seen the flowchart on the guide w.r.t. lost messages?
|
|
[01:04] sig
|
I havn't
|
|
[01:07] indygreg
|
sig: http://zguide.zeromq.org/page:all#Missing-Message-Problem-Solver
|
|
[01:09] sig
|
appreciate, i'll give it a read
|
|
[06:27] sakella
|
anyone online?
|
|
[06:30] sakella
|
i have a q about the push pull internals.
|
|
[14:50] PaoloVictor
|
hello, could anyone help me with a ZMQ_LINGER problem with the C API?
|
|
[14:53] PaoloVictor
|
specifically, why this fails with an "Invalid argument" errno: http://pastebin.com/XHGLHPF4
|
|
[15:00] cremes
|
PaoloVictor: compare your code to the examples in zguide; i'm pretty sure in line 6
|
|
[15:00] cremes
|
that you need to pass 's' by reference like so: &s
|
|
[15:00] cremes
|
hmmm, nope, that isn't it
|
|
[15:02] cremes
|
PaoloVictor: oh, the problem is line 2
|
|
[15:02] cremes
|
you need to pass a context to zmq_socket() but you are passing a transport string
|
|
[15:03] cremes
|
so the subsequent call to zmq_setsockopt fails because publishSocket isn't a valid socket
|
|
[15:03] cremes
|
change line 2: void* publishSocket = zmq_socket(zmqContext, ZMQ_PUB);
|
|
[15:06] PaoloVictor
|
oh, sorry, that was a copy mistake (the original code has some portuguese parts). Here's the corrected version: http://pastebin.com/vyB1Lxdv
|
|
[15:06] PaoloVictor
|
the socket is initialized correctly (after zmq_socket, publishSocket is not NULL and errno != 0)
|
|
[15:08] PaoloVictor
|
I've tried copying the zmq guide example (setting the affinity), but it also fails
|
|
[15:09] PaoloVictor
|
oh, wait
|
|
[15:09] cremes
|
PaoloVictor: what version of 0mq are you using? 2.1.10 or 3.0?
|
|
[15:09] PaoloVictor
|
I guess I've got it
|
|
[15:09] cremes
|
what was the issue?
|
|
[15:10] PaoloVictor
|
my failure condition is: if(*publishSocket == NULL || errno != 0) { return FAILURE; }
|
|
[15:10] cremes
|
yeah, that won't work
|
|
[15:10] PaoloVictor
|
but i should check zmq_setsockopt's return value, instead
|
|
[15:10] cremes
|
right
|
|
[15:10] PaoloVictor
|
but why does it update errno?
|
|
[15:10] cremes
|
errno should be ignored *unless* the return code is -1
|
|
[15:11] PaoloVictor
|
okay, thanks!
|
|
[15:11] cremes
|
you are welcome!
|
|
[15:55] PaoloVictor
|
http://judao.mtv.uol.com.br/televisao/cavaleiros-do-zodiaco-armaduras-de-ouro-de-verdade-voce-ta-chorando-eu-sei/
|
|
[16:02] sig
|
I think I figured out my "lost messages" problem but would like to confirm. I am using ipc socket between multiple processes. Two processes read messages and PUSH them. These two processes call bind(). The other set are the workers, who PULL messages. The problem I suspect is inbound connections from the workers are load balanced, so some workers sit idle. Does this make sense?
|
|
[16:03] sig
|
I'm trying to confirm the need to have a device read from both sockets and send to one or not
|
|
[16:16] indygreg
|
sig: yes, that makes sense to me
|
|
[16:16] indygreg
|
you have to synchronize the workers to ensure both are available before you start sending messages or 1 worker will likely get starved
|
|
[16:17] indygreg
|
this is talked about extensively in the zguide
|
|
[16:21] sig
|
well that's not exactly what I mean. I do take care to synchronize. The problem is process A1 and A2, the PUSH procs both send messages. However the clients are load balanced on connect, correct? So some workers end up connected to A1 and some end up connected to A2?
|
|
[16:21] sig
|
perhaps an example would be better, it make take me a minute to write
|
|
[16:21] PaoloVictor
|
one of the examples does exactly that (sync the workers before sending messages), but I can't find it right now
|
|
[16:22] cremes
|
sig: i don't understand what you are asking about the load balancing
|
|
[16:22] cremes
|
you have A1 and A2 that both bind to different ports/transports, right?
|
|
[16:22] sig
|
k, I will write some example code. I may have a bug too =P
|
|
[16:22] sig
|
A1 and A2 bind to the same socket (ipc)
|
|
[16:22] cremes
|
and then your workers are connecting to *both* A1 and A2, yes?
|
|
[16:22] sig
|
and workers connect to the same socket
|
|
[16:22] cremes
|
um... is that legal?
|
|
[16:22] sig
|
workers connect only once
|
|
[16:22] cremes
|
i don't think you can have 2 sockets bind to the *same* transport
|
|
[16:23] sig
|
they are separate procs
|
|
[16:23] cremes
|
so?
|
|
[16:24] cremes
|
imagine 2 separate procs that both try to bind to localhost:5555
|
|
[16:24] cremes
|
one of them should fail
|
|
[16:24] cremes
|
only 1 socket system-wide should be able to bind a shared resource like a tcp or ipc socket
|
|
[16:25] sig
|
ok, still a newb. I will make them different sockets and use poll from workers
|
|
[16:25] cremes
|
whereas there is no limit to how many sockets can connect to it
|
|
[16:26] sig
|
rgr, makes sense. I mis interpreted something I read elsewhere. I think it was a socket can bind more then once to different transports
|
|
[16:27] cremes
|
yes
|
|
[16:27] cremes
|
you can bind to different inproc, different ipc and different tcp transports
|
|
[16:27] cremes
|
all on the same socket
|
|
[16:28] cremes
|
but 2 sockets can't bind the *same* transport even if they are in other processes (except for inproc... it's
|
|
[16:28] cremes
|
unique to each *context*)
|
|
[16:44] sakella
|
good morning, if I have a push client (connect) and two pull servers (bind), and one of the servers is not running, will push still try to send to both servers? How does it work?
|
|
[17:18] cremes
|
sakella: has the push client connected to both pull servers?
|
|
[17:19] cremes
|
if so, when one of them goes down, the push socket will know not to fair-balance any more messages
|
|
[17:19] cremes
|
to it
|
|
[17:19] cremes
|
if you want to know the mechanics of it, i recommend you read the guide (http://zero.mq/zg) and read the source
|
|
[17:45] jmslagle
|
Hrm!
|
|
[17:48] jmslagle
|
So it's possible to get to session.cpp:98 with msg->content not having been freed yet.
|
|
[17:48] jmslagle
|
Which then calls zmq_msg_init and resets content to 0x20
|
|
[17:48] jmslagle
|
And leaks the former contents..
|
|
[17:54] sig
|
Ugh, I'm not seeing evenly balanced connections with PUSH/PULL
|
|
[17:55] sig
|
If I poll on one side and send without poll on the other, will that cause unbalanced distribution?
|
|
[17:57] cremes
|
poll has nothing to do with load balancing
|
|
[18:08] sakella
|
hi <cremes> push has not connected to both pull servers. When push comes up, already one of the pull servers is down
|
|
[18:08] cremes
|
sakella: ok, then the push server will only send to the pull server that connected
|
|
[18:08] cremes
|
why would you think otherwise?
|
|
[18:08] sakella
|
In this case what happens to the messages at the push client.
|
|
[18:19] jmslagle
|
https://zeromq.jira.com/browse/LIBZMQ-274
|
|
[18:19] jmslagle
|
There
|
|
[18:56] sig
|
cremes: I didn't think so, I have to code up a test case. Still doing something wrong =/
|
|
[19:20] Chris
|
Good evening
|
|
[19:20] Chris
|
is anyone able to help me with zmq java bindings?
|
|
[19:21] Guest40348
|
I'm having some difficulty getting jzmq working on a specific machine
|
|
[19:23] cremes
|
Guest40348: i could help with ruby or C but not for java... :(
|
|
[19:24] Guest40348
|
:( indeed
|
|
[19:24] cremes
|
Guest40348: i recommend you gist the errors you are getting, explain your environment, etc so
|
|
[19:24] cremes
|
other lurkers might be able to chime in and lend a hand
|
|
[19:24] cremes
|
asking if you can ask a question is a good way to never get an answer
|
|
[19:25] Guest40348
|
I am trying to run zmq with the java bindings on a Windows machine
|
|
[19:26] Guest40348
|
currently I am getting unsatisfied link exception (cannot find dependent library)
|
|
[19:26] Guest40348
|
I'm pretty sure I've got the config right
|
|
[19:26] Guest40348
|
as I have 0MQ lib folder in my PATH
|
|
[19:26] Guest40348
|
and -Djava.library.path set to jzmq.dll folder
|
|
[19:39] PaoloVictor
|
I use ZMQ's java bindings, but on linux
|
|
[19:39] PaoloVictor
|
your setup seems right
|
|
[19:39] PaoloVictor
|
how did you build the bindings?
|
|
[19:40] PaoloVictor
|
also, ZMQ's dll also must be on the -Djava.library.path
|
|
[19:50] Guest40348
|
just the ZMQ dll, or also the .lib file?
|
|
[19:50] Guest40348
|
I built the bindings on a different windows PC using VS C++ 2010
|
|
[19:52] Guest40348
|
Also, I used DependencyWalker to check out jzmq.dll
|
|
[19:52] Guest40348
|
everything seems fine - just one warning for MSJAVA.dll which I've read is not a problem
|
|
[19:52] PaoloVictor
|
I don't have much experience on using native libs on windows, but I'd try both
|
|
[19:53] PaoloVictor
|
jzmq does need zmq on linux, though
|
|
[19:53] PaoloVictor
|
*need = depend on
|
|
[19:53] Guest40348
|
yes, I've now dumped all the zmq files into the path specified in -Djava.library.path
|
|
[19:53] Guest40348
|
still the same error
|
|
[19:54] Guest40348
|
....\jzmq.dll: Can't find dependent libraries
|
|
[19:55] PaoloVictor
|
Did you add http://lists.zeromq.org/pipermail/zeromq-dev/2009-March/000435.html
|
|
[19:55] PaoloVictor
|
oops
|
|
[19:55] PaoloVictor
|
I meant, that guy had the same problem
|
|
[19:56] Guest40348
|
yea I had read that post
|
|
[19:56] PaoloVictor
|
you know, more people should let others know when/how they fix their problems
|
|
[19:56] Guest40348
|
I have both jzmq.dll and libzmq.dll in my java.library.path
|
|
[19:56] PaoloVictor
|
"thanks I fixed it" posts with no further information are infuriating
|
|
[19:57] Guest40348
|
indeed
|
|
[19:57] Guest68310
|
should it be possible to create a REQ socket in a secondary thread (using python bindings)?
|
|
[20:01] PaoloVictor
|
have you tried this? http://stackoverflow.com/questions/1087054/jni-dependent-libraries
|
|
[20:02] jmslagle
|
Ug
|
|
[20:02] jmslagle
|
This memory leak is infuriating :)
|
|
[20:02] Guest40348
|
no, looks horrible but will try
|
|
[20:02] PaoloVictor
|
it seems like a common problem with using native libs on windows. yuck
|
|
[20:03] jmslagle
|
I "patched" it but now I get a segv that looks suspciously like a race condition.
|
|
[20:04] Guest40348
|
It works! Thanks for the link Paolo. But this is seriously horrible....looks like for dependent DLL's you need to have them in a directory explicitly referenced on the PATH, java.library.path is not enough
|
|
[20:04] Guest40348
|
windows sucks
|
|
[20:05] jmslagle
|
I actually seem to recall seeing that in a FAQ somewhere
|
|
[20:05] jond
|
jmslagle: patched where? I looked and there are some suspicious comments in one_byte_message_ready and the eight_byte equiva
|
|
[20:06] jmslagle
|
jond: I added a zmq_msg_close in session.cpp right before the zmq_msg_init
|
|
[20:06] jmslagle
|
But, it seems like something else is talking to content after that.
|
|
[20:06] jmslagle
|
(The JNI recv call)
|
|
[20:07] jond
|
jmslagle: are you able to reproduce the problem as a c program , because that's the normally the quickest way to get these matters resolved
|
|
[20:08] jmslagle
|
We're porting it to C++ now
|
|
[20:08] jmslagle
|
I think xreq is going odd things.
|
|
[20:09] PaoloVictor
|
Guest40348: You're welcome. Some guy suggests using System.loadLibrary directly: http://www.eclipsezone.com/eclipse/forums/t64011.html
|
|
[20:09] PaoloVictor
|
Guest40348: just out of curiosity, is there a MSJava.dll on the PATH?
|
|
[20:09] jond
|
jmslagle: that's great , attach it to that issue you've already raised.
|
|
[20:10] jmslagle
|
Will do :)
|
|
[20:10] jmslagle
|
If I can reproduce it in C++ I can likely fix it.
|
|
[20:10] jmslagle
|
Debugging JNI is just impossible.
|
|
[20:12] PaoloVictor
|
is creating a pure Java ZMQ port too much of a crazy idea?
|
|
[20:13] PaoloVictor
|
(or stupid)
|
|
[20:14] jmslagle
|
Actually, a native java lib would be awesome..
|
|
[20:15] Guest40348
|
no, MSJava was nowhere ont he machine
|
|
[20:15] Guest40348
|
but looking at various info pages on DependencyWalker, it seems this should not be a problem
|
|
[20:16] Guest40348
|
as it is a non-mandatory (lazy load) library
|
|
[20:17] Guest40348
|
It would be interesting to compare performance of the JNI zmq implementation and a pure Java version
|
|
[20:17] jmslagle
|
I would give up some performance to ditch JNI :P
|
|
[20:17] Guest40348
|
i.e. does the overhead of a JNI call outweigh the process the JVM needs to go through to access the raw ports
|
|
[20:17] jmslagle
|
If I'm concerned about performance I'm usually not using java anyways ;)
|
|
[20:18] PaoloVictor
|
I started using ZMQ for the performance, stayed for the bindings
|
|
[20:18] Guest40348
|
I always thought there must be a good reason why middleware providers such as Tibco & 29West always shipped a DLL and JNI wrapper rather than pure Java implementation
|
|
[20:18] PaoloVictor
|
I need to to some C <-> Java communication and ZMQ seems to be the most stable/complete solution
|
|
[20:18] jmslagle
|
Yeah. The thing that got me here vs rabbit for instance was the variety of languages
|
|
[20:19] jmslagle
|
With a pure java implementation I could do pl/sql bindings :)
|
|
[20:19] jmslagle
|
Linking C++ into oracle is a PITA :P
|
|
[20:20] PaoloVictor
|
hm, I might just give it a try. At the very least it'd be a nice programming exercise
|
|
[20:20] jmslagle
|
So, ZMQ always has at min 3 threads
|
|
[20:20] jmslagle
|
A reaper, at least 1 IO
|
|
[20:20] Guest40348
|
does the Java socket api allow you to get to a low enough level?
|
|
[20:20] jmslagle
|
What's the other?
|
|
[20:21] jmslagle
|
I presume the main one calling zmq_init
|
|
[20:22] PaoloVictor
|
Guest40348: how low, exactly? There are libraries even for raw ethernet access: http://stackoverflow.com/questions/3574036/java-library-for-raw-ethernet/3585419#3585419
|
|
[20:23] jmslagle
|
Oracle used to ship An OCI driver (JNI), but they recommend the thin driver now which is native java
|
|
[20:24] Guest40348
|
that's pretty cool
|
|
[20:24] crankycoder
|
PaoloVictor: why don't you just use JNA ?
|
|
[20:27] Guest40348
|
surely it's better to avoid native libraries if possible?
|
|
[20:28] crankycoder
|
why?
|
|
[20:28] crankycoder
|
it's just c. what's the big deal?
|
|
[20:29] PaoloVictor
|
crankycoder: not a big deal, really, only that sometimes people have trouble managing dependencies, mainly on windows
|
|
[20:30] PaoloVictor
|
crankycoder: also, it'd be nice not having to compile the library and bindings in order to use it, but it's just a minor annoyance
|
|
[20:30] crankycoder
|
Guest40348: but you're using zeromq already. jna is like python's ctypes - you're just including libffi and then making pure java calls into C
|
|
[20:30] PaoloVictor
|
hm
|
|
[20:31] crankycoder
|
it's hella better than implementing your own codebase.
|
|
[20:31] jmslagle
|
Actually, the bigger reason is the nightmare it creates for instance on tomcat
|
|
[20:31] jmslagle
|
Since Java can only dlload a native lib once.
|
|
[20:32] jmslagle
|
So if you bundle zmq with your app, it does a horrile death if you redeploy
|
|
[20:32] jmslagle
|
There are workarounds, but that's one example of the PITA it causes :)
|
|
[20:32] Guest40348
|
I need to bail, but thanks for your help getting this working. I'll be back another time for such debates:)
|
|
[20:33] crankycoder
|
jmslagle: fair enough. but that seems like an argument to not use 0mq in the first place with java
|
|
[20:34] jmslagle
|
That is true
|
|
[20:35] PaoloVictor
|
funny, I'm using jzmq on tomcat and I didn't have redeploy problems - yet
|
|
[20:35] crankycoder
|
fwiw - i'm looking at jna+0mq myself
|
|
[20:35] crankycoder
|
probably in a month or two
|
|
[20:36] jmslagle
|
The richness in language bindings is the reason I went there
|
|
[20:36] CIA-79
|
jzmq: 03Richard Smith 07master * ra547cf3 10/ (6 files in 2 dirs): Added ZContext, ZFrame and ZMsg classes ...
|
|
[20:36] CIA-79
|
jzmq: 03Richard Smith 07master * r34b4150 10/ src/org/zeromq/ZMQ.java : Merge remote-tracking branch 'upstream/master' - http://git.io/az5igw
|
|
[20:36] CIA-79
|
jzmq: 03Gonzalo Diethelm 07master * r121eed5 10/ (6 files in 2 dirs): Merge pull request #77 from rjsmith/master ...
|
|
[20:36] crankycoder
|
there's bits on github, but i haven't tried it yet, but i've done quite a bit with jna before - so i can polish it off if it's not complete
|
|
[20:36] jmslagle
|
The java ones seem less used/tested though
|
|
[20:36] crankycoder
|
jmslagle: but that's the point. JNA would just use the 'real' 0mq C library
|
|
[20:37] jmslagle
|
Yeah
|
|
[20:38] PaoloVictor
|
I didn't think about using JNA before, thanks for the tip, crankycoder
|
|
[20:55] CIA-79
|
jzmq: 03gonzalo diethelm 07master * r040c4c6 10/ README : Clarified license for jzmq (it is LGPL). - http://git.io/Ibas6g
|
|
[22:13] sig
|
Sigh, My connections are not load balancing. I do not get it. PUSH/PULL should balance PUSH messages, no?
|
|
[22:17] mikko
|
sig: they should be round robined between connected pull sockets
|
|
[22:24] sig
|
Perhaps I miss what exactly it means. Must I call connect each time for it to load balance?
|
|
[22:24] sig
|
I am calling send_pyobj from the side I bind, workers do not seem to balance evenly
|
|
[22:26] mikko
|
sig: it won't be even balancing
|
|
[22:28] sig
|
why not? ZMQ_PUSH says Outgoing routing strategy Load-balanced
|
|
[22:29] jsimmons
|
load balancing isn't even balancing
|
|
[22:29] jsimmons
|
try slowing down your worker artificially and see if it balances work then
|
|
[22:30] sig
|
It does not, that's what i'm doing
|
|
[22:30] jsimmons
|
so the other connected workers are not getting anything?
|
|
[22:30] cremes
|
sig: you are going to have to show some code
|
|
[22:30] cremes
|
push/pull balances great for me
|
|
[22:31] mikko
|
if you haven't got HWM set the results will vary
|
|
[22:31] sig
|
HWM is default atm, I will boil it down into a simple test case
|
|
[22:31] cremes
|
agreed; i usually set HWM to 1 on both the push and pull sockets (even though the docs for pull
|
|
[22:31] cremes
|
say it's ignored...)
|
|
[22:31] sig
|
interesting
|
|
[22:48] sig
|
So if I have 5 workers, would I set HWM to 5 to get the best-balance?
|
|
[22:48] sig
|
Setting hwm definitely affects what i'm seeing
|
|
[22:48] cremes
|
sig: experiment and let us know (really, that's the best way for you to learn this stuff)
|
|
[22:49] sig
|
I'd rather read the behavior somewhere. HWM doesn't exactly say how it affects push/pull sockets
|
|
[23:07] cremes
|
you'd rather read about the behavior? what good is that when you can write some code and see for yourself
|
|
[23:07] cremes
|
docs are sometimes wrong
|
|
[23:22] sig
|
Lets agree to disagree then =P
|
