RE: [jade-develop] Question about JADE/LEAP performance.

Subject: RE: [jade-develop] Question about JADE/LEAP performance.
From: Bellifemine Fabio (Fabio.Bellifemine@TILAB.COM)
Date: Thu Aug 22 2002 - 12:13:15 MET DST

Florent, your results first of all show how difficult is running and analysing a benchmark.
For instance, I am very confused when I compare the results you got with
'1000 msg, J2SE, 2 Hosts (intra), 2JVM' and
'1000 msg, J2SE, 1 Hosts, 2 JVM'.
The only difference is that in the first case you are using a network while in the second case you are using the local loop, but you still get a difference of performance of almost 2 orders of magnitude! Which kind of network are you using and how heavy is the traffic condition? The performance should have improved instead of decreasing because with 2 hosts you double the CPU power available and threads can be scheduled sooner!
 
Regarding the leap between 10 and 11 couples, take care of the statistics. Executing the experiment once and using that value as the average value is a very bad assumption. You should repeat the same experiment a number of times and get the average value and analyse at least the variance in order to avoid rushed and maybe wrong conclusions.
 
Other issues to consider are the time precision you have on the Jeode JVM, optimization of the compilation, just-in-time compilation, ...
 
In a separate e-mail I will try to give you more precise indications and help you in performing this test.
Kindly, Fabio.
 
 

-----Original Message----
From: zze-PRIME Florent FTRD/DIH/LAN [mailto:florent.prime@rd.francetelecom.com]
Sent: 22 August 2002 11:41
To: Bellifemine Fabio; jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Subject: RE: [jade-develop] Question about JADE/LEAP performance.

Hi,
 
in fact, I have tested again with a JADE (without LEAP), and the results are close to those in the paper "Scalability and performance of JADE Message Transport System". I have obtained 129.29 ms for the average RTT, with 15 couples and 1000 message exchanged on 2 hosts.I have therefore a 50 factor between JADE and JADE-LEAP. I have modified a bit the file RoundTripSender in order to run the test on JADE-LEAP.
 
Line 33 replaced "import java.util.Vector;" by "import jade.util.leap.*;"
Line 89 replaced "static Vector times = new Vector();" by "static List times = new ArrayList();"
Line 105 replaced "currentValue = ( (Double)times.elementAt(i) ).doubleValue();" by "currentValue = ( (Double)times.get(i) ).doubleValue();"
 
With this, the bench works on pjava, iPAQ... I have modified the config files too, for example, with 2 agents :
 
#--JADE Properties--
gui = false
icp = jade.imtp.leap.JICP.JICPPeer(4100)
acc = jade.core.FullAcc
mainURL = jicp://10.193.66.17
main = false
agents=s0:benchmark.roundTripTime.RoundTripSender(r0, 1000, null, 2); \
s1:benchmark.roundTripTime.RoundTripSender(r1, 1000, null, 2)
#-END file
 
The result I have obtained are as follows :
 
1000 msg, J2SE, 2 Hosts (intra), 2JVM
# of couples RTT (ms)
1 202
2 410
4 876
5 1397
10 2190 (46 with JADE only)
15 6569 (129 with JADE only)
 
The result are the same if one host is an iPAQ with Jeode VM (pjava).
 
 

1000 msg, J2SE, 1 Hosts, 2 JVM
# of couples RTT (ms)
1 7.5
2 14.7
4 28.8
5 34.23
6 41.57
8 55.4
10 65.443
11 210.37 <---------- !!!
12 224.3
13 247.2
15 294.3
 
I can't explain the leap between the 10th and 11th couples. the progression seems linear beforehand, and linear afterhand, but why is there a leap ?
 
Thank you very much for your help.
 
Florent
 
 

-----Message d'origine-----
De : Bellifemine Fabio [mailto:Fabio.Bellifemine@TILAB.COM]
Envoyé : mercredi 21 août 2002 16:59
À : zze-PRIME Florent FTRD/DIH/LAN; jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Objet : RE: [jade-develop] Question about JADE/LEAP performance.

Florent, LEAP does change the transport mechanism infact.
The paper you mention reports the results for JADE 2.5. Evaluating the results for JADE-LEAP is very important but remind to take appropriate care in understanding what happens, e.g. disabling just-in-time compilation can already be a cause of the problem or using a busy or unreliable network (i.e. several retransmission for the same packeg) can be another or the LEAP protocol can also be or also the CPU power of the iPaq.
If you were interested into using some of your time-resources for implementing a similar benchmark in the JADE-LEAP environment let me know such that someone in the JADE Team can try to help you in the execution of the experiments and analysis of the results.
Kind regards, Fabio.
 

-----Original Message-----
From: zze-PRIME Florent FTRD/DIH/LAN [mailto:florent.prime@rd.francetelecom.com]
Sent: 21 August 2002 15:59
To: jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Subject: [jade-develop] Question about JADE/LEAP performance.

Hi jaders,

I have tried the benchmark addon for JADE, with JADE-LEAP 2.5 and java 1.3.1.03 (without just-in-time compilation). I have run it on a PIII 933 HP Kayak with 100 Mbits ethernet card. When I run the bench with one Jade platform, one host, and one or two JVM, the results I have obtained are very close to the results of the paper "Scalability and performance of JADE Message Transport System". But when I try to run the bench between two hosts, unfortunaly without a proper network for my two hosts, my results (average RTT) are higher, with a 20 factor !!! For example, with 2 hosts, only 15 couples and 1000 messages are exchanged by each couple, my average RTT is 6569 ms ! It should be around 300 ms according to the same paper. I know my network is not dedicated to my test, but the difference is too important. Is LEAP changing the transport mechanism ? I have exactly the same average RTT between my PC and an iPAQ running Jeode VM with 10 Mbits ethernet card ! Does anybody know why ?

Thank you very much for your help.

Florent

====================================================================
CONFIDENTIALITY NOTICE
This message and its attachments are addressed solely to the persons
above and may contain confidential information. If you have received
the message in error, be informed that any use of the content hereof
is prohibited. Please return it immediately to the sender and delete
the message. Should you have any questions, please contact us by
replying to MailAdmin@tilab.com. Thank you
====================================================================

Question about JADE/LEAP performance.

Florent, your results first of all show how difficult is running and analysing a benchmark.
For instance, I am very confused when I compare the results you got with
'1000 msg, J2SE, 2 Hosts (intra), 2JVM' and
'1000 msg, J2SE, 1 Hosts, 2 JVM'.
The only difference is that in the first case you are using a network while in the second case you are using the local loop, but you still get a difference of performance of almost 2 orders of magnitude! Which kind of network are you using and how heavy is the traffic condition? The performance should have improved instead of decreasing because with 2 hosts you double the CPU power available and threads can be scheduled sooner!
 
Regarding the leap between 10 and 11 couples, take care of the statistics. Executing the experiment once and using that value as the average value is a very bad assumption. You should repeat the same experiment a number of times and get the average value and analyse at least the variance in order to avoid rushed and maybe wrong conclusions.
 
Other issues to consider are the time precision you have on the Jeode JVM, optimization of the compilation, just-in-time compilation, ..
 
In a separate e-mail I will try to give you more precise indications and help you in performing this test.
Kindly, Fabio.
 
 
-----Original Message----
From: zze-PRIME Florent FTRD/DIH/LAN [mailto:florent.prime@rd.francetelecom.com]
Sent: 22 August 2002 11:41
To: Bellifemine Fabio; jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Subject: RE: [jade-develop] Question about JADE/LEAP performance.

Hi,
 
in fact, I have tested again with a JADE (without LEAP), and the results are close to those in the paper "Scalability and performance of JADE Message Transport System". I have obtained 129.29 ms for the average RTT, with 15 couples and 1000 message exchanged on 2 hosts.I have therefore a 50 factor between JADE and JADE-LEAP. I have modified a bit the file RoundTripSender in order to run the test on JADE-LEAP. 
 
Line 33 replaced "import java.util.Vector;"  by "import jade.util.leap.*;"
Line 89 replaced "static Vector times = new Vector();" by "static List times = new ArrayList();"
Line 105 replaced "currentValue = ( (Double)times.elementAt(i) ).doubleValue();" by "currentValue = ( (Double)times.get(i) ).doubleValue();"
 
With this, the bench works on pjava, iPAQ... I have modified the config files too, for example, with 2 agents :
 
#--JADE Properties--
gui                  = false
icp                  = jade.imtp.leap.JICP.JICPPeer(4100)
acc                 = jade.core.FullAcc
mainURL         = jicp://10.193.66.17
main               = false
agents=s0:benchmark.roundTripTime.RoundTripSender(r0, 1000, null, 2); \
s1:benchmark.roundTripTime.RoundTripSender(r1, 1000, null, 2)
#-END file
 
The result I have obtained are as follows :
 
1000 msg, J2SE, 2 Hosts (intra), 2JVM
# of couples        RTT (ms)
1                         202
2                         410
4                         876
5                         1397
10                       2190  (46 with JADE only)
15                       6569  (129 with JADE only)
 
The result are the same if one host is an iPAQ with Jeode VM (pjava).
 
 
1000 msg, J2SE, 1 Hosts, 2 JVM
# of couples        RTT (ms)
1                         7.5  
2                         14.7
4                         28.8
5                         34.23
6                         41.57
8                         55.4
10                       65.443
11                       210.37 <---------- !!!
12                       224.3
13                       247.2
15                       294.3
 
I can't explain the leap between the 10th and 11th couples. the progression seems linear beforehand, and linear afterhand, but why is there a leap ?
 
Thank you very much for your help.
 
Florent
 
 
-----Message d'origine-----
De : Bellifemine Fabio [mailto:Fabio.Bellifemine@TILAB.COM]
Envoyé : mercredi 21 août 2002 16:59
À : zze-PRIME Florent FTRD/DIH/LAN; jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Objet : RE: [jade-develop] Question about JADE/LEAP performance.

Florent, LEAP does change the transport mechanism infact. 
The paper you mention reports the results for JADE 2.5. Evaluating the results for JADE-LEAP is very important but remind to take appropriate care in understanding what happens, e.g. disabling just-in-time compilation can already be a cause of the problem or using a busy or unreliable network (i.e. several retransmission for the same packeg) can be another or the LEAP protocol can also be or also the CPU power of the iPaq.
If you were interested into using some of your time-resources for implementing a similar benchmark in the JADE-LEAP environment let me know such that someone in the JADE Team can try to help you in the execution of the experiments and analysis of the results.
Kind regards, Fabio.
 
-----Original Message-----
From: zze-PRIME Florent FTRD/DIH/LAN [mailto:florent.prime@rd.francetelecom.com]
Sent: 21 August 2002 15:59
To: jade-develop@sharon.cselt.it; leap-info@crm.mot.com
Subject: [jade-develop] Question about JADE/LEAP performance.

Hi jaders,

I have tried the benchmark addon for JADE, with JADE-LEAP 2.5 and java 1.3.1.03 (without just-in-time compilation). I have run it on a PIII 933 HP Kayak with 100 Mbits ethernet card. When I run the bench with one Jade platform, one host, and one or two JVM, the results I have obtained are very close to the results of the paper "Scalability and performance of JADE Message Transport System". But when I try to run the bench between two hosts, unfortunaly without a proper network for my two hosts, my results (average RTT) are higher, with a 20 factor !!! For example, with 2 hosts, only 15 couples and 1000 messages are exchanged by each couple, my average RTT is 6569 ms ! It should be around 300 ms according to the same paper. I know my network is not dedicated to my test, but the difference is too important. Is LEAP changing the transport mechanism ? I have exactly the same average RTT between my PC and an iPAQ running Jeode VM with 10 Mbits ethernet card ! Does anybody know why ?

Thank  you very much for your help.

Florent

====================================================================
CONFIDENTIALITY NOTICE
This message and its attachments are addressed solely to the persons
above and may contain confidential information. If you have received
the message in error, be informed that any use of the content hereof
is prohibited. Please return it immediately to the sender and delete
the message. Should you have any questions, please contact us by
replying to MailAdmin@tilab.com. Thank you
====================================================================

_______________________________________________ jade-develop mailing list jade-develop@sharon.cselt.it http://sharon.cselt.it/mailman/listinfo/jade-develop UNSUBSCRIBE INSTRUCTIONS AT http://jade.cselt.it/mailing.htm#unsubscribe

This archive was generated by hypermail 2a22 : Thu Aug 22 2002 - 23:03:11 MET DST