[REPORT] Performance Test Samba vs Windows

[Garming Sam via samba-technical]

There's some interesting observations that you can make about the
overall traffic, without looking at too much detail of the actual runs.

1) In terms of maximum traffic (which in our examples has been quite
LDAP based), Windows is at most 50% faster. This number likely shrinks
depending on whether or not the service levels (e.g. 99% success) at
such high load are acceptable or not.

2) Looking at median packet response times, we generally spend twice as
long on processing a packet. However, the difference between our 95
percentile values is often much smaller than this and some of the mean
values are actually at parity.

3) With small to moderate loads, Samba handles a similar amount of
operations per second to Windows. For the values shown, it actually
seems to get more throughput (although not incredibly so) for a good
chunk of the table shown.

It's actually kind of surprising that Samba doesn't perform all that
much worse than Windows does. It's certainly not off by an order of
magnitude (or two), although this testing has been run in the prefork
mode. Our previous analyses showed much smaller numbers and that's
likely due to the limitations of being stuck with a single-process.


Cheers,

Garming

Hi Team,
I've been trying to make the Samba performance test tool work against
Windows.
After a lot of effort, Garming and I fixed all the errors for
different packets, and finally it's working(fixes are merged into master).
Base on that, we can send traffic to Samba and Windows, get the
maximum load, and compare the performance.
script/traffic_replay -U Administrator%PASSWORD  --realm
krb.samba.site --workgroup KRB --fixed-password FIXED-PASSWORD -r 4 -S
4 traffic-sample-1-model.txt dc1.krb.samba.site 2>&1 >
traffic_replay_stats/dc1_r4_S4.txt
By changing the combination of different -r and -S, we can get test
results and save to files, then parse files to get summary tables as
below.
The number displayed is "*Successful operations per s**econd*", the 0
ones are combinations we think not important and skipped.
All data are attached, hope this can help the team to understand how
is Samba performing with compare to Windows.
--
Joe Guo
Catalyst IT

[William Brown via samba-technical]

As a comment, great work to get this test suite running against samba
and windows! It's awesome to see progress on this, and it will really
help in the future :)

On Mon, 2018-05-14 at 15:16 +1200, Garming Sam via samba-technical

Post by Garming Sam via samba-technical
There's some interesting observations that you can make about the
overall traffic, without looking at too much detail of the actual runs.
1) In terms of maximum traffic (which in our examples has been quite
LDAP based), Windows is at most 50% faster. This number likely
shrinks
depending on whether or not the service levels (e.g. 99% success) at
such high load are acceptable or not.

Great! I think there are lots of ways to improve LDB as well to help
close this gap further. I'll contribute some ideas and patches later
for this.

Post by Garming Sam via samba-technical
2) Looking at median packet response times, we generally spend twice as
long on processing a packet. However, the difference between our 95
percentile values is often much smaller than this and some of the mean
values are actually at parity.
3) With small to moderate loads, Samba handles a similar amount of
operations per second to Windows. For the values shown, it actually
seems to get more throughput (although not incredibly so) for a good
chunk of the table shown.

What about "large" loads?

Post by Garming Sam via samba-technical
It's actually kind of surprising that Samba doesn't perform all that
much worse than Windows does. It's certainly not off by an order of
magnitude (or two), although this testing has been run in the prefork
mode. Our previous analyses showed much smaller numbers and that's
likely due to the limitations of being stuck with a single-process.
Cheers,
Garming

Hi Team,
I've been trying to make the Samba performance test tool work against
Windows.
After a lot of effort, Garming and I fixed all the errors for
different packets, and finally it's working(fixes are merged into master).
Base on that, we can send traffic to Samba and Windows, get the
maximum load, and compare the performance.
script/traffic_replay -U Administrator%PASSWORD --realm
krb.samba.site --workgroup KRB --fixed-password FIXED-PASSWORD -r 4 -S
4 traffic-sample-1-model.txt dc1.krb.samba.site 2>&1 >
traffic_replay_stats/dc1_r4_S4.txt
By changing the combination of different -r and -S, we can get test
results and save to files, then parse files to get summary tables as
below.
The number displayed is "*Successful operations per s**econd*", the 0
ones are combinations we think not important and skipped.
All data are attached, hope this can help the team to understand how
is Samba performing with compare to Windows.
--
Joe Guo
Catalyst IT

[Garming Sam via samba-technical]

Post by William Brown via samba-technical

Post by Garming Sam via samba-technical
2) Looking at median packet response times, we generally spend twice as
long on processing a packet. However, the difference between our 95
percentile values is often much smaller than this and some of the mean
values are actually at parity.
3) With small to moderate loads, Samba handles a similar amount of
operations per second to Windows. For the values shown, it actually
seems to get more throughput (although not incredibly so) for a good
chunk of the table shown.

What about "large" loads?

Joe's original email seems to have been eaten by the list for now. There
was a table he had, which I'll see if I can get him to resend. There's
two axis of load that we generate, more operations along the same
connections + using the same client vs having more clients. As you ramp
up more operations per client, we start falling behind to Windows up to
the 50% number I mentioned. On the other hand, increasing the amount of
clients doesn't seem to have the same behaviour. Windows doesn't seem to
handle more clients (+ more open connections) much better than we do and
the gap between us and Windows does not appear to be quite so bad. After
a certain point, adding more simultaneous clients degrades the maximum
performance. It would be interesting to see what a ridiculous amount of
clients would compare between Samba and Windows.

Cheers,

Garming

Post by William Brown via samba-technical

Post by Garming Sam via samba-technical
It's actually kind of surprising that Samba doesn't perform all that
much worse than Windows does. It's certainly not off by an order of
magnitude (or two), although this testing has been run in the prefork
mode. Our previous analyses showed much smaller numbers and that's
likely due to the limitations of being stuck with a single-process.
Cheers,
Garming

Hi Team,
I've been trying to make the Samba performance test tool work against
Windows.
After a lot of effort, Garming and I fixed all the errors for
different packets, and finally it's working(fixes are merged into master).
Base on that, we can send traffic to Samba and Windows, get the
maximum load, and compare the performance.
script/traffic_replay -U Administrator%PASSWORD --realm
krb.samba.site --workgroup KRB --fixed-password FIXED-PASSWORD -r 4 -S
4 traffic-sample-1-model.txt dc1.krb.samba.site 2>&1 >
traffic_replay_stats/dc1_r4_S4.txt
By changing the combination of different -r and -S, we can get test
results and save to files, then parse files to get summary tables as
below.
The number displayed is "*Successful operations per s**econd*", the 0
ones are combinations we think not important and skipped.
All data are attached, hope this can help the team to understand how
is Samba performing with compare to Windows.
--
Joe Guo
Catalyst IT

[joeg--- via samba-technical]

It seems like some people have trouble to receive my original email
because of the images.

This is the text version:


***@wpts-s1-traffic-runner:~/samba$ python summary.py
 r\S         1        2        3        4        5        6       
7        8        9       10       11       12
  1          0        0        0        0        0        0       
0        0        0        0        0        0
  2          0        0        0        0        0        0       
0        0        0        0        0        0
  3          0        0        0        0        0        0       
0        0        0        0        0        0
  4          0        0        0   41.119   56.732   66.447   72.572  
76.956   95.436  102.396  110.454        0
  5          0        0        0   49.145   70.014   74.849   84.013  
89.707  107.108  116.287  103.106        0
  6          0        0        0   68.731   78.698   93.159   95.705 
115.973  114.349  112.678  130.282        0
  7          0        0        0   71.490   85.902  108.542  108.586 
113.590  126.788  119.140  101.968        0
  8          0        0        0   84.228  103.297  112.420  108.141 
112.146  125.492  109.455  101.155        0
  9          0        0        0   88.669  111.279  121.480  121.692 
138.248  115.464  131.850   98.632        0
 10          0        0        0  107.833  115.134  126.926  132.568 
116.088  136.371   97.090  126.787        0
 11          0        0        0  114.108  111.883  126.213  139.040 
141.352  106.134  124.904  118.583        0
 12          0        0        0  109.988  120.804  142.387  133.999 
124.241  131.333  107.382  111.447        0
 13          0        0        0  133.787  115.635  135.760  144.311 
113.029  109.257  124.848  135.680        0
 14          0        0        0  142.868  118.900  139.201  131.333 
132.071  113.330  116.351  118.728        0
 15          0        0        0  124.355  132.024  146.237  130.092 
107.181  125.418  121.196  126.104        0
 16          0        0        0  123.211  140.214  135.858  109.205 
146.865  137.027  111.136  121.051        0
 17          0        0        0  120.864  149.945  116.799  147.768 
119.295  126.733  119.915  107.164        0
 18          0        0        0  140.311  138.099  108.622  126.043 
127.352  143.128  134.762   93.017        0
 19          0        0        0  145.346  133.715  120.586  139.703 
120.015  118.425   83.349  106.214        0
 20          0        0        0  144.652  130.634  110.613  123.414 
151.610  135.753   93.042   85.515        0
 21          0        0        0  131.350  122.082  121.926  146.542 
132.719  126.720  120.016   91.363        0
 22          0        0        0  140.259  146.325  129.648  142.637 
142.712  132.569  124.193   84.189        0
 23          0        0        0 *169.764  139.553   91.046  137.663 
159.753  138.276  102.191  110.595        0
 24          0        0        0  148.510  142.059  141.010  137.082  
98.231   96.259  109.009   69.462        0
 25          0        0        0        0        0        0       
0        0        0        0        0        0

max: 169.764


***@wpts-s1-traffic-runner-win1:~/samba$ python summary.py
r\S        1        2        3        4        5        6       
7        8        9       10       11       12
 1         0        0        0        0        0        0       
0        0        0        0        0        0
 2         0        0        0        0        0        0       
0        0        0        0        0        0
 3         0        0        0        0        0        0       
0        0        0        0        0        0
 4         0        0        0   37.499   51.081   47.946   78.658  
73.416   92.505   95.842  100.820        0
 5         0        0        0   55.603   57.470   74.442   86.191 
113.089  111.861  115.396  142.818        0
 6         0        0        0   58.732   85.243   92.833  102.397 
108.455  136.802  134.110  136.791        0
 7         0        0        0   66.669   91.372  100.347  132.541 
159.121  137.524  166.432  131.660        0
 8         0        0        0   76.959  113.768  130.787  148.635 
140.318  155.053  185.398  172.079        0
 9         0        0        0  104.442  124.219  135.548  156.073 
159.233  161.505  165.132  111.609        0
10         0        0        0  121.444  128.814  170.306  172.741 
173.592  155.078  137.541  160.373        0
11         0        0        0  119.806  147.530  171.919  175.189 
156.816  189.379  176.344  155.873        0
12         0        0        0  103.682  145.108  174.202  172.044 
206.165  129.995  157.405  150.218        0
13         0        0        0  151.011  151.274  172.072  189.949 
161.779  176.042  137.990  140.948        0
14         0        0        0  146.589  166.755  213.144  189.248 
166.505  145.031  138.344  150.909        0
15         0        0        0  149.442  190.469  200.051  160.482 
143.324  151.437  141.908  130.576        0
16         0        0        0  173.891  202.137  204.066  172.500 
176.371  166.698  127.024  141.724        0
17         0        0        0  170.829  180.184  196.766  181.791 
147.663  155.474   99.551  127.230        0
18         0        0        0  188.009  191.858  155.009  172.549 
147.957  152.508  124.296  129.689        0
19         0        0        0  201.694  225.261  190.230  157.619 
152.969  136.545  130.671  125.775        0
20         0        0        0  203.151  209.354  165.844  166.927 
141.296  140.057  139.265  125.006        0
21         0        0        0  212.496  217.526  171.679  168.568 
168.101  145.787  111.281  106.252        0
22         0        0        0  184.216 *235.421  180.842  156.551 
158.798  149.434  115.148  113.642        0
23         0        0        0  201.699  210.266  191.049  153.199 
153.461  110.562  110.072  112.646        0
24         0        0        0  222.205  150.690  155.598  176.060 
137.402  123.665  115.973  106.362        0
25         0        0        0        0        0        0       
0        0        0        0        0        0

max: 235.421


And I should mention that Samba is running in prefork mode in this test:

    sudo bin/samba -M prefork --option='prefork children = 4'

Post by Garming Sam via samba-technical
There's some interesting observations that you can make about the
overall traffic, without looking at too much detail of the actual runs.
1) In terms of maximum traffic (which in our examples has been quite
LDAP based), Windows is at most 50% faster. This number likely shrinks
depending on whether or not the service levels (e.g. 99% success) at
such high load are acceptable or not.
2) Looking at median packet response times, we generally spend twice as
long on processing a packet. However, the difference between our 95
percentile values is often much smaller than this and some of the mean
values are actually at parity.
3) With small to moderate loads, Samba handles a similar amount of
operations per second to Windows. For the values shown, it actually
seems to get more throughput (although not incredibly so) for a good
chunk of the table shown.
It's actually kind of surprising that Samba doesn't perform all that
much worse than Windows does. It's certainly not off by an order of
magnitude (or two), although this testing has been run in the prefork
mode. Our previous analyses showed much smaller numbers and that's
likely due to the limitations of being stuck with a single-process.
Cheers,
Garming

Hi Team,
I've been trying to make the Samba performance test tool work against
Windows.
After a lot of effort, Garming and I fixed all the errors for
different packets, and finally it's working(fixes are merged into master).
Base on that, we can send traffic to Samba and Windows, get the
maximum load, and compare the performance.
script/traffic_replay -U Administrator%PASSWORD  --realm
krb.samba.site --workgroup KRB --fixed-password FIXED-PASSWORD -r 4 -S
4 traffic-sample-1-model.txt dc1.krb.samba.site 2>&1 >
traffic_replay_stats/dc1_r4_S4.txt
By changing the combination of different -r and -S, we can get test
results and save to files, then parse files to get summary tables as
below.
The number displayed is "*Successful operations per s**econd*", the 0
ones are combinations we think not important and skipped.
All data are attached, hope this can help the team to understand how
is Samba performing with compare to Windows.
--
Joe Guo
Catalyst IT

[joeg--- via samba-technical]

Oops, get wrapped. Attach the table in a text file this time.

Post by joeg--- via samba-technical
It seems like some people have trouble to receive my original email
because of the images.
 r\S         1        2        3        4        5        6       
7        8        9       10       11       12
  1          0        0        0        0        0        0       
0        0        0        0        0        0
  2          0        0        0        0        0        0       
0        0        0        0        0        0
  3          0        0        0        0        0        0       
0        0        0        0        0        0
  4          0        0        0   41.119   56.732   66.447   72.572  
76.956   95.436  102.396  110.454        0
  5          0        0        0   49.145   70.014   74.849   84.013  
89.707  107.108  116.287  103.106        0
  6          0        0        0   68.731   78.698   93.159   95.705 
115.973  114.349  112.678  130.282        0
  7          0        0        0   71.490   85.902  108.542  108.586 
113.590  126.788  119.140  101.968        0
  8          0        0        0   84.228  103.297  112.420  108.141 
112.146  125.492  109.455  101.155        0
  9          0        0        0   88.669  111.279  121.480  121.692 
138.248  115.464  131.850   98.632        0
 10          0        0        0  107.833  115.134  126.926  132.568 
116.088  136.371   97.090  126.787        0
 11          0        0        0  114.108  111.883  126.213  139.040 
141.352  106.134  124.904  118.583        0
 12          0        0        0  109.988  120.804  142.387  133.999 
124.241  131.333  107.382  111.447        0
 13          0        0        0  133.787  115.635  135.760  144.311 
113.029  109.257  124.848  135.680        0
 14          0        0        0  142.868  118.900  139.201  131.333 
132.071  113.330  116.351  118.728        0
 15          0        0        0  124.355  132.024  146.237  130.092 
107.181  125.418  121.196  126.104        0
 16          0        0        0  123.211  140.214  135.858  109.205 
146.865  137.027  111.136  121.051        0
 17          0        0        0  120.864  149.945  116.799  147.768 
119.295  126.733  119.915  107.164        0
 18          0        0        0  140.311  138.099  108.622  126.043 
127.352  143.128  134.762   93.017        0
 19          0        0        0  145.346  133.715  120.586  139.703 
120.015  118.425   83.349  106.214        0
 20          0        0        0  144.652  130.634  110.613  123.414 
151.610  135.753   93.042   85.515        0
 21          0        0        0  131.350  122.082  121.926  146.542 
132.719  126.720  120.016   91.363        0
 22          0        0        0  140.259  146.325  129.648  142.637 
142.712  132.569  124.193   84.189        0
 23          0        0        0 *169.764  139.553   91.046  137.663 
159.753  138.276  102.191  110.595        0
 24          0        0        0  148.510  142.059  141.010  137.082  
98.231   96.259  109.009   69.462        0
 25          0        0        0        0        0        0       
0        0        0        0        0        0
max: 169.764
r\S        1        2        3        4        5        6       
7        8        9       10       11       12
 1         0        0        0        0        0        0       
0        0        0        0        0        0
 2         0        0        0        0        0        0       
0        0        0        0        0        0
 3         0        0        0        0        0        0       
0        0        0        0        0        0
 4         0        0        0   37.499   51.081   47.946   78.658  
73.416   92.505   95.842  100.820        0
 5         0        0        0   55.603   57.470   74.442   86.191 
113.089  111.861  115.396  142.818        0
 6         0        0        0   58.732   85.243   92.833  102.397 
108.455  136.802  134.110  136.791        0
 7         0        0        0   66.669   91.372  100.347  132.541 
159.121  137.524  166.432  131.660        0
 8         0        0        0   76.959  113.768  130.787  148.635 
140.318  155.053  185.398  172.079        0
 9         0        0        0  104.442  124.219  135.548  156.073 
159.233  161.505  165.132  111.609        0
10         0        0        0  121.444  128.814  170.306  172.741 
173.592  155.078  137.541  160.373        0
11         0        0        0  119.806  147.530  171.919  175.189 
156.816  189.379  176.344  155.873        0
12         0        0        0  103.682  145.108  174.202  172.044 
206.165  129.995  157.405  150.218        0
13         0        0        0  151.011  151.274  172.072  189.949 
161.779  176.042  137.990  140.948        0
14         0        0        0  146.589  166.755  213.144  189.248 
166.505  145.031  138.344  150.909        0
15         0        0        0  149.442  190.469  200.051  160.482 
143.324  151.437  141.908  130.576        0
16         0        0        0  173.891  202.137  204.066  172.500 
176.371  166.698  127.024  141.724        0
17         0        0        0  170.829  180.184  196.766  181.791 
147.663  155.474   99.551  127.230        0
18         0        0        0  188.009  191.858  155.009  172.549 
147.957  152.508  124.296  129.689        0
19         0        0        0  201.694  225.261  190.230  157.619 
152.969  136.545  130.671  125.775        0
20         0        0        0  203.151  209.354  165.844  166.927 
141.296  140.057  139.265  125.006        0
21         0        0        0  212.496  217.526  171.679  168.568 
168.101  145.787  111.281  106.252        0
22         0        0        0  184.216 *235.421  180.842  156.551 
158.798  149.434  115.148  113.642        0
23         0        0        0  201.699  210.266  191.049  153.199 
153.461  110.562  110.072  112.646        0
24         0        0        0  222.205  150.690  155.598  176.060 
137.402  123.665  115.973  106.362        0
25         0        0        0        0        0        0       
0        0        0        0        0        0
max: 235.421
    sudo bin/samba -M prefork --option='prefork children = 4'

Post by Garming Sam via samba-technical
There's some interesting observations that you can make about the
overall traffic, without looking at too much detail of the actual runs.
1) In terms of maximum traffic (which in our examples has been quite
LDAP based), Windows is at most 50% faster. This number likely shrinks
depending on whether or not the service levels (e.g. 99% success) at
such high load are acceptable or not.
2) Looking at median packet response times, we generally spend twice as
long on processing a packet. However, the difference between our 95
percentile values is often much smaller than this and some of the mean
values are actually at parity.
3) With small to moderate loads, Samba handles a similar amount of
operations per second to Windows. For the values shown, it actually
seems to get more throughput (although not incredibly so) for a good
chunk of the table shown.
It's actually kind of surprising that Samba doesn't perform all that
much worse than Windows does. It's certainly not off by an order of
magnitude (or two), although this testing has been run in the prefork
mode. Our previous analyses showed much smaller numbers and that's
likely due to the limitations of being stuck with a single-process.
Cheers,
Garming

Hi Team,
I've been trying to make the Samba performance test tool work against
Windows.
After a lot of effort, Garming and I fixed all the errors for
different packets, and finally it's working(fixes are merged into master).
Base on that, we can send traffic to Samba and Windows, get the
maximum load, and compare the performance.
script/traffic_replay -U Administrator%PASSWORD  --realm
krb.samba.site --workgroup KRB --fixed-password FIXED-PASSWORD -r 4 -S
4 traffic-sample-1-model.txt dc1.krb.samba.site 2>&1 >
traffic_replay_stats/dc1_r4_S4.txt
By changing the combination of different -r and -S, we can get test
results and save to files, then parse files to get summary tables as
below.
The number displayed is "*Successful operations per s**econd*", the 0
ones are combinations we think not important and skipped.
All data are attached, hope this can help the team to understand how
is Samba performing with compare to Windows.
--
Joe Guo
Catalyst IT