February 2007 Archives
Tue Feb 20 09:17:17 CET 2007
OOoCon 2007 will be in Barcelona!
The voting of the OpenOffice.org community is now closed and the
conference will be in Barcelona, Spain.
I always enjoyed the city (cathedral, la Rambla, Parc de Montjuic, Corrida, ...) and the region to the north (early mornings watching the sun rise from the beach and especially watching night fireworks in Blanes from the boat with my wife made my love for Costa Brava) so let it be the best conference ever!
... and see you there!
I always enjoyed the city (cathedral, la Rambla, Parc de Montjuic, Corrida, ...) and the region to the north (early mornings watching the sun rise from the beach and especially watching night fireworks in Blanes from the boat with my wife made my love for Costa Brava) so let it be the best conference ever!
... and see you there!
Mon Feb 19 22:06:21 CET 2007
"ccache" effect on repeated builds
Just to proof my experimentally gained feeling, I did the exact
measurement of ccache usage
effect.
I measured five parallel (10/4) builds and the average values is in the following graph:
Build times are one hour, one hour and 3 minutes and 27 minutes. So the worst case with ccache and cold cache where ccache is populating the cache took 3 minutes (1/20th of the total build time).
This is the last part of our "highly accurate measurements of random and inaccurate build times configuration" series (1, 2, 3, 4, 5). I'd again like to thank David Richards for his wonderful service and the possibility to test this machine.
I measured five parallel (10/4) builds and the average values is in the following graph:
Build times are one hour, one hour and 3 minutes and 27 minutes. So the worst case with ccache and cold cache where ccache is populating the cache took 3 minutes (1/20th of the total build time).
This is the last part of our "highly accurate measurements of random and inaccurate build times configuration" series (1, 2, 3, 4, 5). I'd again like to thank David Richards for his wonderful service and the possibility to test this machine.
Sun Feb 18 14:06:56 CET 2007
Build time dependency on memory
The build machine I have for testing is equipped with 16 GB of RAM. It
is a lot of memory. Expensive stuff.
So I decided to measure some parallel builds (build -P10, dmake -P4) with limited amount of memory and graphed it:
The interpretation is very easy - if you have 16GB of RAM, you do not need to limit the available memory to your operating system ;-) The rest is not that easy. It was very parallel build, and it is not efficient with 1G only (it took 41 minutes to build with 1G, 37 minutes with 2G, 32 minutes with 4G).
On the other side, having more than 8G was not needed at all. 8G build was 1731 seconds (average from five builds) and 16G was 1706 seconds. The gain is minimal.
What is remaining? I have to fix build on stock SUSE Linux 10.2 and select the right system based on the money we have now :-)
So I decided to measure some parallel builds (build -P10, dmake -P4) with limited amount of memory and graphed it:
The interpretation is very easy - if you have 16GB of RAM, you do not need to limit the available memory to your operating system ;-) The rest is not that easy. It was very parallel build, and it is not efficient with 1G only (it took 41 minutes to build with 1G, 37 minutes with 2G, 32 minutes with 4G).
On the other side, having more than 8G was not needed at all. 8G build was 1731 seconds (average from five builds) and 16G was 1706 seconds. The gain is minimal.
What is remaining? I have to fix build on stock SUSE Linux 10.2 and select the right system based on the money we have now :-)
Sun Feb 11 14:59:32 CET 2007
Build timing per module
I think that every OOo builder already knows from his daily practice,
that the slowest module to build is binfilter or helpcontent. But have
you ever tried to graph some data from your builds? It is very
interesting...
The following graph shows the complete build time split per module.
There are some interesting data in the graph:
The following graph shows the complete build time split per module.
There are some interesting data in the graph:
- the slowest build is in module binfilter the second slowest module are helpcontent2 and officecfg.
- these three modules together take one quarter of the complete build time
- another quarter of the build time is taken for building top-level modules sw, svx, sc, sd and Mozilla
Sun Feb 11 14:34:33 CET 2007
More build statistics and data
The next idea to try in our journey to get fast builds was to try
building everything in memory. For this purpose, we will compare the
data with our previous results. The fastest build done on disk with 13
parallel directories built at the same time was 1 hour 38 minute
without ccache or 33 minutes with ccache (ccache saving was
approx. 66% when building on disk).
The same builds done in memory (on tmpfs) were 1 hour and 35 minutes (without ccache) or 29 minutes (with ccache). This time, ccache was on the disk, so I tried with ccache cache in memory but the result was the same (I'm building on machine with 16GB or RAM, enough for buffer caches etc.)
To sum up if you are about to build a new build machine for Linux: it is better to have more (even slower) processors/cores than having the most powerful processors available! Instead of throwing your money out of the window, spend your time on optimizing your build parameters for your environment (and you can also help us with fixing parallel build issues ;-). This means that for our new build server, I'll target the slowest possible quad Xeon (5310), one single processor in a dual board.
The tmpfs gain is not that significant if you have enough RAM so it is not worth the effort. And to be able to build in RAM, you must have enough RAM anyway ;-)
If you have other ideas what to measure/compare on this machine, just ping me on IRC or via mail.
In the next blog entry, I'll analyze the overall build data a bit.
The same builds done in memory (on tmpfs) were 1 hour and 35 minutes (without ccache) or 29 minutes (with ccache). This time, ccache was on the disk, so I tried with ccache cache in memory but the result was the same (I'm building on machine with 16GB or RAM, enough for buffer caches etc.)
To sum up if you are about to build a new build machine for Linux: it is better to have more (even slower) processors/cores than having the most powerful processors available! Instead of throwing your money out of the window, spend your time on optimizing your build parameters for your environment (and you can also help us with fixing parallel build issues ;-). This means that for our new build server, I'll target the slowest possible quad Xeon (5310), one single processor in a dual board.
The tmpfs gain is not that significant if you have enough RAM so it is not worth the effort. And to be able to build in RAM, you must have enough RAM anyway ;-)
If you have other ideas what to measure/compare on this machine, just ping me on IRC or via mail.
In the next blog entry, I'll analyze the overall build data a bit.
Thu Feb 8 21:47:30 CET 2007
Benchmarking another server
After publishing my previous blog entries about dual Xeon machine, I
was offered to do similar benchmarking of the server (thanks to city
of Largo and Dave Richards) with four Xeons 3 GHz. Every CPU is
running (or least Linux kernel thinks so) at 2.4 GHz. We still do not
know why not at 3GHz. The Xeons are of older generation, based on
Pentium IV architecture though, not based on new Core architecture (at
least this is what I can read from the Linux kernel). The system is
equipped with 16GB of RAM.
The system is installed with openSUSE 10.2 so I was able to spot several issues that people using this (very good, BTW) version will see. I fixed them all in my build system (I'll fix them properly in the following days) and I was finally able to build without manual intervention (this time both DEBs and RPMs).
The first two builds were done to get some overview of the speed and to get motivated to get the total time down ;-)) Both were done with cold ccache cache.
The system was preconfigured with hyperthreading turned on, so the first idea was to compare the effect of this BIOS option on the build time. It was done with only one process, and the result was terrible 8) Full build took 8 hours and 1 minute. With hyperthreading off, it took only 7 hours and 9 minutes. So by turning hyperthreading off, the build was actually faster - we saved 52 minutes! But uniprocess build is probably the worst possible scenario for HT (for more info and benchmarks about HT, see IBM developerWorks' excellent article Hyper-Threading speeds Linux).
The next set of build numbers was generated using hyperthreading turned off and with ccache already populated with the previous builds and using -P argument to build (not dmake). As I read today, it is not module parallelism but directory parallelism.
I run 16 builds and measured the time of all builds (done with -P1, -P2, ..., -P16). The result is in the following graph:
There are several important points about the graph - the best
performance was generated using -P13 or -P14 in this configuration
(four dual processors). Adding more processes doesn't make sense.
The build time of the fastest build was 33 minutes (both DEBs and RPMs instsets and en-US language pack in both formats).
For the fun of it, I meassured the same statistics on my regular build server with single Pentium IV 2.8GHz machine:
So even on single processor machine, it does make sense to use more
parallel directory builds.
I'll collect the same statistics from these two machines, bus using the other parallel build method - in directory parallelism (dmake -P) and will blog about it tomorrow.
(And as always: this is not accurate statistics, I'm not filtering the data, I'm not doing repetitive measurements, etc.)
The system is installed with openSUSE 10.2 so I was able to spot several issues that people using this (very good, BTW) version will see. I fixed them all in my build system (I'll fix them properly in the following days) and I was finally able to build without manual intervention (this time both DEBs and RPMs).
The first two builds were done to get some overview of the speed and to get motivated to get the total time down ;-)) Both were done with cold ccache cache.
The system was preconfigured with hyperthreading turned on, so the first idea was to compare the effect of this BIOS option on the build time. It was done with only one process, and the result was terrible 8) Full build took 8 hours and 1 minute. With hyperthreading off, it took only 7 hours and 9 minutes. So by turning hyperthreading off, the build was actually faster - we saved 52 minutes! But uniprocess build is probably the worst possible scenario for HT (for more info and benchmarks about HT, see IBM developerWorks' excellent article Hyper-Threading speeds Linux).
The next set of build numbers was generated using hyperthreading turned off and with ccache already populated with the previous builds and using -P argument to build (not dmake). As I read today, it is not module parallelism but directory parallelism.
I run 16 builds and measured the time of all builds (done with -P1, -P2, ..., -P16). The result is in the following graph:
The build time of the fastest build was 33 minutes (both DEBs and RPMs instsets and en-US language pack in both formats).
For the fun of it, I meassured the same statistics on my regular build server with single Pentium IV 2.8GHz machine:
I'll collect the same statistics from these two machines, bus using the other parallel build method - in directory parallelism (dmake -P) and will blog about it tomorrow.
(And as always: this is not accurate statistics, I'm not filtering the data, I'm not doing repetitive measurements, etc.)
Mon Feb 5 11:18:01 CET 2007
New toy for testing purposes (building OpenOffice.org in 30 minutes)
We are about to buy new system for the community build server I'm
using to build OpenOffice.org. We have some money and as a pure
coincidence I have had access to a machine that would be close to our
"dream machine" over the weekend. But it is a bit over our budget ;-)
The machine is a dual Xeon 5160 3GHz (two dual core Xeons) with 4GB RAM and fast disks (15k RPM). The system was Debian with gcc 4.1, gcj.
I collected some data from my build system (no system stuff, with binfilter, with mozilla from source, with SDK, build --all in transex3 and then build --all in instsetoo_native, all on unxlngx6.pro environment, building DEB packages only).
en-US standard build without ccache: 3 hours 51 minutes.
The same build with hot ccache cache: 1 hour 41 minutes.
At this time we started to be a bit nervous because it was actually slower than our current Opteron based system which builds one language with hot ccache cache in 1 hour and 35 minutes even with slower disk subsystem and slower memory. But the cold ccache build is still twice as slow as on the Xeon based machine.
As the testing machine has four cores, we started to experiment with multi process builds. We first raised MAXPROCESS to 2 to allow two processes to run in parallel. This resulted in 1 hour and 26 minutes. Not that interesting gain...
So I decided to test module parallelism instead of in-module parallelism. This was more interesting :-)
With up to 2 modules built in parallel, the time was 55 minutes.
With up to 4 modules built in parallel, the time was 38 minutes.
With up to 8 modules built in parallel, the time was 34 minutes.
With up to 12 modules built in parallel, the time was 30 minutes.
The fastest build I was able to do was 29 minutes 59 seconds.
The effect of in-module parallelism was not that interesting, so "optimal" configuration was up-to-12 modules built in parallel. To confirm this, I started the build with ccache cache removed: 1 hour 13 minutes. The next build with hot ccache cache was again 30 minutes long.
To sum up: you *can* build OpenOffice.org in half an hour!.
The machine is a dual Xeon 5160 3GHz (two dual core Xeons) with 4GB RAM and fast disks (15k RPM). The system was Debian with gcc 4.1, gcj.
I collected some data from my build system (no system stuff, with binfilter, with mozilla from source, with SDK, build --all in transex3 and then build --all in instsetoo_native, all on unxlngx6.pro environment, building DEB packages only).
en-US standard build without ccache: 3 hours 51 minutes.
The same build with hot ccache cache: 1 hour 41 minutes.
At this time we started to be a bit nervous because it was actually slower than our current Opteron based system which builds one language with hot ccache cache in 1 hour and 35 minutes even with slower disk subsystem and slower memory. But the cold ccache build is still twice as slow as on the Xeon based machine.
As the testing machine has four cores, we started to experiment with multi process builds. We first raised MAXPROCESS to 2 to allow two processes to run in parallel. This resulted in 1 hour and 26 minutes. Not that interesting gain...
So I decided to test module parallelism instead of in-module parallelism. This was more interesting :-)
With up to 2 modules built in parallel, the time was 55 minutes.
With up to 4 modules built in parallel, the time was 38 minutes.
With up to 8 modules built in parallel, the time was 34 minutes.
With up to 12 modules built in parallel, the time was 30 minutes.
The fastest build I was able to do was 29 minutes 59 seconds.
The effect of in-module parallelism was not that interesting, so "optimal" configuration was up-to-12 modules built in parallel. To confirm this, I started the build with ccache cache removed: 1 hour 13 minutes. The next build with hot ccache cache was again 30 minutes long.
To sum up: you *can* build OpenOffice.org in half an hour!.