Apache OpenOffice (AOO) Bugzilla – Issue 4190
Distributed make - lets speed things up!
Last modified: 2002-06-17 19:08:33 UTC
Currently, passing -P# to build or make when building a module will allow you to build that module in parallel. Some modules will cause grief when built in parallel, therefore need to be built sequentially. I have been investigating the matter and have compiled some patches that will eventually enable developers to use dmake to its full potential. We have seen dramatic results on IRIX performing a distributed make on a multiprocessor machine. I have identified most modules and directories that must be built sequentially in order to prevent the build from coming to a screeching halt, and have added a simple macro to enable the directory to be built sequentially. To force a sequential build, all that is simply needed is .SEQUENTIAL: {Target to be built sequentially} I have placed a conditional in solenv/inc/settings.mk, in which a sequential build is forced if the macro 'FORCE_SEQUENTIAL' is set to TRUE within the makefile including settings.mk. I realise that it would make more sense for this macro to be included in target.mk rather than settings, but it needs to be set early on, otherwise it wont work in some cases. I have created quite a few patches so far, and I expect to be creating more as I go along.
Adding ooo1.0 keyword
Ause, please review !
could you attach the first couple of patches? i wondering how the targets needed for .SEQUENTIAL get defined.
Created attachment 1431 [details] Patch for solenv/inc/settings.mk
Created attachment 1432 [details] Patch for berkleydb/makefile.mk
Created attachment 1433 [details] Patch for dtrans/source/cnttype/makefile.mk
Created attachment 1434 [details] Patch for fileacces/source/makefile.mk
Created attachment 1435 [details] Patch for framework/source/makefile.mk
Created attachment 1436 [details] Patch for framework/util/makefile.mk
Created attachment 1437 [details] Patch for i18npool/source/localedata/ascii/makefile.mk
Created attachment 1438 [details] Patch for i18npool/cource/localedata/CJK/makefile.mk
Created attachment 1439 [details] Patch for i18npool/util/makefile.mk
Created attachment 1440 [details] Patch for padmin/source/makefile.mk
Created attachment 1441 [details] Patch for product/examples/cpp/counter/makefile.mk
Created attachment 1442 [details] Patch for product/examples/cpp/remoteclient/makefile.mk
Created attachment 1443 [details] Patch for product/util/makefile.mk
Created attachment 1444 [details] Patch for stoc/source/corereflection/makefile.mk
Created attachment 1445 [details] Patch for stoc/source/defaultregistry/makefile.mk
I think a large number of these are just bugs in the makefile where sufficent dependencies are not defined. stoc and framework definately have worked in full pararllel mode previously.
Created attachment 1447 [details] Patch for stoc/source/implementationregistration/makefile.mk
Created attachment 1448 [details] Patch for stoc/source/inspect/makefile.mk
I suspect you are right Sander, but for now we are just trying to get a distributed make running without having to 'nurse' the build. Once this objective is met, we will focus on refinement, as we just want to get something in, so that we can build OOO_STABLE_1 in parallel 'out of the box' without any build issues.
Created attachment 1449 [details] Patch for stoc/source/invocation/adapterfactory/makefile.mk
Created attachment 1450 [details] Patch for stoc/source/invocation/makefile.mk
Created attachment 1451 [details] Patch for stoc/source/javaloader/makefile.mk
Created attachment 1452 [details] Patch for stoc/source/javavm/makefile.mk
Created attachment 1453 [details] Patch for stoc/source/loader/makefile.mk
Created attachment 1454 [details] Patch for stoc/source/namingservice/makefile.mk
Created attachment 1455 [details] Patch for stoc/source/proxy/factory/makefile.mk
Created attachment 1456 [details] Patch for stoc/source/registry/tdprovider/makefile.mk
Created attachment 1457 [details] Patch for stoc/source/servicemanager/makefile.mk
Created attachment 1458 [details] Patch for stoc/source/simpleregistry/makefile.mk
Created attachment 1459 [details] Patch for stoc/source/tdmanager/makefile.mk
Created attachment 1460 [details] Patch for stoc/source/typeconv/makefile.mk
Created attachment 1461 [details] Patch for stoc/test/excomp/makefile.mk
Created attachment 1462 [details] Patch for stoc/test/javavm/makefile.mk
Created attachment 1463 [details] Patch for stoc/test/makefile.mk
Created attachment 1464 [details] Patch for stoc/test/testsmgr/cpnt/makefile.mk
Created attachment 1465 [details] Patch for ucb/source/ucp/package/makefile.mk
Created attachment 1466 [details] Patch for vcl/source/gdi/makefile.mk
Well, thats all I have for now, but there will be more to follow. One thing to note is that a lot of these modules/directories will build in parallel when built singularly from their respective locations, but tend to break the build when a top level dmake is invoked.
Created attachment 1467 [details] Patch for connectivity/prj/build.lst
looking at the first couple of patches, it looks like you're treating the symptoms, not the cause. SLOTARGET itself is quite unlikely to cause problems when using "-Pn", searching for USE_SHLTARGET revealed no use of this variable (maybe USE_SHLnTARGET n=1..9 was intended). also in this case the problem is probably located somewhere in the dependency chain of the prerequisites, but in the dependencies of SHLnTARGET itself. on the other hand, these two parts of the build process (compiling and linking) would have the biggest advantage of using "-Pn". thus poluting the makefiles with extra variables which hide the problems for the cost of reduced build speed is not the approach i would prefer. if you have problems when building with "-Pn" i would like to help you tracking down the cause and fix it.
George, Ause, I don't think that this is a must for the ooo1.0 release, I would like reset the ooo1.0 keyword.
I am quite sure that USE_SHLTARGET is being used, as this does make a difference. If you take a look into solenv/inc/_tg_shl.mk, approx line 4260, you will see that the macro is assigned the name of the shared library being built. As for SLOTARGET, there is only one case so far where it is needed, (i forgot where now), but there was no shared library being built but one of those dummy .lib files. You are right in saying that I am treating the symptoms and not the cause, but I am in a hurry to at least get OO to compile in parallel without having to nurse the build. My plan was to first identify all areas where a distributed make would fail before I gave any further attention to each problem. If you would like to assist in tracking these problems down your help will be greatly appreciated. I guess the best starting point is to go through all directories where I have put the FORCE_SEQUENTIAL macro, and identify the problem. For now I will continue working in the same manner, but will not post any more patches, but indicate where there are any further problems. George
i would like to close this one and star new issues for each upcomming problem. that would be easier to handle for me. any comments?
since there were no complains i'll close this issue, looking forward to the seperate issues.
-