Apache OpenOffice (AOO) Bugzilla – Issue 53148
"dmake -Pn" captures arbitrary output when using "$(shell ...)"
Last modified: 2013-08-07 15:34:52 UTC
there are mainly two flavours of this issue: - capturing stdout from another process - capture stdout from previous commands of the same target for examples see #i9818#, #i51873#
while hunting another issue, i noticed that STDOUT and STDERR are not flushed before doing a fork. this may lead to having unflushed output beeing copied to the child process and thus get captured by $(shell ...). didn't check yet...
Thanks! dmake/unix/runargv.c - line 152 calls fork without flushing stdout first. <http://aplawrence.com/Bofcusm/1819.html> gives a nice explanation of the problem. Definitely a flush() is missing Something like: if( flush() ) Fatal("%s: %s", argv[0], strerror( errno )); is missing in line 151.
Would have been nice if that would have worked. I tried inserting fflush(NULL) and fflush(stdout), no luck :( On the bright side I now have a testcase for that problem: --- start makefile.mk --- AA=X$(shell +echo shellexec)X a1 : +@echo a0 +@echo a1 +@echo a2 +@echo A$(AA)A > a1 --- end makefile.mk --- With dmake.exe -P2 leading to: $ cat a1 AXshellexec a1 a2XA Whereas no -P# leads to: $ cat a1 AXshellexecXA
The fflush() was a red herring, _exec_shell in function.c plays with dup(1) and something with the redirection is not correct for nestlevel > 0. To be continued ...
Created attachment 29314 [details] Patch for dmake
Whoever had the idea of redirecting stdout before forking deserves to be beaten! The previous patch solves the problem for my testcase. I introduce the use of waitpid for unix builds and I hope it exists whenever wait exists. Committed to CWS dmake43p01. @cmc: Can you check this dmake (you only need the dmake module from cws_src680_dmake43p01, there are no changes in filter and solenv) on one of your multi parallel monsters?
I applied the dmake part of the workspace and did a test spin, I get an error of... 1 module(s): sysui need(s) to be rebuilt Reason(s): ERROR: error 65280 occurred while making /usr/src/build/609589-i386/BUILD/SRC680_m127/sysui/desktop/freedesktop Attention: if you build and deliver the above module(s) you may prolongue your the build issuing command "build --from sysui" the mixed up output from the very parallel build around the error point is... cp ../../../../../unxlngi6.pro/misc/registry/res/gu-IN/org/openoffice/Office/SFX.tmp ../../../../../unxlngi6.pro/misc/registry/res/gu-IN/org/openoffice/Office/SFX.xcu rm -f ../../../../../unxlngi6.pro/misc/registry/res/gu-IN/org/openoffice/Office/SFX.tmp > /dev/nulli386-redhat-linux-gcc -Wreturn-type -fmessage-length=0 -c -I. -I. -I../inc -I../inc -I../unx/inc -I../unxlngi6.pro/inc -I. -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solver/680/unxlngi6.pro/inc/dont_use_stl -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solver/680/unxlngi6.pro/inc/external -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solver/680/unxlngi6.pro/inc -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solenv/unxlngi6/inc -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solenv/inc -I/usr/src/build/609589-i386/BUILD/SRC680_m127/res -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solver/680/unxlngi6.pro/inc/dont_use_stl -I/usr/src/build/609589-i386/BUILD/SRC680_m127/solenv/inc/Xp31 -I/usr/include -I/usr/X11R6/include -I. -I../res -I. -Os -fno-strict-aliasing -Wuninitialized -pipe -Wp,-D_FORTIFY_SOURCE=2 -g -fpic -DLINUX -DUNX -DVCL -DGCC -DC341 -DINTEL -DGXX_INCLUDE_PATH=/usr/lib/gcc/i386-redhat-linux/4.0.1/../../../../include/c++/4.0.1 -DCVER=C341 -D_USE_NAMESPACE -DGLIBC=2 -DX86 -D_PTHREADS -D_REENTRANT -DNEW_SOLAR -D_USE_NAMESPACE=1 -DSTLPORT_VERSION=400 -DHAVE_GCC_VISIBILITY_FEATURE -D__DMAKE -DUNIX -DCPPU_ENV=gcc3 -DSUPD=680 -DPRODUCT -DNDEBUG -DPRODUCT_FULL -DOSL_DEBUG_LEVEL=0 -DOPTIMIZE -DEXCEPTIONS_OFF -DCUI -DSOLAR_JAVA -DSRC680 -DSHAREDLIB -D_DLL_ -o ../unxlngi6.pro/slo/utl_dflt_version.o ../unxlngi6.pro/misc/utl_dflt_version.c if ( -e ../unxlngi6.pro/slo/utl_dflt_version.o) touch ../unxlngi6.pro/slo/utl_dflt_version.obj ------------------------------ dmake: Executing shell macro: echo $$$$ dmake: Executing shell macro: cd $(MISC)$/$(TARGET)/BUILD; pwd -------------+ creating locale dependent entries Making: ../unxlngi6.pro/slo/utl_dflt_version.obj ../../unxlngi6.pro/misc/freedesktop/BUILD: No such file or directory. dmake: Error code 1, while making 'Shell escape' mkdir -p ../../../../../../unxlngi6.pro/misc/registry/res/sl/org/openoffice/Office/UI/ /usr/bin/xsltproc -o ../../../../../../unxlngi6.pro/misc/registry/res/sl/org/openoffice/Office/UI/BibliographyCommands.tmp \ perhaps this simply points to a problem in the freedesktop makefile.mk
Drat! I know where the problem comes from: -=-=-=- aaa: (1) @foo (2) @mkdir bar (3) @baz $(shell cd bar) (4) -=-=-=- The _shell_exec stuff is made in a way that it is executed as a separate target. See the example makefile from "comments from vq Tue Aug 30 20:04:51 -0700 2005". This means that line 3 from the makefile above might well be executed after the shell escape is done. This problem is not introduced by my dmake patch but the latent design flaw was propably uncoverd by the slightly changed process order. As a workaround: The freedesktop makefile PACKAGE_RPM should depend on a separate target that creates $(MISC)$/$(TARGET)$/BUILD. Maybe you should file an issue about this.
Created attachment 29330 [details] Additional patch making sure to execute shell escapes after all previous recipe lines from the same target
I also commtted the second patch. @cmc: Can you please try again? You have to either checkout dmake from dmake43p01 or apply both patches.
Did I mention that redirecting in the parent is evil? @#$% <BLOG> Guess what happens now that the shell escape in done in the correct order: aaa : +echo b0 (a) +echo b1 (b) +echo b2 (c) +echo A$(shell echo aaa)A (d) leads to: ... echo Ab1 b2 aaaA Ab1 b2 aaaA What happens is that (a) is executed, (b) and (c) are pushed behind (a) in the queue. Then line (d) triggers the redirection of stdout *in the parent* to capture the output of the $(shell ..) macro. Then we wait for the previous recipe lines to finish. This is where we capture their stdout and the we start our shell escape. I'll move the redirection to the child now. </BLOG>
Created attachment 29336 [details] Move _exec_shell redirection into child
Committed to dmake43p01. @cmc: This works for me, Caolan can you test?
I tried with the Sep 5 10:21:02 patch and on one platform got three broken modules, while on another platform the build hung mysteriously overnight. I'll try again with the 13:54:41 patch
getting this in localedata with latest checkout of that dmake workspace ERROR: error 65280 occurred while making /usr/src/build/609874-ppc/BUILD/SRC680_m127/i18npool/source/localedata/data Checking DLL ../../../unxlngppc.pro/lib/check_libdict_zh.so ...dmake: Executing shell macro: $(FIND) . -name "{$(subst,$($(WINVERSIONNAMES)_MAJOR),* $(subst,$(UPD)$(DLLPOSTFIX), $(SHL4TARGET)))}.xml" -rwxr-xr-x 1 bhcompile bhcompile 2272366 Sep 6 03:46 ../../../unxlngppc.pro/lib/libdict_zh.so ------------------------------ Making: ../../../unxlngppc.pro/lib/idict_ja.lib no ImportLibs on Mac and *ix mkdir -p ../../../../../../unxlngppc.pro/misc/merge/org/openoffice/Office/UI/ cfgex -p officecfg -i DbuCommands.xcu -o ../../../../../../unxlngppc.pro/misc/merge/org/openoffice/Office/UI/DbuCommands.xcu -m localize.sdf -l all ../../../unxlngppc.pro/bin/saxparser en_AU en_AU.xml ../../../unxlngppc.pro/misc/localedata_en_AU.cxx ../../../unxlngppc.pro/bin/localedata_en_AU.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_AU.rdb ../../../unxlngppc.pro/bin/saxparser en_BZ en_BZ.xml ../../../unxlngppc.pro/misc/localedata_en_BZ.cxx ../../../unxlngppc.pro/bin/localedata_en_BZ.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_BZ.rdb ../../../unxlngppc.pro/bin/saxparser en_CA en_CA.xml ../../../unxlngppc.pro/misc/localedata_en_CA.cxx ../../../unxlngppc.pro/bin/localedata_en_CA.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_CA.rdb ../../../unxlngppc.pro/bin/saxparser en_CB en_CB.xml ../../../unxlngppc.pro/misc/localedata_en_CB.cxx ../../../unxlngppc.pro/bin/localedata_en_CB.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_CB.rdb ../../../unxlngppc.pro/bin/saxparser en_GB en_GB.xml ../../../unxlngppc.pro/misc/localedata_en_GB.cxx ../../../unxlngppc.pro/bin/localedata_en_GB.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_GB.rdb ../../../unxlngppc.pro/bin/saxparser en_IE en_IE.xml ../../../unxlngppc.pro/misc/localedata_en_IE.cxx ../../../unxlngppc.pro/bin/localedata_en_IE.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_IE.rdb ../../../unxlngppc.pro/bin/saxparser en_JM en_JM.xml ../../../unxlngppc.pro/misc/localedata_en_JM.cxx ../../../unxlngppc.pro/bin/localedata_en_JM.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_JM.rdb ../../../unxlngppc.pro/bin/saxparser en_NZ en_NZ.xml ../../../unxlngppc.pro/misc/localedata_en_NZ.cxx ../../../unxlngppc.pro/bin/localedata_en_NZ.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb rm -f ../../../unxlngppc.pro/bin/localedata_en_NZ.rdb ../../../unxlngppc.pro/bin/saxparser en_PH en_PH.xml ../../../unxlngppc.pro/misc/localedata_en_PH.cxx ../../../unxlngppc.pro/bin/localedata_en_PH.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb Exception on createRegistryServiceFactory dmake: Error code 1, while making '../../../unxlngppc.pro/misc/localedata_en_BZ.cxx' Exception on createRegistryServiceFactory '---* tg_merge.mk *---'
on the other platform it appears to have stalled permantly on... /usr/src/build/609873-i386/BUILD/SRC680_m127/shell/source/backends/gconfbe Running processes: 8 dmake: Executing shell macro: date +%d%m%Y cp -f gconfbe.xml ../../../unxlngi6.pro/misc/gconfbe.xml touch ../../../unxlngi6.pro/misc/gconfbe.mk echo XML2MK_FILES += gconfbe >> ../../../unxlngi6.pro/misc/gconfbe.mk dmake: Executing shell macro: xml2cmp -types stdout $(MISC)$/$(COMP1TYPELIST)$($(WINVERSIONNAMES)_MAJOR).xml dmake: Executing shell macro: $(PKGCONFIG) $(PKGCONFIG_PREFIX) --cflags $(PKGCONFIG_MODULES) dmake: Executing shell macro: $(PKGCONFIG) $(PKGCONFIG_PREFIX) --libs $(PKGCONFIG_MODULES) /usr/src/build/609873-i386/BUILD/SRC680_m127/shell/source/backends/localebe Running processes: 8 dmake: Executing shell macro: date +%d%m%Y cp -f localebe.xml ../../../unxlngi6.pro/misc/localebe.xml touch ../../../unxlngi6.pro/misc/localebe.mk echo XML2MK_FILES += localebe >> ../../../unxlngi6.pro/misc/localebe.mk dmake: Executing shell macro: xml2cmp -types stdout $(MISC)$/$(COMP1TYPELIST)$($(WINVERSIONNAMES)_MAJOR).xml
The cryptic error from saxparser means that ::rtl::OUString::createFromAscii(argv[x]) failed on x={4,5}, I guess that means that one of ../../../unxlngppc.pro/bin/localedata_en_BZ.rdb /usr/src/build/609874-ppc/BUILD/SRC680_m127/solver/680/unxlngppc.pro/bin/types.rdb are missing. Propably a missing dependency somewhere. Ause? I have no clue concerning the hang :(
Created attachment 29359 [details] Make sure _attach_cmd can not be used for _exec_shell calls. Even though this should not happen there is a slim chance that it fixes the hang. Committed.
For the record: I can reproduce a STATUS_ACCESS_VIOLATION in sysintf.c line 657 with $ build -- -P2 in io/.
Created attachment 29631 [details] Make sure that the command started from _exec_shell really finished before returning. Committed.
I guess I nailed the hang. Beside the fact that the windows build was never before build with more than one process and therefore has some issues the current dmake from CWS dmake43p01 seems to work OK. Caolan, Pavel: Can one of you give it a spin on a non-Windows machine with more than one processor for an OOo build? Thanks!
looks good, worked on 8x ppc linux build
Reassign to verify
Seems to work, please verify.
While testing dmake43p01 in a W32-4nt configuration I realized that my moving the _exec_shell redirection into the child broke the $(shell ..) macro for the native W32 build. The following patch restores the old behaviour for non-*NIX dmake versions. (Tested for W32-4nt, untested for unixy systems ATM)
Created attachment 30760 [details] Patch to restore $(shell ..) for VC builds
Committed to dmake43p01.
.