Page 2 of 3

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 6:17 pm
by sledgehammer_999
Dayman wrote: The other bad thing that happens is that make install fails fot Qt with gcc, currently fixing with this:

I never did a "make install" on Qt. Just wait and see how I use it with qt creator to build qbittorrent. Maybe you'll get some ideas.
(I currently compile qt. It will some time)
Dayman wrote:Also: Building Qt with -flto -fuse-linker-plugin -fwhole-program in attachment.
[me=Dayman]remembers building wxWidgets monolithic library with gcc. 3 Gigs ate by ld and can't allocate memory comes next. I hope current mingw versions are built with LARGEADDRESSAWARE in mind.[/me]
Yesterday when I used only flto it compiled/linked on my system which has 2GB RAM. I didn't watch how much it used though, it probably used the swap too at some brief moment.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 7:17 pm
by Dayman
Confirming -fuse-linker-plugin is bugged in mingw64 (actually in binutils port shipped with mingw, since that's ld's fault) and probably in i686 version too. OpenSSL fails to run tests with this, and Qt doesn't even build (during uic.exe link: 3+ screens filled with multiple definition errors and bad reloc in .rdata at the end).
Currently using -fno-use-linker-plugin in case this option is on by default.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 7:58 pm
by sledgehammer_999
Qbittorrent git master at 76f959599fe

Download and install Qt Creator from www.qt-project.org

1. Launch it and select Tools->Options...->Tool Chains
2. There add a new Tool Chain and point it to g++.exe and gbd.exe of your mingw toolchain (and set the appropriate ABI 32bit for 32bit and 64bit for 64bit duh)
3. Then switch to "Qt Versions". There add a new version and name it appropriately. Point this new version to your just compiled qmake.exe.
Note: If qmake.exe depends on some gcc dlls (like stdc++ or winpthread-1.dll) and gcc isn't in the PATH then nothing will happen when you select your qmake.exe

4. Then select File->Open File or Project and select "qbittorrent.pro" from qbt's source folder.
5. Select the Qt version you just configured
6. Open winconf/winconf.pri and change the paths according to the comments there
7. Open winconf-mingw/winconf-mingw.pri and AFTER line 14 delete the mentioned boost libs and only add the boost.system lib eg libboost_system-mgw47-mt-s-1_51 (without the .a)
8. Also delete the other LIBS+= lines and this

Code: Select all

LIBS += libwsock32
LIBS += libpowrprof libwinpthread

CONFIG += exceptions rtti sse sse2 mmx 3dnow
For some reason you need to specifically tell qmake to enable exceptions rtti sse sse2 mmx 3dnow for gcc (although you don't have to for msvc)

9. Then you have to fix the libs that QtNetwork passes to the linker. It has the wrong names for the ssl libs and it doesn't mention some others. Go to your Qt folder and navigate to the "lib" subfolder. There open the "QtNetwork.prl" file in an editor(Qt Creator).
10. Look for this line QMAKE_PRL_LIBS = -lQtCore -lole32 -luuid -lws2_32 -ladvapi32 -lshell32 -luser32 -lkernel32 -lz -lz -lssleay32 -llibeay32 at the end. Change it to QMAKE_PRL_LIBS = -lQtCore -lole32 -luuid -lws2_32 -ladvapi32 -lshell32 -luser32 -lkernel32 -lz -lssl -lcrypto -lgdi32 -lcrypt32
libssl HAS to be before libcrypto. Also libdgi32 and crypt32 HAVE to be after the openssl libs.

11. Download the GeoIP database and extract it inside src/geoip

12. Hit build and wait. The linking phase will take ages(on my amd x2 2.2GHZ machine it takes about an hour).

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 9:18 pm
by sledgehammer_999
Compilation finished.

qbittorrent.exe LTO size: 30.7MB
--strip-all yields the same size.
qbittorrent.exe without LTO size: 20.2MB

NOTE: I add the following to the QMAKE_CFLAGS_RELEASE and QMAKE_LFLAGS_RELEASE of mkspecs "-fuse-linker-plugin" when compiling qbt. It seems to work for 32bit.

(for non LTO build I just drop the lto flags from mkspecs when building qbt source)

I cannot explain the increase in size with LTO.... :S

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 10:05 pm
by Dayman
Strange, maybe static build has something to do with it.
I'm getting different results for different libs. Libtorrent did shrink in side, OpenSSL hasn't changed, boost_system grew by 2 kilos. Will finish other libs tomorrow.

P.S. Why mingw builds all of a sudden?
P.P.S. It might be interesting to compare results with MSVC with /O1 /Ob1.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sun Oct 21, 2012 10:40 pm
by Onemoar
good to see you using mingw personally I think VC is terrible the larger binary size is illreavent com paired to the proably speed gains mingw should give definitely looks like you have some static linking issues tho are you using -O3
you should ALWAYS use at least -O1 with minigw symbols are also HUGE so there is that ... I perfer to use -O3 -msse -msse2 -mss3 -mpfmath=sse

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 8:10 am
by Dayman
[quote="Onemoar"]
good to see you using mingw personally I think VC is terrible the larger binary size is illreavent com paired to the proably speed gains mingw should give definitely looks like you have some static linking issues tho are you using -O3
you should ALWAYS use at least -O1 with minigw symbols are also HUGE so there is that ... I perfer to use -O3 -msse -msse2 -mss3 -mpfmath=sse
[/quote]
VC is at least fast at compiling and supports AMD64 well.
Depending on the situation -Os might be better than -O3, allowing more cache-hits on CPUs with small-medium cache sizes. On the other hand -Os can break stuff in very rare cases (only happened to me on linux - pulseaudio segfaults when built with -Os).
I've built everything with lto and I can say that it's completely useless on windows (probably due to bugged -fuse-linker-plugin or silently accepting unimplemented/OS-incompatible flags). Qt and qBt build with lto, but lto1 and lto-wrapper are not launched during link phase even though cmd line is correct. The only binary/lib which did benefit from lto was libtorrent (around 15% less space consumed), everything else grew by a few kilos (1-10).
I'm also not sure how you've managed to build static libtorrent. For me it always dies during link phase (https://code.google.com/p/libtorrent/is ... =1&q=mingw). Looks like a weird arg overflow.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 12:10 pm
by sledgehammer_999
[quote="Dayman"]
P.S. Why mingw builds all of a sudden?[/quote]

I was just exploring the possibility to go back to mingw builds if there was some negligible size difference between MSVC and mingw binaries due to LTO. But this turned out to be a disappointment. I think the issues that forced me to switch to MSVC are gone now. Mostly fixed in libtorrent and the unicode support seems to work fine with 4.7.x series. Maybe an LTO dynamic build should be considered now. I don't know.

For the time being it is easier to compile with MSVC(and faster).

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 12:13 pm
by sledgehammer_999
[quote="Onemoar"]
good to see you using mingw personally I think VC is terrible the larger binary size is illreavent com paired to the proably speed gains mingw should give definitely looks like you have some static linking issues tho are you using -O3
you should ALWAYS use at least -O1 with minigw symbols are also HUGE so there is that ... I perfer to use -O3 -msse -msse2 -mss3 -mpfmath=sse
[/quote]

I am not sure that mingw are faster than MSVC. And I don't think a bittorrent program needs "more" optimized code. I mean it just downloads/uploads data. It doesn't do something computationally intensive with them.

It would be an interesting idea to provide optimized mingw and MSVC builds and let the users conduct some experiments and them maybe switch to mingw if it is indeed faster.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 12:24 pm
by sledgehammer_999
[quote="Dayman"]Qt and qBt build with lto, but lto1 and lto-wrapper are not launched during link phase even though cmd line is correct. The only binary/lib which did benefit from lto was libtorrent (around 15% less space consumed), everything else grew by a few kilos (1-10).[/quote]
During linking of qbt lto1.exe was launced on my machine.

Which mingw build do you use? Original mingw, Mingw-w64, TDM-GCC and mingw builds. I use the last (the variant with posix threading).

[quote="Dayman"]I'm also not sure how you've managed to build static libtorrent. For me it always dies during link phase (https://code.google.com/p/libtorrent/is ... =1&q=mingw). Looks like a weird arg overflow.[/quote]

Zlib 0.16.x uses its own zlib and you cannot specify an external zlib. Furthermore, it dies for me at the last stage but I relaunch the command and redirect then output to a text file. From there I copy paste the individual commands into MSYS and works.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 3:01 pm
by Dayman
[quote="sledgehammer_999"]
[quote="Dayman"]Qt and qBt build with lto, but lto1 and lto-wrapper are not launched during link phase even though cmd line is correct. The only binary/lib which did benefit from lto was libtorrent (around 15% less space consumed), everything else grew by a few kilos (1-10).[/quote]
During linking of qbt lto1.exe was launced on my machine.

Which mingw build do you use? Original mingw, Mingw-w64, TDM-GCC and mingw builds. I use the last (the variant with posix threading).
[/quote]
The last one, but 4.7.0 with win32 threads.
[quote="sledgehammer_999"]
[quote="Dayman"]I'm also not sure how you've managed to build static libtorrent. For me it always dies during link phase (https://code.google.com/p/libtorrent/is ... =1&q=mingw). Looks like a weird arg overflow.[/quote]

Zlib 0.16.x uses its own zlib and you cannot specify an external zlib. Furthermore, it dies for me at the last stage but I relaunch the command and redirect then output to a text file. From there I copy paste the individual commands into MSYS and works.
[/quote]
That sucks. I mostly deal with automated building: heavy patch, sed and bat files usage, no QtCreator - everything is done with qmake. So this kind of workaround doesn't really suit me.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 3:47 pm
by sledgehammer_999
[quote="Dayman"]That sucks. I mostly deal with automated building: heavy patch, sed and bat files usage, no QtCreator - everything is done with qmake. So this kind of workaround doesn't really suit me.
[/quote]

I meant bjam/b2 for libtorrent compilation. Maybe the problem I am describing doesn't exist if you build using autotools.
Qt Creator is not needed for qbt compilation, if you know your way around qmake. You just may just need to edit winconf.pri and winconf-mingw.pri. Never tried it through command line and qmake though.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Oct 22, 2012 9:49 pm
by Dayman
[quote="sledgehammer_999"]
[quote="Dayman"]That sucks. I mostly deal with automated building: heavy patch, sed and bat files usage, no QtCreator - everything is done with qmake. So this kind of workaround doesn't really suit me.
[/quote]

I meant bjam/b2 for libtorrent compilation. Maybe the problem I am describing doesn't exist if you build using autotools.
[/quote]
I just wanted to say that I prefer minimal manual control over the process. Basically I just do everything from bat files (or shell scripts in case of OpenSSL build under MSYS).

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Sat Nov 03, 2012 6:41 pm
by Dayman
Interesting. qBt, compiled with MSVC with /O1 just died after adding magnet while the one built with gcc works fine.

Re: Trying out new gcc 4.7.x builds with lto and size optimizations

Posted: Mon Nov 19, 2012 6:36 pm
by Dayman
An interesting interview I found, when searching on the subject of speed vs. size optimizations: http://www.gamearchitect.net/Articles/C ... OfMA2.html
The article describes steps taken to optimize MA2 game, which was released on the 'Old' Xbox (733 MHz P3 CPU with 32 KB L1 and 128 KB L2 cache).

So, it looks like speed optimization is almost always better for post 2K CPUs.

Nice, builds with -fstack-protector-all -Wl,--nxcompat -Wl,--dynamicbase
ASLR is now on.