compile_build.txt

Compiling/building Software on Unix

Building here is a process of preprocessing, compiling,
assembly-ing and linking.

Big software project may depend on hundreds of source
files and libraries. Many developers can work in parallel
on different pieces of it. Then all these pieces are
combined together during the build process. 

The GNU project provides a set of standard tools which 
makes the build process simplier

Standard steps:
   configure
   make build
   make test
   make install

# -----------------------------------------------
Here are some other tools:
# -----------------------------------------------
OS X SCons and CMake
  - https://www.scons.org/
  SCons is an Open Source software construction tool. 
  Think of SCons as an improved, cross-platform substitute 
  for the classic Make utility with integrated functionality 
  similar to autoconf/automake and compiler caches such as ccache. 
  In short, SCons is an easier, more reliable and faster way to build software.

# -----------------------------------------------
Ansible 
  - https://www.ansible.com/
  - https://github.com/ansible/ansible
  Ansible is a radically simple IT automation platform that makes 
  your applications and systems easier to deploy.

# -----------------------------------------------
CFEngine
  - https://cfengine.com/
  - https://github.com/cfengine/core
  CFEngine 3 is a popular open source configuration management system

# -----------------------------------------------
Chef
  - http://www.chef.io/chef/
  - https://github.com/chef/chef
  A systems integration framework ..

# -----------------------------------------------
Puppet
  - https://puppet.com/
  - https://github.com/puppetlabs/puppet
  Puppet, an automated administrative engine ...

# -----------------------------------------------
Salt
  - https://www.saltstack.com/
  - https://github.com/saltstack/salt
  Software to automate the management and configuration of any 
  infrastructure or application at scale

# -----------------------------------------------
Here are some standard tools/files:

* gcc - gnu compiler collection

* gcc (= cc) - C-compiler

* g++ (= c++) - C++ compiler

* cc1, cc1plus - C/C++ compilers

* cpp - preprocessor

* as - assembler (produces binary code from assembly
       language code and puts it in an object file *.o)

* ld - linker -  produces an executable binary file from
       object files and libraries

* etc. etc.

* configure - a long executable shell script (usually
  generated by auto-tools listed below) which in many cases
  you run as a first step of the build. 
  configure script generates a custom Makefile 
  to fit the system it is being compiled on

* gmake - a utility to execute the Makefile

* autoconf - utility to create a configure script. 
  Accepts configure.in or configure.ac

* autoheader - tool to help manage  C header files

* autoscan - tool to  create an initial input file for
  autoconf

* ifnames - tool to create a list of C pre-processor
  identifiers used in the program

* automake - to create portable Makefiles (takes its input
  as Makefile.am, and turns it into Makefile.in, which is
  used by the configure script to generate the file Makefile
  output)

* libtool - tool to manage the creation of static and
  dynamic libraries on various flavors of unix (Linux,
  Solaris, etc.)

* gnulib - simplifies the process of making software that
  uses autoconf and automake portable to a wide range of
  systems

* ar - archiver - creates an *.a file by archiving 
  several .o files into it, for example: 
    ar rcs libclass.a class1.o class2.o class3.o

Note - there are several formats of .a files, 
  the most common are GNU and BSD - see
    http://en.wikipedia.org/wiki/Ar_(Unix)

* gettext - translate a text string into the user’s 
  native language

* pkg-config - retrieve information about installed
  libraries in the system

* autoreconf - Update generated configuration files

* ldd - print shared library dependencies
        ldd /bin/time
        ldd `which xterm`

* nm - lists the symbols from object files

* binutils - ld, as, ar, nm, strings, strip, gprof,
  c++filt, etc.

* gdb - GNU debugger

* ncurses - (new curses) a programming library 
  to write "GUI-like" apps to run in terminal

For C and C++ projects, the common file type are:

* .c, .cc - source C or C++ files
* .i, .ii - preprocessed source (generated by gcc or g++)
* .s - assembly language source (assembled by "as")
* .o - compiled object files (modules) ready for linking
* .a - library of object modules (ar arname member1 member2 ...)
* .so - shared (dynamically loaded) libraries (.dylib on Mac X).
        http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

* .lo, .la - library files created by libtool

Note: The user working on a project can generate the
  ./configure file by running shell script (autogen.sh) 
  which runs a series of commands:
     aclocal; autoconf; autoheader; automake

# -----------------------------------------------
   gcc usage
# -----------------------------------------------
cc -E hello.c

view hello.i - does only preprocessing and shows the result

cc -S hello.c 

view hello.s - generates assembler code

cc -g -c hello.s

view hello.o - generates object code

dis -L -t .data hello.o >hello.dis - disassemble 
  the object file into a text file

cc -o hello hello.c - does all steps (pre-processing,
  compiling, assembling, linking) - and generates 
  output executable "hello"

Here are some common compiler options:

-c - compile only (do not link)

-S - stop after making assempler code

-E - stop after preprocessing stage

-o file - define the output file

-v - verbouse (print commands executed on STDERR, also
print compiler version, etc.)

-l - to specify libraries to link 
  ( -lxyz means to link libxyz.a)

-L - to specify directories to search for libraries

-I - to specify directories to search for include files

-B/some/path - prefix to find  executables,
  libraries, include files, and data files of the compiler
  itself. 
  Default is /usr/lib/gcc/, /usr/local/lib/gcc-lib/, and $PATH

-static - tells compiler to include shared libraries into
  the executable (thus making it larger) - even though the
  libraries are available (usually in /lib) and can be linked
  dynamically during execution (on systems that support
  dynamic linking)

-shared - produce a shared object which can then be linked
  with other objects  to  form an executable.  Only a few
  systems support this option

-symbolic - bind  references to global symbols when
  building a shared object, warn about any unresolved
  references - only if supported

-On, where n=0,1,2,or 3 - sets level of optimization

-b machine - to specify the type of machine for which to
  compile

-V version - to specify which version of compiler to run

-g  Produce debugging information in the operating system's
  native format

there are many more options (100s) - see man pages for the
compiler


# -----------------------------------------------
Search Paths
# -----------------------------------------------
To see gcc default search paths, run gcc -print-search-dirs

It will show where gcc is installed, and list of
directories where it looks for its binaries or libraries.

The search path for headers is different than that for
libraries. 

For headers gcc will look into "include" subdirectories, for example:
/usr/include/
/usr/include/c++/ - for C++

On normal Unix system gcc will look for header files in
directories similar to these:

  /usr/local/include/
  /usr/libexec/gcc/sometarget/someversion/include/
  /usr/lib/gcc/sometarget/someversion/include/
  /usr/sometarget/include/
  /usr/include/

(where, for example, sometarget/someversion = i386-redhat-
linux/3.4.6)

The search for libraries themselves may be like this:

  /usr/lib/gcc/sometarget/someversion/
  /usr/sometarget/lib/sometarget/someversion/
  /usr/sometarget/lib/
  /usr/lib/sometarget/someversion/
  /usr/lib/
  /lib/sometarget/someversion/
  /lib/

Note: main library for C is libc.a, 
                   for C++ -  libstdc++.a

Note: you can use the following gcc options:

  -I - to specify directories to search for include files
  -L - to specify directories to search for libraries
  -nostdinc option - to prevent GCC from searching any of
       the default directories

Read more details here: 
  http://gcc.gnu.org/onlinedocs/gcc-4.3.0//cpp/Search-Path.html#Search-Path

Note: you can use GCC with a number of different C
libraries. Here are some examples:

* http://www.gnu.org/software/libc/
* http://www.fefe.de/dietlibc/
* http://sources.redhat.com/newlib/
* http://www.uclibc.org (http://www.buildroot.org)


# -----------------------------------------------
Environment
# -----------------------------------------------
GCC_INCLUDE_DIR, LD_LIBRARY_PATH, LD_PRELOAD, etc.

David Barr "Why LD_LIBRARY_PATH is bad":
   http://xahlee.org/UnixResource_dir/_/ldpath.html

"""
LD_LIBRARY_PATH is one of those insidious things that once
 it gets set globally for a user, things tend to happen
which cause people to rely on it being set. Eventually when
LD_LIBRARY_PATH needs to be changed or removed, mass
breakage will occur!
"""


# -----------------------------------------------
Linking
# -----------------------------------------------
Static linking - include libraries into the executable itself. 
  Advantage    - no dependencies on external libraries.
  Disadvantage - executable gets larger, all utilities/extensions 
                 will be also larger.

Dynamic linking - using external shared libraries. This is
the default linking option. 
  Advantage    - smaller executables. 
  Disadvantage - we need to make sure that executable will 
                 find the libraries and that these libraries
                 has the right version to match the executables.

The search for shared libraries can be influenced by a
number of factors:

    * How the executable was linked and with which options
    * setguid or setuid?
    * permissions?
    * bitness (32 or 64 bit)
    * etc.

/lib/ld.so - shared library loader
# -----------------------------------------------