Commit 93c22dd9 authored by alalazo's avatar alalazo
Browse files

Merge branch 'develop' of https://github.com/LLNL/spack into qa/coding_standard

parents 1664d1b5 69f8d8fa
......@@ -59,7 +59,8 @@ can join it here:
At the moment, contributing to Spack is relatively simple. Just send us
a [pull request](https://help.github.com/articles/using-pull-requests/).
When you send your request, make ``develop`` the destination branch.
When you send your request, make ``develop`` the destination branch on the
[Spack repository](https://github.com/LLNL/spack).
Spack is using a rough approximation of the [Git
Flow](http://nvie.com/posts/a-successful-git-branching-model/)
......
# -------------------------------------------------------------------------
# This is the default spack module files generation configuration.
#
# Changes to this file will affect all users of this spack install,
# although users can override these settings in their ~/.spack/modules.yaml.
# -------------------------------------------------------------------------
modules:
enable: ['tcl', 'dotkit']
......@@ -149,26 +149,46 @@ customize an installation in :ref:`sec-specs`.
``spack uninstall``
~~~~~~~~~~~~~~~~~~~~~
To uninstall a package, type ``spack uninstall <package>``. This will
completely remove the directory in which the package was installed.
To uninstall a package, type ``spack uninstall <package>``. This will ask the user for
confirmation, and in case will completely remove the directory in which the package was installed.
.. code-block:: sh
spack uninstall mpich
If there are still installed packages that depend on the package to be
uninstalled, spack will refuse to uninstall it. You can override this
behavior with ``spack uninstall -f <package>``, but you risk breaking
other installed packages. In general, it is safer to remove dependent
packages *before* removing their dependencies.
uninstalled, spack will refuse to uninstall it.
A line like ``spack uninstall mpich`` may be ambiguous, if multiple
``mpich`` configurations are installed. For example, if both
To uninstall a package and every package that depends on it, you may give the
`--dependents` option.
.. code-block:: sh
spack uninstall --dependents mpich
will display a list of all the packages that depends on `mpich` and, upon confirmation,
will uninstall them in the right order.
A line like
.. code-block:: sh
spack uninstall mpich
may be ambiguous, if multiple ``mpich`` configurations are installed. For example, if both
``mpich@3.0.2`` and ``mpich@3.1`` are installed, ``mpich`` could refer
to either one. Because it cannot determine which one to uninstall,
Spack will ask you to provide a version number to remove the
ambiguity. As an example, ``spack uninstall mpich@3.1`` is
unambiguous in this scenario.
Spack will ask you either to provide a version number to remove the
ambiguity or use the ``--all`` option to uninstall all of the matching packages.
You may force uninstall a package with the `--force` option
.. code-block:: sh
spack uninstall --force mpich
but you risk breaking other installed packages. In general, it is safer to remove dependent
packages *before* removing their dependencies or use the `--dependents` option.
Seeing installed packages
......@@ -774,6 +794,34 @@ Environment modules
Spack provides some limited integration with environment module
systems to make it easier to use the packages it provides.
Installing Environment Modules
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to use Spack's generated environment modules, you must have
installed the *Environment Modules* package. On many Linux
distributions, this can be installed from the vendor's repository.
For example: ```yum install environment-modules``
(Fedora/RHEL/CentOS). If your Linux distribution does not have
Environment Modules, you can get it with Spack:
1. Install with::
spack install environment-modules
2. Activate with::
MODULES_HOME=`spack location -i environment-modules`
MODULES_VERSION=`ls -1 $MODULES_HOME/Modules | head -1`
${MODULES_HOME}/Modules/${MODULES_VERSION}/bin/add.modules
This adds to your ``.bashrc`` (or similar) files, enabling Environment
Modules when you log in. It will ask your permission before changing
any files.
Spack and Environment Modules
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can enable shell support by sourcing some files in the
``/share/spack`` directory.
......
......@@ -39,7 +39,7 @@
#
# This is the list of environment variables that need to be set before
# the script runs. They are set by routines in spack.build_environment
# the script runs. They are set by routines in spack.build_environment
# as part of spack.package.Package.do_install().
parameters="
SPACK_PREFIX
......@@ -50,7 +50,7 @@ SPACK_SHORT_SPEC"
# The compiler input variables are checked for sanity later:
# SPACK_CC, SPACK_CXX, SPACK_F77, SPACK_FC
# Debug flag is optional; set to true for debug logging:
# Debug flag is optional; set to "TRUE" for debug logging:
# SPACK_DEBUG
# Test command is used to unit test the compiler script.
# SPACK_TEST_COMMAND
......@@ -65,12 +65,11 @@ function die {
}
for param in $parameters; do
if [ -z "${!param}" ]; then
die "Spack compiler must be run from spack! Input $param was missing!"
if [[ -z ${!param} ]]; then
die "Spack compiler must be run from Spack! Input '$param' is missing."
fi
done
#
# Figure out the type of compiler, the language, and the mode so that
# the compiler script knows what to do.
#
......@@ -78,14 +77,18 @@ done
# 'command' is set based on the input command to $SPACK_[CC|CXX|F77|F90]
#
# 'mode' is set to one of:
# vcheck version check
# cpp preprocess
# cc compile
# as assemble
# ld link
# ccld compile & link
# cpp preprocessor
# vcheck version check
#
command=$(basename "$0")
case "$command" in
cpp)
mode=cpp
;;
cc|c89|c99|gcc|clang|icc|pgcc|xlc)
command="$SPACK_CC"
language="C"
......@@ -102,9 +105,6 @@ case "$command" in
command="$SPACK_F77"
language="Fortran 77"
;;
cpp)
mode=cpp
;;
ld)
mode=ld
;;
......@@ -113,10 +113,12 @@ case "$command" in
;;
esac
# If any of the arguments below is present then the mode is vcheck. In vcheck mode nothing is added in terms of extra search paths or libraries
if [ -z "$mode" ]; then
# If any of the arguments below are present, then the mode is vcheck.
# In vcheck mode, nothing is added in terms of extra search paths or
# libraries.
if [[ -z $mode ]]; then
for arg in "$@"; do
if [ "$arg" = -v -o "$arg" = -V -o "$arg" = --version -o "$arg" = -dumpversion ]; then
if [[ $arg == -v || $arg == -V || $arg == --version || $arg == -dumpversion ]]; then
mode=vcheck
break
fi
......@@ -124,14 +126,16 @@ if [ -z "$mode" ]; then
fi
# Finish setting up the mode.
if [ -z "$mode" ]; then
if [[ -z $mode ]]; then
mode=ccld
for arg in "$@"; do
if [ "$arg" = -E ]; then
if [[ $arg == -E ]]; then
mode=cpp
break
elif [ "$arg" = -c ]; then
elif [[ $arg == -S ]]; then
mode=as
break
elif [[ $arg == -c ]]; then
mode=cc
break
fi
......@@ -139,175 +143,76 @@ if [ -z "$mode" ]; then
fi
# Dump the version and exit if we're in testing mode.
if [ "$SPACK_TEST_COMMAND" = "dump-mode" ]; then
if [[ $SPACK_TEST_COMMAND == dump-mode ]]; then
echo "$mode"
exit
fi
# Check that at least one of the real commands was actually selected,
# otherwise we don't know what to execute.
if [ -z "$command" ]; then
if [[ -z $command ]]; then
die "ERROR: Compiler '$SPACK_COMPILER_SPEC' does not support compiling $language programs."
fi
# Save original command for debug logging
input_command="$@"
if [ "$mode" == vcheck ] ; then
if [[ $mode == vcheck ]]; then
exec ${command} "$@"
fi
#
# Now do real parsing of the command line args, trying hard to keep
# non-rpath linker arguments in the proper order w.r.t. other command
# line arguments. This is important for things like groups.
#
includes=()
libraries=()
libs=()
rpaths=()
other_args=()
while [ -n "$1" ]; do
case "$1" in
-I*)
arg="${1#-I}"
if [ -z "$arg" ]; then shift; arg="$1"; fi
includes+=("$arg")
;;
-L*)
arg="${1#-L}"
if [ -z "$arg" ]; then shift; arg="$1"; fi
libraries+=("$arg")
;;
-l*)
arg="${1#-l}"
if [ -z "$arg" ]; then shift; arg="$1"; fi
libs+=("$arg")
;;
-Wl,*)
arg="${1#-Wl,}"
# TODO: Handle multiple -Wl, continuations of -Wl,-rpath
if [[ $arg == -rpath=* ]]; then
arg="${arg#-rpath=}"
for rpath in ${arg//,/ }; do
rpaths+=("$rpath")
done
elif [[ $arg == -rpath,* ]]; then
arg="${arg#-rpath,}"
for rpath in ${arg//,/ }; do
rpaths+=("$rpath")
done
elif [[ $arg == -rpath ]]; then
shift; arg="$1"
if [[ $arg != '-Wl,'* ]]; then
die "-Wl,-rpath was not followed by -Wl,*"
fi
arg="${arg#-Wl,}"
for rpath in ${arg//,/ }; do
rpaths+=("$rpath")
done
else
other_args+=("-Wl,$arg")
fi
;;
-Xlinker)
shift; arg="$1";
if [[ $arg = -rpath=* ]]; then
rpaths+=("${arg#-rpath=}")
elif [[ $arg = -rpath ]]; then
shift; arg="$1"
if [[ $arg != -Xlinker ]]; then
die "-Xlinker -rpath was not followed by -Xlinker <arg>"
fi
shift; arg="$1"
rpaths+=("$arg")
else
other_args+=("-Xlinker")
other_args+=("$arg")
fi
;;
*)
other_args+=("$1")
;;
esac
shift
done
# Dump parsed values for unit testing if asked for
if [ -n "$SPACK_TEST_COMMAND" ]; then
IFS=$'\n'
case "$SPACK_TEST_COMMAND" in
dump-includes) echo "${includes[*]}";;
dump-libraries) echo "${libraries[*]}";;
dump-libs) echo "${libs[*]}";;
dump-rpaths) echo "${rpaths[*]}";;
dump-other-args) echo "${other_args[*]}";;
dump-all)
echo "INCLUDES:"
echo "${includes[*]}"
echo
echo "LIBRARIES:"
echo "${libraries[*]}"
echo
echo "LIBS:"
echo "${libs[*]}"
echo
echo "RPATHS:"
echo "${rpaths[*]}"
echo
echo "ARGS:"
echo "${other_args[*]}"
;;
*)
echo "ERROR: Unknown test command"
exit 1 ;;
esac
exit
# Darwin's linker has a -r argument that merges object files together.
# It doesn't work with -rpath.
# This variable controls whether they are added.
add_rpaths=true
if [[ mode == ld && $OSTYPE == darwin* ]]; then
for arg in "$@"; do
if [[ $arg == -r ]]; then
add_rpaths=false
break
fi
done
fi
# Save original command for debug logging
input_command="$@"
args=("$@")
# Read spack dependencies from the path environment variable
IFS=':' read -ra deps <<< "$SPACK_DEPENDENCIES"
for dep in "${deps[@]}"; do
if [ -d "$dep/include" ]; then
includes+=("$dep/include")
# Prepend include directories
if [[ -d $dep/include ]]; then
if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then
args=("-I$dep/include" "${args[@]}")
fi
fi
if [ -d "$dep/lib" ]; then
libraries+=("$dep/lib")
rpaths+=("$dep/lib")
# Prepend lib and RPATH directories
if [[ -d $dep/lib ]]; then
if [[ $mode == ccld ]]; then
$add_rpaths && args=("-Wl,-rpath,$dep/lib" "${args[@]}")
args=("-L$dep/lib" "${args[@]}")
elif [[ $mode == ld ]]; then
$add_rpaths && args=("-rpath" "$dep/lib" "${args[@]}")
args=("-L$dep/lib" "${args[@]}")
fi
fi
if [ -d "$dep/lib64" ]; then
libraries+=("$dep/lib64")
rpaths+=("$dep/lib64")
# Prepend lib64 and RPATH directories
if [[ -d $dep/lib64 ]]; then
if [[ $mode == ccld ]]; then
$add_rpaths && args=("-Wl,-rpath,$dep/lib64" "${args[@]}")
args=("-L$dep/lib64" "${args[@]}")
elif [[ $mode == ld ]]; then
$add_rpaths && args=("-rpath" "$dep/lib64" "${args[@]}")
args=("-L$dep/lib64" "${args[@]}")
fi
fi
done
# Include all -L's and prefix/whatever dirs in rpath
for dir in "${libraries[@]}"; do
[[ dir = $SPACK_INSTALL* ]] && rpaths+=("$dir")
done
rpaths+=("$SPACK_PREFIX/lib")
rpaths+=("$SPACK_PREFIX/lib64")
# Put the arguments together
args=()
for dir in "${includes[@]}"; do args+=("-I$dir"); done
args+=("${other_args[@]}")
for dir in "${libraries[@]}"; do args+=("-L$dir"); done
for lib in "${libs[@]}"; do args+=("-l$lib"); done
if [ "$mode" = ccld ]; then
for dir in "${rpaths[@]}"; do
args+=("-Wl,-rpath")
args+=("-Wl,$dir");
done
elif [ "$mode" = ld ]; then
for dir in "${rpaths[@]}"; do
args+=("-rpath")
args+=("$dir");
done
if [[ $mode == ccld ]]; then
$add_rpaths && args=("-Wl,-rpath,$SPACK_PREFIX/lib" "-Wl,-rpath,$SPACK_PREFIX/lib64" "${args[@]}")
elif [[ $mode == ld ]]; then
$add_rpaths && args=("-rpath" "$SPACK_PREFIX/lib" "-rpath" "$SPACK_PREFIX/lib64" "${args[@]}")
fi
#
......@@ -323,34 +228,40 @@ unset DYLD_LIBRARY_PATH
#
IFS=':' read -ra env_path <<< "$PATH"
IFS=':' read -ra spack_env_dirs <<< "$SPACK_ENV_PATH"
spack_env_dirs+=(".")
spack_env_dirs+=("" ".")
PATH=""
for dir in "${env_path[@]}"; do
remove=""
for rm_dir in "${spack_env_dirs[@]}"; do
if [ "$dir" = "$rm_dir" ]; then remove=True; fi
done
if [ -z "$remove" ]; then
if [ -z "$PATH" ]; then
PATH="$dir"
else
PATH="$PATH:$dir"
addpath=true
for env_dir in "${spack_env_dirs[@]}"; do
if [[ $dir == $env_dir ]]; then
addpath=false
break
fi
done
if $addpath; then
PATH="${PATH:+$PATH:}$dir"
fi
done
export PATH
full_command=("$command")
full_command+=("${args[@]}")
full_command=("$command" "${args[@]}")
# In test command mode, write out full command for Spack tests.
if [[ $SPACK_TEST_COMMAND == dump-args ]]; then
echo "${full_command[@]}"
exit
elif [[ -n $SPACK_TEST_COMMAND ]]; then
die "ERROR: Unknown test command"
fi
#
# Write the input and output commands to debug logs if it's asked for.
#
if [ "$SPACK_DEBUG" = "TRUE" ]; then
if [[ $SPACK_DEBUG == TRUE ]]; then
input_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_SHORT_SPEC.in.log"
output_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_SHORT_SPEC.out.log"
echo "$input_command" >> $input_log
echo "$mode ${full_command[@]}" >> $output_log
echo "[$mode] $command $input_command" >> $input_log
echo "[$mode] ${full_command[@]}" >> $output_log
fi
exec "${full_command[@]}"
......@@ -27,9 +27,11 @@
'force_remove', 'join_path', 'ancestor', 'can_access', 'filter_file',
'FileFilter', 'change_sed_delimiter', 'is_exe', 'force_symlink',
'set_executable', 'copy_mode', 'unset_executable_mode',
'remove_dead_links', 'remove_linked_tree']
'remove_dead_links', 'remove_linked_tree', 'find_library_path',
'fix_darwin_install_name']
import os
import glob
import sys
import re
import shutil
......@@ -38,6 +40,7 @@
import getpass
from contextlib import contextmanager, closing
from tempfile import NamedTemporaryFile
import subprocess
import llnl.util.tty as tty
from spack.util.compression import ALLOWED_ARCHIVE_TYPES
......@@ -392,3 +395,44 @@ def remove_linked_tree(path):
os.unlink(path)
else:
shutil.rmtree(path, True)
def fix_darwin_install_name(path):
"""
Fix install name of dynamic libraries on Darwin to have full path.
There are two parts of this task:
(i) use install_name('-id',...) to change install name of a single lib;
(ii) use install_name('-change',...) to change the cross linking between libs.
The function assumes that all libraries are in one folder and currently won't
follow subfolders.
Args:
path: directory in which .dylib files are alocated
"""
libs = glob.glob(join_path(path,"*.dylib"))
for lib in libs:
# fix install name first:
subprocess.Popen(["install_name_tool", "-id",lib,lib], stdout=subprocess.PIPE).communicate()[0]
long_deps = subprocess.Popen(["otool", "-L",lib], stdout=subprocess.PIPE).communicate()[0].split('\n')
deps = [dep.partition(' ')[0][1::] for dep in long_deps[2:-1]]
# fix all dependencies:
for dep in deps:
for loc in libs:
if dep == os.path.basename(loc):
subprocess.Popen(["install_name_tool", "-change",dep,loc,lib], stdout=subprocess.PIPE).communicate()[0]
break
def find_library_path(libname, *paths):
"""Searches for a file called <libname> in each path.
Return:
directory where the library was found, if found. None otherwise.
"""
for path in paths:
library = join_path(path, libname)
if os.path.exists(library):
return path
return None
......@@ -117,7 +117,8 @@ def caller_locals():
scope. Yes, this is some black magic, and yes it's useful
for implementing things like depends_on and provides.
"""
stack = inspect.stack()
# Passing zero here skips line context for speed.
stack = inspect.stack(0)
try:
return stack[2][0].f_locals
finally:
......@@ -128,7 +129,8 @@ def get_calling_module_name():
"""Make sure that the caller is a class definition, and return the
enclosing module's name.
"""
stack = inspect.stack()
# Passing zero here skips line context for speed.
stack = inspect.stack(0)
try:
# Make sure locals contain __module__
caller_locals = stack[2][0].f_locals
......
......@@ -136,9 +136,7 @@
# don't add a second username if it's already unique by user.
if not _tmp_user in path:
tmp_dirs.append(join_path(path, '%u', 'spack-stage'))
for path in _tmp_candidates:
if not path in tmp_dirs:
else:
tmp_dirs.append(join_path(path, 'spack-stage'))
# Whether spack should allow installation of unsafe versions of
......
......@@ -59,6 +59,11 @@
SPACK_DEBUG_LOG_DIR = 'SPACK_DEBUG_LOG_DIR'
# Platform-specific library suffix.
dso_suffix = 'dylib' if sys.platform == 'darwin' else 'so'
class MakeExecutable(Executable):
"""Special callable executable object for make so the user can
specify parallel or not on a per-invocation basis. Using
......@@ -208,7 +213,7 @@ def set_module_variables_for_package(pkg, module):
# TODO: of build dependencies, as opposed to link dependencies.
# TODO: Currently, everything is a link dependency, but tools like
# TODO: this shouldn't be.
m.cmake = which("cmake")
m.cmake = Executable('cmake')
# standard CMake arguments
m.std_cmake_args = ['-DCMAKE_INSTALL_PREFIX=%s' % pkg.prefix,
......@@ -246,6 +251,9 @@ def set_module_variables_for_package(pkg, module):
# a Prefix object.
m.prefix = pkg.prefix
# Platform-specific library suffix.
m.dso_suffix = dso_suffix
def get_rpaths(pkg):
"""Get a list of all the rpaths for a package."""
......@@ -270,21 +278,6 @@ def parent_class_modules(cls):
return result