Homebrew, Mac OS 10.11, Xcode 8, Clang, and the problem with thread_local

The version of the clang compiler that comes with Xcode 7 does not recognize the thread_local keyword.  Packages installed with Homebrew that use certain c++11 or higher commands may terminate during compilation with errors like "thread-local storage is not supported for the current target".  Xcode 8, which contains a suitable clang, cannot be run on MacOS 10.11.  Catch-22.

To work around, install a new version of gcc using Homebrew:

brew install gcc;

Now figure out which version of gcc was installed by Homebrew:

brew list gcc;

If gcc 8 was installed you will see something like: /usr/local/Cellar/gcc/8.2.0/bin/gcc-8
in the list of brew-installed gcc programs.  Now, tell Homebrew to use this newer version of gcc, instead of the old version of clang from xcode, to install your program.  As an example, I use the package poppler:

brew install --cc=gcc-8 poppler;

Quick create swap file on Rocks 6 cluster

swapon -s; #check existing swap
dd if=/dev/zero of=/state/partition1/swapfile bs=1024 count=200000k; #create swapfile, 200000k x 1kb = 200GB
mkswap /state/partition1/swapfile; #define swapfile
swapon /state/partition1/swapfile; #activate swapfile
cp /etc/fstab /etc/fstabORIG; #back up current fstab
echo "/state/partition1/swapfile          swap            swap    defaults        0 0" >> /etc/fstab; #make permanent

Quick create RAID 0 scratch drive

Assume two new drives have been installed which show up as /dev/sdb and /dev/sdc in fdisk -l.

mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 /dev/sdb /dev/sdc;
parted -s -a optimal /dev/md0 mklabel gpt; #assign a partition tree type
parted -s -a optimal /dev/md0 mkpart primary 0% 100%; #create a single partition containing the entire disk
parted /dev/md0 print; #show some specs
mkfs.ext4 /dev/md0; #define the file system for the partition /dev/md0
mkdir /scratch; #the drive will be called 'scratch'
mount -t ext4 /dev/md0 /scratch; #mount drive for all users.
chmod -R 777 /scratch; #allow rwx access to everybody
aa=$(blkid /dev/md0 | awk '{print $2}' | sed 's/\"//g'); #get UUID of new raid array /dev/md0
cp /etc/fstab /etc/fstabORIG; #backup original fstab
echo "$aa /scratch ext4 defaults 0 0" >> /etc/fstab; #add a line to fstab to automount
umount /dev/md0; #unmount the raid array to test fstab
mount -a; #run /etc/fstab to remount
df; #make sure /dev/md0 is there

fsck and repair on reboot

Problem: Input/output errors observed on disk access. dmesg or cat /var/log/messages shows "Unrecovered read error".
Solution: Your hard disk is bad. At a minimum it has some bad sectors. You can try to repair it by forcing the utility fsck to run on reboot. You must be root to do this.

su;
touch /forcefsck; #the presence of this file tells system to fsck on boot
echo "-y" > /fsckoptions; #option to automatically repair errors encountered
reboot;

The system will remove these files after completing the fsck.

The Rocks version looks like this:

su;
ssh compute-0-1 'touch /forcefsck';
ssh compute-0-1 'echo -y > /fsckoptions';
ssh compute-0-1 'reboot';
exit;

If there are problems and the fsck won't complete, you may need to boot using a live disk and remove forcefsck and fsckoptions manually. This because the Rocks admin password may not work on a compute node.

Another approach is to search for bad blocks and write them to the 'bad block inode', so they will not be used in the future. This can be done interactively, e.g:
umount /dev/sda5;
e2fsck -ck /dev/sda5; #use badblocks read-only test to find bad blocks, faster.
or
e2fsck -cck /dev/sda5; #use badblocks read/write test to find bad blocks, slow.

Add new RAID 5 hard drive array to Rocks cluster without reboot, using MegaCli

These are some basic notes on how to add storage to your Rocks cluster.  Everybody has a different situation, mine was Rocks 6.2, CentOS 6, five unoccupied hard drive bays in the head node.  This procedure has been lifted from various places on the internet.

lspci | grep RAID; #determine if you have an LSI Logic RAID bus controller, if not stop here
fdisk -l; #list all devices, make a note of output

    #add identical hard drives to all available bays
    #login as root
cd /var/tmp;
wget http://techedemic.com/wp-content/uploads/2015/10/8-07-14_MegaCLI.zip; #download MegaCli, a command line utility to set up RAID
unzip 8-07-14_MegaCLI.zip;
cd Linux;
rpm -Uvh MegaCli-8.07.14-1.noarch.rpm; #install the package
alias MegaCli="/opt/MegaRAID/MegaCli/MegaCli64"; #set an alias to the executable for this session, add it to the root user .bashrc if you want it to be permanent

    #some useful MegaCli commands to describe the situation
MegaCli -PDlist -aAll; #describe eligible drives
MegaCli -PDlist -aAll | grep -A2 "Enclosure Device"; #list the enclosure, slots, and their assignment to DiskGroups, i.e. Virtual drive for eligible drives
MegaCli -PDlist -aAll | grep 'Firmware state'; #status of eligible drives
MegaCli -LDInfo -Lall -a0; #list the virtual drives that have been defined thus far on adapter 0
MegaCli -PDlist -aAll | grep 'Foreign State'; #determine if any disks are in the foreign state, this may mean they were previously used on another machine, if so, clear them
MegaCli -CfgForeign -Clear -aAll; #clear any foreign configurations, verify using above command to check foreign state

    #show mapping of physical drives to virtual drives
MegaCli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g';

    #from the above commands compile a list of values like:
Adapter ID: 0
Enclosure ID: 32
Physical Drive IDS (slots): 1,2,3,4,5
Raid Level: 5

    #Basic command syntax to create the RAID is:
    #MegaCli -CfgLdAdd -rX[enclosure_id:physical_id,enclosure_id:physical_id] -aN; #where X=RAID level, N=Adapter ID
    #for the situation described above use the following to create a RAID 5 array from drives in slots 1-5 (slot 0 held the boot drive, don't mess with it)
MegaCli -CfgLdAdd -r5[32:1,32:2,32:3,32:4,32:5] -a0;
    #confirm that you now have a new virtual drive, containing the physical drives you specified
MegaCli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g'
MegaCli -LDInfo -Lall -a0; #list the virtual drives that have been defined thus far on adapter 0

fdisk -l; #list all devices, you should have a new one,  mine appeared as /dev/sdc

#delete the virtual drive later if necessary using: MegaCli -CfgLdDel -Lx -aN

    #make a partition
parted /dev/sdc print; #view disk specs
parted -s -a optimal /dev/sdc mklabel gpt; #assign a partition tree type
parted /dev/sdc print;
parted -s -a optimal /dev/sdc mkpart primary 0% 100%; #create a single partition containing the entire disk
parted /dev/sdc print;
mkfs.ext4 /dev/sdc1; #define the file system for the partition sdc1
parted /dev/sdc print;

    #mount the drive to a shared location. Usually this is /export, which is generally just a symlink to /state/partition1.  Those paths are used interchangeably below:
mkdir /export/space; #the drive will be called 'space'
mount -t ext4 /dev/sdc1 /export/space; #mount drive for all users. to unmount: umount /export/space
chown -R root:google-otp /export/space; #change ownership to google-otp group, which should include root and all users automatically
chmod -R 777 /export/space; #allow rwx access to everybody

    #share drive to nodes, it will be accessible at /share/space:
cp /etc/exports /etc/exportsORIG; #preserve the original /etc/exports file
echo '/state/partition1/space 10.1.1.1(rw,async,no_root_squash) 10.1.0.0/255.255.0.0(rw,async)' >> /etc/exports; #add the shared drive description to /etc/exports
/etc/rc.d/init.d/nfs restart; #restart nfs
cp /etc/auto.share /etc/auto.shareORIG; #preserve the original /etc/auto.share file

echo 'space YOURHEADNODENAME.local:/state/partition1/&' >> /etc/auto.share; #where YOURHEADNODENAME is just that

make -C /var/411; #update the 411 configuration

Enable user ssh to compute nodes to use GNU parallel on a Rocks cluster running Slurm

Problem: When using Slurm as your scheduler for a Rocks cluster users cannot ssh to compute nodes unless they have a job running there.  This means you cannot use GNU Parallel's --sshloginfile option to parallelize across the cluster.  There are some crazy workarounds out there to use GNU Parallel under Slurm, like sending each Parallel command to a separate srun command, but that is a hassle when the functionality is already built in to Parallel.
Solution: Slurm uses a PAM (Pluggable Authentication Module) to control user access to some features.  To permit all users to freely ssh to any compute node, turn off the Slurm PAM:
1. Verify the setting for the Rocks attribute 'slurm_pam_enable':
rocks list attr
It should be true.
2. Make it false, and verify:
rocks set attr slurm_pam_enable false
rocks list attr
3. Send the setting to all the compute nodes:
rocks sync slurm
4. Verify:
rocks list host attr | grep slurm
You should see something like:

compute-0-0:     slurm_pam_enable            false          G    
compute-0-0:     slurm_pam_enable_old     true           G    
compute-0-1:     slurm_pam_enable            false          G    
compute-0-1:     slurm_pam_enable_old     true           G    

The old attribute has been renamed slurm_pam_enable_old and retains the old setting, true.
5. If rocks sync slurm didn't take, kickstart the nodes and verify the change as above:
rocks run host '/boot/kickstart/cluster-kickstart'
6. Log out and log in under your user name, not root, and test ssh, e.g.:
ssh compute-0-0
You should arrive at the login prompt for the node.
7. Try out GNU Parallel with --sshloginfile now.  Go Ole!

(You probably shouldn't do this if you have a bunch of users.)

Add shared SSD drive to Rocks cluster

Problem: You might want to add an SSD drive to speed up procedures that have a lot of disk I/O.  What follows is a description of just one way to do this.  If you are clever, you can probably manage without any restarts.  I am not so clever.  The description assumes that the cluster is named "MyCluster" and the SSD drive is named "SSDscratch". You will need to be root.
Solution:
1. Plug in the drive to the head node. Reboot.
2. List disks and their partitions:
/sbin/fdisk -l
3. Find your new SSD drive in the output. In my case it was called "/dev/sdb". It probably needs to be partitioned and formatted so fdisk may say something like "Disk /dev/sdb doesn't contain a valid partition table."
4. Partition the disk
/sbin/fdisk /dev/sdb
     Follow menu items:
     >n (create a new partition)
        >p (primary partition)
           >1 (partition number)
              >accept defaults to make the entire disk a single partition
     >w (write the new partition)
5. Verify that the partition table for the SSD is how you want it:
/sbin/fdisk -l
6. Find out what disk format other volumes on the head node are using (ext3, ext4, etc).
df -T
7. Format the SSD similarly.  I will use ext3.
/sbin/mkfs.ext3 /dev/sdb1
8. Create a mount point, make it writable by everyone, mount the disk, verify it mounted. It's a good idea to keep it in the /export/home directory since that is a place that Rocks likes to share.
mkdir /export/home/SSDscratch
chmod a+w /export/home/SSDscratch
mount /dev/sdb1 /export/home/SSDscratch
df -T

9. Modify /etc/exports so NFS shares the new mount:

cp /etc/exports /etc/exportsORIG #backup the original file

vi /etc/exports

     Using vi (or whatever you like), add line like:"/export/home/SSDscratch 10.1.1.1(rw,async,no_root_squash) 10.1.0.0/255.255.0.0(rw,async)"

10. Modify /etc/auto.share:

cp /etc/auto.share /etc/auto.shareORIG

vi /etc/auto.share

     Add line like:"SSDscratch MyCluster.local:/export/home/SSDscratch"

11. Modify /etc/auto.home:

cp /etc/auto.home /etc/auto.homeORIG

vi /etc/auto.home

     Add line like:"SSDscratch    -nfsvers=3      MyCluster.local:/export/home/SSDscratch"

12. Modify /etc/fstab so it automatically mounts:

cp /etc/fstab /etc/fstabORIG

vi /etc/fstab

     Add line like:"/dev/sdb1     /export/home/SSDscratch     ext3     defaults     0 0"

13. Restart NFS and sync cluster. This sends the modified files to all nodes:

/sbin/service nfs restart

rocks sync users

14. To load the new settings on the nodes, I had to reboot them:

rocks run host 'reboot'

15. To verify that the SSD is mounted automatically, reboot the head node. (If it won't reboot, the problem is likely in the /etc/fstab file. Revert /etc/fstab to /etc/fstabORIG by booting from a live disk and try again.):
reboot
16. Verify that the SSD directory is mounted on all nodes. It should mount at /home/SSDscratch:

rocks run host compute 'hostname; ls -l /home/SSDscratch'

     If not, rocks sync users again and reboot. You may have to reboot twice for unknown reasons.

Congratulate yourself. That was a lot of work.