MacBook Air 2020 SMC PRAM reset

MacBook Air 2020, unresponsive if monitor is closed and reopened. MacBook Air 2020, charger not recognized after monitor is closed and reopened.

SMC Reset:  Plug in charger.  Press right shift, left control, left option keys for 7 sec, then press power key. Hold all four keys for 7 more seconds.  Machine -may- restart on its own after a period of time.

PRAM Reset:  Hold option, command, P, and R for 20 seconds until second chime or Apple logo appears and disappears a second time.

LC_ALL=C grep -F to search large fasta files for specific read names

PLINK STRUCTURE output to NTSYS PCOMC input

#create input for NTSYS/PCOMC from *strct_in files
f="ApolloLeafCASDENfbpQ30snp.str0505.recode.strct_in"; # STRUCTURE output file from PLINK

ncl=$(head -1 "$f" | awk -F' ' '{print NF}'); #calculate number of loci from 1st line of *strct_in
nrw=$(tail -n +3 "$f" | wc -l); #number of samples

#make the .ntb file
ntbfn="ALLCS"; #NTSYS input file name
echo "*simgend d=row c=band o="$ntbfn".nts r=Dist"$ntbfn".nts
*dcenter o=Dist"$ntbfn".nts  r=Dcent"$ntbfn".nts
*eigen o=Dcent"$ntbfn".nts  n="$nrw" r=proj"$ntbfn".nts" > "$ntbfn".ntb;
unix2dos "$ntbfn".ntb;

#make the .nts file
#create header
r="1 "$nrw"B $(( $ncl * 2 ))L 1 -9"; #matrix description line
s=""; #locus description line
for i in $(seq 1 1 $ncl);
  do s+=$i"a "$i"b ";
  done;
header=$(echo "$r"; echo "$s" | sed 's/ $//');    

#create data matrix
labs=$(tail -n +3 "$f" | cut -d' ' -f1);
matx=$(tail -n +3 "$f" | cut -d' ' -f3- | sed 's/0/-9/g');
matxx=$(paste -d' ' <(echo "$labs") <(echo "$matx"));

echo "$header"$'\n'"$matxx" > "$ntbfn".nts; #assemble the .nts file
unix2dos "$ntbfn".nts;

Bash one-liner to calculate md5sum by line, using Python

python -c "exec(\"import hashlib, sys\nfor line in sys.stdin:\n\tprint hashlib.md5(line).hexdigest()\")" <<<123$'\n'456


Output (includes newline):
ba1f2511fc30423bdbb183fe33f3dd0f
d2d362cdc6579390f1c0617d74a7913d
 

Convert EMF file to PDF, PNG using ImageMagick

Install ImageMagick on a Windows machine.
magick infile.EMF outfile.pdf
magick infile.EMF outfile.png

Regex to extract citations and other parenthetical expressions from text

Print out parenthetical expressions using sed:
sed 's/(/\n(/g' InputFile.txt | sed 's/)/)\n/g' | grep "^(";

Remove parenthetical expressions using perl: 
perl -pe 's|(\(.*?\))||g' InputFile.txt;

Homebrew, Mac OS 10.11, Xcode 8, Clang, and the problem with thread_local

The version of the clang compiler that comes with Xcode 7 does not recognize the thread_local keyword.  Packages installed with Homebrew that use certain c++11 or higher commands may terminate during compilation with errors like "thread-local storage is not supported for the current target".  Xcode 8, which contains a suitable clang, cannot be run on MacOS 10.11.  Catch-22.

To work around, install a new version of gcc using Homebrew:

brew install gcc;

Now figure out which version of gcc was installed by Homebrew:

brew list gcc;

If gcc 8 was installed you will see something like: /usr/local/Cellar/gcc/8.2.0/bin/gcc-8
in the list of brew-installed gcc programs.  Now, tell Homebrew to use this newer version of gcc, instead of the old version of clang from xcode, to install your program.  As an example, I use the package poppler:

brew install --cc=gcc-8 poppler;

Quick create swap file on Rocks 6 cluster

swapon -s; #check existing swap
dd if=/dev/zero of=/state/partition1/swapfile bs=1024 count=200000k; #create swapfile, 200000k x 1kb = 200GB
mkswap /state/partition1/swapfile; #define swapfile
swapon /state/partition1/swapfile; #activate swapfile
cp /etc/fstab /etc/fstabORIG; #back up current fstab
echo "/state/partition1/swapfile          swap            swap    defaults        0 0" >> /etc/fstab; #make permanent

Quick create RAID 0 scratch drive

Assume two new drives have been installed which show up as /dev/sdb and /dev/sdc in fdisk -l.

mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 /dev/sdb /dev/sdc;
parted -s -a optimal /dev/md0 mklabel gpt; #assign a partition tree type
parted -s -a optimal /dev/md0 mkpart primary 0% 100%; #create a single partition containing the entire disk
parted /dev/md0 print; #show some specs
mkfs.ext4 /dev/md0; #define the file system for the partition /dev/md0
mkdir /scratch; #the drive will be called 'scratch'
mount -t ext4 /dev/md0 /scratch; #mount drive for all users.
chmod -R 777 /scratch; #allow rwx access to everybody
aa=$(blkid /dev/md0 | awk '{print $2}' | sed 's/\"//g'); #get UUID of new raid array /dev/md0
cp /etc/fstab /etc/fstabORIG; #backup original fstab
echo "$aa /scratch ext4 defaults 0 0" >> /etc/fstab; #add a line to fstab to automount
umount /dev/md0; #unmount the raid array to test fstab
mount -a; #run /etc/fstab to remount
df; #make sure /dev/md0 is there

fsck and repair on reboot

Problem: Input/output errors observed on disk access. dmesg or cat /var/log/messages shows "Unrecovered read error".
Solution: Your hard disk is bad. At a minimum it has some bad sectors. You can try to repair it by forcing the utility fsck to run on reboot. You must be root to do this.

su;
touch /forcefsck; #the presence of this file tells system to fsck on boot
echo "-y" > /fsckoptions; #option to automatically repair errors encountered
reboot;

The system will remove these files after completing the fsck.

The Rocks version looks like this:

su;
ssh compute-0-1 'touch /forcefsck';
ssh compute-0-1 'echo -y > /fsckoptions';
ssh compute-0-1 'reboot';
exit;

If there are problems and the fsck won't complete, you may need to boot using a live disk and remove forcefsck and fsckoptions manually. This because the Rocks admin password may not work on a compute node.

Another approach is to search for bad blocks and write them to the 'bad block inode', so they will not be used in the future. This can be done interactively, e.g:
umount /dev/sda5;
e2fsck -ck /dev/sda5; #use badblocks read-only test to find bad blocks, faster.
or
e2fsck -cck /dev/sda5; #use badblocks read/write test to find bad blocks, slow.

Add new RAID 5 hard drive array to Rocks cluster without reboot, using MegaCli

These are some basic notes on how to add storage to your Rocks cluster.  Everybody has a different situation, mine was Rocks 6.2, CentOS 6, five unoccupied hard drive bays in the head node.  This procedure has been lifted from various places on the internet.

lspci | grep RAID; #determine if you have an LSI Logic RAID bus controller, if not stop here
fdisk -l; #list all devices, make a note of output

    #add identical hard drives to all available bays
    #login as root
cd /var/tmp;
wget http://techedemic.com/wp-content/uploads/2015/10/8-07-14_MegaCLI.zip; #download MegaCli, a command line utility to set up RAID
unzip 8-07-14_MegaCLI.zip;
cd Linux;
rpm -Uvh MegaCli-8.07.14-1.noarch.rpm; #install the package
alias MegaCli="/opt/MegaRAID/MegaCli/MegaCli64"; #set an alias to the executable for this session, add it to the root user .bashrc if you want it to be permanent

    #some useful MegaCli commands to describe the situation
MegaCli -PDlist -aAll; #describe eligible drives
MegaCli -PDlist -aAll | grep -A2 "Enclosure Device"; #list the enclosure, slots, and their assignment to DiskGroups, i.e. Virtual drive for eligible drives
MegaCli -PDlist -aAll | grep 'Firmware state'; #status of eligible drives
MegaCli -LDInfo -Lall -a0; #list the virtual drives that have been defined thus far on adapter 0
MegaCli -PDlist -aAll | grep 'Foreign State'; #determine if any disks are in the foreign state, this may mean they were previously used on another machine, if so, clear them
MegaCli -CfgForeign -Clear -aAll; #clear any foreign configurations, verify using above command to check foreign state

    #show mapping of physical drives to virtual drives
MegaCli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g';

    #from the above commands compile a list of values like:
Adapter ID: 0
Enclosure ID: 32
Physical Drive IDS (slots): 1,2,3,4,5
Raid Level: 5

    #Basic command syntax to create the RAID is:
    #MegaCli -CfgLdAdd -rX[enclosure_id:physical_id,enclosure_id:physical_id] -aN; #where X=RAID level, N=Adapter ID
    #for the situation described above use the following to create a RAID 5 array from drives in slots 1-5 (slot 0 held the boot drive, don't mess with it)
MegaCli -CfgLdAdd -r5[32:1,32:2,32:3,32:4,32:5] -a0;
    #confirm that you now have a new virtual drive, containing the physical drives you specified
MegaCli -LdPdInfo -a0 | grep -E "Virtual Drive:|Slot Number:" | xargs | sed -r 's/(Slot Number:)(\s[0-9]+)/\2,/g' | sed 's/(Target Id: .)/Physical Drives ids:/g' | sed 's/Virtual Drive:/\nVirtual Drive:/g'
MegaCli -LDInfo -Lall -a0; #list the virtual drives that have been defined thus far on adapter 0

fdisk -l; #list all devices, you should have a new one,  mine appeared as /dev/sdc

#delete the virtual drive later if necessary using: MegaCli -CfgLdDel -Lx -aN

    #make a partition
parted /dev/sdc print; #view disk specs
parted -s -a optimal /dev/sdc mklabel gpt; #assign a partition tree type
parted /dev/sdc print;
parted -s -a optimal /dev/sdc mkpart primary 0% 100%; #create a single partition containing the entire disk
parted /dev/sdc print;
mkfs.ext4 /dev/sdc1; #define the file system for the partition sdc1
parted /dev/sdc print;

    #mount the drive to a shared location. Usually this is /export, which is generally just a symlink to /state/partition1.  Those paths are used interchangeably below:
mkdir /export/space; #the drive will be called 'space'
mount -t ext4 /dev/sdc1 /export/space; #mount drive for all users. to unmount: umount /export/space
chown -R root:google-otp /export/space; #change ownership to google-otp group, which should include root and all users automatically
chmod -R 777 /export/space; #allow rwx access to everybody

    #share drive to nodes, it will be accessible at /share/space:
cp /etc/exports /etc/exportsORIG; #preserve the original /etc/exports file
echo '/state/partition1/space 10.1.1.1(rw,async,no_root_squash) 10.1.0.0/255.255.0.0(rw,async)' >> /etc/exports; #add the shared drive description to /etc/exports
/etc/rc.d/init.d/nfs restart; #restart nfs
cp /etc/auto.share /etc/auto.shareORIG; #preserve the original /etc/auto.share file

echo 'space YOURHEADNODENAME.local:/state/partition1/&' >> /etc/auto.share; #where YOURHEADNODENAME is just that

make -C /var/411; #update the 411 configuration

Enable user ssh to compute nodes to use GNU parallel on a Rocks cluster running Slurm

Problem: When using Slurm as your scheduler for a Rocks cluster users cannot ssh to compute nodes unless they have a job running there.  This means you cannot use GNU Parallel's --sshloginfile option to parallelize across the cluster.  There are some crazy workarounds out there to use GNU Parallel under Slurm, like sending each Parallel command to a separate srun command, but that is a hassle when the functionality is already built in to Parallel.
Solution: Slurm uses a PAM (Pluggable Authentication Module) to control user access to some features.  To permit all users to freely ssh to any compute node, turn off the Slurm PAM:
1. Verify the setting for the Rocks attribute 'slurm_pam_enable':
rocks list attr
It should be true.
2. Make it false, and verify:
rocks set attr slurm_pam_enable false
rocks list attr
3. Send the setting to all the compute nodes:
rocks sync slurm
4. Verify:
rocks list host attr | grep slurm
You should see something like:

compute-0-0:     slurm_pam_enable            false          G    
compute-0-0:     slurm_pam_enable_old     true           G    
compute-0-1:     slurm_pam_enable            false          G    
compute-0-1:     slurm_pam_enable_old     true           G    

The old attribute has been renamed slurm_pam_enable_old and retains the old setting, true.
5. If rocks sync slurm didn't take, kickstart the nodes and verify the change as above:
rocks run host '/boot/kickstart/cluster-kickstart'
6. Log out and log in under your user name, not root, and test ssh, e.g.:
ssh compute-0-0
You should arrive at the login prompt for the node.
7. Try out GNU Parallel with --sshloginfile now.  Go Ole!

(You probably shouldn't do this if you have a bunch of users.)

Add shared SSD drive to Rocks cluster

Problem: You might want to add an SSD drive to speed up procedures that have a lot of disk I/O.  What follows is a description of just one way to do this.  If you are clever, you can probably manage without any restarts.  I am not so clever.  The description assumes that the cluster is named "MyCluster" and the SSD drive is named "SSDscratch". You will need to be root.
Solution:
1. Plug in the drive to the head node. Reboot.
2. List disks and their partitions:
/sbin/fdisk -l
3. Find your new SSD drive in the output. In my case it was called "/dev/sdb". It probably needs to be partitioned and formatted so fdisk may say something like "Disk /dev/sdb doesn't contain a valid partition table."
4. Partition the disk
/sbin/fdisk /dev/sdb
     Follow menu items:
     >n (create a new partition)
        >p (primary partition)
           >1 (partition number)
              >accept defaults to make the entire disk a single partition
     >w (write the new partition)
5. Verify that the partition table for the SSD is how you want it:
/sbin/fdisk -l
6. Find out what disk format other volumes on the head node are using (ext3, ext4, etc).
df -T
7. Format the SSD similarly.  I will use ext3.
/sbin/mkfs.ext3 /dev/sdb1
8. Create a mount point, make it writable by everyone, mount the disk, verify it mounted. It's a good idea to keep it in the /export/home directory since that is a place that Rocks likes to share.
mkdir /export/home/SSDscratch
chmod a+w /export/home/SSDscratch
mount /dev/sdb1 /export/home/SSDscratch
df -T

9. Modify /etc/exports so NFS shares the new mount:

cp /etc/exports /etc/exportsORIG #backup the original file

vi /etc/exports

     Using vi (or whatever you like), add line like:"/export/home/SSDscratch 10.1.1.1(rw,async,no_root_squash) 10.1.0.0/255.255.0.0(rw,async)"

10. Modify /etc/auto.share:

cp /etc/auto.share /etc/auto.shareORIG

vi /etc/auto.share

     Add line like:"SSDscratch MyCluster.local:/export/home/SSDscratch"

11. Modify /etc/auto.home:

cp /etc/auto.home /etc/auto.homeORIG

vi /etc/auto.home

     Add line like:"SSDscratch    -nfsvers=3      MyCluster.local:/export/home/SSDscratch"

12. Modify /etc/fstab so it automatically mounts:

cp /etc/fstab /etc/fstabORIG

vi /etc/fstab

     Add line like:"/dev/sdb1     /export/home/SSDscratch     ext3     defaults     0 0"

13. Restart NFS and sync cluster. This sends the modified files to all nodes:

/sbin/service nfs restart

rocks sync users

14. To load the new settings on the nodes, I had to reboot them:

rocks run host 'reboot'

15. To verify that the SSD is mounted automatically, reboot the head node. (If it won't reboot, the problem is likely in the /etc/fstab file. Revert /etc/fstab to /etc/fstabORIG by booting from a live disk and try again.):
reboot
16. Verify that the SSD directory is mounted on all nodes. It should mount at /home/SSDscratch:

rocks run host compute 'hostname; ls -l /home/SSDscratch'

     If not, rocks sync users again and reboot. You may have to reboot twice for unknown reasons.

Congratulate yourself. That was a lot of work.

MPI_Bcast large dynamic char array

SysAdmins hate it when all your MPI procs access their precious disks.  The preferred procedure is to read from the disk once, using a single proc, then pass the data to all of the other procs.  In principle MPI_Bcast makes this easy.  In practice...you decide.

I needed to do this for arbitrary file sizes of a gigabyte or more containing string data.  The process is conceptually simple:

1. Use the root process (proc 0) to get the file size.
2. MPI_Bcast this file size to all procs.
3. Initialize and size a char array to contain the file data on all procs.
4. Use proc 0 to read the data file.
5. MPI_Bcast the file data to all procs.

Here is a prototype.
Save:  testmpi.cpp.
Compile:  mpic++ -o t testmpi.cpp.
Run:  mpirun -np 8 t yourbigfile.txt
-------
#include <algorithm>
#include <fstream>
#include <mpi.h>
#include <stdio.h>
#include <string.h>

using namespace std;

//determine the size of a file
std::ifstream::pos_type filesize(const char* filename)
{
std::ifstream in(filename, std::ifstream::ate | std::ifstream::binary);
return in.tellg(); 
}

//quickly reads a large .dat file into a memory buffer
char * MyBigRead(char* DatFilePath)
{
FILE * pFile;
unsigned long long lSize;
char * buffer;
size_t result;

pFile = fopen ( DatFilePath , "r" );
if (pFile==NULL) {fputs ("File error",stderr); exit (1);}

// obtain file size:
fseek (pFile , 0 , SEEK_END);
lSize = ftell (pFile);
rewind (pFile);

// allocate memory to contain the whole file:
buffer = (char*) malloc (sizeof(char)*lSize);
if (buffer == NULL) {fputs ("Memory error",stderr); exit (2);}

// copy the file into the buffer:
result = fread (buffer,1,lSize,pFile);
if (result != lSize) {fputs ("Reading error",stderr); exit (3);}

fclose (pFile);
return buffer;
}

int main(int argc, char* argv[]) {

MPI::Init ();
int procid = MPI::COMM_WORLD.Get_rank ( );  //Get the individual process ID.
int nprocs = MPI::COMM_WORLD.Get_size ( );

unsigned long long f = 0; //dat file size

//get the size of the dat file using proc 0
if (procid == 0)
{
f = (unsigned long long)filesize(argv[1]); 
f = f + 1; //increase by 1 to accomodate \0 string terminator
}

//broadcast file size to all procs
MPI_Bcast(&f, 1, MPI_UNSIGNED_LONG_LONG, 0, MPI_COMM_WORLD);

//initialize and size the data structure to hold contents of dat file
char * DatFileBuffer = (char*)malloc(f);

//report before MPI_Bcast'ing the data
printf("[proc%d]Before: snippet of DatFileBuffer:>%.5s<, size:%d\n", procid, DatFileBuffer, f);

//read the dat file from disk using proc 0
if (procid == 0)
{
char * d = MyBigRead(argv[1]);

//convert the char* to a char array
strcpy(DatFileBuffer, d); //copy data read into char * d to the pre-sized DatFileBuffer
}

//broadcast the dat file contents to all procs
MPI_Bcast(&DatFileBuffer[0], f, MPI_CHAR, 0, MPI_COMM_WORLD);

//report after MPI_Bcast'ing the data
printf("[proc%d]After: snippet of DatFileBuffer:>%.10s<, size:%d\n", procid, DatFileBuffer, f);

MPI_Finalize();
return 0;
}
-------

Convert PLINK to fastPHASE with BASH and GNU Parallel

Problem: The converter built in to PLINK converts nucleotide encoded data to a 0/1 binary format automatically when using --recode-fastphase to create a fastPHASE input file. This causes irretrievable loss of information.

Solution:  I can't find any way around this other than to make a new converter.
1. Using PLINK, recode .ped file such that each chromosome has its own .ped/.map set of files.  Something like:
./plink --file mydata --chr 1 --recode --out malc1;
./plink --file mydata --chr 2 --recode --out malc2;

2. Use the following script to create fastPHASE input files. Enter your values for 'nchr', 'iroot', and 'oroot' after the function dbl().

dbl() {

i=$1;

f="$infile"".ped";

nrow=$(wc -l "$f" | awk '{print $1}');

echo "splitting row $i/$nrow";

# split lines containing genotypes, by allele, for each individual

#odd alleles

odd=$(sed -n $i"p" "$f" | tr ' ' '\n' | tail -n +7 | sed -n 'p;n' | tr '\n' ' ' | sed 's/ $//g');

#even alleles

even=$(sed -n $i"p" "$f" | tr ' ' '\n' | tail -n +7 | sed -n 'g;n;p' | tr '\n' ' ' | sed 's/ $//g');

echo "$odd" > "$i.tmp";

echo "$even" >> "$i.tmp";

}

export -f dbl;

nchr=17; #number of chromosomes

iroot="malc"; #root string of input files for each chromosome

oroot="chr"; #root string for output files, in fastPHASE format

for ((i=1;i<=$nchr;i++));

  do echo "chr"$i;

    export infile="$iroot""$i";

    h1=$(wc -l "$infile".ped | awk '{print $1}'); #n samples for fastPHASE header line 1

    h2a=$(head -1 "$infile".ped | awk '{print NF}');

    h2=$((($h2a-6)/2)); #n loci for fastPHASE header line 2

    h3a=$(awk '{print $4}' "$infile".map);

    h3=$(echo "P $h3a" | tr "\n" " "); #locus positions for fastPHASE header line 3

    s=$(awk '{print $2}' "$infile".ped);

    

    #split single line per sample plink format into two line per sample fastPHASE format

    seq 1 $h1 | parallel --env dbl --env infile dbl;

    #Reassemble into a single file:

    outfile="$oroot""$i".inp;

    echo "$h1" > "$outfile";

    echo "$h2" >> "$outfile";

    echo "$h3" >> "$outfile";

    for ((j=1;j<=$h1;j++));

      do sample=$(echo "$s" | sed -n $j"p")" ";

        echo "writing chr $i sample $sample $j/$h1";

        echo '>'"$sample" >> "$outfile";

        head -1 "$j.tmp" | sed 's/0/?/g' >> "$outfile";

        tail -n +2 "$j.tmp" | sed 's/0/?/g' >> "$outfile";

        rm "$j.tmp";

       done;

  done;

3. Phase away.

Transpose large data matrix using BASH. II. GNU Parallel.

In a prior post, I presented a low memory BASH solution for transposing large data matrices.  Here is a way to speed that basic procedure using parallel processing on an HPC.

1. Generate a large data table for testing (~2GB, ~1E9 elements):
ncol=2472;
nrow=404627;
seq -s' ' 1 $ncol > m.txt;
foo=$(for ((i=1; i<=$ncol; i++));
do
   echo $[ 1 + $[ RANDOM % 4 ]]; 
done;);
foo=$(echo $foo | tr "\n" " ");
export nrow;
export foo;
perl -e 'for($i=0;$i<$ENV{nrow};$i++){print "$ENV{foo}\n"}' >> m.txt;

Notes: In the 3rd line, a header is created such that columns will be labeled consecutively.  These become important later.  Watch this step, some Linux versions add a linebreak, others do not.  You want the linebreak.

2. Run on HPC using GNU Parallel:
InputFile="m.txt";
seq 1 $ncol | parallel --sshloginfile ~/machines --jobs 24 "cut -d' ' -f{} $InputFile | tr '\n' ' ' | sed 's/ $/\n/g' > ~/{}.txt; echo Col {};";

Notes: The method above works as follows. First, seq delivers a set of numbers (from 1 to the total number of columns in the input matrix) to GNU Parallel.  GNU Parallel then distributes $ncol jobs among nodes specified in the file ~/machines.  The option --jobs 24 specifies that each node has 24 cores.  This approach cuts a single column from the input file, transposes it, then writes it to disk.  I had no luck with the GNU Parallel option --keep-order, which would presumably allow one to avoid this intermediate write step.

3. Fuse the output files together:
> mrot.txt;
for ((i=1; i<=$ncol; i++));
do
   cat "$i.txt" >> mrot.txt;
   rm "$i.txt";
done;

Fun with triangular matrices and bash

val="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15";

j=1; k=0;
v=$(echo $val | wc -w); #number of elements in $val
r=$(printf "%.0f" $(echo "sqrt(2*$v+(1/4))-(1/2)" | bc -l)); #number of rows needed in matrix to accomodate $v elements

#Lower triangle, by row
for ((i=1;i<=$r;i++));
  do m=$(($j+$k));
    j=$(($j+$k)); k=$(($k+1));
    n=$(printf "%.0f" $(echo "($k+1)*($k/2)" | bc -l));
    echo $val | cut -d' ' -f$m-$n;
done;

1

2 3

4 5 6

7 8 9 10

11 12 13 14 15

#Upper triangle, by row, tab-delimited
nblanks=0;
nvals=$(($r - 1));
m=1;
mat=$(
for ((i=1;i<=$r+1;i++));
  do 
    #write blanks
    for ((j=1;j<=$nblanks;j++)); do echo -n $'X\t'; done;

    #write zero on diagonal
    echo -n $'0\t';
    
    #write values from input
    n=$(($m + $nvals));
    echo -n $val | cut -d' ' -f$m-$n | tr " " "\t";
    
    #update
    nblanks=$(($nblanks + 1));
    nvals=$(($nvals - 1));
    m=$(($n + 1));
  done;
);
echo "$mat";  

0     1     2     3     4     5

X     0     6     7     8     9

X     X     0     10    11    12

X     X     X     0     13    14

X     X     X     X     0     15

X     X     X     X     X     0

#Lower triangle, by column, comma-delimited
  #rotate the matrix
  p=$(echo "$mat" | head -1 | awk '{print NF}');
  for (( i=1; i<=$p; i++ ));
    do
     foo=$(echo "$mat" | cut -d$'\t' -f$i);
     echo $foo | tr " " "," | sed 's/X//g';
    done  

0,,,,,

1,0,,,,

2,6,0,,,

3,7,10,0,,

4,8,11,13,0,

5,9,12,14,15,0

Windows 7 will not connect to Mac OS 10.11 El Capitan

1. Verify that you can ping the El Capitan server from the Windows 7 machine.
2. Portscan the El Capitan server, make sure port 445 is responding.

All good?  Ok, here's where it gets ugly.

3. Open the command window, CMD.EXE, on Windows. Type 'regedit'.
4. Expand folders:
     HKEY_LOCAL_MACHINE
          SYSTEM
               CurrentControlSet
                    Control
5. Select the folder 'Lsa' within 'CurrentControlSet'. A list of keys should appear to the right in the Registry Editor application.
6. Select the key called "lmcompatibilitylevel". Choose "Modify" from the Edit menu.
7. Set the "Value data:" field to "3". Click "OK".
8. Quit Registry Editor.
9. Try connecting again.

Calculate variance and standard deviation using Welford's algorithm and AWK

boo="1 2 3 4 5 6 0.257143 0.278571 0.285714 0.271429 0.235714";
awk '{
  sum = 0;
  M = 0;
  S = 0;
  for (k=1; k <= NF; k++) {
    sum += $k;
    x = $k;
    oldM = M;
    M = M + ((x - M)/k);
    S = S + (x - M)*(x - oldM);
  }
  var = S/(NF - 1);
  print "n=" NF " mean=" sum/(NF) " var=" var " sd=" sqrt(var);
}' <<< $boo


n=11 mean=2.02987 var=4.60305 sd=2.14547

GNU Parallel and Rocks clusters I. Distributing BASH scripts across nodes

1. Install GNU Parallel as user:
(wget -O - pi.dk/3 || curl pi.dk/3/) | bash

Path to executable should now be:  ~/bin/parallel
Modify PATH if necessary so that the GNU Parallel you just installed is used preferentially to any other GNU parallel on the system. I did this by putting my ~/bin directory ahead of the other entries in PATH in my ~/.bash_profile file.  Like so:

# User specific environment and startup programs
PATH=$HOME/bin:$PATH
export PATH


2. Test:
rocks run host command="~/bin/parallel ::: hostname"

The output should look something like:
compute-0-0: down
compute-0-1.local
compute-0-2.local
compute-0-5.local
compute-0-4.local

Here each compute node should receive a request (via "rocks run host") to print its name to standard output ("hostname") which is executed via GNU Parallel.  My cluster happens to be missing compute-0-3, and has a permanently dead node called compute-0-0, which explains the weirdness in the output above.


3. A more complex test.  Here we want to parallelize a simple subroutine across all nodes in the cluster.  We need a file that names all nodes we want to send jobs to.  Let's call it ~/machines. It should look something like:

cat ~/machines
compute-0-1
compute-0-2
compute-0-4
compute-0-5

In the BASH script below, the simple subroutine ("subr()") creates a couple variables ("a" and "b") from the arguments sent to it by GNU Parallel.  It then echoes those variables plus the hostname of the node running it.

subr() needs to be exported to the shell so that GNU parallel can access it ("export -f subr") from the various independent shells it creates.

The "parallel" command is assembled as follows:
parallel #call to the executable, you may need to use the full path ~/bin/parallel
--env subr #pass the exported subroutine to the new shell
--sshloginfile machines #path to the machines file with list of node, note this assumes passwordless ssh, the norm on Rocks clusters.
--jobs 24 #the number of cores available on the compute node
subr #a call to the subroutine
::: $a ::: $b #GNU Parallel syntax to manage the variable lists


4. Paste the following into the terminal:

subr() {
a=$1;
b=$2;
echo -n $a $b" ";
hostname;
}
export -f subr; #necessary for gnu parallel to work

a="1 2 3";
b="x y z";
parallel --env subr --sshloginfile machines --jobs 24 subr ::: $a ::: $b;

The output should look something like:
1 x compute-0-1.local
1 y compute-0-5.local
1 z compute-0-4.local
2 y compute-0-1.local
2 x compute-0-2.local
2 z compute-0-5.local
3 z compute-0-1.local
3 x compute-0-4.local
3 y compute-0-2.local

So each pairwise combination of variables has been "echoed" precisely one time, using the nodes specified in the "machines" file, as controlled by GNU Parallel. Go Ole!