Um blog sobre nada

Um conjunto de inutilidades que podem vir a ser úteis

Multiple GitHub Accounts on Windows with “wincred”

Posted by Diego em Dezembro 10, 2018

One of the first things I do when using git on windows is to cache my password so I don’t have to type it every time.  That is done by running:

git config --global credential.helper wincred

That works fine if there is only one account. If I tried to “use a second account” to clone a repo that the first one didn’t have access, I’d get a “FATAL: Repository not found.” Message.  Not really using the second account at this point because windows (or git) wouldn’t prompt me for credentials – so it was still using the first one. That happens because windows credential manager stores credentials by domain only.

In order to fix it, you need to set the credential.useHttpPath global config to change the credential manager’s behaviour to store credentials using the full HTTP path instead of the domain only.

git config --global credential.useHttpPath true

Posted in Git, I.T. | Leave a Comment »

Fixing Git – Anonymous access to /.git denied.

Posted by Diego em Março 15, 2018

Error when pushing to git:

remote: Anonymous access to <user>/<repo>.git denied.
fatal: Authentication failed for ‘https://github.com/<user>/<repo&gt;.git/’

One possible Solution (caching password):


git config --global credential.helper wincred

Source

Posted in Git, I.T. | Leave a Comment »

Resizing EBS Volumes on Unix EC2 Instances

Posted by Diego em Fevereiro 21, 2018

Simply resizing a volume on the AWS console (or using any other option to do so) is not enough to increase the space available.
What is shown on the console after the resize is complete, wont be inline with the output of df –h (and the actual space available).

In fact, even though the lsblk command will notice the newly added space, it is still not available as this is what you’d expect to see after issuing the command (in my case from 300Gb –> 500Gb)

image

To fix that, just run:

sudo growpart /dev/xvda 1

and the lsblk command confirms that the partition 1 fills the available space on the volume xvda.  Then run:

sudo resize2fs /dev/xvda1

to resize each file system to the new volume capacity. Now df –h should be outputting the expected value.

Posted in AWS | Leave a Comment »

Installing tensorflow with GPU support on windows

Posted by Diego em Novembro 1, 2017

Installing tensorflow with GPU support on windows can be challenging. There are quite a few moving pieces and each one of those pieces have a specific version and will only work with that version. I spent a few hours trying to get that right, mainly because when I say that “cuda” was a dependency, the first thing I did was to install the latest version; Very soon I learned that it was not a good idea. So, if you are getting any of the errors bellow, this guide is for you.

This guide was written on October 2017, so the versions mentioned are related to that date.

ImportError: DLL load failed: The specified module could not be found.                   

ModuleNotFoundError: No module named ‘_pywrap_tensorflow_internal’

Failed to load the native TensorFlow runtime.

First of all, this script is your best friend during the process (TensorFlow on Windows self-check). It will show which pieces you are missing. Eventually this is what we are looking for:

image

I will go through the process but it all boils down to having the correct (matching) versions of tensorflow, CUDA and cuDNN (the last two being more problematic)

1)    Install Visual Studio:

I was trying this process on a brand new laptop so the first thing I had to do is install Visual Studio (I presume it is for the C++ compiler), which can be downloaded from here.

2)    Download the NVIDIA CUDA Toolkit (1.4 Gb):

Be careful, if you go to the CUDA downloads page (https://developer.nvidia.com/cuda-downloadsy) you will end up downloading the most recent version (CUDA 9 at the moment), and tensorflow uses CUDA 8.

With CUDA 8 there will also be a data patchTo get CUDA 8, go to https://developer.nvidia.com/cuda-toolkit-archive


FYI: I made the mistake of downloading and installing CUDA 9 before realizing 8 was the correct version so I ended up having two versions installed. I didn’t seem to have mattered but I did however get this message when installed version 8:

image

 

3)    Download cudNN

cuDNN is a wrapper of NVIDIA’s cuDNN library, which is an optimized library for CUDA containing various fast GPU implementations, such as for convolutional networks and RNN modules. (Installation guide: http://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#install-windows0)   

Here I made another mistake. I went straight to the frameworks page (https://developer.nvidia.com/deep-learning-frameworks) and clicked the link associated with tensorflow. That led me to download cuDNN v7 but tensorflow needs v5.1 (or 6).

The link you want is this: https://developer.nvidia.com/rdp/cudnn-download

And select version:

·       “cuDNN v5.1 (Jan 20, 2017), for CUDA 8.0” if you have tensorflow version <1.3

·       “cuDNN v6.0 (April 27, 2017), for CUDA 8.0”  if you have tensorflow version 1.3

               Then download and unzip the file and copy the files to the respective locations on the CUDA folder:

· bin\cudnn64_7.dll to \CUDA\v8.0\bin.

· nclude\cudnn.h to \CUDA\v8.0\include.

· lib\x64\cudnn.lib to  \CUDA\v8.0\lib\x64.

 

That should be enough to get you up and running.

Few other useful links:

·       https://developer.nvidia.com/rdp/cudnn-download

·       https://www.tensorflow.org/install/install_windows#CommonInstallationProblems

·       http://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/

Posted in Data Science, tensorflow | Leave a Comment »

Installing awscli on Cygwin

Posted by Diego em Junho 21, 2017

The normal way of installing the aws-cli is simply by running pip install awscli
However, If you do that from cygwin, it will install awscli in Window’s Anaconda Python installation, instead of in Cygwin’s Python (which is what we want). Then, when you run aws configure, you will get an error that the aws executable can’t be found. Like the one bellow (I have my python installed at c:\Anaconda2) :

 

can't open file '/cygdrive/c/Anaconda2/Scripts/aws': [Errno 2] No such file or directory

 

If I use the  which command to find out where python is installed, I can see it is looking at my windows installation:

 

 

 

The solution is to try the following from a cygwin shell:

wget rawgit.com/transcode-open/apt-cyg/master/apt-cyg
install apt-cyg /bin
apt-cyg install python
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py

 

At this point you can verify that python is installed in cygwin

 and then run:

pip install awscli

 


Posted in AWS, I.T., Python | Leave a Comment »

Using Spyder with the interpreter set by conda environment

Posted by Diego em Maio 31, 2017

 

Using anaconda, you can create an environment named "python3env" for example, and install the latest version of Python 3 as follows:

 

conda create -n python3env python=3 anaconda

activate python3env

 

After activating the environment, by just typing spyder, you will launch it using the 3.x interpreter:

 

clip_image001

clip_image003

 

More info: https://www.continuum.io/blog/developer-blog/python-3-support-anaconda

Posted in Python | Leave a Comment »

Easiest way to install xgboost on windows (download binaries – no need to compile )

Posted by Diego em Abril 5, 2017

1) (I am assuming both git and Anaconda are already installed).

2) Choose a place to have the installer files and clone the git repo:

 

git clone https://github.com/dmlc/xgboost.git xgboost_install

clip_image002

 

 

3) Download the libxgboost.dll file from here and copy it to the xgboost folder on: <install_dir>\python-package\xgboost\

 

clip_image004

 

 

4) Navigate to the python_package folder and run:

python setup.py install

clip_image006

That should work fine.

If, however, you get the error bellow – like I did – when trying to import the library:

 

 

WindowsError: [Error 126] The specified module could not be found

 

here’s what I recommend:

After some debugging I found out the problem was on the from .core import DMatrix, Booster command, more specifically, on the “_load_lib()” function inside Core trying to run this line:

 

lib = ctypes.cdll.LoadLibrary(lib_path[0])

 

where lib_path[0] was precisely the file path for the libxgboost.dll I had just copied to the xgboost folder.

Since I was sure the file existed, I realized that maybe the DLL depended on other DLLs that could not be found. To check that, I downloaded dependency walker from this link, which showed me that the required VCOMP140.DLL was missing:

 

 

image

 

 

 

After some goggling, I discovered that the most common cause for that is that my machine did not have the C++ runtime installed, which I downloaded from here and eventually solved my problem:

image

 

Posted in Data Science, Machine Learning, Python, xgboost | Leave a Comment »

Scripting body and signature of Functions (Actian Matrix \ Redshift)

Posted by Diego em Fevereiro 19, 2017

 

Useful if you want to automatically manage permissions on the functions since you have to include the function signature on the grant\revoke statement.

 

SELECT proname, n.nspname||'.'||p.proname||'('||pg_catalog.oidvectortypes(p.proargtypes) ||')' as signature, prosrc as body
FROM pg_catalog.pg_namespace n 
JOIN pg_catalog.pg_proc p ON pronamespace = n.oid 

 

 

image

Posted in AWS Redshitft \ Actian Matrix, I.T., SQL | Leave a Comment »

How to move a VM file to another location

Posted by Diego em Novembro 25, 2016

1) Copy the "VirtualBox VMs" folder from its current location to the new location you desire;

3) Change the "Default Machine Folder" to the new location (go to File -> preferences -> General):

clip_image001

 

4) On the Virtual Box Manager, right click on your VM and click "Remove" -> “Remove only”.

clip_image002

5) Close and then reopen VM Manager

6) Go to “Machine” -> “Add” (it should default to the new folder) and re-add the VM

Posted in Uncategorized | Leave a Comment »

How to connect to PostgreSQL running on an Ubuntu VM

Posted by Diego em Setembro 19, 2016

 

FYI, PostgreSQL can be installed using:

 

                                

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

 

 

Optionally, install pgAdminIII (https://www.pgadmin.org/download/windows.php) .to test the connectivity.

 

Virtual Box creates virtual machines with the NAT network type by default. If you want to run server software inside a virtual machine, you’ll need to change its network type or forward ports through the virtual NAT.

With the NAT network type, your host operating system performs network address translation. The virtual machine shares your host computer’s IP address and won’t receive any incoming traffic. You can use bridged networking mode instead — in bridged mode, the virtual machine will appear as a separate device on your network and have its own IP address.

To change a virtual machine’s network type in VirtualBox, right-click a virtual machine and select Settings, go to “network” and change the “attached to” option to “Bridged Adapter”

 

clip_image001[4]

 

You can check your VM’s new IP by typing “ifconfig” or clicking on the top right corner icon -> “System Settings” -> “Network”:

 

clip_image003[4]

 

 

Then, navigate to Postgres’ installation folder (normally on: /etc/postgresql/9.5/main) and edit the postgresql.conf file setting it to whatever suits you (I set it to all):

 

                           

sudo vi postgresql.conf

clip_image005[4]

                      

sudo systemctl restart postgresql  #restart

 

That will make PostgreSQL to listen to all IPs, as we can see on this before\after:

clip_image007[4]

 

(That will allow you to connect locally using the server’s IP address, before this step you’d be able to connect only using “localhost”).

At last, edit the pg_hba.conf (https://www.postgresql.org/docs/9.5/static/auth-pg-hba-conf.html ) file, which controls client authentication, and add a row that allows all connections:

 

                   

sudo vi pg_hba.conf

clip_image009[4]

 

                   

sudo systemctl restart postgresql

 

 

By doing so, you should be able to access PostgreSQL from outside your VM.

Posted in PSQL | Leave a Comment »