The rattled cough of a Hyper-Empathetic PM

This post is just record-keeping for me to remember why I abandoned Ansible on Windows a few months back.

Here’s what I did to enable me to use Ansible for automation

• Since I’m using a Windows box as my primary desktop for now, I wanted to see if I could avoid running a Linux VM just to manage other *NIX boxes
• I thought I should be able to get away with using the Git for Windows environment rather than the full Cygwin environment that Jeff Geerling documents here (spoiler alert: I couldn’t, but it’s instructive to document how far I got and where it broke down)

Install and troubleshoot Ansible on Git for Windows

• I followed installation instructions from step (3) (including using those legacy versions of PyYAML and Jinja2)
• I skipped step (6) since I already had SSH keys

• I ran ansible –version, and saw errors like:

   $ ansible --version
   Traceback (most recent call last):
     File "C:/Users/Mike/code/ansible/bin/ansible", line 46, in <module>
      from ansible.utils.display import Display
     File "C:\Users\Mike\code\ansible\lib\ansible\utils\display.py", line 21, in <module>
      import fcntl
   ImportError: No module named 'fcntl'

• Tried resolving that dependency via http://stackoverflow.com/questions/1422368/fcntl-substitute-on-windows
• This just led to the following (which seemed a hint that I might encounter many such issues, not worth it):

   $ ansible --version
   Traceback (most recent call last):
     File "C:/Users/Mike/code/ansible/bin/ansible", line 46, in <module>
      from ansible.utils.display import Display
     File "C:\Users\Mike\code\ansible\lib\ansible\utils\display.py", line 33, in <module>
      from termios import TIOCGWINSZ
   ImportError: No module named 'termios'

• Gave in, uninstalled Python 3.5.1 and jumped back to latest Python 2.x (2.7.1)
  • Turned back the time machine to 2004 which appears to be the target Python for Ansible control systems
• Still couldn’t get past that error, so I gave up and went to Cygwin

Install and troubleshoot Ansible on Cygwin64

• http://www.jeffgeerling.com/blog/running-ansible-within-windows with Cygwin as a start
• different error to ansible –version:

  AttributeError: 'module' object has no attribute '_vendor'

Then I found this article, followed a combo of the top answer:

• apt-cyg remove python-cryptography
• git clone –depth 1 git://github.com/ansible/ansible
• apt-cyg install git python-{jinja2,six,yaml}
• wget rawgit.com/transcode-open/apt-cyg/master/apt-cyg && install apt-cyg /bin
• That results in this output:

   $ ansible --version
   ansible 2.2.0 (devel 37737ca6c1) last updated 2016/05/11 14:27:18 (GMT -700)
     lib/ansible/modules/core:  not found - use git submodule update --init lib/ansible/modules/core
     lib/ansible/modules/extras:  not found - use git submodule update --init lib/ansible/modules/extras
     config file =
     configured module search path = ['/opt/ansible/library']

• Then running this command from within the local ansible repo (e.g. from /opt/ansible) gets the basic modules you’ll need to get started (e.g. the “ping” module for testing)
  git submodule update –init –recursive
• Then I decided to test ansible’s ability to connect to the target host based on the command recommended in Ansible’s installation docs:

   $ ansible all -m ping --ask-pass
   SSH password:
   192.168.1.13 | FAILED! => {
      "failed": true,
      "msg": "to use the 'ssh' connection type with passwords, you must install the sshpass program"
   }

• OK, let’s get sshpass installed on my control device [Windows box] so that we can quickly bootstrap the SSH keys to the target…
• Since sshpass isn’t an available package in Cygwin package directory, the only way to get sshpass is to install it from source
  •  NOTE: I discovered this the hard way, after trying every trick I could think of to make all this work without having to deal with a C compiler
• Found this article for OS X users and followed it in my Windows environment (NOTE: I had to install the “make” and “gcccore” packages from Cygwin setup – and then having to re-run apt-cyg remove python-cryptography from the Cygwin terminal again because it gets automatically reinstalled by Cygwin setup)
• This time when running the command I get a different error – which means I must’ve got sshpass stuffed in the right location:

   $ ansible all -m ping --ask-pass
   SSH password:
   192.168.1.13 | FAILED! => {
      "failed": true,
      "msg": "Using a SSH password instead of a key is not possible because Host Key checking is enabled and sshpass does not support this.  Please add this host's fingerprint to your known_hosts file to manage this host."
   }

• I’m guessing this means I need to obtain the target host’s SSH public key (and not that the target host refused to connect because it didn’t trust the control host).  So then I had to harvest the target host’s fingerprint

   $ ssh -o StrictHostKeyChecking=ask -l mike 192.168.1.13
   The authenticity of host '192.168.1.13 (192.168.1.13)' can't be established.
   ECDSA key fingerprint is SHA256:RqBcq8aCAgohFeiTlPeYd8hLDfTz1A25ZPlzyxlDrqI.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added '192.168.1.13' (ECDSA) to the list of known hosts.
   mike@192.168.1.13's password:

• This time around when running the ping command I saw this error:

   $ ansible all -m ping --ask-pass
   SSH password:
   192.168.1.13 | UNREACHABLE! => {
      "changed": false,
      "msg": "Authentication failure.",
      "unreachable": true
   }

• I finally got the bright idea to poke around the target’s log files and see if there were any clues – sure enough, /var/log/auth.log made it clear enough to me:

   Invalid user Mike from 192.168.1.10
   input_userauth_request: invalid user Mike [preauth]
    pam_unix(sshd:auth): check pass; user unknown
    pam_unix(sshd:auth): authentication failure...
    Failed password for invalid user Mike from 192.168.1.10 port 58665 ssh2
    Connection closed by 192.168.1.10 [preauth]

• (Boy do I hate *NIX case-sensitivity at times like this – my Windows username on the control device is “Mike” and the Linux username on the target is “mike”)
• Tried this same command but adding the root user (ansible all -m ping –user root –ask-pass) but kept getting the same error back, and saw /var/log/auth.log reported “Failed password for root from 192.168.1.10”)
• Re-ran su root in bash on the target just to make sure I wasn’t forgetting the password (I wasn’t)
• Tried a basic ssh connection with that same password (to eliminate other variables):

   $ ssh root@192.168.1.13
   root@192.168.1.13's password:
   Permission denied, please try again.
   root@192.168.1.13's password:
   Permission denied, please try again.
   root@192.168.1.13's password:
   Permission denied (publickey,password).

• Tried the same thing with my “mike” user, success:

   $ ssh mike@192.168.1.13
   mike@192.168.1.13's password:

   The programs included with the Debian GNU/Linux system are free software;
   the exact distribution terms for each program are described in the
   individual files in /usr/share/doc/*/copyright.

   Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
   permitted by applicable law.
   Last login: Sat May 14 16:10:05 2016 from 192.168.1.10
   mike@debianbox:~$

• One more try with the ansible/SSH/password approach:

   $ ansible all -m ping --user mike --ask-pass
   SSH password:
   192.168.1.13 | UNREACHABLE! => {
      "changed": false,
      "msg": "Failed to connect to the host via ssh.",
      "unreachable": true
   }

• So close!!  Search led to this possibility, so I re-ran with -vvvv param to get this:

   $ ansible all -m ping --user mike --ask-pass -vvvv
   No config file found; using defaults
   SSH password:
   Loaded callback minimal of type stdout, v2.0
   <192.168.1.13> ESTABLISH SSH CONNECTION FOR USER: mike
   <192.168.1.13> SSH: EXEC sshpass -d44 ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=mike -o ConnectTimeout=10 -o ControlPath=/home/Mike/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.13 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1463273031.8-128175022355609 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1463273031.8-128175022355609 `" )'"'"''
   <192.168.1.13> PUT /tmp/tmp948_ec TO /home/mike/.ansible/tmp/ansible-tmp-1463273031.8-128175022355609/ping
   <192.168.1.13> SSH: EXEC sshpass -d44 sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o User=mike -o ConnectTimeout=10 -o ControlPath=/home/Mike/.ansible/cp/ansible-ssh-%h-%p-%r '[192.168.1.13]'
   192.168.1.13 | UNREACHABLE! => {
      "changed": false,
      "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
      "unreachable": true
   }

• I can definitely see the ansible-tmp-1463273031.8-128175022355609 file under ~/.ansible/tmp on the target system, so Ansible is getting authenticated and can run the initial shell commands
• But I’m not seeing the /ping file under that directory, and I’m wondering if there’s something preventing sftp from connecting to the target host (since that’s the final command being run just before I get back the error). The “sftp” program is available on the controller though.
• Digging around in Wireshark, I can see the SSHv2 traffic between controller and target, but after the initial key exchange I see exactly 3 encrypted packets sent from the controller (and encrypted responses from the target), and then no further communication between the two thereafter.  The only other activity I see on either system that’s unexplained is the target device ARP’ing for the local router three times, once every second, after the SSH traffic dies off
• After the first two initial successes in getting the tmp files pushed via ssh, I’ve since only had failures “Failed to connect to the host via ssh.”  Using ProcExp.exe to verify that there is actual network traffic being sent to the target’s IP, and using Wireshark to get some idea what’s getting through and what’s not (but Wireshark is acting up and no longer showing me traffic from controller to target, only target to controller responses, so it’s getting a little nuts at this point)
• I’ve added “PTR” records to the hosts files on both the controller and the target to resolve the IP address for each other to a defined name, but I’m still getting “failed to connect…” (even though I can confirm that the tools are using the newly-registered name, since I even tried substituting the falsified name in the ansible_hosts file for the previously-used IP address)
• I tried the advice from here to switch to SCP if SFTP might not be working, but that didn’t help (so I emptied out the .ansible.cfg file again)
• I don’t know where to look for log files to see exactly what errors are occurring locally, so I’m pretty much stumped at this point
• !!!!!! I pushed my ssh key to the target host, and the very next run of ansible all -m ping succeeded!!!! 😦
• CONCLUSION: ssh-pass doesn’t seem compatible with the Windows setup I’ve been using all weekend
• EPILOGUE: “Failed to connect…” error is back again when running ansible from Windows – I can see successful auth in the target’s /var/log/auth.log, but even -m ping fails (e.g. ansible all -m ping -u root)

Other References

http://everythingshouldbevirtual.com/ansible-using-ansible-on-windows-via-cygwin

https://help.ubuntu.com/community/SSH/OpenSSH/Keys