So…do not make fun of me…however I have some really old deployment pipelines lurking around. One of them is for my personal websites. The site actually contains three target environments. Definitely overkill for the volume of traffic this website receieves however even now my webiste servers as a testbed for researching the software devleopment lifecycle for relatievely static sites.

The deployment pipeline broke. When I check in the source repository it generates all of the static assets. From there the assets are checked into another repository for the alpha stage. Once pushed the repository uses a post-commit hook to unpack the resources into a directory. The assets are then served by a webserver.

According to my instance of Janky Jenkins it looks like the SSH key is no longer being accepted for some reason. Recently I noticed for some reason there were a bunch of ssh-agent processes being abandoned. In my infinite wisdom I terminated the sessions to reclaim some processes. I hope I did not do something janky like setup an ssh-agent process to explicitly add the key. Either way, the target account has the key enabled.

Next up is ensuring the target host is who I expect. Using a quick dig does confirm the correct target host is contained within the A record responses. Nothing in the ~/.ssh/config entry for the user. I don not override the SSH_AGENT_SOCK anywhere.

Manual SSH

Next up is to try manually pushing. I feel like breaking automated seals is a bad thing however sometimes there is no better method to diagnose the issues. Using something like su - jenkins -l -s /bin/bash gets you into the user. When manually tring to do ssh -vvv deployer@alpha.meschbach.com I got the following entry:

debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey                                                                                                                                                                                                                                         
debug1: Offering public key: /home/jenkins/.ssh/id_rsa RSA SHA256:{hash}
debug3: send packet: type 50        
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password

This generally indicates the key isn’t authroized for the target account. Next up is to check the SSH server logs.

Feb 26 19:36:38 kal sshd[28470]: User deployer not allowed because account is locked

Well, that would be a really good reason why I can no longer deploy to it. Now to figure out why the account has been locked and how to unlock it. From the build logs the user stopped working between November and January. Apparently I need to update my site more often!

$ sudo passwd --status deployer
deployer L 04/15/2013 -1 -1 -1 -1

I’ve infered the L confirms the user is actually locked. Although according to this reference it should show as LK which it definitely does not. I wonder if that is a distro difference. Unforunately my logs are not kept that long. In the future I might look at increase my log space since I have drives I just need to move data off of. Well that is a bit terrifying though. Hopefully the account was locked as a result of the upgrade.

Unlocking the Account

There were two parts. Apparently in the upgrade it decided to lock the account since there was not a password associated with the account since there shouldn’t have been password based logins. After fixing that issue and restoring the original machinery issuing passwd -u deployer worked like a charm.