Recently I updated some code in my wandrlust project and was going to publish to live hosting the same way that I've done many times before. With everything setup properly, this is usually as easy as entering the command flyctl deploy, but this time (even though this worked just fine the day before) I was met by a warning and then an error message.
TLDR; What finally fixed it was this:
fly wireguard list
fly wireguard reset
fly wireguard removeThe error
WARN Failed to start remote builder heartbeat: failed building options: failed probing "personal": context deadline exceededError: failed to fetch an image or build from source: failed building options: failed probing "personal": context deadline exceeded
After a quick search, I realized that I was not the only one that encountered this issue so I had a few things to try. I did find it a bit worrying that the error just appeared out of nowhere since it worked just fine the day before and I did not change anything in regards to settings or really anything I can think of that could cause it.
What I tried
One of the first post I encountered on fly.io's own support forums suggested that I simply ran fly wireguard reset. This created a new WireGuard peer, but that alone did not solve the issue. Some reported that it was all it took, but no such luck for me.
Another post in fly.io's support forum suggested fly wireguard websockets enable since there had been communication issues before and this had helped a lot of people. For me however, again no luck - the error persisted.
Some of the posts suggested flyctl doctor and while it did not actually solve anything, I got some more information after a very long timeout:
$ fly doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)At this point I tried to upgrade the version of flyctl with the command flyctl version upgrade just in case that could have something to do with it. I was one minor version behind, but upgrading did nothing.
Now I went into the "Have you tried turning it off and on again?" territory and tried logout/login (fly auth logout followed by fly auth login), but again - nothing.
Yet another post suggested using SUDO. I'm on windows, but used WSL and tried it - no go.
Some users reported that this issue simply fixed itself some days later, but I did not want to wait.
All of the commands above had helped atleast somebody at some point in time, but none of them worked for me. I tried even more searching and finally ended up at this post that gave me the answer.
The idea was to show all the WireGuard peers, create a new one and then remove all older ones. I started with fly wireguard list and during all of my half-assed debugging and testing I had created 13 of them apparently... In one post I read that it should not matter if you had one or many in that list, but it turns out that it does. Anyway - I just deleted away and with only one left, I started over from the beginning.
$ fly wireguard list
automatically selected personal organization: [MyOrg]
+-----------------------------------------------------------------------------+--------+------------------------------+
| NAME | REGION | PEER IP |
+-----------------------------------------------------------------------------+--------+------------------------------+
| interactive-agent-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | ams | fdaa:3:cb7d:a7b:16a9:0:a:d02 |
+-----------------------------------------------------------------------------+--------+------------------------------+
$ fly wireguard reset
automatically selected personal organization: [MyOrg]
New WireGuard peer for organization "personal": "interactive-agent-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
$ fly wireguard remove
automatically selected personal organization: [MyOrg]
? Select peer: interactive-agent-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Removing WireGuard peer "interactive-agent-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" for organization personal
Removed peer.
$ fly wireguard list
automatically selected personal organization: [MyOrg]
+-----------------------------------------------------------------------------+--------+------------------------------+
| NAME | REGION | PEER IP |
+-----------------------------------------------------------------------------+--------+------------------------------+
| interactive-agent-YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY | ams | fdaa:3:cb7d:a7b:16a9:0:a:e02 |
+-----------------------------------------------------------------------------+--------+------------------------------+
$ flyctl doctor
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED
Testing WireGuard DNS... PASSED
Testing WireGuard Flaps... PASSED
App specific checks for wandrlust:
Checking that app has ip addresses allocated... PASSED
Checking A record for XXXXXXXXXXXXXXXX.fly.dev... PASSED
Checking AAAA record for XXXXXXXXXXXXXXXX.fly.dev... PASSED
Build checks for wandrlust:
Checking docker context size (this may take little bit)... PASSED (907 kB)
Checking for .dockerignore... PASSEDI have replaced some things in the terminal-dump above, but that was the final solution. After this - I simply ran flyctl deploy and it completed without any issues.
Summary
I tried a lot of things, but in the end I had to cleanup the list of WireGuard peers and make sure that I only had one newly created remaining. Some of the answers in the forum suggests that all of the things I tried above has helped somebody at some point so it might be a good idea to try them all if this happens again.
