Published: Jul 18, 2023 by Isaac Johnson

I may throw shade at XFinity a lot. I have plenty of reasons to gripe about them; for instance, I have to pay extra to use my own cable modem and extra to remove home internet data caps. That said, they were never as bad as USWest➡Qwest➡CentryLink➡Lumen ; At the very least they left my Home IP address alone.

This allowed me to host quite a lot locally out of my home network. Recently, out of the blue, the IP changed. It had been two years and plenty of power outages, etc. But just one day, poof, externally, I resolve to something else.

In this post I’ll walk through how I found and fixed, then used Ansible to correct it and lastly create a scheduled auto-remediation (should this happen again).

Uh Oh

I became aware that morning when I went to post last week’s blog and it failed to turn on my build lightbulb

Checking impact

I can see how big an impact we have. Granted, many of these systems are offline and represent a blog entry from some point in time the past. But it’s far easier to just hunt down all the matching and deal with in a single fell swoop

$ aws route53 list-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP | jq '.ResourceRecordSets[] | select(.ResourceRecords[0].Value=="73.242.50.46")' | jq .Name
"aksdemo.freshbrewed.science."
"argocd.freshbrewed.science."
"argoci.freshbrewed.science."
"awx.freshbrewed.science."
"awx2.freshbrewed.science."
"awxvm.freshbrewed.science."
"azurevote.freshbrewed.science."
"cloudcustodian.freshbrewed.science."
"core.freshbrewed.science."
"foo2.freshbrewed.science."
"foo4.freshbrewed.science."
"foo5.freshbrewed.science."
"gbwebui.freshbrewed.science."
"guestbook.freshbrewed.science."
"harbor.freshbrewed.science."
"core.harbor.freshbrewed.science."
"notary.harbor.freshbrewed.science."
"homek8s.freshbrewed.science."
"jenkins.freshbrewed.science."
"junkins.freshbrewed.science."
"kasarest.freshbrewed.science."
"loft.freshbrewed.science."
"logstream.freshbrewed.science."
"notary.freshbrewed.science."
"portainer.freshbrewed.science."
"publish.freshbrewed.science."
"python-crfunction.freshbrewed.science."
"rancher.freshbrewed.science."
"registry.freshbrewed.science."
"rundeck.freshbrewed.science."
"signoz.freshbrewed.science."
"sonarqube.freshbrewed.science."
"test.freshbrewed.science."
"uptime.freshbrewed.science."
"vault.freshbrewed.science."
"waypoint.freshbrewed.science."
"workflows.freshbrewed.science."
"workflows2.freshbrewed.science."
"wpblog.freshbrewed.science."
"zabbix.freshbrewed.science."

Now I can query all my records and replace them at the same time. This is going to create a full changeset of all records that used to point to my former home IP address (73.242.50.46) to the new one (75.72.238.228).

$ aws route53 list-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP | jq '.ResourceRecordSets[] | select(.ResourceRecords[0].Value=="73.242.50.46")' | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed 's/73.242.50.46/75.72.238.228/' | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to 75.72.238.228", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq
{
  "Comment": "Update All Homed IPs to 75.72.238.228",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "aksdemo.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "argocd.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "argoci.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "awx.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "awx2.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "awxvm.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "azurevote.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "cloudcustodian.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "core.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "foo2.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "foo4.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "foo5.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "gbwebui.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "guestbook.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "harbor.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "core.harbor.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "notary.harbor.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "homek8s.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "jenkins.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "junkins.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "kasarest.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "loft.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "logstream.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "notary.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "portainer.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "publish.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "python-crfunction.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "rancher.freshbrewed.science.",
        "Type": "A",
        "TTL": 86400,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "registry.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "rundeck.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "signoz.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "sonarqube.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "test.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "uptime.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "vault.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "waypoint.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "workflows.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "workflows2.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "wpblog.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "zabbix.freshbrewed.science.",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [
          {
            "Value": "75.72.238.228"
          }
        ]
      }
    }
  ]
}

The last step, and likely most dangerous, is to save this to a JSON and apply it

$ aws route53 list-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP | jq '.ResourceRecordSets[] | select(.ResourceRecords[0].Value=="73.242.50.46")' | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed 's/73.242.50.46/75.72.238.228/' | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to 75.72.238.228", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq > MYHugeChangeset.json

$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://MYHugeChangeset.json
{
    "ChangeInfo": {
        "Id": "/change/C07770883OD4SJBWPV709",
        "Status": "PENDING",
        "SubmittedAt": "2023-07-13T15:44:19.779Z",
        "Comment": "Update All Homed IPs to 75.72.238.228"
    }
}

I can now check R53 to see if it took

I can test my kasarest API

and Harbor is back

Automating

Let’s first create an automated check

$ cat testNewIp.sh
#!/bin/bash

MYIP="`host -4 myip.opendns.com resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"
HARBOR="`host -4 harbor.freshbrewed.science resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"

if [ "$MYIP" = "$HARBOR" ]; then
   echo "Strings are Equal . $MYIP matches $HARBOR"
else
   echo "IP CHANGED!  Harbor $HARBOR does not match local $MYIP"
   wget 'https://kasarest.freshbrewed.science/on?devip=192.168.1.24&apikey=MYSECRETKEY'
   exit 1
fi

I can run to test

$ ./testNewIp.sh
Strings are Equal . 75.72.238.228 matches 75.72.238.228

And i can change the URL and see the error condition (and it turns on an alert lightbulb at my desk)

$ ./testNewIp.sh
host: '.harbor.freshbrewed.science' is not a legal name (empty label)
IP CHANGED!  Harbor  does not match local 75.72.238.228
--2023-07-13 19:38:50--  https://kasarest.freshbrewed.science/on?devip=192.168.1.24&apikey=MYSECRETNOT
Resolving kasarest.freshbrewed.science (kasarest.freshbrewed.science)... 75.72.238.228
Connecting to kasarest.freshbrewed.science (kasarest.freshbrewed.science)|75.72.238.228|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 111 [text/html]
Saving to: ‘on?devip=192.168.1.24&apikey=MYSECRETNOT’

on?devip=192.168.1.24&apikey=MYSECRETNOT 100%[======================================================================>]     111  --.-KB/s    in 0s

2023-07-13 19:38:51 (46.1 MB/s) - ‘on?devip=192.168.1.24&apikey=MYSECRETNOT’ saved [111/111]

I almost checked that in verbatim till it dawned on me that my ansible playbooks are public. I parameterized that APIKEY

$ cat testNewIp.sh
#!/bin/bash

MYIP="`host -4 myip.opendns.com resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"
HARBOR="`host -4 harbor.freshbrewed.science resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"

if [ "$MYIP" = "$HARBOR" ]; then
   echo "Strings are Equal . $MYIP matches $HARBOR"
else
   echo "IP CHANGED!  Harbor $HARBOR does not match local $MYIP"
   wget "https://kasarest.freshbrewed.science/on?devip=192.168.1.24&apikey=$1"
   exit 1
fi

$ cat testNewIp.yaml
- name: Check if IP Changed
  hosts: all

  tasks:
  - name: Transfer the script
    copy: src=testNewIp.sh dest=/tmp mode=0755

  - name: Run Script
    command: sh /tmp/testNewIp.sh 

I pushed it to main

$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 16 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 712 bytes | 712.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/idjohnson/ansible-playbooks.git
   1580e06..21c6c85  main -> main

and added as a new template

Now when I launch

I can see it checked successfully

Scheduled Job

I want to test this more often. This last change was the first in a couple years, but I think I nightly check might be in order

Go to schedules, which is a tab on the template

There we can add a new schedule, such as a daily check at 3:50am CT

Once I click save, I can see the planned next 10 runs

The next day I saw it ran

Playbook to fix

Chances are I will forget the steps next time this happens. While I don’t quite want to update automatically just yet, I do want a playbook that would just do it for me.

I’ll write a script that would take in all the steps and optionally run the AWS Route53 update

$ cat updateRoute53.sh
#!/bin/bash
set -x

export FROMHZ=$1
export FROMHN=$2

export AWSKEY=$3
export AWSSECRET=$4

export MYIP="`host -4 myip.opendns.com resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"
export FROMIP="`host -4 $FROMHN resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"

export AWS_ACCESS_KEY_ID=$AWSKEY
export AWS_SECRET_ACCESS_KEY=$AWSSECRET

# Just show the output
aws route53 list-resource-record-sets --hosted-zone-id $FROMHZ | jq ".ResourceRecordSets[] | select(.ResourceRecords[0].Value==\"$FROMIP\")" | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed "s/$FROMIP/$MYIP/" | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to New IP", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq

# if final parmam is "DOIT" then do it.
if [ "$5" = "DOIT" ]; then
   echo "Modifying HZ"
   echo "==================================================="
   rm -f /tmp/MYHugeChangeset.json || true
   aws route53 list-resource-record-sets --hosted-zone-id $FROMHZ | jq ".ResourceRecordSets[] | select(.ResourceRecords[0].Value==\"$FROMIP\")" | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed "s/$FROMIP/$MYIP/" | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to New IP", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq > /tmp/MYHugeChangeset.json
   aws route53 change-resource-record-sets --hosted-zone-id $FROMHZ --change-batch file://tmp/MYHugeChangeset.json

Then a playbook that will call it.

$ cat updateRoute53.yaml
# e.g.
# HZ: Z39E8QFU0F9PZP
# DNS to check: harbor.freshbrewed.science
- name: Update R53
  hosts: all

  tasks:
  - name: Transfer the script
    copy: src=updateRoute53.sh dest=/tmp mode=0755

  # use ACTION=dryrun or ACTION=DOIT
  - name: Run Script
    command: sh /tmp/updateRoute53.sh     

This does assume a host with a working AWS CLI. I could go as far as to install that on the fly. But let’s not get carried away just yet.

Like before, I’ll push that up

builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git add updateRoute53.sh
builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git add updateRoute53.yaml
builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git commit -m"R53 update script"
[main 77a8299] R53 update script
 2 files changed, 40 insertions(+)
 create mode 100755 updateRoute53.sh
 create mode 100644 updateRoute53.yaml
builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 16 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.10 KiB | 1.10 MiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/idjohnson/ansible-playbooks.git
   ccdacf2..77a8299  main -> main

Refresh the project

Then add the template

Lastly, I can run it with DryRun and see it has the outputs I want

My last step is to allow it to prompt for variables on launch

Then type in DOIT for the action

And Launch

I saw an odd error

I decided to tweak the script just a bit

$ git diff updateRoute53.sh
diff --git a/updateRoute53.sh b/updateRoute53.sh
index 3c7b324..8d08f5e 100755
--- a/updateRoute53.sh
+++ b/updateRoute53.sh
@@ -22,5 +22,6 @@ if [ "$5" = "DOIT" ]; then
    echo "==================================================="
    rm -f /tmp/MYHugeChangeset.json || true
    aws route53 list-resource-record-sets --hosted-zone-id $FROMHZ | jq ".ResourceRecordSets[] | select(.ResourceRecords[0].Value==\"$FROMIP\")" | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed "s/$FROMIP/$MYIP/" | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to New IP", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq > /tmp/MYHugeChangeset.json
-   aws route53 change-resource-record-sets --hosted-zone-id $FROMHZ --change-batch file://tmp/MYHugeChangeset.json
-fi
\ No newline at end of file
+   cd /tmp
+   aws route53 change-resource-record-sets --hosted-zone-id $FROMHZ --change-batch file://MYHugeChangeset.json
+fi

upload

builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git add updateRoute53.sh
builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git commit -m "change script path"
[main 3486366] change script path
 1 file changed, 3 insertions(+), 2 deletions(-)
builder@DESKTOP-QADGF36:~/Workspaces/ansible-playbooks$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 347 bytes | 347.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/idjohnson/ansible-playbooks.git
   77a8299..443fc02  main -> main

And try again (though this should be a No-Op as the from and to are the same)

I can see it did the needful

Now one could bring them all together

$ cat checkAndUpdateR53.sh
#!/bin/bash
set -x

export FROMHZ=$1
export FROMHN=$2
export APIKEY=$3

export AWSKEY=$4
export AWSSECRET=$5

export MYIP="`host -4 myip.opendns.com resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"
export FROMIP="`host -4 $FROMHN resolver1.opendns.com | tail -n1 | sed 's/.* //' | tr -d '\n'`"

export AWS_ACCESS_KEY_ID=$AWSKEY
export AWS_SECRET_ACCESS_KEY=$AWSSECRET

# Just show the output
# aws route53 list-resource-record-sets --hosted-zone-id $FROMHZ | jq ".ResourceRecordSets[] | select(.ResourceRecords[0].Value==\"$FROMIP\")" | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed "s/$FROMIP/$MYIP/" | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to New IP", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq

if [ "$MYIP" = "$FROMIP" ]; then
   echo "Strings are Equal . $MYIP matches $FROMIP"
else
   echo "IP CHANGED!  Harbor $HARBOR does not match local $MYIP"
   wget "https://kasarest.freshbrewed.science/on?devip=192.168.1.24&apikey=$APIKEY"

   echo "Modifying HZ"
   echo "==================================================="
   rm -f /tmp/MYHugeChangeset.json || true
   aws route53 list-resource-record-sets --hosted-zone-id $FROMHZ | jq ".ResourceRecordSets[] | select(.ResourceRecords[0].Value==\"$FROMIP\")" | sed 's/^{/{ "Action": "UPSERT", "ResourceRecordSet": {/g' | sed 's/^}/} },/g' | sed "s/$FROMIP/$MYIP/" | tr -d '\n' | xargs -0 -I {} echo ' {"Comment": "Update All Homed IPs to New IP", "Changes": [ {}] }' | sed 's/},] }/} ] }/g' | jq > /tmp/MYHugeChangeset.json
   cd /tmp
   aws route53 change-resource-record-sets --hosted-zone-id $FROMHZ --change-batch file://MYHugeChangeset.json
fi

With the playbook

$ cat checkAndUpdateR53.yaml
- name: Update R53
  hosts: all

  tasks:
  - name: Transfer the script
    copy: src=checkAndUpdateR53.sh dest=/tmp mode=0755

  - name: Run Script
    command: sh /tmp/checkAndUpdateR53.sh     

Then add the playbook

which worked just fine

Lastly, let’s just schedule this auto-remediation to happen earlier in the night so there is still value to our 3:50am check we did earlier.

And as before, we can see the schedule

Summary

We found out our home IP changed when the Github runners couldn’t use Kasa REST to turn on a local lightbulb. This triggered a panicked check into my IP and DNS entries. We fixed the AWS Route53 with a bit of bash, jq and sed. We then banged out some Ansible Playbooks to check and fix, then do both on a schedule to automatically remediate into the future.

I could see reason to solve the secrets (like APIKEY, AWS Secret and Access Keys) in a better way in Ansible. That will likely be a follo-wup (as in other places I use AKV to back secrets in AWX).

Pro-Tip on XFinity Data Caps: They will not advertise you can pay to remove them. You will see, if like me, your bills grow month over month with overages. When you call them, and ask to remove the data cap, they will try to talk you out of it. “Most customers never reach the xxxx GB data cap, your plan…”. Do not be dissuaded. Simply ask “Is there option to remove the data cap for a charge”. In my case, it made my base bill go up $30/month but I was already paying $50+/mo in overages so it was net cheaper.

Many of us simply have no option but to use a provider like XFinity with a data cap. I do not be-grudge companies for trying to make money. I do take issue with hiding it, or turning on/changing caps without notice.