Gitea went kaput. Just up and stopped working one day. I came back from the BWCA and noticed none of my automated builds had succeeded:
The errors were all about “back pack header”
I tried many things - restarting Kubernetes, docker, upgrading the container image. All the things that usually ‘unstuck’ a Gitea instance just weren’t.
Any local pull or clone would fail
builder@DESKTOP-QADGF36:~/Workspaces/fbsnew$ git pull
fatal: protocol error: bad pack header
It wasn’t disk space
builder@builder-T100:~/GiteaNew$ du -chs ./gitea/
du: cannot read directory './gitea/ssh': Permission denied
2.8G ./gitea/
2.8G total
builder@builder-T100:~/GiteaNew$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 766M 16M 751M 2% /run
/dev/sda2 234G 148G 74G 67% /
The file system was fine
$ docker exec -it gitea /bin/bash
ac6bc572c240:/# cd /data/git/
ac6bc572c240:/data/git# cd repositories/builder
ac6bc572c240:/data/git/repositories/builder# ls
fbsnew.git mejwebsite.git perlMCP.git perlmcp.git testrepo.git testtest123.git
ac6bc572c240:/data/git/repositories/builder# cd fbsnew.git/
ac6bc572c240:/data/git/repositories/builder/fbsnew.git# git config --global --add safe.directory /data/git/repositories/builder/fbsnew.git
ac6bc572c240:/data/git/repositories/builder/fbsnew.git# git fsck --full --strict
Checking ref database: 100% (1/1), done.
Checking object directories: 100% (256/256), done.
Checking objects: 100% (205/205), done.
I was (and am) just out of ideas.
New Gitea
I need to quickly make a new Gitea instance so I can start to migrate and backup work
I made a new A Record in Route 53:
$ cat r53-gitea2.json
{
"Comment": "CREATE gitea2 fb.s A record ",
"Changes": [
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "gitea2.freshbrewed.science",
"Type": "A",
"TTL": 300,
"ResourceRecords": [
{
"Value": "76.156.69.232"
}
]
}
}
]
}
$ aws route53 change-resource-record-sets --hosted-zone-id Z39E8QFU0F9PZP --change-batch file://r53-gitea2.json
I then created a new docker instance on a new port (4224 instead of 4220) on the same docker host…
builder@builder-T100:~$ cd Gitea202608/
builder@builder-T100:~/Gitea202608$ ls
docker-compose.yaml gitea
builder@builder-T100:~/Gitea202608$ cat docker-compose.yaml
version: "3"
networks:
gitea202608:
external: false
services:
server:
#image: gitea/gitea:1.21.7
#image: gitea/gitea:1.26.4
image: gitea/gitea:1.27.2
container_name: gitea202608
environment:
- USER_UID=1000
- USER_GID=1000
restart: always
networks:
- gitea202608
volumes:
- ./gitea:/data
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
ports:
- "4224:3000"
- "4225:22"
I just needed to use docker compose up -d to fire it up and check when it was running
builder@builder-T100:~/Gitea202608$ docker compose up -d
[+] Building 0.0s (0/0)
[+] Running 2/2
✔ Network gitea202608_gitea202608 Created 0.1s
✔ Container gitea202608 Started
$ docker ps | grep gitea202608
4a8f5af0659f gitea/gitea:1.27.2
"/usr/bin/entrypoint…" 45 hours ago Up 45 hours
0.0.0.0:4225->22/tcp, :::4225->22/tcp,
0.0.0.0:4224->3000/tcp, :::4224->3000/tcp gitea202608
Lastly, we need an ingress, service and endpoint to forward traffic from Kubernetes to the Dockerhost
$ cat gitea2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
ingress.kubernetes.io/proxy-body-size: "0"
ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.org/client-max-body-size: "0"
nginx.org/proxy-connect-timeout: "3600"
nginx.org/proxy-read-timeout: "3600"
nginx.org/websocket-services: gitea2-external-ip
labels:
app.kubernetes.io/instance: gitea2ingress
name: gitea2ingress
spec:
rules:
- host: gitea2.freshbrewed.science
http:
paths:
- backend:
service:
name: gitea2-external-ip
port:
number: 80
path: /
pathType: ImplementationSpecific
tls:
- hosts:
- gitea2.freshbrewed.science
secretName: gitea2-tls
---
apiVersion: v1
kind: Service
metadata:
name: gitea2-external-ip
spec:
ports:
- name: gitea2p
port: 80
protocol: TCP
targetPort: 4224
type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
name: gitea2-external-ip
subsets:
- addresses:
- ip: 192.168.1.99
ports:
- name: gitea2p
port: 4224
protocol: TCP
I just did a kubectl apply -f gitea2.yaml to apply.
When I saw the cert was ready
$ kubectl get cert | grep -i gitea
gitea-tls True gitea-tls 51d
gitea2-tls True gitea2-tls 44h
I logged into gitea2.freshbrewed.science and configured the initial user (which also becomes the admin user).
Backing Up the old Gitea
Before we do anything harsher, let’s ensure we can at least save the important repos.
On the docker host, I’ll create a backup dir and pull down the repos:
builder@builder-T100:~/GiteaNew$ mkdir /tmp/repo_transfer
builder@builder-T100:~/GiteaNew$ docker cp gitea:/data/git/repositories/builder/mejwebsite.git /tmp/repo_transfer/mejwebsite.git
When done, I have the 3 repositories mastered in Gitea. I do not care about test repos or remote syncs (those are easy to just recreate)
builder@builder-T100:~/GiteaNew$ ls /tmp/repo_transfer/
fbsnew.git mejwebsite.git perlMCP.git
I then copied over a couple of the repos from the old Gitea to the new one (Gitea202608). I focused on those for which I didn’t have local up to date clones already.
builder@builder-T100:/tmp/repo_transfer$ docker cp /tmp/repo_transfer/mejwebsite.git gitea202608:/data/git/repositories/builder/mejwebsite.git
Successfully copied 2.27GB to gitea202608:/data/git/repositories/builder/mejwebsite.git
I need to fix the ownership (and check it)
builder@builder-T100:/tmp/repo_transfer$ docker exec -it gitea202608 chown -R git:git /data/git/repositories/builder
builder@builder-T100:/tmp/repo_transfer$ docker exec -it gitea202608 ls -l /data/git/repositories/builder
total 8
drwxr-xr-x 7 git git 4096 Aug 17 09:53 fbsnew.git
drwxr-xr-x 7 git git 4096 Aug 17 08:58 mejwebsite.git
I’ll then go to the new Gitea instance and in Site Administration, go to “Code Assets”, Repositories and click “Unadopted Repositories”
I left it blank to find all
It found one to adopt so I clicked “Adopt Files”
Then confirmed
One of them would not copy down.. only giving me hooks
builder@builder-T100:/tmp/repo_transfer$ docker cp gitea:/data/git/repositories/builder/perlMCP.git /tmp/repo_transfer/perlMCP.git
Successfully copied 3.07kB to /tmp/repo_transfer/perlMCP.git
builder@builder-T100:/tmp/repo_transfer$ ls perlMCP.git/
hooks
This matched the local data files as well
builder@builder-T100:~/GiteaNew/gitea/git/repositories/builder$ cd perlMCP.git/
builder@builder-T100:~/GiteaNew/gitea/git/repositories/builder/perlMCP.git$ ls
hooks
Yet, I can clearly see the files
Alternate approach
I cannot clone as we mentioned earlier because this Gitea is fooo
builder@DESKTOP-QADGF36:~/Workspaces$ git clone https://gitea.freshbrewed.science/builder/perlMCP.git perlmcp-new
Cloning into 'perlmcp-new'...
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
I can, however, download the TGZ or Bundle
Since I’m on Copilot OS, it dumps those into a folder in Downloads
I’m going to use the bundle for this next step as i can “clone” from the bundle into a local clone:
builder@DESKTOP-QADGF36:~/Workspaces/tmp_perlMCP$ git clone /mnt/c/Users/isaac/Downloads/perlMCP-main.bundle ./perlmcp
Cloning into './perlmcp'...
Receiving objects: 100% (19/19), 5.32 KiB | 5.32 MiB/s, done.
Resolving deltas: 100% (4/4), done.
In the new Gitea, I’ll make a blank repo for it
It exists, but is uninitialized
I have it locally and can see it points it’s origin to the local file system
builder@DESKTOP-QADGF36:~/Workspaces/tmp_perlMCP/perlmcp$ git remote show origin
* remote origin
Fetch URL: /mnt/c/Users/isaac/Downloads/perlMCP-main.bundle
Push URL: /mnt/c/Users/isaac/Downloads/perlMCP-main.bundle
HEAD branch: bundle
Remote branch:
bundle tracked
Local branch configured for 'git pull':
bundle merges with remote bundle
Local ref configured for 'git push':
bundle pushes to bundle (up to date)
So I add the new origin and push (had I been thinking, I would have used git co -b main first)
builder@DESKTOP-QADGF36:~/Workspaces/tmp_perlMCP/perlmcp$ git remote add origin2 https://gitea2.freshbrewed.science/builder/perlMCP.git
builder@DESKTOP-QADGF36:~/Workspaces/tmp_perlMCP/perlmcp$ git push -u origin2 --all
Enumerating objects: 19, done.
Counting objects: 100% (19/19), done.
Delta compression using up to 16 threads
Compressing objects: 100% (14/14), done.
Writing objects: 100% (19/19), 5.32 KiB | 5.32 MiB/s, done.
Total 19 (delta 4), reused 19 (delta 4), pack-reused 0
To https://gitea2.freshbrewed.science/builder/perlMCP.git
* [new branch] bundle -> bundle
Branch 'bundle' set up to track remote branch 'bundle' from 'origin2'.
Because I neglected to rename the branch before pushing, I’ll go to branches in gitea next
And rename it
to “main”
And the last of my repositories now exists
Actions
On a docker host i just needed to add a new runner with an updated token. Note that I needed to give it a unique name (gitearunner2) as I’m still trying to debug/fix the old one
builder@bosgamerz9:~/gitearunner2$ cat start.sh
docker run \
-e GITEA_INSTANCE_URL=https://gitea2.freshbrewed.science/ \
-e GITEA_RUNNER_REGISTRATION_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxx \
-e GITEA_RUNNER_NAME=bosgamerz9 \
--name gitearunner2 \
--restart always \
-v /var/run/docker.sock:/var/run/docker.sock \
-d docker.io/gitea/act_runner:latest
# For gitea2.freshbrewed.science
The one part that sucks - and I don’t know any good way to solve it - is that the secrets are not extractable. So if you have Gitea Actions secrets, you’re just going to need to figure them out and set them in the new Gitea instance
I might suggest using a secret store and leaving yourself hints (i did for some).
New Syncs
Part of the reason I didn’t have nearly as much fear when Gitea crashed was that the key repos are synced to external systems.
Here we see the MEJ website backup in Forgejo:
I’ll now make new Sync destination repos in Forgejo
Then in the new Gitea, I’ll add a Push Mirror to this new Sync repo
I’ll then force a sync
We could also add something external, like an Azure DevOps Azure Repo as a sync destination
In AzDO, instead of a PAT, we can generate GIT credentials in the Repos page so we have narrowly scoped credentials in use
I found for the larger repos, this force push sync approach was not working. Instead I would mirror as as a pull
One last approach is a Sync Mirror in Gitea/Forgejo
I’ll start by making a new Read Only PAT in the new Gitea
Once I have that token
I can set it up as a Mirror in Forgejo
This starts the sync process
and I can see it is a mirror based on the icon in the UI
By default it synchronizes every 8 hours but you can always adjust that
I can now feel much safer knowing my key repos are in at least two if not three places
The old Gitea….
I tried a lot. I am a bit stumped on what to do next.
As everything is fully migrated, albeit it on a new URL, I might as well power it down. As it stands the nightly build errors are mounting up
which is properly caught by Datadog
Which keeps alerting me
I’ll leave myself a note and stop the old service
uilder@builder-T100:~$ cd GiteaNew/
builder@builder-T100:~/GiteaNew$ touch THIS_IS_DEAD_USE_Gitea202608_Instead
builder@builder-T100:~/GiteaNew$ docker compose down
[+] Running 2/2
✔ Container gitea Removed 0.4s
✔ Network giteanew_gitea Removed
Summary
So Gitea died. It was up for a long long time. It was actually hijacked by spammers for a while who learned they could stash metadata in their profiles without repo access. I cleaned everything up but just could not get this thing working again.
The next steps were to back up the raw GIT files. Had the system been totally corrupted, we would have instead relied on local clones and/or syncs in other GIT systems like Codeberg, Forgejo and Github.
We spun up a new Gitea and Runner and started with the fbsnew repo (this blog) so I could get posts going again. As only three real repositories were hosted in Gitea, it was easy to migrate the three and get things moving again.
I set up new syncs in Forgejo and Azure DevOps and lastly stopped the old service and left myself a note.
My next steps will be to go to my various laptops and update the local “fbsnew” git clones so I don’t accidentally use a defunct old repository. If I give up on trying to fix this Gitea, I’ll also then need to remove the service, endpoint and ingress from Kubernetes.