Script to push (mirror) a large repo to GitHub

push-big-repo

#!/bin/bash -ex
#
# Push the current repository to GitHub, in small enough chunks that it
# won't exceed the pack-size limit

# Commit to start with, counting from the oldest.  If the process fails,
# you can change this variable to restart from where it failed.
START_COMMIT=1000

# Number of commits to push at a time, counting from the oldest.  If a
# push fails because the pack file is too big, try using a smaller number.
COMMIT_STEP=1000

git log --pretty=%H | ruby -e 'puts ARGF.each_line.to_a.reverse' > commits
COMMIT_COUNT=$(wc -l commits | cut -d' ' -f1)

for i in `seq $START_COMMIT $COMMIT_STEP $COMMIT_COUNT`; do
	echo ====== $i
	COMMIT=$(git show $(head -$i commits | tail -1) | head -1 | cut -d' ' -f2)
	git tag -d foo || true
	git tag foo $COMMIT
	git push -f origin foo
done

git tag -d foo
git push origin HEAD
git push --mirror

Hello,

I had issues with

• tags protected, i had to name a unique name
• macos bash syntax
• larges files, i had to use git lfs migrate, like for example

git lfs migrate info
migrate: Sorting commits: ..., done.
migrate: Examining commits: 100% (13065/13065), done.
*.mp4      	5.4 GB	    390/390 files	100%
*.apk      	2.3 GB	      23/23 files	100%
*.jpg      	1.1 GB	  4173/4173 files	100%
*.strings  	854 MB	  5621/5621 files	100%
*.png      	548 MB	19789/19789 files	100%

LFS Objects	0 B   	       0/17 files	  0%
❯ git lfs migrate import --include="*.mp4"
etc...

here's an updated script

#!/bin/bash

set -e

# Push the current repository to GitHub in chunks to avoid pack size limits

START_COMMIT=1000
COMMIT_STEP=1000

# Reverse the git log (tac not on macOS by default)
git log --pretty=%H | ruby -e 'puts ARGF.readlines.reverse' > commits

COMMIT_COUNT=$(wc -l < commits | tr -d '[:space:]')

i=$START_COMMIT
while [ "$i" -le "$COMMIT_COUNT" ]; do
  echo "====== $i"
  COMMIT=$(sed -n "${i}p" commits)
  TAG="foo-$(echo $COMMIT | cut -c1-7)"  # unique tag per commit

  echo "Pushing commit: $COMMIT as tag: $TAG"
  git tag -f "$TAG" "$COMMIT"
  echo "Pushing tag $TAG to origin"
  GIT_TRACE=1 git push origin +$TAG

  i=$((i + COMMIT_STEP))
done

git push origin HEAD
git push --mirror origin