Script to push (mirror) a large repo to GitHub

push-big-repo

#!/bin/bash -ex
#
# Push the current repository to GitHub, in small enough chunks that it
# won't exceed the pack-size limit

# Commit to start with, counting from the oldest.  If the process fails,
# you can change this variable to restart from where it failed.
START_COMMIT=1000

# Number of commits to push at a time, counting from the oldest.  If a
# push fails because the pack file is too big, try using a smaller number.
COMMIT_STEP=1000

git log --pretty=%H | ruby -e 'puts ARGF.each_line.to_a.reverse' > commits
COMMIT_COUNT=$(wc -l commits | cut -d' ' -f1)

for i in `seq $START_COMMIT $COMMIT_STEP $COMMIT_COUNT`; do
	echo ====== $i
	COMMIT=$(git show $(head -$i commits | tail -1) | head -1 | cut -d' ' -f2)
	git tag -d foo || true
	git tag foo $COMMIT
	git push -f origin foo
done

git tag -d foo
git push origin HEAD
git push --mirror

Good catch, @nkitagawa-venn. Fixed it!

I get this error, any idea? I'm trying to push 2.6gb size repo to empty repository that rides on a private org with team plan

error: tag 'foo' not found.
fatal: --mirror can't be combined with refspecs
Enumerating objects: 17288, done.
Counting objects: 100% (17288/17288), done.
Delta compression using up to 12 threads
Compressing objects: 100% (11057/11057), done.
Writing objects: 100% (17288/17288), 2.48 GiB | 185.90 MiB/s, done.
Total 17288 (delta 5173), reused 17288 (delta 5173), pack-reused 0 (from 0)
error: RPC failed; HTTP 500 curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
Everything up-to-date

@mejuliver Is the repository you're pushing a mirror of something else? You can run git config remote.origin.mirror to find out. If it's true, you need to run git config --unset remote.origin.mirror to set it not to be.

You don't seem to be running the script with bash -ex that's in the #! line. If you do, it will show each command as it runs and will stop at the first error, which is also helpful for debugging.

@mejuliver Is the repository you're pushing a mirror of something else? You can run git config remote.origin.mirror to find out. If it's true, you need to run git config --unset remote.origin.mirror to set it not to be.

You don't seem to be running the script with bash -ex that's in the #! line. If you do, it will show each command as it runs and will stop at the first error, which is also helpful for debugging.

using this git config remote.origin.mirror returns true so its a mirror repo

Hello,

I had issues with

• tags protected, i had to name a unique name
• macos bash syntax
• larges files, i had to use git lfs migrate, like for example

git lfs migrate info
migrate: Sorting commits: ..., done.
migrate: Examining commits: 100% (13065/13065), done.
*.mp4      	5.4 GB	    390/390 files	100%
*.apk      	2.3 GB	      23/23 files	100%
*.jpg      	1.1 GB	  4173/4173 files	100%
*.strings  	854 MB	  5621/5621 files	100%
*.png      	548 MB	19789/19789 files	100%

LFS Objects	0 B   	       0/17 files	  0%
❯ git lfs migrate import --include="*.mp4"
etc...

here's an updated script

#!/bin/bash

set -e

# Push the current repository to GitHub in chunks to avoid pack size limits

START_COMMIT=1000
COMMIT_STEP=1000

# Reverse the git log (tac not on macOS by default)
git log --pretty=%H | ruby -e 'puts ARGF.readlines.reverse' > commits

COMMIT_COUNT=$(wc -l < commits | tr -d '[:space:]')

i=$START_COMMIT
while [ "$i" -le "$COMMIT_COUNT" ]; do
  echo "====== $i"
  COMMIT=$(sed -n "${i}p" commits)
  TAG="foo-$(echo $COMMIT | cut -c1-7)"  # unique tag per commit

  echo "Pushing commit: $COMMIT as tag: $TAG"
  git tag -f "$TAG" "$COMMIT"
  echo "Pushing tag $TAG to origin"
  GIT_TRACE=1 git push origin +$TAG

  i=$((i + COMMIT_STEP))
done

git push origin HEAD
git push --mirror origin