Navigate back to the homepage

Clone any Private or Public repo from Organization

Have you ever wanted to clone all repos under your account or specific organization or even from the company where you work?

Ahmed Abdulrahman
Feb 3rd, 2020 • ☕️ 1 min read
bashshellgitjq

Say you want to clone or backup of all github repositories under your account or specific organization or you want to contribute to an open source project. You successfully do that copy & paste repo URL and then clone them manually 😫. But there might be plenty of repos and you dont want to repeat yourself. what do you do if there plenty of repos? you will need to repeat yourself again and again 😣.

To automate this process, you can use github API together with jq processor to clone all repos with one-liner shell script.

Prerequisites

GitHub API returns really nicely formatted JSON but there is a pretty fair amount of JSON in the response!. We’re primarily only interested to get the ssh_url, and not all the other output.

Simply we can use jq to “reformat” the JSON output from API response and include only that information. That’s done with this snippet:

1curl -s "https://api.github.com/orgs/twitter/repos?per_page=10" \
2 | jq '.[].ssh_url'

This will provide output that will look something like this:

1"git@github.com:twitter/hadoop-lzo.git"
2"git@github.com:twitter/thrift_client.git"
3"git@github.com:twitter/twurl.git"
4"git@github.com:twitter/elephant-bird.git"
5"git@github.com:twitter/ostrich.git"
6"git@github.com:twitter/scala_school.git"
7"git@github.com:twitter/finagle.git"
8"git@github.com:twitter/joauth.git"
9"git@github.com:twitter/twitter.github.io.git"
10"git@github.com:twitter/util.git"

This makes it really easy to parse out the specific information you need. Now that you’re armed with ssh_url, we need to convert the output to raw strings, not JSON texts by passing -r flag to jq then we can loop over the list and clone all repos:

1curl -s "https://api.github.com/orgs/twitter/repos?per_page=10" \
2 | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
3 | while read url; do git clone "$url"; done

Boom! You got all repos now 🔥! if you needed to clone private repos, then you will need to:

  • Add &type=private on the URL.
  • Create a personal access token as described here

snippet will look like this:

1curl -H "Authorization: token YOUR_ACCESS_TOKEN" -s \
2 "https://api.github.com/orgs/twitter/repos?per_page=10" \
3 | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
4 | while read url; do git clone "$url"; done

Bonus

You create a shell function inside your .bashrc or .zshrc file:

1function cloner {
2 # you can store your access token as env variable inside `.bashrc` or `.zshrc`
3 # ex: export GITHUB_TOKEN= , and replace $1 with $GITHUB_TOKEN and $2 with $1
4 curl -H "Authorization: token $1" -s \
5 "https://api.github.com/orgs/{$2}/repos?per_page=10" \
6 | jq '.[].ssh_url' -r \ # output raw strings, not JSON texts
7 | while read url; do git clone "$url"; done
8}

and use it like this:

1# with ENV variable
2cloner your-organization
3
4# without ENV variable
5cloner <your-access-token> your-organization

Did you find this useful? Buy me a coffee to give my brain a hug☕️.

Hope you liked this article. If you did, then share it. It means a lot.🙌 Thanks for reading!


Discuss on TwitterFollow @_ahmed_ab

Other things I've written

How to Automatically Sort Import Modules with ESLint

In this artcile, you’ll learn how configure ESLint to imports for us automatically.

Jan 5th, 2020 · 1 min read

How to Start Vim temporarily without settings or plugins

Sometime you want temporarily to disable your vimrc without loading any of my plugins or settings. In this article, we will see how can we do that

Jan 4th, 2020 · 1 min read

© 2018–2020
Ahmed Abdulrahman

Contact
hello@aadev.me

Link to $https://twitter.com/_ahmed_abLink to $https://www.behance.net/ahmedabdulrahmanLink to $https://github.com/AhmedAbdulrahman