Re: Cleaning up a git repository

Thanks Marcus for the very useful information about git!

 

Another piece of advice I would like to give to those who start with to git is to make sure not to store passwords, credentials, API keys, or other confidential information in git.

 

While having 'build/' stored in git is an annoyance, having secrets and confidential information in git can be a very serious security risk, especially if those credentials allow access to a much wider (and perhaps expensive) set of resources than those you are actually using (I am thinking AWS tokens and similar secrets).

Once they are out on the Internet for everyone to grab, there's no going back; the best thing to do is to change those passwords/secrets immediately and hope for the best.

 

The safest thing to do is to store those confidential information outside of your git tree.

I also always append 'private' to my '.gitignore' (even if I don't use it), and use a directory called 'private' to store personal notes and other pieces of possibly related code that I don't want to share.

 

I also have the habit of running 'git status -u' before a commit, just to make sure I am not leaking something that shouldn't be out there.

 

Franco

 

On 06/09/2024 8:04 AM EDT Marcus Müller <mmueller@gnuradio.org> wrote:

 

 

Hi,

so as promised, my email about git.

Unless you and your advisors have explicitly said you should do that for now to debug some build issue, you should not have your build/ folder in your git (and you should not use git add --all / git commit --all unless you know you're only commiting the files you worked on). The build directory is regenerated by your CMake and Compiler runs, and is useless for other people. On the contrary, if I check out your module, and try to build it, I suddenly have git conflicts in build/!

So, to remove that from git's index, without deleting if from disk, you'd

cd gr-fec-_dev  git rm -r --cached build

Because you don't want to see all the changes in there all the time, you would add it to the "git ignore" file. That's easy:

echo build/ >> .gitignore

in the same directory. Now, because you're using VS Code, and other tools, and macos leaves .DS_Store files all over the place, my whole approach to getting rid of the files from the list of tracked files that you shouldn't be tracking in git:
Let someone else be diligent, and get a ready-made .gitigignore from them:

curl 'https://www.toptal.com/developers/gitignore/api/VisualStudioCode,cmake,python,macos' | grep -v '^lib/' > .gitignore  # ^                                                      ^                                   ^     ^  ^      ^ ^  # |                                                      |                                   |     |  |      | |  # \------use the "curl" program to download "something"  |                                   |     |  |      | |  #        from a webserver to standard output             \-- the "something" being the URL   |     |  |      | |  #                                                         of  a  service  that  generates    |     |  |      | |  #                                                         .gitigone files for                |     |  |      | |  #                                                                                            |     |  |      | |  #   we use "|", the "pipe" symbol, to pipe the output of curl into the input of "grep". -----/     |  |      | |  #   grep is a "regular expression filter";  you can tell it what to look for in a line,            |  |      | |  #   and it will only output lines that match that.                                                 |  |      | |  #                                                                                                  |  |      | |  #   we're looking for lines starting with "lib/" -----------------------------------------------------/      | |  #                                                                                                  |         | |  #   actually, we inVert that match, so we're looking for lines NOT starting with "lib/" -----------/         | |  #   (the generated .gitignore erroneously ignores "lib/", and we need to undo that)                          | |  #                                                                                                            | |  #   We redirect ">" the output of grep into the file ".gitignore".  -----------------------------------------+-/                                                                                             

Finally, I'd check that I'm now ignoring the files I want to ignore:

git ls-files --ignored --cached --exclude-from=.gitignore | grep -v '^build/'  #   ^        ^         ^        ^                            ^^^^^^^^^  #   |        |         |        |                            |  #   \-----------------------------------------------------------  git: list files  #            |         |        |                            |  #            \--------------------------------------------------  … which are ignored  #                      |        |                            |  #                      \----------------------------------------  … but only from the index (not from disk)  #                               |                            |  #                               \-------------------------------  and read the ignored files from the .gitignore file  #                                                            |  #                                                            \---  filter out all lines starting with "build/", because  #                                                                  we already know that we don't care about build/*  

Great! That looks like this to me:

.DS_Store  docs/doxygen/doxyxml/__pycache__/__init__.cpython-312.pyc  docs/doxygen/doxyxml/__pycache__/base.cpython-312.pyc  docs/doxygen/doxyxml/__pycache__/doxyindex.cpython-312.pyc  docs/doxygen/doxyxml/__pycache__/text.cpython-312.pyc  docs/doxygen/doxyxml/generated/__pycache__/__init__.cpython-312.pyc  docs/doxygen/doxyxml/generated/__pycache__/compound.cpython-312.pyc  docs/doxygen/doxyxml/generated/__pycache__/compoundsuper.cpython-312.pyc  docs/doxygen/doxyxml/generated/__pycache__/index.cpython-312.pyc  docs/doxygen/doxyxml/generated/__pycache__/indexsuper.cpython-312.pyc  include/gnuradio/.DS_Store  python/.DS_Store  

OK, since there's nothing in here that needs to be tracked by git, let's remove these files all from the index:

git ls-files --ignored --cached --exclude-from=.gitignore -z | xargs -0 git rm --cached  

 

Hope that helps (and also, honestly, helps future readers that find this mail via a search engine, especially students).

Best regards,
Marcus