------------------------------
Title: Migrate ☁️ GCP projects across organizations, with gcloud
[content]
Nel mezzo del cammin di nostra vita, mi ritrovai per una selva oscura, ché la diritta via era smarrita”— Dante Alighieri(*), Divine Comedy(*) the Italian version of Shakespeare, just betterTranslated for non-🇮🇹: some day I was encouraged by some external entity to move a lot of projects from 5 of my organizations (source) to another organization (destination).TL;DR If you find this article too long and you want to jump to the code, click on this 🐙😺 gist.My two boys in front of The Gates of Hell, Musée Rodin, Paris, 2022.Since I’m the best kind of Software Engineer — the bad coder but lazy kind 🌝 — I decided to script all my move and document for posterity. Also showing you the mistakes I found along the way might be helpful (maybe?).This is explained in this GCP page(s): https://cloud.google.com/resource-manager/docs/project-migrationPlease remember to dump your current IAM config for the 2+ organisations in case you want to recreate the previous config you just changed. Having a local JSON config for your Org is always a good thing to do. 😉 For instance:$ gcloud alpha organizations get-iam-policy $ORG_ID \ - format json > out/org-$DOMAIN-iam.jsonYou can also add specific policies you care about (see below why this is important):$ gcloud resource-manager org-policies describe constraints/iam.allowedPolicyMemberDomains \ --organization $ORG_ID | tee -a out/org-$DOMAIN-policies.logKnow the current state of the worldYou need to know the state of the world (your world). The best way for me to understand where I stand is this:Use this awesome repo https://github.com/palladius/org-folder-projects-graph I wrote a few years ago. Note: It launches gcloudonce per folder, it’s super-inefficient (uses bash over APIs), but has the nice aspect that it keeps all JSON from gcloud locally. This is a big plus in every case except today: ▶️ after the move you might need a make cache-clean before re-launching it.Launch the script for one Org. Sample output:Output from an Org dump script written by an ItalianIf you don’t care about the folder structure at all, a simpler way is to just invoke something like this:# Option 1: Flat structure (simple)$ gcloud projects list - filter \  "parent.id=$ORG_ID AND parent.type=organization"For example:flat taxonomy from script1Note this won’t work if you have a complex hierarchy, which is why you probably want to use my script to get all the ramification of your org.You can also play with GCP Asset Inventory (see great stackoverflow tip) to get the list without the 🍕 pizza slices, but you’d be sadder 😫 , I’m sure.# Option 2: Tree structure, with AssetInventory.$ gcloud beta asset search-all-resources \  --asset-types='cloudresourcemanager.googleapis.com/Project' \  --scope="organizations/$ORGID" | egrep '^name:' | cut -d/ -f 5Whatever your workflow, I’m pretty sure you’ll have exceptions, which is why having a spreadsheet (https://spreadsheet.new/ ) will help you big time. I state the source org, destination org, source folder,. source project id, expected destination folder, and notes for my exceptions.Please also read this https://cloud.google.com/resource-manager/docs/handle-special-cases for special cases, like VPC-SC or GCS Bucket lock.Let’s now find out the IAM state for your org. This is pretty sweet and took me a while to research and adapt:# Beautiful query which flattens (multiple roles <=> multiple identities).$ gcloud organizations get-iam-policy "$ORG_ID" \  -flatten='bindings[].members' \  -format='table(bindings.role,bindings.members)' |   tee "t.org-iam-policy.$ORG_DOMAIN.txt"ROLE MEMBERS[..]roles/billing.admin         group:gcp-billing-admins@example.orgroles/billing.admin         user:ricc@example.orgroles/billing.creator       domain:example.orgroles/browser               group:gcp-billing-admins@example.orgroles/cloudasset.owner      user:palladius@example.orgroles/cloudasset.owner      user:ricc@example.orgroles/orgpolicy.policyAdmin group:gcp-org-administrators@example.orgroles/orgpolicy.policyAdmin user:riccardo@example.orgroles/resourcemanager.projectCreator user:palladiusbonton@example.orgroles/resourcemanager.projectMover   user:ricc@example.orgI don’t want to read the docs: Let’s first try brute force!I hate it when someone provides me with a working solution and I apply it and don’t know the mystical secrets that led to it. (I invented the half derivative when I was 16, just so you know — and yes, as you can imagine, it’s useless).So let’s try to smash our head over it, one mistake at a time.Error 1 : “The caller does not have permission”Let’s try it, the command is simple — but won’t work the first 27 times you try it. You have my word. But still, try it. I’ll use the same ids as in public docs:# Your Org Admin account, and gcloud already working for it:export POWER_ACCOUNT='my-power-account@gmail.com'# SRC ORG: 12345678901 source.example.com export SRC_ORG_ID="12345678901"export SRC_ORG_DOMAIN='source.example.com'# DST ORG: 45678901234 destination.example.comexport DST_ORG_ID="45678901234"export DST_ORG_DOMAIN='destination.example.com'$ gcloud beta projects move my-project-in-src-org \  --organization=$DST_ORG_ID --quiet- '@type': type.googleapis.com/google.rpc.DebugInfo  detail: |-    [ORIGINAL ERROR] generic::permission_denied: The caller does not have permission    com.google.apps.framework.request.StatusException: <eye3 title='PERMISSION_DENIED'/> generic::PERMISSION_DENIED: The caller does not have permission [google.rpc.error_details_ext] { code: 7 message: "The caller does not have permission" }# Computer says no!The most powerful IAM permission you can have on a GCP Organization is organizationAdmin. You’d think that if you have that power, you can do anything in that universe, right? Think twice! There are additional powers that role doesn’t have (and I’m kind of relieved it is this way).One of them is the OrgPolicyAdmin, which means that as OrgAdmin, you first need to give yourself that power before you can use it (not very UNIX, I know, but definitely safer).# Set IAM for SrcOrg$ gcloud organizations add-iam-policy-binding $ORG_ID \  --member='user:$ACCOUNT' \  --role='roles/resourcemanager.organizationAdmin' \  --role='roles/resourcemanager.projectIamAdmin' \  --role='roles/resourcemanager.projectMover'  \  --role='roles/orgpolicy.policyAdmin'  \  --condition=NoneFor this command to work, you need the issuer (gcloud config get account) to have the power to grant — but this is beside the scope of this doc.In the UI it will look something like thisNote: Org Policies are documented here.Error 2: Constraint `constraints/resourcemanager.allowedExportDestinations`Let’s try to migrate a project now:This is NOT where I would have expected a Input/Output error! 😆Next step: Let me configure the allowedPolicyMemberDomains policy first. (Why? It failed but I forgot to capture the error, my bad. For this error you just need to trust me).# This might fail..$ gcloud beta projects move "my-project-in-src-org" \  --organization="$DST_ORG_ID" --quiet# This sets target customer_id!$ gcloud resource-manager org-policies allow --organization "$SRC_ORG_ID" \  iam.allowedPolicyMemberDomains 'C04abcdef' # gives access to SrcOrg to Dest idNote. The “C0..” id is the Directory Customer Id you get from gcloud organization list for your destination org.- I’m ready… Now it’s going to work, I’m sure! - Oh wait… 🙈Damn, another error!What does it mean? It means that you need to configure your “Org Firewall” to allow projects out of your Source Organization. If you’re a Cisco Certified engineer — you already know what error is coming next.. Allowlisting palooza 😜!# Aloow SRC to export to DSTgcloud resource-manager org-policies allow --organization "$SRC_ORG_ID" \  resourcemanager.allowedExportDestinations \  "under:organizations/$DST_ORG_ID"# Note the "under:" needed to pass the policy all the way down from that folderError 3: constraints/resourcemanager.allowedImportSourceThis is similar to the error above, but it’s from the receiving end.To fix: damn I did it on the UI. My apologies. I would presume it’s really the symmetrical opposite, like:# This is me guessing but I should be too far. # TODO(ricc): remove powers from DST and try againgcloud resource-manager org-policies allow --organization "$DST_ORG_ID" \  resourcemanager.allowedImportSource \  "under:organizations/$SRC_ORG_ID"[optional] Refining the securityNote that to make things simple I gave a single user OrgAdmin for two organizations.simple/insecure (pet project). The simple way is to have a single powerful account (gcloud account, eg “poweruser@company.com”) which has access to both SRC and DST orgs. This worked for me, as I was the sole owner of them all and I was under time pressure.complex/more secure (enterprises). In a more enterprise scenario, you’d have a less pressing time concern and you would probably be obliged to have smaller IAM permissions on both sides, probably having two individuals to carry on these two tasks. Which is why it’s nice to have them gcloud-ed.In the second case, follow the docs:Source Org :IAM: roles/resourcemanager.projectIamAdmin in the project to move (but if I want to do it at scale I’ll set it up in the org)IAM: roles/resourcemanager.projectMover in the parent resource.Again, if you’re lazy you can just set these two at Org level and get it done once and for all. But then remember to remove this access at the end of the org move. You can also a time-bound IAM constraint ending tomorrow — this is for lazy people like me who don’t trust themselves to remember to close a parenthesis tomorrow and rather close it as you open it.OrgPolicy: constraints/resourcemanager.allowedExportDestinations. You can set the domain id (“C0somethin”) as allowed destination or you can be lazy as me and just set them ALL, depending if you’re an enterprise who wantgs to do it well or you just want to get it done soon and you don’t care much of the org you leave behind.On the destination organization:IAM: ProjectMoverOrg Policy: constraints/resourcemanager.allowedImportSourceMeta-Scripts and tidy migrationIf you migrate 300 projects from 5 orgs into one, you might want to use Folders or it’ll be a very nasty place tomorrow (“where does this project come from again?”). You can track pre-migration state in a few ways: you can dump the state on a lot of TXT in local file (but I usually remove them with my ruthless make cleans), on a Google Spreadsheet, or have your business logic do the job for you. Let’s explore this one..Here’s the migration in a nutshell: all the Source folder structure complexity is flattened in the Destination under a single special folder.I had some fun coding in Ruby (in a private repo which is in no shape to be publicized unless you make a lot of noise in the comments 😉).Nothing better than a script which writes a script for you. This is useful when you want to iterate through 10 migrations but you know that ONE project doesnt fit into it -> I create a 10 line blurb and then I just comment out the one I dont need. It’s also good for documenting which commands you ran on local file. Let me show you.$ bin/13-gcloud-move-projects-to-joonix-under-domain.rb palladius.eu palladius-eu-infra prova-enonomai-palldius-eu  gcpprojects2sheets [...]gcloud beta projects move 'palladius-eu-infra' --folder='1038774614299' --quiet       # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.eugcloud beta projects move 'prova-enonomai-palldius-eu' --folder='1038774614299' --quiet       # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.eugcloud beta projects move 'gcpprojects2sheets' --folder='1038774614299' --quiet       # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.euNote, this commands are dumped into a deterministically-named bash script which contains the commands, so I can comment out the projects I don’t want to move (yet); the deterministic naming means that if I run it again, it will re-apply the same logic to the same file, which is good if you have it in version control and if the logic is not trivial (eg, find all projects under a folder 😉).Let’s now call my 🍕 pizza software (TM) after a make cache-clean to see if the migration worked:$ git clone https://github.com/palladius/org-folder-projects-graph/ $ cd org-folder-projects-graph/(base) ricc@mbp:🏡~/org-folder-projects-graph$ ./recurse_folders.rb palladius.joonix.net[...]├─ 📂 133363080569 (Spostamenti CrossOrg)    ├─ 📂 737267668168 (SPECIAL_ORPHANS)        ├─ 🍕 cicd-platinum-test041 (713582791007)    ├─ 📂 1038774614299 (palladius-eu)        ├─ 🍕 original-bot-383907 (642248137380)        ├─ 🍕 prova-enonomai-palldius-eu (898096115869)        ├─ 🍕 gcpprojects2sheets (383820962508)        ├─ 🍕 palladius-eu-infra (845199147079)[..]And it did! A round of applause.Plenty of magic here: my script accepts a FolderId ( 🇮🇹 “CrossOrg Movemements”) and it searches for the subfolder named like the domain (creating it if its not found). Wow!My script was configured to generate a folder called like the org (changing dot with dash for obvious reasons): 133363080569. My script would create a subfolder if needed, get that folder id and create the scripts for me:gcloud beta projects move 'palladius-eu-infra' - folder='1038774614299' - quiet # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.eugcloud beta projects move 'prova-enonomai-palldius-eu' - folder='1038774614299' - quiet # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.eugcloud beta projects move 'gcpprojects2sheets' - folder='1038774614299' - quiet # migrate project given via CLI to proper subfolder of palladius.joonix.net under palladius.euAfter a lot of lost hours debugging, I wrote my bash code-writing ruby script to add meaningful comments aside the command. This was a stroke of genius (for my limited coding abilities) since all this movement includes non-mnemonic numbers, so using the domain names after the comment helped me understand what i was doing.Another error: bad chars in project name (!)Now, I don’t know how many of you will find this error, but if you do, Google is your friend (and mine too) and you’ll find this article, I’m sure 😄I had project ids since 2012, and I’ve given them names which included parenthesis. For instance, the project id palladiusbonton-nobilling had an astonishing description (Project Name): “PalladiuBontonTest (No Billing)”. Now this was never a problem in my life until some day… some of them gave me error. Interestingly enough, most came from one single error type: I gave my project ids a bad name: I should have known better than Bon Jovi!Solution: I scripted in 💎 ruby a project description change which just removes the parenthesis (and the regex is DRY in case I find something new). To be on safe side, I change bad chars into empty spaces 🙂, as I love Queen more than Bon Jovi.$ bin/11-migrate-orphans.rb [..]goliardia2                     Goliardia2 (no billing uffi)              576689998257palladiusbonton                palladiusbonton (billing disabled)        606248867298palladiusbontontest-nobilling  PalladiuBontonTest (No Billing)           690861376600riccardo-chatroom1             riccardo-chatroom-one (billing disabled)  941891472972# EXECUTE this to fix the project: gcloud projects update 'palladiusbonton' --name='palladiusbonton  billing disabled ' # Former description was: 'palladiusbonton (billing disabled)'palladiusbontontest-nobilling // 690861376600 ### 'PalladiuBontonTest (No Billing)'# EXECUTE this to fix the project: gcloud projects update 'palladiusbontontest-nobilling' --name='PalladiuBontonTest  No Billing ' # Former description was: 'PalladiuBontonTest (No Billing)'riccardo-chatroom1 // 941891472972 ### 'riccardo-chatroom-one (billing disabled)'# EXECUTE this to fix the project: gcloud projects update 'riccardo-chatroom1' --name='riccardo-chatroom-one  billing disabled ' # Former description was: 'riccardo-chatroom-one (billing disabled)'Note the pattern: my 11th script creates an output with (1) the hopefully right command and (2) a human-readable description.How do Igcloud it?gcloud projects update 'project-id-or-number' \  --name='New Description without Parenthesis and stuff'..and here its how it looks if you have good eyes or monitor.Wait: what about the orphans?I noticed that not all projects were under an organization, some were orphans (ie, not attached to any organization).A few orphan project ids, stranded in Google CloudNow the migration script is the same as per other projects (documentation):gcloud beta projects move $PROJECT_ID \--{folder|organization} $DESTINATION_IDHowever, finding org-less projects require a different search, which took me a while. Let me Stackoverflow it for you:gcloud projects list --filter="parent.id.yesno(yes='Yes', no='No')=No"# This returns the list of ptroject that current $ACCOUNT has access toNote that the definition of orphan depends on your identity (foo.bar@gmail.com), not from an org. Different users perceive different orphans, based on their access to projects. To make sure you see them all, you should iterate through all your accounts (code snippet).Beware! While Org-ful projects are likely to all belong to you or your team, the Org-less projects could belong to anyone else! I moved by mistakes projects where I was owner but which belonged to another team. Make sure you look at permissions on those projects before migrating. If you’re not the sole owner, consider adding these projects to some “dispute” spreadsheet where you get the ok for the respective owners before moving (or any workflow which works for you).ConclusionsSuccess! I was able to migrate hundreds of projects from one org to anotherA few lesson learnt:Being OrgAdmin is not enough, sometimes you need to call the Org Police 👮.You need to allowlist the source domain to destination domain, and vice versa. Just like in a network firewall with two untrustworthy parts, which makes sense.I believe nobody documented this gcloud approach before (prove me wrong — or maybe not, as I’ve spent 72+ hours doing this😛). Researching the right gcloud filters is sometime a tedious work.Nothing helps more than a Migration Spreadsheet.Write code which writes and documents code, particularly where you have to do with boring numeric ids.Migrating N source orgs into N deterministically-named folders of a target destination org sub-folder was a really good idea.; by deterministically I mean: take the org domain and change dots with dashes (since dot are illegal): eg “example-com”.Here’s the migration in a nutshell: all the Source folder structure complexity is flattened in the Destination under a single special folder.Please let me know if you find mistakes, or you find a failure domains I haven’t described that can be gclouded , or any missing note. Thanks in advance. 😙Next Stepsopen source my code:recreate the source folder structure in the destination (!)Bonus point to copy/preserve IAM and Org Policies across folders structure (although this probably needs some source/destination IAM mapping which is beside the scope of this article).Put some code in a GitHub Gist:<a href="https://medium.com/media/3c408b048fe9b49c8ea3a56ab8dfd0d0/href">https://medium.com/media/3c408b048fe9b49c8ea3a56ab8dfd0d0/href</a>URLographyPages I used to find answers to this:www.google.comhttps://github.com/palladius/org-folder-projects-graph/ Amazing script to dump your orghttps://cloud.google.com/resource-manager/docs/project-migration (official). It took me a while to understand there are 6 more pages to navigate on the left (see image below).https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints Org policies explained.https://medium.com/google-cloud/migrating-a-project-from-one-organization-to-another-gcp-4b37a86dd9e6 Yogesh wrote the article before me, but no gcloud :)https://medium.com/google-cloud/setting-up-google-cloud-identity-for-multinational-companies-4afcbb18dee1 this guy is solving another problem, but he’s using gcloudhttps://stackoverflow.com/questions/73411088/unable-to-migrate-gcp-projecthttps://stackoverflow.com/questions/61244749/how-to-list-all-projects-in-gcp-that-belongs-to-a-specific-organizationhttps://stackoverflow.com/questions/72781977/move-project-to-a-different-organizationMigrate ☁️ GCP projects across organizations, with gcloud was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
[/content]

Author: Riccardo Carlesso
PublishedDate: 2023-04-18
Category: Blogs
NewsPaper: Riccardo Carlesso - Medium
Tags: gcp-security-operations, google-cloud-platform, migration