GitHub Enterprise Server Administration Best practices from GitHub Support

[djdefi]

Every day, the GitHub Enterprise Support team assists GitHub Enterprise Server administrators in managing, securing, and supporting the most critical and central hub of their organization's development lifecycle.

Adhering to best practices ensures your GitHub Enterprise Server remains secure, reliable, and high-performing. These practices not only help maintain operational stability but also allow you to get the most value from your environment.

Here are some essential practices you can adopt to enhance the management of your GitHub Enterprise Server.

High Availability and Disaster Recovery (Backups)

• Use GitHub Enterprise Backup Utilities to schedule regular backups. Test restores in a staging or test environment to ensure reliability in case of failures.
• Configure at least one High Availability Replica for redundancy and ease of recovery in the event of a primary appliance failure.
• Place replicas and GitHub Enterprise Backup Utilities hosts in different regions or data centers from the primary appliance to ensure geographic redundancy.
• Add Geo-replicas or Repository Cache nodes only in response to specific identified needs. Low latency and high bandwidth links are essential to avoid bottlenecks or management overhead.

Test / Staging Environments

• Keep your appliance up to date with the latest release to take advantage of security updates and bug fixes. Testing upgrades in a staging environment is particularly important when upgrading to a new feature release.
• Use one or more GitHub Enterprise Server Staging Environments to test and familiarize yourself with procedures such as upgrading the appliance, configuring and failing over in High Availability scenarios, and restoring backups.
• Use your GitHub Enterprise Server license to set up any number of non-production environments. These are strongly encouraged for testing upgrades or other changes to your environment.

Implement Policies for Rate Limiting and Abuse

• Set enforceable, policy-backed API rate limits. Exceptions should be rare and used for specific integrations or service accounts. For example, you may want to set a primary API rate limit policy similar to GitHub's established 5,000 requests per hour per user limit.
• Use log forwarding and collectd metrics forwarding to detect increases in request rates, bad actors, and patterns.
• Regularly review CLI usage reporting tools like ghe-governor-summary and ghe-btop for outlying or abusive behaviors.
• Suspend bad actors or block them at a network or firewall level.

Be a Good CI or API Citizen

• Use webhooks instead of cron-style polling to reduce unnecessary API load.
• If cron-style polling is unavoidable, spread out and skew fetches to minimize parallel requests.
• Utilize conditional requests with the GitHub API. For example, use the If-None-Match header to avoid downloading unchanged data.
• Keep CI plugins, such as GitHub Actions, up to date with caching features enabled.
• Avoid storing build artifacts, logs, or data backups in Git (especially in refs/heads). Instead, use Git LFS for large files to maintain Git's performance.
• Minimize frequent automated pushes, batch multiple commits into a single push, and limit the use of scheduled automated pushes.

Provision Hardware for Growth

• Expand your root volume beyond the default. Provision 2x or 4x the default size for busy systems to allow for natural growth.
• Use fast SSD-based storage with low latency. For example, consider NVMe drives with at least 10,000 IOPS for high-performance environments. Selecting storage with configurable IOPS allows for future scaling.
• Allocate sufficient memory, such as 64GB or more, to optimize disk caching and GitHub worker/service scaling.
• Ensure adequate CPU, Memory, and Disk and resources for your GitHub Enterprise Backup Utilities host to speed up backup and restore processes.

Join the Conversation!

We’d love to hear how you’re managing your GitHub Enterprise Server environment. What are some of your tried-and-true strategies?

Share your insights, tips, or questions below and join the discussion to help others enhance their server environments!

[MattSnow-amd]

@djdefi - Given that GHES is a virtual machine deployed on a few different hypervisors, can you share any more details for specific hardware that is used with success?

I would love to be able to use GHES on Linux w/ Libvirt or something like oVirt, but this hypervisor is not listed.
Additionally are there any recommendations or experiences you can speak to on hyperconverged infrastructure, such as VMware vSAN or oVirt+GlusterFS ? I feel like HCI is the likely way to go for on-premise GHES environments for quicker recovery in the event of hardware failure.

[djdefi]

Hey @MattSnow-xlnx

In support we often have quite a bit more insight into the hardware cloud platform than on-prem deployments just due to those instance types having more readily available documentation and benchmark data to reference. In the supported cloud platforms, we find that "memory optimized" instances which reflect a 1CPU to 8GB RAM ratio often fit workloads the best. Additionally, faster CPU clock speed offerings often perform better during heavy Git operations than ones that advertise power saving or other CPU stepping features.

For on premises hardware, we actually see a lot of variance in what is used, and what can be successful. In general you want the fastest CPU clock speeds, a fast SSD disk, lots of RAM, and the fewest possible NUMA nodes for the CPU+RAM combination.

For both cloud and on premises, in many cases we have found that large, active systems with a smaller number of NUMA nodes (ideally 2) perform better than systems that have 4 or more nodes. Also enabling Performance mode / disabling power saving features in the BIOS may be recommended to maximize and stabilize the potential hardware performance.

We don't have a lot of specific benchmarks with regards to VMware vSAN or oVirt+GlusterFS currently, but in general we do see that high storage latency can become a limiting factor or bottleneck. For this reason our documentation recommends fast locally attached SSD, but many modern storage vendors and systems can likely provide equivalent performance. One caveat is that we often we do find network type storage overhead, such as with NFS can often perform poorly with GitHub Enterprise and many tiny files contained within repositories on the appliance. For that reason, we would recommend making sure that any software based storage solution has roughly equivalent latency and throughput as a local SSD to the extent possible.

While we don't specifically mention Libvirt or oVirt in our documentation (just OpenStack KVM), we are fairly confident that these platforms would work and be easily supportable given enough interest. We use a vanilla KVM / oVirt based hypervisor for a large portion of CI/CD, so this is already is rather well tested path. If you are interested in official Libvirt or oVirt support specifically, I would suggest raising a support ticket or contacting your GitHub account manager to "officially" log a feature request.

I hope this helps!