Ok, here's my first go at it. This includes the necessary changes to build a working Gem library on macOS. The build works with both MacPorts and Homebrew now, but prefers Homebrew if it is installed. (As a working libquicktime is only available on Homebrew, the QT4L backends are disabled in the MacPorts build for now, but they work -- kind of -- in the Homebrew build.) All in all, this works quite well actually, but there are some known quirks, see below.
I also updated the Homebrew instructions in the toplevel README. And the MR contains a backport from upstream (vanilla) which improves diagnostics of library loading. While this is unrelated, it helped a lot to diagnose library issues which prevented this to work at first.
The gemmacoswindow (which is the default on macOS) doesn't work at present, at least not on anything that's more recent than Sierra. This is a known bug, see https://github.com/umlaeute/Gem/issues/257. The window backend which worked best for me (on Mojave at least) is the SDL2 window, so we (build and) default to that. Of course, this means that SDL2 must be installed (it's listed in the updated README). The SDL2 window does have some quirks of its own on the Mac, though. You can't close it, but you can open different patches in succession and have them render to the same window.
Video loading and recording via QT4L works (in the Homebrew build), but the options are extremely limited. For one thing, it seems that there's no mp4v codec in the libquicktime from Homebrew, so no MPEG4 playback. In fact, there doesn't seem to be much of any of the video formats that people commonly use. Maybe we can fix that by building our own libquicktime or port one of the existing versions that are known to work, such as the one from Ubuntu. But that's something for another day.
On the bright side, basic OpenGL functionality, images, 3D models, and video capture all seem to work fine, so it's certainly usable (and better than not having Gem supported on the Mac at all).
I tested this with both Homebrew and MacPorts, both builds work fine for me. Test builds on Linux and Windows also continue to work.