My impression from reading the documentation for semver is that version changes correspond with API compatibility, not necessarily the content of the framework.
Here's an FAQ from the documentation.
What should I do if I update my own dependencies without changing the public API?
That would be considered compatible since it does not affect the public API. Software that explicitly depends on the same dependencies as your package should have their own dependency specifications and the author will notice any conflicts. Determining whether the change is a patch level or minor level modification depends on whether you updated your dependencies in order to fix a bug or introduce new functionality. I would usually expect additional code for the latter instance, in which case it's obviously a minor level increment.
@Copying It's important to note the distinction between being "buildable" and "compatible." If you make a change that causes the screen origin to be at the bottom-left corner of the screen instead of the top-left, that change would probably allow for older games to build just fine; however, they would be "broken" by that API change, because they were originally developed under the assumption that the origin was at the top-left.
Just use common sense when making changes. If you're worried about breaking compatibility, just ask us and we'll take a look at it. The easiest way would probably to just share a link to your commit on Github, like so: