In Fluxbox, you can use Alt+F11 to make all X applications to switch to "fullscreen". Therefore, just press Alt+F11 and your Urxvt should switch to fullscreen. I found this inadvertently while trying to switch another application to fullscreen. I didn't realize that Urxvt was under the "focus" at that point. Nonetheless, it's very useful because I don't need additional package to make it work.
The "fullscreen" moniker here is rather tentative because what happens is Fluxbox switch the window that's currently in focus to occupy the entire screen (a.k.a maximized) and at the same time disabling its window decoration. This is effectively the same as what you would expect from a "fullscreen-ed" application, i.e. applications that supports fullscreen mode such as video player.
As a bonus, this is my .Xresources snippet that shows how to make Urxvt "transparent" and at the sametime removes the scrollbar (which you won't need if you're using tmux or screen):
URxvt.depth: 32 URxvt.foreground: rgba:eeee/eeee/eeee/ffff URxvt.background: rgba:0000/0000/0000/cccc URxvt.cursorBlink: True URxvt.cursorUnderline: True URxvt.scrollBar: False